Hey everyone! I decided to dive into NLP (Natural Language Processing) and found a dataset containing real and fabricated news articles for a starting project. My goal was to train a model to distinguish fake news from authentic publications.
For implementation, I used a combination of TF-IDF Vectorizer and Logistic Regression from scikit-learn. On the test set, the model achieved an impressive 97.25% accuracy. However, things got really interesting when testing the exact same fabricated story phrased in four different ways:
“BREAKING: Germany Bans Smartphones in All Schools and Universities Starting 2027” → Real
“Germany to Introduce Nationwide Smartphone Ban in Schools and Universities in 2027” → Real
“Federal Government Announces New Restrictions on Smartphone Use Across German Education System” → Real
“Germany Plans to Eliminate Personal Smartphone Use in Educational Institutions Starting in 2027” → Fake
🖼️ confusion_matrix.png
Notice that despite the high accuracy score, the model remains vulnerable. Its decision is heavily influenced by key phrases and stylistic cues:
"BREAKING" and "Nationwide" mimic the style of breaking news alerts.
"Federal Government Announces" sounds like an official press release.
"Plans to Eliminate" uses word combinations more frequently associated with fake samples in the training set.
Consequently, the classifier evaluates writing style and n-grams while completely missing actual factual manipulation, numbers, or dates.
🖼️ word_count_distribution.png
🖼️ subjects_breakdown.png
In conclusion, it is impossible to reliably determine whether a news story is true based solely on its phrasing. A full-fledged fake news detection system must include fact-checking capabilities and cross-source verification.
The source code for fake_news_detector.py is available on GitHub. Feel free to check it out and share your thoughts!
