Does Truora generalize to real-world news?

Every model in this project was trained and tested on a single combined corpus built from the ISOT Fake News dataset and the LIAR dataset. Strong performance on a held-out split of that same corpus doesn’t, by itself, prove a model has learned genuine indicators of misinformation rather than dataset-specific artifacts — writing style, source formatting, publication conventions.

To test whether the best-performing model (fine-tuned BERT) generalizes beyond its training distribution, we built a case study from freshly scraped, genuinely real articles that never appeared in training. This page walks through the case study as it actually unfolded: an initial evaluation that failed badly, a diagnostic investigation into why, a targeted fix, a re-evaluation, and a further probe into what the fix did and didn’t solve.

543 real-news articles were scraped independently of the ISOT/LIAR training corpus, all labelled Real. Because every ground-truth label is Real, the relevant metric is the False Positive Rate (FPR) — the share of genuinely real articles the model incorrectly flags as Fake.

Despite strong in-distribution performance on the ISOT/LIAR held-out split, the fine-tuned BERT model misclassified almost every real article once the data came from an unfamiliar publisher and time period.

The model wasn’t merely wrong — it was confidently wrong. Predicted P(Fake) for misclassified articles clustered close to 1.0 rather than near the 0.5 decision boundary, pointing to a systematic rather than a marginal failure.

ISOT’s Real class is sourced almost entirely from Reuters wire articles, which follow a highly distinctive, repetitive format (e.g. WASHINGTON (Reuters) --). If the model learned “looks like a Reuters wire story” as a shortcut proxy for Real, any article lacking that exact formatting would default toward Fake — regardless of its actual publisher or truthfulness.

Two prefix variants were tested on the 30 most confidently misclassified real articles: a fabricated Reuters dateline, and a length-matched neutral control with no wire-service signal.

The result was unambiguous: a fabricated Reuters dateline flipped every single misclassified article back to Real. The neutral control had almost no effect — the model was reacting specifically to Reuters wire-service formatting, not to added context in general.

Repeating the same diagnostic on the classical models confirmed the artifact lives in the training data itself, not just BERT’s architecture — though BERT was far more prone to fixating on it.

Guided by this diagnosis, the preprocessing pipeline was revised before retraining:

• Wire-service dateline removal — a regex matching the CITY (Reuters) -- pattern stripped the formatting cue out of the training text itself.
• Exact and near-duplicate removal — exact-text duplicates dropped, followed by a similarity-based near-duplicate pass at a 0.95 cosine-similarity threshold.
• Junk-row filtering — degenerate scraped fragments (e.g. “Watch:”, truncated captions) removed using empirically justified length thresholds.
• Systematic artifact-skew check — a class-conditional term-skew analysis surfaced other source-composition artifacts (image-credit boilerplate, pic.twitter.com links), closed the same way.

BERT was retrained from scratch on the cleaned corpus. One additional change was bundled into the same retrain: max input length was increased from 256 to 512 tokens, since roughly half of training articles were being truncated at 256.

Headline metrics dropped slightly after the fix — most notably recall, from a near-perfect 0.9997 to 0.9485. That’s not a regression; it’s evidence the fix worked. The original near-perfect recall was itself a symptom of the leak.

The identical 543-article case study was re-scored on the retrained model:

Kathmandu Post and OnlineKhabar reached a perfect 0.00% false positive rate. BBC retained a residual 10.93%, concentrated unevenly across sections like world/australia and health.

Because the retrain bundled two changes (the leak fix and the longer context window), a follow-up ablation isolated their individual contributions:

Roughly 88% of the total improvement is attributable to the leak-removal fix itself, and about 12% to the increased context window.

To probe this, three entirely fabricated news articles were hand-written — describing events that never happened, but composed in a measured, professionally-styled register consistent with real journalism.

All three were classified as Real with greater than 99.8% confidence. The fix closed one specific, identifiable shortcut (Reuters-style formatting) without resolving the model’s broader reliance on stylistic and structural surface cues as a proxy for factual accuracy. It improved reliability as a filter against false alarms on genuine news — it did not meaningfully improve detection of well-written misinformation.

This whole investigation started with one observation: a genuine, verified article reporting Nepal’s 2026 general election result was submitted to the pre-fix model and classified as Fake with 100% confidence. The same article was re-submitted to the post-fix model as a final check.

The post-fix prediction is correct — but only barely, sitting almost exactly on the decision boundary. Reduced bias, not confident understanding.

• Initial false positive rate on 543 unseen real articles: 99.63%.
• Root cause: the model learned to associate Reuters-style datelines with the Real label, not genuine indicators of factuality.
• After a targeted preprocessing fix and retrain, FPR fell to 4.97% — with two of three publishers reaching a perfect 0.00%.
• The fix does not generalize to detecting sophisticated, professionally-styled fabricated content — all three synthetic test articles were still classified Real with high confidence.
• Reliability against false alarms on genuine news improved substantially and measurably; the ability to detect deceptive, well-formatted misinformation remains an open problem.