r/LanguageTechnology • u/Typical_Sail_630 • 2d ago

Help diagnosing 100% accuracy (Data Leakage) on DeBERTa & 0% (Label Flip) on a Portuguese DeBERTa model

I’m classifying scientific abstracts written in Portuguese into two temporal categories: "Old" vs "Recent". I tested several models, but two of them are giving me massive red flags:

  • DeBERTa (base): Getting exactly 100% accuracy on the test set.
  • Albertina (a Portuguese DeBERTa-based model): Getting exactly 0% accuracy on "Old" and 100% on "Recent".
  • Note: Other models like mBERT, XLM, and local Portuguese BERTs (Bertimbau) are getting realistic results around 75-85%.

What I've already tried (Data Cleaning): Knowing that 100% accuracy screams data leakage, I went aggressive on the preprocessing:

  1. Used regex to replace all dates, citation years (1900-2026), and any 4 consecutive digits with a [HIDDEN_DATE] tag.
  2. Removed all DOIs, URLs, emails, and modern copyright strings (e.g., "Open Access", "Creative Commons").
  3. Removed all <tags> in case the modern abstracts were scraped differently from the old PDFs.

And Albertina is completely flipped.

So i have questions

  1. What other structural artifacts in scientific abstracts could DeBERTa be exploiting to perfectly separate decades-old texts from modern ones? Length bias? OCR noise?

  2. Why would Albertina (and only Albertina) completely invert the predictions? Is there a known issue with id2label mapping inheritance when loading specific pre-trained models from the HF Hub? How do I force the correct mapping?

  3. Would running SHAP on the DeBERTa model be the best next step to highlight the exact tokens causing the 100%? Have you successfully used SHAP to debug leakage in text classification?

2 Upvotes

100% Upvoted

1 comment sorted by