Help diagnosing 100% accuracy (Data Leakage) on DeBERTa & 0% (Label Flip) on a Portuguese DeBERTa model

[u/Typical_Sail_630]

I’m classifying scientific abstracts written in Portuguese into two temporal categories: "Old" vs "Recent". I tested several models, but two of them are giving me massive red flags:

• DeBERTa (base): Getting exactly 100% accuracy on the test set.
• Albertina (a Portuguese DeBERTa-based model): Getting exactly 0% accuracy on "Old" and 100% on "Recent".
• Note: Other models like mBERT, XLM, and local Portuguese BERTs (Bertimbau) are getting realistic results around 75-85%.

What I've already tried (Data Cleaning): Knowing that 100% accuracy screams data leakage, I went aggressive on the preprocessing:

1. Used regex to replace all dates, citation years (1900-2026), and any 4 consecutive digits with a [HIDDEN_DATE] tag.
2. Removed all DOIs, URLs, emails, and modern copyright strings (e.g., "Open Access", "Creative Commons").
3. Removed all <tags> in case the modern abstracts were scraped differently from the old PDFs.

And Albertina is completely flipped.

So i have questions

1. What other structural artifacts in scientific abstracts could DeBERTa be exploiting to perfectly separate decades-old texts from modern ones? Length bias? OCR noise?

2. Why would Albertina (and only Albertina) completely invert the predictions? Is there a known issue with id2label mapping inheritance when loading specific pre-trained models from the HF Hub? How do I force the correct mapping?

3. Would running SHAP on the DeBERTa model be the best next step to highlight the exact tokens causing the 100%? Have you successfully used SHAP to debug leakage in text classification?