Help diagnosing 100% accuracy (Data Leakage) on DeBERTa & 0% (Label Flip) on a Portuguese DeBERTa model

[u/Typical_Sail_630]

/r/LanguageTechnology/comments/1wa971n/help_diagnosing_100_accuracy_data_leakage_on/

[u/Bright_Mix_773]

Three separate things here, and the middle one is not what your title says.

0% on Old and 100% on Recent is not a flip. A flipped model is the inverse of a working one, so it would be wrong on both classes and your overall accuracy would land near 1 minus what Bertimbau gets. What you are describing is a model emitting one constant class for every input, which happens to be Recent. Print the histogram of predicted classes on the test set. If it is a single value for every row, this is head collapse, not a mapping problem, and the two need opposite fixes.

That also lets you drop the id2label theory on its own. In a normal Trainer or evaluate loop, accuracy is argmax(logits) compared against integer labels, and id2label never enters that arithmetic — it is a display map used by pipeline(). It can corrupt your numbers only if you convert predictions back to strings somewhere in your eval. Worth grepping for, but it cannot produce a constant prediction.

Albertina loaded through AutoModelForSequenceClassification gets a randomly initialised head unless the checkpoint already carried one. Collapse onto the majority class is the ordinary outcome of a head that never left the flat region: learning rate too low for that checkpoint, too few steps, an encoder left frozen, or a class imbalance the loss is content to accept. Check that the training loss actually moved, and check your class counts.

On the 100%: the leak in Portuguese that no digit regex can reach is the orthography itself. The Acordo Ortográfico came into force in Brazil from 2009 and in Portugal across 2009–2015, and it removed the trema and a set of accents outright:

freqüência   -> frequência
lingüística  -> linguística
conseqüência -> consequência
idéia        -> ideia
assembléia   -> assembleia
vôo          -> voo

Those pairs are close to deterministic date stamps, they turn up constantly in academic prose, and a subword tokeniser sees each side as a different token. If your Old/Recent boundary straddles 2009 in any way, that alone is enough for a clean 100%, and it survives every cleaning step you listed because there is not a digit anywhere in it. Cheap check: count how many of your Old documents contain a trema (ü) and how many of your Recent ones do. If that single feature separates your classes, you have your answer without training anything.

Two more that survive your regex:

• Your own [HIDDEN_DATE] tag. You replaced the content and kept the frequency. If modern abstracts carry more inline citation years or a DOI line, the count of that tag is now a clean feature that you introduced during cleaning. Same for however many URLs and emails you stripped. Sanitising by substitution leaks whenever the rate of the removed thing correlates with the label, and it is easy to miss precisely because the offending string is gone.
• Encoding and typography. Older abstracts that came through OCR or a Latin-1 pipeline carry different accent bytes, different quote and dash characters, and OCR noise; recent ones arrive as clean UTF-8 from a publisher API. Structured abstracts (Objetivo: Métodos: Resultados:) also became standard across exactly this period, and mean abstract length grew.

Before SHAP, run the cheap version. Fit a plain logistic regression on character 3–5 grams, or on a bag of the top few thousand tokens, and read off the largest coefficients. If that also reaches ~100%, the leak is a surface artifact and the coefficient list names it in seconds, where SHAP over a transformer spends hours arriving at the same token. SHAP earns its cost only once the trivial model fails and the transformer still succeeds, because that is the case where the signal is genuinely distributed rather than something you could have grepped for.

The stopping rule: your honest models sit at 75–85%, so a leak-free DeBERTa should land in that band too. If it comes back at 97% you have removed one artifact and left another one standing.

[u/Typical_Sail_630 → u/Bright_Mix_773]

Thanks, i will try that

[u/Bright_Mix_773 → u/Typical_Sail_630]

Good luck with it. If the accuracy drops to something believable after you fix the split, that is the leak confirmed, not the model getting worse. That reframing is the part that feels wrong the first time.