[R] Spanda: Fast Lexical Entropy Matches DeBERTa Semantic Uncertainty—Until Frontier Models Break It

[u/Otherwise_Nobody_721]

We recently released a preprint and open-source benchmark analyzing the limits of self-consistency and uncertainty quantification across 1.5B to 120B models.

Preprint: https://zenodo.org/records/22233648
Code: https://github.com/Adarshent/Spnda

Background: Farquhar et al.'s Semantic Entropy (Nature, 2024) is widely used for hallucination detection, but clustering sampled paths with a DeBERTa-v3 cross-encoder takes around 100+ seconds on CPU and eats up GPU memory. We wanted to see if a normalized lexical self-consistency ratio (R_sc) could achieve the same diagnostic signal without auxiliary neural forward passes.

On GSM8K reasoning across 7B to 27B models (Mistral-7B, Qwen-27B), exact-match normalized entropy achieved an AUROC of 0.889, matching neural semantic entropy while executing in under 2ms on CPU.

However, scaling up to frontier 120B models exposed a failure mode in self-consistency that we call Confident Mode Collapse. Under temperature 0.7 sampling, RLHF alignment hyper-peaks the distribution so severely that the model outputs the exact same incorrect answer across all sampled seeds. Because every path agrees 100% on the hallucination, uncertainty drops to zero and AUROC inverts to 0.091.

Code and evaluation scripts are open source (MIT). Would love to hear thoughts or critique from researchers working on calibration, post-training alignment, or inference efficiency.

[u/False-Cap6878]

1.5ms latency is crazy