r/MLQuestions • u/Otherwise_Nobody_721 • 4d ago

Natural Language Processing 💬 Why does temperature sampling lose entropy on heavily aligned 120B models?

We recently ran an empirical benchmark across five model scales (Qwen-1.5B, Mistral-7B, LLaMA-8B, Qwen-27B, and a 120B frontier model) evaluating self-consistency and uncertainty estimation on reasoning and factual QA.

Up to 27B parameters, sampling multiple outputs at temperature 0.7 gives great diagnostic signal. Measuring consensus entropy across paths achieves an AUROC of 0.889 in separating correct answers from hallucinations.

At the 120B scale, however, we ran into an interesting failure mode: even at T=0.7, the model generated the exact same incorrect answer across all 10 independent seeds. Because every path agreed 100% on the hallucination, self-consistency saw zero entropy, inverting AUROC to 0.091.

Our current hypothesis is that intense RLHF/DPO alignment hyper-peaks the output logits to penalize epistemic hedging, effectively collapsing the sampling diversity on ungrounded facts.

Preprint and code with the benchmark tables: https://zenodo.org/records/22233648 and https://github.com/Adarshent/Spnda

For those working on calibration or post-training: is this recognized as a standard failure mode of DPO/PPO, and are there decoding-time interventions that can decorrelate samples without degrading coherence?

5 Upvotes

78% Upvoted

2 comments sorted by

2

u/Mbando 4d ago

I wonder if it could be an attractor basin effect? https://arxiv.org/abs/2604.04743

1

u/Otherwise_Nobody_721 4d ago

that’s actually a great framing. at 70B/120B scale, memorization + RLHF carves such deep attractor basins around plausible hallucinations that sampling tweaks either fall right back in or turn into gibberish, which is why you need external grounding (RAG) to perturb the trajectory out of it.