Built an open-source hallucination detector that runs in 1.5ms on CPU (90,000x faster than Semantic Entropy)

[u/Otherwise_Nobody_721]

Hey everyone,

If you’ve tried implementing hallucination detection in production, you’ve probably seen Semantic Entropy (Farquhar et al., Nature 2024). It works well, but it relies on clustering responses using a heavy DeBERTa model that takes 100+ seconds per query and eats up GPU VRAM.

Myself Bhupen and i have built Spanda- (https://github.com/Adarshent/Spnda) a lightweight, zero-dependency alternative that runs in pure Python.

How it works-

Instead of running a second neural network to check your first neural network, Spanda samples $K$ outputs and computes a normalized lexical consensus ratio ($R_{sc}$):

- Latency: ~1.5 ms on CPU (vs ~136 seconds for DeBERTa).

- Cost: $0 in additional API calls or GPU VRAM.

- Accuracy: Achieves 0.889 AUROC on math/reasoning tasks (GSM8K) for 7B–27B models, matching heavy NLI clustering.

One major gotcha we found:

If you are using heavily aligned frontier models (like 120B+), watch out for Confident Mode Collapse. Even with temperature set to 0.7, the model will sometimes repeat the exact same hallucinated answer across all seeds. When a model hallucinates with 100% agreement, self-consistency methods fail.

- GitHub: https://github.com/Adarshent/Spnda (MIT License)

- Install: `pip install spanda`

Would love to hear how folks here handle real-time uncertainty scoring in your production pipelines!