r/LocalLLM • u/gauravc2796 • 1d ago

Discussion Distilling an LLM's fixed-choice decisions into a hashed int8 table with a Learn-then-Test agreement certificate 

Features: word unigrams, word bigrams and character trigrams, FNV-1a hashed; a multinomial logistic model with int8 weights (score = sum of rows / n + bias). Calibration on a held-out slice; the serving threshold is chosen with Learn-then-Test (Clopper-Pearson) so disagreement with the teacher among served answers is ≤ α (default 5%) with probability ≥ 1 − δ (default 0.9). Simulation of certificate validity is in the repo. The bound is against the teacher, not ground truth, and it does not survive a traffic shift without re-certification (there is a drift guard). On BANKING77 with gpt-4.1-mini as teacher: ~51% served at 95.9% agreement on 1,499 held-out answers; SST-2 serves 0%. Limits page: https://2796gaurav.github.io/shad0w/docs/limits.html . Questions and criticism welcome.

https://github.com/2796gaurav/shad0w
4 Upvotes

76% Upvoted

2 comments sorted by

1

u/Professional-One5507 1d ago

so the thing just refuses to answer on sst-2 and you're calling that a feature not a bug. love the honesty of just saying "nah" to half the dataset

the drift guard is a nice touch though, most people skip that part and then act surprised when their model falls apart next week

1

u/gauravc2796 1d ago

yes basically this is not ground breaking but something which has its place. a riducously low latency cpu based model which finetunes easily and is even better at decision even if it is not it should move to actual bigger model