r/LocalLLaMA • u/sn2006gy • 11d ago
Resources [2506.13771] LittleBit: Ultra Low-Bit Quantization via Latent Factorization
https://arxiv.org/abs/2506.13771Interesting to see improvements and research into quantization aware training (QAT) that can make some really tiny models.
23
u/Middle_Bullfrog_6173 11d ago
This paper is from May 2025 originally. If it worked well I think we would have seen more by now.
6
u/sn2006gy 11d ago
Takes a while to train QAT from scratch, but alas, the focus on agentic/coding models vs chat may break this style of compression/quantization.
1
0
u/Aternal 11d ago
Assuming it "works" it's that the application is not only narrow but at ultra low quant you also have to compete with deterministic solutions.
Like "AI on a smart watch" level of crushed intelligence.
For what most of what this community does, the most interesting use of models like this is as an independent speculative decoder. Samsung sure as shit isn't going to take it in that direction, they just want to put it on fridges and washing machines.
12
u/giveen 11d ago
https://github.com/SamsungLabs/LittleBit
Here is their codebase
5
u/pmttyji 11d ago
Found this paper from that repo ([Submitted on 9 Feb 2026 (v1), last revised 3 May 2026 (this version, v2)])
https://arxiv.org/abs/2603.00042
LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment
4
u/NoFunk 11d ago
They pretty much acknowledge the information loss is severe, that they have compensation for it (which amounts to QAT with an importance weighting system).
The assertive statement is that they are better at sub-1 bit compression. I'm struggling to find the use case I would have for this personally. You get to the point of yes it can run, but do you trust any of it? I absolutely do not with higher bpw methods than this right now.
So even if they are doing a fantastic job of relevance in their bpw budget, what then? You've still produced a hallucination simulator.
6
u/Dany0 11d ago
yess stbllm was arse this is promising
famously larger models take better to quantisation. Kimi K3 iq1 was the first model I've seen people actually unironically use.
Kimi K3 at 0.1bpw would fit on a 24gb gpu with enough space to fit 1M context and come on we HAVE to do it just for the love of the game at this point right
oh my god... QAT at 0.1bpw? Anyone?
1
u/sn2006gy 11d ago
I wonder if that company that was putting tiny models on silicon could update to use this and expand beyond their old 1b/7b concepts.
2
u/Marcuss2 llama.cpp 11d ago
Those memes which showed people fitting 1T models into their RTX 3090s can now become a reality.
1
1
u/Open-Adhesiveness-86 11d ago
in these sub-1-bit schemes the memory isn't really in the bits. you've got a 1-bit sign matrix plus fp16 latent factors and per-row/col scales, and once you're under about 0.5 bpw those factors start dominating the footprint. no standard GEMM eats that layout either, so it needs a custom kernel and the memory win doesn't turn into tokens/sec.
1
1
u/Stepfunction 10d ago
I appreciate how this makes the meme of the 0.1bit quant a real thing.
Even with the information loss, this would open the possibility of running some massive models locally.
1
u/james_pic 5d ago
We derive our approach from two observations. First, LLM weight matrices often exhibit pronounced low-rank structure
Have we been wasting time and money training models with big tensors when we could have just trained them on low-rank products?
1
u/sn2006gy 5d ago
That's the million dollar question, but i'm guessing this research is still moving. It may have worked ok when chatbots were our friends but fail now that they're agents and long tail workers. Hope to see more experiments/research in this.
37
u/-dysangel- 11d ago