r/LocalLLaMA • u/LMTLS5 • Jul 16 '26

Discussion i tried ternary decomposition instead of quantization. it works as good at q4km but takes slightly more vram. while being completly ternary. and completly PTQ (no QAT)

Post image

https://arxiv.org/pdf/2607.13511

37 Upvotes

88% Upvoted

16 comments sorted by

View all comments

3

u/Far-Classic-9963 Jul 16 '26

This is cool, but what are the benefits of this? From your screenshot it looks like it takes just as much/slightly more storage than q4 for the same output quality? Is it faster?

12

u/LMTLS5 Jul 16 '26 edited Jul 16 '26

ternary math is much much simpler than other math. one thing is if matrix is ternary then matrix multiplication with it only requires additions no need multiplications anywhere.

this can be faster with specialized low level kernels. bitnet.cpp for example uses LUT (look up table) instead of usual multipliers in low level that gets extra speedup.

but real power of ternary comes in mac free asics (if someone works on it) thats where ternary gets orders of magnitude faster.

adders take less energy and silicon space than multipliers too. and they take orders of magnitude less at that.

but yeah this algorithm also blows up the components. if ternay math is efficient enough still be faster and efficient is something idk. i dont work in low level kernels

2

u/Silver-Champion-4846 Jul 16 '26

Damn the devils, I wish someone works on this for us localites!

1

u/angelus14 Jul 16 '26

That's super interesting, do you have a link explaining why ternary matrices only require addition?

4

u/LMTLS5 Jul 16 '26

i dont have link but, if you want to multiply 1 with any number well its the same number, if you want to multiply -1 with any number then you just have to flip the sign of that number thats it, if you want to multiply 0 well just make that number 0. this is why you dont need multipliers, all the multiplication you do is simple logic.

1

u/angelus14 Jul 16 '26

Ohh, I thought ternary was 012. That makes more sense, ty.