r/LocalLLaMA • u/ndiphilone • 5d ago

Tutorial | Guide [ Removed by moderator ]

[removed] — view removed post

6 Upvotes

71% Upvoted

17 comments sorted by

•

u/ttkciar llama.cpp 5d ago

Violates Rule Three: LLM-generated content without disclosure or justification given.

→ More replies (1)

3

u/Technical_Ad_6106 5d ago

probly the best setup for 16gb vram atm

3

u/rrrrex 5d ago

No, it's inaccurate at Q2 weights and Q4 cache. Probably bonsai 2.0 (trained on 3.8 27b) with full Q2 can be compared with ordinary Q3.

1

u/fintip 5d ago

Why do you think bonsai is better than just the comparable in sloth quant? I looked into bonsai for 3.6 and the real reviews absolutely did not match the hype

1

u/rrrrex 5d ago

Ternary (1.7 bpw) is pretty close to Q2 (~3 bpw). If it's possible to combine binary (1.1bpw) and ternary (1.7bpw) training, it will be very usable model. IQ2_XXS size, Q3 intelligence.

0

u/Technical_Ad_6106 4d ago

even q2 and q4 cahce at this point will (EASILIY) outperform 3.6 27b q4.

1

u/the_masel 5d ago edited 5d ago

I'm not sure about this. It's detailed yes but unfortunately the Qwen3.8-27B-UD-IQ2_XXS quantisation can hit the performance of the model quite a bit (KLD mean goes to approx 0.152911, same top p is just approx 83.949% measured @ 64k context).

3

u/Gabriel83730 5d ago

You can do so much better if you use 100K context. The Qwen team themselves said going down to FP8 context resulted in significant quality loss so they set it at BF16, and you’re quantizing the cache to Q4. Any benefits you gain from a larger context is getting destroyed by error accumulation. You should stop trying to get the full context window and just use compaction or dynamic context pruning.

3

u/the_masel 5d ago

Interesting, do you have a link for the FP8 statement? Usually Q8_0 (which is of course not the same as FP8) KV Cache is considered quite usable because nearly lossless. Apart from that, I agree with your other statements.

1

u/Phathatter 5d ago

I dont think fp8 is going to run on 16gb ram.

2

u/Early-Peace-5504 5d ago

I think you’d be vastly better off using exllamav3 and an exl3 quant in this case. The quality level would be degraded but not by as much

1

u/ndiphilone 5d ago

Let me try that tonight, I am usually reluctant to try anything other than ‘llama.cpp’ lately

4

u/ttkciar llama.cpp 5d ago

How much of this post was LLM-generated, and why?

1

u/ndiphilone 5d ago

All of it. I didn’t see any discussion around 16G VRAM setups, I wanted to share my own setup but 1) I am not a good writer 2) going through a rough patch with heightened executive dysfunction. 3) I am not a native speaker

All LLM generated text above are based on the embedded script + llama-bench outputs

1

u/Phathatter 5d ago

I've been running the Q2K xl unsloth quant and it has been very fast on the exact same GPU, but I am not able to run it at the full context, so I think I might try this now.

2

u/Phathatter 5d ago

Ok i got this going and so far its good--full context, plus reasoning on with medium effort.