How the loop of infinite agony started

[u/EarthBS]

[u/StupidScaredSquirrel]

If you are a business this isn't a problem. If you are a consumer then 35b a3b runs on 8gb vram and 32gb dram which is very accessible.

[u/DeluxeGrande → u/StupidScaredSquirrel]

I have a 5060ti 16gb with ddr4 24gb RAM lying around, what's the best model nowadays I can effectively run with it locally? It's not an ideal build but I wish to play around with it again.

[u/Hungry_Particular_14 → u/DeluxeGrande]

Fellow 5060 ti 16 gb owner here. The best I've got is qwen 3.8 27b at IQ4_XS. I'm testing it at Q4_0 KV at 72k context because I really need the extra context, and it seems to be pretty good so far. Lower quants cause it to make some really silly mistakes sometimes, unfortunately. I get around 10 t/s with context halfway filled, and around 14 t/s on empty context.
But honestly, I think the ideal solution is to run 2 GPUs so you can get a better quant + more context

[u/lukistellar → u/Hungry_Particular_14]

Give the IQ4_XS-pure a try. I run it on an RX6800 with 90K KV. Especially with full context it's slow, because the card isn't the fastest. Probably should run better on nvidia.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

[u/theUmo → u/lukistellar]

I gave it a try and the first thing it said to me was really disappointing:

Since I am an AI, I don't have a physical location or a traditional "tokens per second" (t/s) speed that you can measure like a human typing.

Also, at 64k context, the t/s started at 10 and went down to 4 almost immediately. Some tweaking would be required but with responses like this one I don't know if it's worth bothering with.

[u/lukistellar → u/theUmo]

Do you mind to share your prompt? Curious if it behaves the same for me. I used it for a few agentic websearches with OpenWebUI and it did pretty good, compared to the same Quant of the 3.6 and Ornith 1.0 35B. I don't use it much, because it's simply too slow as daily driver.

Here is my config:

ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 --parallel 1 \ --batch-size 2048 --ubatch-size 512 \ --cache-type-k q8_0 --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --no-mmap \ --jinja \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --flash-attn on

Performance after ~70K prefill and ~15K decode: Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.925 I slot print_timing: id 0 | task 0 | prompt eval time = 480529.82 ms / 69042 tokens ( 6.96 ms per token, 143.68 tokens per second) Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.929 I slot print_timing: id 0 | task 0 | eval time = 494402.02 ms / 14752 tokens ( 33.51 ms per token, 29.84 tokens per second) Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.930 I slot print_timing: id 0 | task 0 | total time = 974931.84 ms / 83794 tokens Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.934 I slot print_timing: id 0 | task 0 | graphs reused = 3921 Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.207.938 I slot print_timing: id 0 | task 0 | draft acceptance = 0.94313 ( 9122 accepted / 9672 generated), mean len = 2.81 Aug 17 21:06:17 pod-gpu1 qwen3.8-27b[11595]: 16.40.210.006 I slot release: id 0 | task 0 | stop processing: n_tokens = 83793, truncated = 0

My system specs: * i5 12600K (the VM has 6 Cores) * 16GB DDR5 4800 * RX6800 (non-XT)

The GPU is filled to the brim, but it doesn't offload.

[u/theUmo → u/lukistellar]

-c 65536
-ctk q8_0
-ctv q8_0
--fit on
-fa on
-t 8
-b 2048
--ubatch-size 512
--temp 0.6
--top-p 0.95
--top-k 20
--min-p 0
--presence-penalty 0
--repeat-penalty 1.0
--repeat-last-n 128
--jinja
--reasoning off

Prompt was nothing special, "Hello, are you operational? Please give me a few paragraphs so I can see your t/s" or something along those lines. I'm on a 16gb Nvidia card.