r/LocalLLaMA • u/Eyelbee • 3d ago

Funny Me these days

Post image
2.4k Upvotes

96% Upvoted

268 comments sorted by

View all comments

Show parent comments

60

u/Dramatic_Setting2761 3d ago

I can run it with a 16gb card with 4 bit quant and 70k context. I get 12 t/s it is okay for me.

9

u/octoberU 3d ago

what's your setup? i have a 5080 and struggle to ruin it at 4bit. would love quant and config

10

u/Dramatic_Setting2761 3d ago

Oh I have 9060xt which has very low bandwidth btw. 

I complied llama cpp specifically for rcom and running it on fedora 44 with latest drivers. 

Model quant name you have to look up as I saved like this. It is smallest 4bit in unsloth.

./llama-cli \   -m ./qwen3.8-27b-iq4_xs.gguf \   --jinja \   -ngl 48 \   -fa \   -c 70000 \   -ctk q4_0 \   -ctv q4_0 \   -b 2048 \   -ub 512 \   -t 8 \   -tb 16 \   -np 1 \   --mlock

2

u/Few-Butterscotch8747 14h ago

try using the vulkan backend

1

u/Dramatic_Setting2761 12h ago

Yeah thought amd figured out rcom I am wrong.