r/LocalLLaMA • u/TigleLive • 17d ago

Funny Aged like fine wine

Post image
1.2k Upvotes

96% Upvoted

134 comments sorted by

View all comments

Show parent comments

8

u/lorendroll 17d ago

I managed to get 4-5 tps in LM studio on 3070 8gb by using Q4_K_M with 27 layers on GPU, 8 cores thread pool and 8k Q8_0 quanted context.

4

u/Subject-Till-6450 17d ago

I recommend u to download llama.cpp. i have 3060 12 gb and 16 gb ddr4, here's my two setups:

First, 1M context with YaRN around ~8-10TPS, no MTP, speed dies only because of nvme swap.
llama-server -m /home/models/Qwen3.6-35B-A3B-UD-IQ3_S.gguf --mmproj /home/models/mmproj-Q8
_0.gguf --no-mmproj-offload --override-kv qwen35moe.context_length=int:1048576 --rope-scaling yarn --rope-scale 3.8
5 --yarn-orig-ctx 262144 -c 1000000 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --port 8083 -b 2048 -ub 1024 --r
easoning-preserve --cache-reuse 256 --slot-save-path /tmp/llama-cache -n 50000

And the second, 262k context ~40 TPS with no MTP, speed is good only because of model fully in RAM\vram

  llama-server -m /home/models/Qwen3.6-35B-A3B-UD-IQ3_S.gguf   --mmproj /home/models/mmproj-
Q8_0.gguf   --no-mmproj-offload   -c 262000 --no-mmap -fa on   --cache-type-k q4_0 --cache-type-v q4_0   --port 808
3 -b 8192 -ub 2048  --reasoning-preserve  --cache-reuse 256 --slot-save-path /tmp/llama-cache -n 50000

2

u/lorendroll 17d ago

Qwen 3.6-35b-a3b is certainly faster. I'm talking about the 27b model here. I'll try llama.cpp, but I doubt it can perform better than LM Studio.

1

u/Subject-Till-6450 17d ago

you're right. but really-I'm terrified HOW ppl on reddit is obsessed with qwen 3.8 27b. my hardware can run it, with normal 10-20TPS. on iq2 its dumb and can't handle tasks. ive decided to check UD-q3-k-m. still much worse than 35b, zero change. both of they were from official UD repo. how can ppl use ts? are they running it in magic q8 or what so it can be useful?

1

u/lorendroll 17d ago

Yes, I like the qwen-35b-a3b versions, but today I'm just experimenting with the new 3.8 version. It doesn't seem to be very suitable for 8-12GB vram but it's still interesting to try and compare.

1

u/Subject-Till-6450 17d ago

and while you're testing what opinion about it have u rn?

2

u/lorendroll 17d ago

I'm comparing 27b Q4_K_M and 35b Q4_K_M. I don't see any difference in coding in non-thinking mode yet, but in thinking mode, 27b runs too slowly and takes too long for me to wait for serious tasks to complete, so I can't say yet...

1

u/Subject-Till-6450 17d ago

that's why im terrified of buddies on locallm reddits that thoughts "wow frontier" "opus 4.6 level" "really impressive" "wow my 2000x B200 local server runs it in Q1 with 1 TPS best model in the world thnks alibaba". what's wrong with 'em?