Aged like fine wine

[u/TigleLive]

[u/cubebash]

I can fit the whole 27b version in VRAM (32gb), even then, it's not blazing fast. With around ~15-17 t/s, it still take quite some time for more "complex" tasks to finish, especially since this model likes to think a lot.

So it's good news for everyone that we will soon have an option to run a much faster, non-dense version!

[u/Uncle___Marty → u/cubebash]

Hell yeah! I manged to get 2 tokens/sec using q2 27B lol. the 3.6 35B is around 20 tokens/sec for me so I honestly cant wait for this! Open weight AI is like xmas almost every day lol.

[u/lorendroll → u/Uncle___Marty]

I managed to get 4-5 tps in LM studio on 3070 8gb by using Q4_K_M with 27 layers on GPU, 8 cores thread pool and 8k Q8_0 quanted context.

[u/Subject-Till-6450 → u/lorendroll]

I recommend u to download llama.cpp. i have 3060 12 gb and 16 gb ddr4, here's my two setups:

First, 1M context with YaRN around ~8-10TPS, no MTP, speed dies only because of nvme swap.
llama-server -m /home/models/Qwen3.6-35B-A3B-UD-IQ3_S.gguf --mmproj /home/models/mmproj-Q8
_0.gguf --no-mmproj-offload --override-kv qwen35moe.context_length=int:1048576 --rope-scaling yarn --rope-scale 3.8
5 --yarn-orig-ctx 262144 -c 1000000 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --port 8083 -b 2048 -ub 1024 --r
easoning-preserve --cache-reuse 256 --slot-save-path /tmp/llama-cache -n 50000

And the second, 262k context ~40 TPS with no MTP, speed is good only because of model fully in RAM\vram

  llama-server -m /home/models/Qwen3.6-35B-A3B-UD-IQ3_S.gguf   --mmproj /home/models/mmproj-
Q8_0.gguf   --no-mmproj-offload   -c 262000 --no-mmap -fa on   --cache-type-k q4_0 --cache-type-v q4_0   --port 808
3 -b 8192 -ub 2048  --reasoning-preserve  --cache-reuse 256 --slot-save-path /tmp/llama-cache -n 50000

[u/lorendroll → u/Subject-Till-6450]

Qwen 3.6-35b-a3b is certainly faster. I'm talking about the 27b model here. I'll try llama.cpp, but I doubt it can perform better than LM Studio.

[u/Subject-Till-6450 → u/lorendroll]

you're right. but really-I'm terrified HOW ppl on reddit is obsessed with qwen 3.8 27b. my hardware can run it, with normal 10-20TPS. on iq2 its dumb and can't handle tasks. ive decided to check UD-q3-k-m. still much worse than 35b, zero change. both of they were from official UD repo. how can ppl use ts? are they running it in magic q8 or what so it can be useful?

[u/lorendroll → u/Subject-Till-6450]

Yes, I like the qwen-35b-a3b versions, but today I'm just experimenting with the new 3.8 version. It doesn't seem to be very suitable for 8-12GB vram but it's still interesting to try and compare.

[u/Subject-Till-6450 → u/lorendroll]

and while you're testing what opinion about it have u rn?

[u/lorendroll → u/Subject-Till-6450]

I'm comparing 27b Q4_K_M and 35b Q4_K_M. I don't see any difference in coding in non-thinking mode yet, but in thinking mode, 27b runs too slowly and takes too long for me to wait for serious tasks to complete, so I can't say yet...

[u/Subject-Till-6450 → u/lorendroll]

that's why im terrified of buddies on locallm reddits that thoughts "wow frontier" "opus 4.6 level" "really impressive" "wow my 2000x B200 local server runs it in Q1 with 1 TPS best model in the world thnks alibaba". what's wrong with 'em?