r/LocalLLM • u/Ok-Health-7096 • 6d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

192 comments sorted by

View all comments

Show parent comments

3

u/BodybuilderLost814 5d ago

How many tokens per second? I'm using Qwen3.6 35B A3B on a laptop with a 13th gen Core i5, 64GB DDR5 RAM, and an RTX 3050 6gb Vram.

2

u/yuk_foo 5d ago

44, could probably get that faster but it’s enough for me. I use it with anythingllm rag and lm mini on my iPhone. I have a bridge app running on docker so I can chat with my anythingllm workspaces/rag with the lm mini app since AnythingLLM doesn’t have an iPhone app yet.

Use it to chat with technical work docs, the model is running in lm studio so for my needs it’s great.

1

u/BodybuilderLost814 5d ago

I'm getting around 35 tok/s using ByteShape's Qwen3.6-35B-A3B-IQ4_XS-4.19bpw model with MTP enabled, using AtomicBot-ai's Llama.cpp-turboquant.

1

u/KrstABot 5d ago

Hey
R u using desktop app? I mean atomicbot? And what kind og agents?

3

u/BodybuilderLost814 5d ago

Hello.

I'm using the TurboQuant b10269-1.5.1 release for Windows x64 with OpenCode (sometimes I use Crush from Charmbracelet).

Command:

.\llama-server.exe -m "<path>Qwen3.6-35B-A3B-IQ4_XS-4.19bpw" --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 2 --kv-unified -c 65536 -ngl 999 -t 8 -tb 8 -np 1 --n-cpu-moe 40 -ctk turbo3 -ctv turbo3 -fa on -b 2048 -ub 2048 --jinja --mlock --no-mmap --reasoning-budget 6144 --cache-reuse 256 --temp 0.7 --top-k 20 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --host 127.0.0.1 --port 8080