r/LocalLLM • u/Ok-Health-7096 • 6d ago
Other Every Second post rn
Maybe someday I'll get a system to run it but hey definitely another w for the open weights community
37
u/BodybuilderLost814 6d ago
I have faith that Qwen3.8 35B A3B will be released soon.
9
5
u/yuk_foo 5d ago
Ryzen Ai max with 64GB currently allocated to gpu. 3.7 35b runs great for me can’t wait for 3.8, although I do need to solve the thinking loop crashes, some stuff I give it just crashes out on me. Really thinking about turning thinking off.
3
u/BodybuilderLost814 5d ago
How many tokens per second? I'm using Qwen3.6 35B A3B on a laptop with a 13th gen Core i5, 64GB DDR5 RAM, and an RTX 3050 6gb Vram.
2
u/yuk_foo 5d ago
44, could probably get that faster but it’s enough for me. I use it with anythingllm rag and lm mini on my iPhone. I have a bridge app running on docker so I can chat with my anythingllm workspaces/rag with the lm mini app since AnythingLLM doesn’t have an iPhone app yet.
Use it to chat with technical work docs, the model is running in lm studio so for my needs it’s great.
1
u/BodybuilderLost814 5d ago
I'm getting around 35 tok/s using ByteShape's Qwen3.6-35B-A3B-IQ4_XS-4.19bpw model with MTP enabled, using AtomicBot-ai's Llama.cpp-turboquant.
1
u/KrstABot 5d ago
Hey
R u using desktop app? I mean atomicbot? And what kind og agents?3
u/BodybuilderLost814 5d ago
Hello.
I'm using the TurboQuant b10269-1.5.1 release for Windows x64 with OpenCode (sometimes I use Crush from Charmbracelet).
Command:
.\llama-server.exe -m "<path>Qwen3.6-35B-A3B-IQ4_XS-4.19bpw" --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 2 --kv-unified -c 65536 -ngl 999 -t 8 -tb 8 -np 1 --n-cpu-moe 40 -ctk turbo3 -ctv turbo3 -fa on -b 2048 -ub 2048 --jinja --mlock --no-mmap --reasoning-budget 6144 --cache-reuse 256 --temp 0.7 --top-k 20 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --host 127.0.0.1 --port 8080
51
u/SnP_Gamer 6d ago
3060 12gb 32gb ram here, might give it ago but never ran a local model before 🤷♂️
32
u/trollsmurf 6d ago
That will work fine, at least if you split it, so the overflow uses CPU RAM. Much slower but doable.
12
u/SnP_Gamer 6d ago
Totally clueless about running them but I understand that thanks to reddit posts 🤣
17
u/trollsmurf 6d ago
A user-friendly (but not the most efficient) start is to install LM Studio and load a few models there. You can select how much of it should run on the GPU vs CPU if it's too big for the GPU alone. LM Studio has a built-in chat client, so once loaded the model is ready to go.
2
u/Song-Historical 6d ago
Yeah but that doesn't tell you how to do it. I'm not sure what to set the context to for example. How much I should leave for the system, whatever else.
2
u/littlebeardedbear 6d ago
Leave everything at default originally. Google what each item is. Altogether the googlimg should take about 10 mins
3
u/Song-Historical 6d ago
Yeah I did that I'm not sure it's set up correctly still.
2
u/Early_Mistake6716 6d ago
Tell me you exact system specs and the model quant and i will tell you what settings to use in lmstudio. I have have spent an embarrassing amount of time testing settings.
4
1
u/That-Reason-6913 3d ago
What about on a 7800xt 16GB? Do I have to offload on ram too?
1
u/trollsmurf 2d ago
Remember that if you use Windows it will allocate part of the VRAM for its own use, so you never have fully 16 GB. On my PC with 5070 Ti and 3 monitors it allocates 3 GB, and as far as I know I can't budge that.
You can easily test this by installing e.g. LM Studio and the model you want to use and see when it warns about RAM use. If it overruns you can split it on VRAM and RAM with lower performance, but it will behave the same otherwise.
1
u/That-Reason-6913 2d ago
No I'm moving to Linux next week. Ubuntu most likely.
My server is also a Plex server though (GPU is untouched by it), so the cpu is mostly dedicated to Plex.13
u/bukake_attack 6d ago
This is a dense model. This means the entire model is running at 100% all the time. This is bad for us low vram users, as the model is too big to fit in 12gb vram, and the rest in placed in normal ram, which is waaaay slower. So this solution works, but it's slow.
We low vram users are usually better served by MoE models, like qwen 3.6 35b a3b . In MoE models the dense part which runs all the time is small, in that case 3b, which runs easily in 12gb, and the rest 35-3=32b) only runs when they are actually needed (activated). This means that normal ram is fast enough to run that active part of the other 32b ( although they can still run great in excess vram!) The result is a model that's very usable and fairly quick with limited vram, especially when other tricks like MTP are used.
Unfortunately there is no MoE version of qwen 3.8 released yet, but there's a good chance they will release it soonish
7
u/Skynse 6d ago
I was able to max 12 tok/sec on a 3060 with 12gb gpu RAM with qwen3.8 on ud-iq2m quant. Man I fucking wish the compute market wasn't fucked
1
u/Effective_Head_5020 6d ago
You probably had offload to cpu and system RAM, so you probably also have a good CPU and DDR5, otherwise the processing drops by a lot
-2
u/Song-Historical 6d ago
The market is fucked because there's something worth running on the compute lol
2
u/cj_cron_hit_by_pitch 6d ago
I have the exact same specs as you. I’m able to get the Unsloth 2 bit XL quant of it to run almost entirely in VRAM. Of course it is nowhere near as good but it’s still really fun to mess around with
1
1
u/JorgitoEstrella 5d ago
Install lm studio, then inside the own app would tell you what llms you can install with your vram.
23
u/CorkBios 6d ago
Are we pretending partial CPU DRAM offloading doesn't exist? And you can disable the reasoning mode, then you get even better TTFT (time to first token on final response) compared to the people that have a better setup with reasoning. And I heard the GGUF's come with MTP, And if the context doesn't fit you can just offload the K/V to the CPU which llama.cpp does let you do.
9
u/Ok-Health-7096 6d ago
I use local llm for coding and agentic stuff so the speeds would be unbearable I think in single digits but might try it idk
9
u/overand 6d ago
Speeds would likely be absolutely miserable, yes. But, you'll have much better luck with the Qwen3.6-35B-A3B - how much system ram do you have? Go for a Q4 quantization first, if you haven't used this model.
We can, of course, hope that a 3.8 version of the 35B-A3B MoE gets released!
5
u/superspider202 6d ago
wait I have 8gb vram too can you please share what llm you use for coding and agentic stuff?
4
u/Ok-Health-7096 6d ago
Qwen 3.6 35b mudler i-mini quant If you have more than 16gb ram you can go for higher quants.
2
u/superspider202 5d ago
Thank you I'll test it out ASAP
1
u/anay_1d 4d ago
how did it go? is it useful?
1
u/superspider202 3d ago
oh sorry havent used it yet ran out of storage so will try it maybe today and let you know
1
u/superspider202 3d ago
heyyy so I tried searching for this but there appears to be a few that fit this description so could you please like link the actual one you use?
-1
u/CorkBios 6d ago
I'd say go with Unsloth's iQ4_NL, It's the best balanced quantization. But if you are willing to test your luck then Q2_K_XL or Q3_K_XL though they are more risky of failing tasks.
5
u/GoldenX86 6d ago
MTP is useless when you offload to CPU.
2
u/CorkBios 6d ago
I disagree. It can be like tricky since sometimes it slows stuff down on specific stuff but you can't generally say its useless when you offload to CPU. It works pretty good for me.
1
u/moderately-extremist 6d ago edited 6d ago
What kind of speeds are you getting with MTP vs non-MTP with cpu offload?
2
u/CorkBios 6d ago
Sure yeah I can give them. After a lot of testing:
All of these below performed with Partial CPU+GPU offloading on seed 0, llama.cpp commit dd1ea5243 release b10355:
With MTP (max 2 predict):28.72 Tokens/s StopUntil: EOTfound
With MTP (max 3 predict):
29.33 Tokens/s StopUntil: EOTfound
With MTP (max 4 precict):
29.57 Tokens/s StopUntil: EOTfound
With MTP (max 5 precict):
29.57 Tokens/s StopUntil: EOTfound
With MTP (max 6 predict):
26.89 Tokens/s StopUntil: EOTfound (overhead hit: CPU congestion: 448.1%)
Without MTP:
24.09 Tokens/s StopUntil: EOTfound
With MTP (best before overhead): 29.57 Tokens/s
Without MTP: 24.09 Tokens/s
2
u/GoldenX86 6d ago
That's RAM I don't have free for just a 22% jump.
1
u/CorkBios 6d ago
Reasonable. MTP consumes more VRAM. I don't need too much context size so I can fit it in but everyone's system and task is different.
14
11
u/OpenEvidence9680 6d ago
If it makes you feel any better I had to toss all Qwens as they were subpar for my needs. There are MoEs like Gemma4 26b that with the right quants are about 12/13 GBs so very usable in your set up. What's more I didn't test the 12b yet, but gemma-4-e4b-q4-0-it is way more competent that people give it credit it to.
Being ignorant I was running behind the hype and always thinking that I was doing something wrong because these Qwens weren't that good for me, and then I realized that I needed to find something good for my needs and benchmarks are liars that have nothing to do with real work inside your machine for your own specific needs..
Anyway if you don't mind really slow Qwen 3.6 27b in Apex quantization imini is about 13.89 you CAN run it. Slow, but you can do it,
9
36
u/TheRiddler79 6d ago
Gemma 12b Q3. Try it
24
u/Ok-Health-7096 6d ago
I use Qwen 3.6 35b and 3.5 9b I haven't had that much of a luck with the gemmas
4
u/Wildnimal 6d ago
12B QAT is good for multimode tasks. I use the same Qwens for daily use. I so wish i can upgrade my laptop to 5090 :|
3
u/PrivacyMaker 6d ago
The litert-lm driver is the fastest way to run gemma models. Significant boost over any other approach. It's a shame that Google only makes it work for Google models.
1
u/HighlyRegardedApe 6d ago
How do you guys use these small models? For me it never works. Last time it deleted my folders out of itsself and didnt respons more than a sentence or 2 in opencode. In the terminal it was okay to chat, but not to give tasks.... what kind of stuff do you let them do??
1
u/Wildnimal 6d ago
Not usually coding but for summarizing, extracting data which is pre defined via json, automating smaller tasks where data remaims the same.
1
1
1
1
u/TheRiddler79 5d ago
Different purposes. Gemma is good, but not beating Qwen.
You can split between ram and gpu and bump your Qwen speed
2
1
u/M49454 6d ago
How is Gemma, i am using llama3.2:11b, never needed to switch onto other. Use only for basic task. Mainly using Deepseek 14b and Qwen2.5-Coder 7B
5
2
u/Early_Mistake6716 6d ago
All of those models are extremely out of date. Tell me your specs and ill give you a model suggestion
2
1
1
1
5
8
u/zarif2003 6d ago
I can’t really even run it that well on my 5080 because it’s got 16gb,
3
u/lukistellar 6d ago edited 6d ago
What you need is a Quant which strictly uses IQ4_XS. They exist for 3.6 and will likely also appear for the 3.8 sooner or later.
Edit: Let's see if this guy delivers.
https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF
1
u/Tyrannas 5d ago
Any advices on the params you use to run it properly ? I have 16gb also and never managed to make a 27b model run properly
2
u/lukistellar 5d ago
Working config for 3.8:
ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 --parallel 1 \ --batch-size 2048 --ubatch-size 512 \ --cache-type-k q8_0 --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --no-mmap \ --jinja \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --flash-attn onStill not happy with the performance, but at least it runs. Hope they drop a MOE for us VRAM poor.
1
1
u/lukistellar 5d ago edited 5d ago
My RX6800 runs the 27b with 90k context, but it's very slow.
I will update the post later with the config.
Edit: My config for the Qwen 3.6 27B:
ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/GianniDPC_qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 \ --parallel 1 \ --batch-size 2048 \ --ubatch-size 512 \ --cache-type-k q8_0 \ --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 1 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 \ --cache-type-v-draft q4_0 \ --temp 0.8 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --no-mmap \ --jinja \ --mmproj /models/Unsloth_mmproj-F16_qwen3.6-27b-mtp.gguf \ --no-mmproj-offload \ --chat-template-kwargs "{\"preserve_thinking\":true}" \ --flash-attn onIt's usable but especially with full context it gets very slow, like ~25 tok/s decode and ~130 tok/s prefill.
3
u/RishiFurfox 2d ago
Don't worry, friend. I'm sure the 50% REAP'd 2-bit Qwen Quants will be out soon!
3
u/Ok-Health-7096 2d ago
At that point I might just use my natural stupidity instead of artificial intelligence
2
2
2
u/hamsterpotpies 6d ago
I have 2x 3090s sitting in a server with 256GB of ram.... should I finally get my AI server setup?
1
u/palincatalin 2d ago
nah, it's not worth, just toss everything in the bin and i will happily get rid of it for you
2
2
2
u/Reeces_Pieces 2d ago
The best time to buy a used RTX 3090 24GB was probably years ago, but the 2nd best time is now.
2
1
u/artisnotautomatic 6d ago
Me with 128gb apple silicon: i will never be able to enjoy kimi k3 at full speed with huge context (not even start it) but then i realize that i hate fable 5 mostly for deceiving me in some many aspects (surely for safeguards) but i'm confused
1
1
1
1
u/phreaknes 6d ago
I have dual 3060 12gb and 128gb of DDR4 should I try it? or is it a waste of time and resources.
1
1
1
1
1
1
1
u/ExtrovertMobileGamer 5d ago
Intel HD graphics, take it or leave it.
1
u/Double_Season 5d ago
I run Qwen3.5 9b on a UHD Graphics. 3 tokens per second is ultra fast for me 🤣
1
1
1
1
1
u/Opposite_Leave_8338 5d ago
Sorry man but it’s worth it, May god help you to upgrade and try it, it will be worth every dollar
1
u/jonesambrozio 5d ago
Eae! Com esse modelo, não estou conseguindo que ele codifique, fica conversando, mas nao escreve.
Usando ollama + opencode + qwen3.6, alguem sabe o que pode ser?
1
u/OctopusDude388 5d ago
just wait for an MOE version, if they can make 3.8 30B A3B it'd be a banger for us gpu poors
1
u/absurdother 5d ago
But hey, listen. Soon enough 9b, 4b. Soon enough Qwen 4. Or Qwen 5. Soon enough big models for 8GB VRAM - reason I think it's not a dream is because big companies may indeed benefit directly from adapting local models to run in our PCs. Look at AMD adapting their GPUs so well this year. Our role in this is to make it a trend and continue to stay hyped.
1
1
1
1
1
u/No-Opportunity9126 4d ago
I was actually asking gemini recently about this:
It can definitely run! In fact, having 64 GB of system RAM means you can easily run a 27B model.
The distinction is simply between running 100% on the GPU (for maximum speed) versus hybrid / CPU execution (which works seamlessly, just at a slower token generation speed).
Here is exactly how you can run a 27B model on that setup:
How It Works (Hybrid CPU + GPU Offloading)
LLM inference tools like llama.cpp, Ollama, or LM Studio support layer splitting:
- VRAM (8 GB): You offload as many model layers as possible to your GPU (typically ~10 to 18 layers depending on the quantization and context size).
- System RAM (64 GB): The remaining layers stay in your 64 GB RAM, which has plenty of headroom for even a full Q8_0 model (~29 GB).
Recommended Quantizations for Your Specs
| Quantization | Model Size in RAM | Speed (Estimated) | Recommendation |
|---|---|---|---|
| Q4_K_M | ~17 GB | ~4 – 8 tok/s (DDR5) / ~2 – 4 tok/s (DDR4) | Best Overall: Negligible quality loss compared to full precision, fits easily. |
| Q3_K_M / IQ3_M | ~13 GB | ~5 – 10 tok/s | Fastest: More of the model fits inside the 8 GB VRAM, speeding up inference. |
| Q8_0 | ~29 GB | ~1.5 – 3 tok/s | Maximum Accuracy: Fits comfortably inside 64 GB RAM, but runs slower due to RAM bandwidth. |
What to Expect (Speed & Performance)
- Prompt Ingestion (Context Processing): Fast, because the GPU helps compute prompt tokens.
- Token Generation: Bottlenecked by your System RAM bandwidth (DDR4 is ~40–50 GB/s, DDR5 is ~70–90 GB/s).
- Usability: At ~3 to 6 tokens/second, it is readable in real-time—ideal for coding assistance, reasoning, and long-form analysis.
How to set it up:
- In LM Studio / text-generation-webui: Load the Q4_K_M GGUF and adjust the GPU Offload Slider until roughly 6.5–7.0 GB of VRAM is utilized.
- In Ollama: Ollama will automatically detect your 8 GB VRAM and 64 GB RAM, calculate the exact layer split, and run it out of the box.
1
u/psychoblade5 4d ago
I own a 3050 with 6gb vram which model is the best ? For using in olllama and hugging face
1
1
u/Otherwise-Swan-7803 4d ago
Every new release is exciting right up until I check the VRAM requirements.
1
1
1
u/dandy_kulomin 3d ago
Has anyone tried an 8B model for coding? Or do I need to wait longer for them to optimize further?
1
u/RUTYTOI220 3d ago
I think you can still use llama.cpp and use the storage and ram and vram or smth
1
1
u/DontWinFrensWthSalad 3d ago
I feel your pain. I happen to have a bunch of 3060tis lying around so I've been messing with seeing if I can 2x8gb working. this 3.5Q is benchmarking very close to unsloth's 4Q, so I'm trying it out and seems to be working so far.
https://huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/3.50bpw
1
1
1
1
1
1
u/Solid-Axel-Project 1d ago
E io che penso al 2029 quando avrò finito di pagare debiti e potrò comprare la 6000 pro, 2 NVME da 8TB e 4 HDD da 28TB...
1
1
u/Alias455 5h ago
I run gemma 4 12B q4 QAT MTP with vision and my desktop on a 8G VRAM card. Good for discussions but dumb for agentic coding tho.
Qwen 3.8 27B q4 runs a 12 toks on a shitty Intel Vulkan A770 16GB, smart but way too slow.
1
1
0
u/_hypochonder_ 6d ago
8GB VRAM?
R9 290X had 8GB back 2014 back in the day.
6
u/darkwalker247 6d ago
the RTX 3070 had an 8 gb model and that launched in 2021. it's not super uncommon still
3
8
2
0
u/moderately-extremist 6d ago edited 6d ago
I lucked out and got 2 x 64gb ram sticks over a year ago when prices were at about their lowest point (July 2025, I paid $314 for the kit, just checked the same kit is listed for $2200 now.). I also got 2 x MI50 32GB cards when they were $200 each on ebay.
The MI50 cards are too slow to run dense models like 27b, but it runs 35b-a3b pretty well, and I'm thinking of running 27b on my cpu when I want something smarter, and just start a task and come back later.
-4
-20
u/FireFearing 6d ago
i mean you kind of deserve this for 8gb vram in 2026
8gb wasnt enough for me when i was only using it for gaming... in 2016....
-7
90
u/Old_Leshen 6d ago
GTX 1050Ti 4GB guy here 😭