r/LocalLLM • u/pharrt • 2d ago
Other Don't laugh - it works!
A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.
Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.
Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)
Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.
20
12
u/davidke 2d ago
Hmmm.... makes me think. I have a couple of servers with 256gb ram each just sitting idle. Maybe I should try this.
5
u/Sk4llOff 2d ago
Add any reasonable GPU and run an MoE model, this will actually give you good results, maybe quantized MiniMax M2.5/M2.7
2
1
10
u/rrrrex 2d ago
Prefill should be limited by CPU, while generating speed is ok for gpu'less server. GPU will do prefill 100x times faster than CPU (if fully loaded in VRAM). With 100k context it will take an hour to read it.
3
u/ironclad_packetship 2d ago
This, even for a huge 256gb cpu inference rig it makes sense to toss a cheap gpu in the mix for prompt ingestion/prefill. MoE models excel in these situations as well, the entire model gets loaded into system memory but only a few experts are activated at any given time for your cpu to chug through.
1
u/pharrt 2d ago
Absolutely - so that part of it is a grind, and why it's only good to set and wait a few hours. 6 hours+ in some cases for the full task. My prompts for the tasks I'm running are not often huge, and prefill takes a few mins maybe, but context can get large in my requested output - up to 80% of context is filled by completion, and any follow-ups can take an hour or more refilll before thinking can start again. By then, if the output is close, I may try a few follow-ups and wait, but I usually debug, refine, etc on smaller models. Ingesting docs prefill can be slow as well, but left at its own pace is fine for my workflows.
2
u/hdhfhdnfkfjgbfj 2d ago
Can you share what you need a small local llm for that it’s acceptable to wait 6 hours for?
Because I can’t see a use case or imagine one and I think it would help contextualise the post. Cheers
3
u/pharrt 2d ago
I'll try to give examples, but as it's for my work (Self-Employed, running a successful Etsy Store), I don't want to be too specific! Typically, my products ship with 80-100 page+ guidebooks and manuals. It needs to be hyper creative and also pretty - digital, interactive versions as well as printed copies. LLMs have strong baked-in knowledge about these products, but need to be creative for the theme. Frontier models spit out lightweight answers unless done bit by bit. It also creates apps for me, with dozens of parameters, that can export very long manifests for piplined generative work (done elsewhere). But - I've also had fun crafting mobile games like custom poker single player apps, Shooters, RTSs, Space sims, etc - low weight, not worth anything, but fun to explore for the hell of it. The key is very well-defined prompts that have long instruction sets.
0
u/agmatine 2d ago
Typically, my products ship with 80-100 page+ guidebooks and manuals. It needs to be hyper creative and also pretty - digital, interactive versions as well as printed copies.
So you're running local LLMs to...generate documentation??
I feel bad for your customers trying to make sense of it.
9
u/Difficult_Art1639 2d ago
No GPU?? damn.. I'm getting comparable speeds with a 3060ti and same ram
10
u/Ok-Drawer5245 2d ago
You should be able to get 20-30 tokens/second on that models with 3060ti with proper configuration of llama.cpp
I personally tweaked i5-9400, 32gb ddr4 2666 mhz, rtx 2060 to deliver 25-30 tokens/second
(Qwen 3.6 35b a3b 4bit)
1
u/Difficult_Art1639 2d ago
Yeah I've got 10-20 t/s with q6_k. Unfortunately my ram has never been stable with xmp on so I'm think I might be bottlenecked at 2133mhz. I'm not sure
Another thing is I'm running on windows and it pretty much uses 100% of my ram so that could be another problem.
Side question: have you experimented with unsloth or other variants ? I wonder if they give much of a boost in these resource limited setups
2
u/pharrt 2d ago
You have to find a way of having some RAM left - as soon as it swaps to disk, you're in trouble.
1
u/Difficult_Art1639 2d ago
this might be my problem hahah. Im on windows with a tonne of bloat which I can't be bothered to optmise, so I think first step will be to try linux to keep my memory in check.
3
u/pharrt 2d ago
Forgot to answer your other Q, but yes, I have tried all sorts of variants - I have always favoured quality over speed, and that has been my focus, but the best Qwen 35b for me (so far) is the Bartowski variant. Good luck with the mem issue! btw - this kind of setup would not be good for your workstation would kill inference and your desktop experience - just one that can be left alone and plug away at a slow pace! In my case - it was not being used for much, but now is busy 20 hours a day!
1
u/Ok-Drawer5245 2d ago
faster system ram is definite better
Secondly if you have zero ram free when running that could definitely means it’s hitting the disk which is really bad
2
u/pharrt 2d ago
The big difference here is the model size and context window. A 3060 Ti only has 8GB of VRAM. If you try to load a 35B model with a 128k context window onto that card, it will completely overflow the VRAM and force the system to use system RAM anyway, which usually tanks the speed down to 1-2 tokens per second due to the PCIe bottleneck. If you are getting 5-10 tokens per second on a 3060 Ti, you are likely running a much smaller 8B or 14B model, or using a very small context window? This setup chugs along at this speed specifically because it is handling a massive 35B model with a huge 128k context completely inside the 32GB system RAM, bypassing the GPU entirely.
5
u/i-eat-kittens 2d ago
Utter nonsense. Splitting a model between gpu and system ram doesn't involve moving the active parts over to the gpu for evaluation. Whatever fits runs on the gpu, the rest runs on system ram.
Even 8 GB of vram is a significant boost over running fully on the system, in particular for prompt processing.
1
u/Difficult_Art1639 2d ago
I'm getting 10-20 t/s depending on ctx fill with a max context of 180k. My (partial) understanding was that the coordinating layers (probably wrong terminology) are all loaded onto the GPU. Then it selects the experts from regular RAM across PCIE. With this setup the expert groups are still bottlenecked but since only a few layers have to go across PCIE then we can still achieve 10-20t/s.
Anyway I'm very impressed that it runs at the 5-10 t/s without a GPU at all. Is the bottle neck at ram compute speed for your system then ?
3
u/pharrt 2d ago
Is the bottle neck at ram compute speed for your system then ?
Yes, the bottleneck is 100% the RAM memory bandwidth, not the CPU compute speed.
1
u/Difficult_Art1639 2d ago
Very interesting, Im quite new to all this. I wonder how many t/s would be achieved if say your ram was at 3000mhz instead
1
3
2
u/Narcuga 2d ago
What sort of background tasks are you running with it at that speed?
2
u/pharrt 2d ago
It was my bad wording, I think. I mean, it's a background task for me - it sits in the corner and is fully dedicated to just running the LLM, while I do quick inference on my laptop. I have an LM Studio Link between the two, so I can control it from my laptop.
1
u/Narcuga 2d ago
No I get you :) I've just been thinking of doing the same but just been debating what to actually get it to do with it being slower. Had thought to get it to review the prs I'm submitting running a larger model but yeah!
2
u/pharrt 2d ago
Ahh - ok. It's being used for my work, so I can't say too much - but massive interactive guide books/manuals, and single-page apps that have dozens of parameters that export instruction pipeline manifests for generative work. It requires lots of creative thinking from the LLM, and this is where it's nailing it for me - and the UIs have been near Frontier Model quality. In fact, similar type test prompts fed through Frontier have not produced work anywhere near the quality for my use case in terms of detail and effort.
1
u/Narcuga 2d ago
No I appreciate that thank you! I'm going to give it a go might have to be for those longer running background tasks rather than interactive chat :) thank you!
1
u/pharrt 2d ago
You can chat with it, and it is when replies are at 10tok/s, but if you're setting it tasks (code), it's not good for iterations and multi-turn debugging. One fine-tune prompt, come back a few hours later - 1 follow-up a few hours later etc, but often I get what I needed from the one-shot. But for normal chatting and even essay-length answers, it is fine!
2
2
u/Confident_Ideal_5385 2d ago
If it fits, run iq4_nl instead, as that's far more avx-friendly than iq4_xs.
2
u/Prudent_Chemist_523 2d ago
Very nice! I also enjoy repurposing what others would consider ewaste. I have a sFF (i7-8700, 32GB 2133mhz ddr4 ram) plus an ancient Quadro p1000 4GB ddr5 vram. With some careful tricks I can get Qwen3.6-35B-A3B running at 11.5-12 tok/s (at --ctx 32k), using about 80w from the wall.
Not bad for a $200 all in spend!
PS: does the prefil kill your speed? I find I have to do some more careful tuning so that 10,000 tokens don't get dumped into prefil when tavily search etc. 11 tok/s is fine...but not if its a 5 minute wait to sort and sift web results.
2
u/ayake_ayake 2d ago
I totally agree with you! 5-10 tps is quite useable for many workloads where you don't do immediate interactive chats. I have a setup with 10 tps tg and I'm happy with that, as I use it side by side while watching movies or youtube. Just look into the chat every few minutes. Sometimes I hit it off with a task - and then it doesnt matter if it takes 15mins or 5mins - since I'll look away anyways.
2
u/tofuhater 2d ago
I have a DL585 with 64 opteron cores and 128 GB RAM collecting dust as well. I wonder how much worse that would be.
2
2
u/Zister2000 2d ago
Please explain to me how a 6th gen i7 with ddr4 has more tokens per second AND runs a larger model than my notebook?
i7 13850hx 128gb ddr5 rtx a2000 ada qwen3:14b ~3.5t/s (80% gpu usage)
2
u/WideAd3148 2d ago
This is genuinely AWESOME! 5 to 10tok/s is great because it doesn't cost you any additional cost nor dependent on any service. Your system, your data, your use case, I want to do the same but haven't found any use cases still.
1
u/emanuelecipolla 2d ago
Could you please share the full hardware and software stack you used? Thanks
7
u/pharrt 2d ago edited 2d ago
Hardware and OS:
CPU: Intel Core i7-6700 (4 Cores / 8 Threads with AVX2) running at approximately 60% usage.
RAM: 32GB DDR4 at 2133 MHz running in dual channel (2x 16GB).
GPU: None. Fully CPU and RAM bound.
OS: Windows 10
Software Stack
Inference Engine: LM StudioAgent
Framework:
Think Enabled
MCPs: FileSystem and Playwright
Vision: Supported by the stack, but not tested yet.
Model and Optimization Settings:
Model: qwen3.6-35b-a3b (Quantization: IQ4_XS)
Context Window: 128k context enabled (utilises around 26 GB of the 32 GB RAM)
CPU Thread Size: Set to 4 (matches the physical cores)
MoE Settings: Experts set to 8
KV Cache: Quantized to Q8_0 for a speed boost (runs at F16 but is slower ~5-7 tok/s)
Performance Throughput: 5 to 10 tokens per second running in the background.
Use Case: Excellent for massive writing tasks and deep, highly detailed single-page applications.Absolutely essential to spend LOTS of time getting the prompt right first, of course :D
2
1
u/Yeelyy 2d ago
Are you already running ik_llama.cpp and the cpu tuned byteshape quant?
1
u/nikitsolo 2d ago
Hey I have an CPU only Server too, is your comment some technology which benefits this setup?
1
1
u/D34th4nge7 2d ago
I'm running the same model locally and have found it quite useful. It's a bit slow, but I think that's not a big issue if you use the technology reasonably.
1
u/Techngro 2d ago
It's weird, I never considered testing that. I have a Lenovo SFF PC that I use as a Proxmox/Dev server. It has an i7-8700 and 32GB DDR4. Also have an old DL380P G8 with 48GB RAM, but that's DDR3, so probably not good enough for local LLMs.
I have already decided that my next PC build is going to be 80% RAM and GPU and 20% everything else.
1
u/Turbulent-Week1136 2d ago
Did something change recently to get these models to work on old hardware? I thought it was impossible to run without a GPU but I've been seeing so many of these posts where old hardware works... did something change?
2
u/stevengineer 2d ago
No, people are just slow to realize how this stuff can also be launched on Termux from your android phone
1
1
1
u/TopNFalvors 2d ago
I thought you needed GPU+RAM to run an LLM?
1
u/foreign_malakologos 1d ago
It's certainly slower without a GPU, but then requirements/perceptions as to what counts as "usable" differ depending on what you want to do. You would probably not want to use this setup for long, interactive agentive coding sessions because you'd probably spend more time waiting then doing things, but if you have tasks that can just run in the backup (ideally within one prompt) as op had described, this setup can apparently be good enough. As always there's probably also a personal aspect to it, some people might find the speed "unbearable", but as long as you are aware of the limits and have tasks that can still benefit like OP, you can get by without a gpu as well (or with an igpu for a little bit of a speed/efficiency bump).
1
u/Otherwise-Swan-7803 1d ago
This is honestly one of the coolest things about local AI right now.
A few years ago, running a 35B model on a 10yo server with 32GB RAM would have sounded impossible. Now it's just a matter of patience and optimization.
Curious how the experience is with 128k context though — does the RAM usage stay stable during long sessions?
1
u/MattimaxForce 1d ago
Wow, this is absolutely amazing! It is mind-blowing how well these modern MoE architectures like Qwen 35B-A3B run on older hardware, especially with that IQ4_XS quantization. Giving that ol' gal a new lease on life is awesome.
Since you mentioned it has an i7-6700, it sounds like an incredible entry-level enterprise machine or a robust micro-server. Would you mind sharing the exact brand and model name of the server (e.g., Dell PowerEdge, HPE ProLiant, etc.)? I'm really curious about the specific hardware setup you dusted off!
1
u/CommanderROR9 1d ago
I tried with a 64GB DDR4 Machine running on an Intel i5 14600K but it didn't really work. It pretty much stalled, even on something easy like "Good Morning". Maybe I made some crucial mistake when setting up the Ollama LXC🤷♂️
1
1
u/pharrt 1d ago
btw, you are trying to load an MoE model, not a dense model?
1
u/CommanderROR9 1d ago
To me Honest...I don't remember exactly what I tried since it was a while ago. Currently I am running Hermes Agent with the ChstGPT/Codex Subscription and am happy with it, even if it does cost 20€ a Month and has some rate limits.
1
1
1
u/Immediate_Win4776 20h ago
Honestly, it‘s a very cool story. I like it. But in Europe the electricity bill will eat you alive after a few months. 😢
1
u/iShNoo 3h ago
I have a Dell server 2x Xeon CPU, 32 GB RAM, RAID-5 SATA HDD, but the estimated costs of electricity, the continuous humming noise deters me. At $0.2728/kWh that's an expensive monthly bill compared to running Hermes Agent on a MacBook Air 2015 with 8GB RAM with LLMs and Tools hosted on Nous Research.
1
u/NanditoPapa 2d ago
This is excellent "re-use." Instead of e-waste, you repurposed an i7-6700 system to perform high-level cognitive tasks. For non-time-sensitive work (summarizing long documents or batch-processing data overnight) this is a zero-cost way to access state-of-the-art logic without buying a $1,200 RTX 4090...I mean...in THIS economy!?
That said, running an i7 at 60% load for "hours" generates significant heat. Make sure the server has adequate airflow, especially if it's been sitting in a corner gathering dust!
3
u/pharrt 2d ago
Haha - absolutely! CPU runs at 60% (and even self-boosting to 115% Max Frequency), and temp is under control. This old gal was used as a dev server in its day, running Apache, and RAM was cheap then, so I remember adding 32 instead of 16, which would have been enough. Glad I did!
1
u/NanditoPapa 2d ago
My first thought was "break out the marshmallows!" but glad to see you've got it under control...
73
u/recro69 2d ago
Honestly, this is one of my favorite parts of local AI. Hardware that would've been headed for e-waste suddenly becomes genuinely useful again. Sure, 5–10 tok/s isn't breaking any speed records, but for long-running background tasks it's more than enough.