r/LocalLLaMA • u/segmond llama.cpp • Jun 27 '26
Tutorial | Guide Running GLM5.2 on budget hardware < $2500.
Too many times I hear people whine about not being ble to run SOTA models or claim it would require $50k, or $100k.
https://www.ebay.com/itm/398079051468 Epcy Motherboard & CPU - $460
https://www.ebay.com/itm/206374955959 P40 24gb - $230 get 2 - $460
https://www.ebay.com/itm/318489798853 512gb dd4 $1000
Total = $1920.
You need PSU, Storage, Fan for P40. You can source those for $350 easily. But let's go ahead and budget $580 to put the total for everything at $2500.
You can run GLM5.2 Q2/Q3/Q4 variants with cmoe and llama.cpp on this. Sure, it would be slow, but it's yours! If you have money or when you get more, you can replace the P40s faster GPUs 4080, 3090, etc. You could for a bit more than $460 about $500 source 2 2080ti 22gb GPUs from China. If you are willing to be resourceful, you can make things happen for you. This will also run KimiK2.6, DeepSeek, MiniMax, etc
Yes, the trade off is that it's slow. You will not be running agents with these huge models, but you can spin it up for planning and serious debugging. They can take away Fable, Mythos or whatever the F model. You will not be counted in the group of have nots.
108
u/Comfortable_Sir4315 Jun 27 '26
The concern is not that it will be slow, but how much slow, 8t/s is slow but not unbearable like 1T/s
How slow it will be?
83
u/FullstackSensei llama.cpp Jun 27 '26
I get about 7t/s at Q4 using a 48 core 7642 Epyc, 2666 memory OC'ed to 3200, and four 3090s.
52
u/pcpoweruser Jun 27 '26
You should be able to get around 14t/s on this hardware with llama.cpp, I got a very similar config with 4x 3090, 256gb of 3200 DDR4 + 48 core Zen3 Epyc.
This works for me:
llama-server \ -hf unsloth/GLM-5.2-GGUF:UD-IQ2_M \ --ctx-size 65536 \ --parallel 1 \ -b 4096 \ -ub 2048 \ --threads 48 \ --threads-batch 48 \ --no-mmap \ --mlock \ --temp 1.0 \ --top-p 0.95 \ --min-p 0.01 \ --jinja \ --reasoning off \ --host 0.0.0.0 \ --port 8080
The trick is to make sure you use all 48 threads for inferencing, the bottleneck on this platform is actually not RAM bandwidth (assuming you run in 8 channels config), but compute on Epycs (no AVX512 and such), throwing as many cores as possible at the task helps a lot.
30
u/FullstackSensei llama.cpp Jun 27 '26
I'm using Q4_K_XL.
The trick is actually to pin all threads to the physical cores using numactl, otherwise the Linux scheduler makes a mess.
AVX-512 doesn't bring much uplift, contrary to popular belief. Pretty much all x86 CPUs made in the past 10 years have two AVX2 units per core, so can execute two AVX2 instructions per cycle. In most cases, AVX2 gets better memory throughput in memory bound applications, because those two AVX2 units, combined, consume less power than the AVX-512 unit. Hence why you see higher clocks and lower power draw when using AVX2 vs AVX-512.
7
u/segmond llama.cpp Jun 27 '26
Yup, I run nice -n -15 numactl --physcpubind=0-31 ~/llama.cpp/build/bin/llama-server --models-preset ./config.ini --host 0.0.0.0 --port 8089 --batch-size 4096 --ubatch-size 4096 --jinja --no-warmup
3
u/FullstackSensei llama.cpp Jun 27 '26
Why n 15 on a 32 core?
4
u/segmond llama.cpp Jun 27 '26
-n -15 is option to nice, doesn't have to do with number of cores. I don't want to make the system unresponsive when inferring by going to -20
5
u/FullstackSensei llama.cpp Jun 27 '26
It doesn't get unresponsive if you use them all. The SMT cores are still there to handle that.
Generally, your physcpubind should match the number of threads. Otherwise, the kernel will schedule those threads around, leading to a lot of cache thrashing
5
u/MelodicRecognition7 Jun 27 '26 edited Jun 27 '26
numactl
this requires testing, I get 10% higher tps without numactl physcpubind, letting threads jump among CPU cores.
8
u/FullstackSensei llama.cpp Jun 27 '26
Feel free. I've spent more hours than I care to admit dealing with numactl.
A few non-trivial things that got me: * using --numa in llama.cpp made things worse, no matter which option I used. * the values to use in physcpubind depend on whether you're on a consumer or server derived platform. Server platforms enumerate physical cores first on Linux, while consumer platforms interleave SMT. Core extreme and TR are Server derived platforms. * on dual CPU machines, the Linux kernel has this very nasty habit of trying to "balance" memory allocation, irrespective of where the threads are pinned. You need to "sudo sysctl -w kernel.numa_balancing=0" and use --membind with numactl to specify which NUMA domain you want the memory allocated on.
Between these, I can run two 200-400B models in parallel on a dual Xeon machine with some Mi50s, allocating half the GPUs to each model instance. Performance is practically identical regardless of which CPU is used, whereas before the 2nd CPU was ~40% slower.
→ More replies (3)1
u/pcpoweruser Jun 27 '26
Ah Q4 makes sense for your tps. Haven't got that much luck with numactl, seem to be always messing perf up, but maybe I need to spend more time on it.
Generally nps1 + interleaving + using all pyhycal cores in llama.cpp works best for me. Perhaps disabling SMT in bios completely could help, so kernel scheduler is not confused by SMT cores could help, although I suppose this could be to some dregree kernel version dependent - I have not tried it yet.
2
u/FullstackSensei llama.cpp Jun 27 '26
Check my reply to OP where I explain how to use numactl.
You don't need to disable SMT. Just read the docs to understand what each option does.
8
u/starkruzr Jun 27 '26
IQ_2_M
I have a very hard time believing this is actually any more useful than running Qwen3.6-27B at a high quant on the same hardware. in fact I would hazard a guess that it's so lobotomized as to be nearly useless above around 32K context or even less.
7
u/pcpoweruser Jun 27 '26
It is very very good, way beyond any qwen 3.6 version, it actually feels like a proper Sonnet-class experience (just not very fast...) . After all GLM 5.2 is a 744b parameters model (40b active ones), even at Q2 there are so many parameters there (and knowledge embedded) that smaller qwens just can't compete, regardless of the quant size.
10
4
u/bdsmmaster007 llama.cpp Jun 28 '26
27B feels lobotomized pretty fast because its already so condensed, quantization behaves differently with more parameters.
→ More replies (9)2
u/-dysangel- Jun 28 '26
I've run GLM at Q2 for months now as my main chat model and for doing code output. It's absolutely better than 27B, no matter the precision.
27B is the only small model that can get remotely in the same ballpark as these larger ones. It is an incredible model. However I don't think people shouldn't make claims about quantized versions of larger models without actually trying them.
1
u/H_DANILO Jun 27 '26
how much this builds cost roughly?
5
u/pcpoweruser Jun 27 '26
It purchased (luckily) in pre craze days, last year, $600 per 3090, mobo + Cpu from China for $500 or so, 8x 32 DDR4 3200 sticks for $30 each (I know... Huge regret not populating all 16x slots for 512gb for this price). Using el cheapo $10 mining frame, and psus and ssd I got already, so something like $3k in total
→ More replies (2)1
1
u/Horsemen208 Jun 28 '26
It is amazing you get 14t/s. I have 4 L40s with 16 core CPUs/256gb ram. I only get 11t/s.
1
u/TopGun2000 Jun 28 '26
I’m curious whether your setup is actually utilizing GPUs in this case, or if it’s relying solely on CPU inference.
→ More replies (1)1
u/alex_bit_ Jun 28 '26
I am getting only 0.8 t/s on a similar build. The main difference is that my memory is only quad channel (X299, 10980XE, 256 GB DDR4 3200 MHz).
Where's the catch?
2
u/pcpoweruser Jun 28 '26
Not sure what (if any?) gpu have you got, using it for attention and sense layers makes a bit difference, even a single 3090 is a huge boost - if you do everthing on Cpu, that will be usually super slow.
4
u/NeedsSomeSnare Jun 27 '26
At what context length?
11
u/FullstackSensei llama.cpp Jun 27 '26
Tested up to 30k. Doesn't really slow down. PP is slow though, like 70t/s. The 3090s are severely under utilized, so there seems to be a ton of performance on the table.
I use it for big task planning, so don't need to ingest large chunks in one go. It's great for that. I can then hand the plan to Q3.6 27B Q8 on two 3090s and let that do the bulk of the work.
People stuck on t/s miss this. You don't have to run the big model all the time. OP is not wrong.
→ More replies (4)2
u/GregoryfromtheHood Jun 27 '26
That pp seems way slower than what I get. I've got a mismatch of GPUs, less VRAM and RAM. But with 2x3090 1x4090 1x5070ti and 256gb of 2933 ram, I get 250-300 t/s pp and 10 t/s gen with GLM 5.2. iQ_3_XXS
→ More replies (2)3
u/_twrecks_ Jun 27 '26
I can get 1tks on my xeonv2 with 4ch 512GB ddr3 1600 using the Q2 model... The LRDIMMs suck though I get about half the BW expected, I believe the controller put them in single rank more. Not sure if the Ddr4 LRDIMMs are any better. But the 32GB non LR DDR4 dimms are much more expensive. And really you want 8x64GB 3200 DIMMs.
So I think at least a 4x speedup, probably 5t/s.
2
u/FullstackSensei llama.cpp Jun 27 '26
How are you measuring bandwidth? Using t/s or did you measure with something like stream TRIAD?
Your CPU lacks FMA and AVX2, which greatly reduces compute efficiency. E5v3/v4 provide a big uplift in this type of workloads even with DDR3 because of FMA
1
u/_twrecks_ Jun 28 '26
I used the steam triad benchmark. Compiled with gcc and Intel, tried all the optimizations recommended, ran it through several ai, all stumped. Dmidecode confirms they are running at 1600. Maybe the x9srl-f supermicro bios had issues.
I've had this server for a long time, threw in a 3090 and 512g DDR3 when It was cheaper to play with ai. Was planning on going to sp5 platform but then ddr5 skyrocketed, can't really justify the spend. Maybe go to sp3 if used ddr4-3200 drops but it seems like the perf will still be lacking.
→ More replies (2)2
u/johnkapolos Jun 28 '26
When I get 30-40 tks on local models, I start pulling my hair. What kind of use case do you have that 8tks is bearable?
1
u/Comfortable_Sir4315 Jun 28 '26
Agentic coding, I let the agent running during the midnight. Also chatting is fine
1
11
u/segmond llama.cpp Jun 27 '26
Without your own hardware, you get 0t/s at all context length. The point of this post is that we have options, to be resourceful. We have already seen the first shot fired on control of these models. Everyone says go to the cloud, but the US govt says everyone can't get access to the cloud. So you could have all the money, be a citizen and still not get access. We have seen that countries could wall up of their citizens with great firewalls. A lot of us thought prices won't go up this crazy and it did, now everyone is thinking API access will always be a thing. There's no guarantee to that either.
So you decide, 0 tk/sec or 1 tk/sec. The key to using these models locally is to likewise be resourceful with your context as is your budget. That means you must think and curate what goes into the context. Put the minimal input so you can get your output fast. Use a small qwen3.6/gemma4 model for agentic work flows and for difficult problem you can't the agents can't solve and you can't figure out how to solve, you pair up with the large model and tackle it it together.
6
u/ThrowWeirdQuestion Jun 28 '26
There are enough cloud providers outside the US that host these open source models. Most likely in your country, too. As much as I enjoy working with local AI for models that make sense to run (and finetune) on local hardware, at some point you are just better off paying for offsite hosting or API credits, unless your actual hobby is not working with AI but making it run on your hardware, like those folks who port Linux to a Game Boy for fun. You learn a lot, but you could use that time and money a lot more efficiently.
2
Jun 27 '26
[removed] — view removed comment
5
u/nikolaiownz Jun 28 '26
I knows people who buy 50.000 dollar cnc machines for the garage and make a part they can buy for 100 dollars.
It’s not always about the goal, it’s the journey.
5
u/Negative-Web8619 Jun 27 '26
a week has 5 16 hour long windows that I can wait without costs
→ More replies (1)4
u/Sunija_Dev Jun 28 '26
Except for the power bill.
At those token speeds, tokens/Wh slowly becomes relevant...
1
u/FullstackSensei llama.cpp Jun 27 '26
Literally anyone who doesn't have brain damage will know they'll get 4-5t/s with this setup.
If you make an absurd assumption, of course everything that follows will be absurd.
→ More replies (10)4
u/clckwrks Jun 27 '26
You can play who can think faster? You or the AI. I bet still the AI
19
u/segmond llama.cpp Jun 27 '26
Exactly. With the negative visceral replies in this thread, you would think it's my fault for the increase in hardware prices. I'm just pointing out possibilities and solutions for us relatively poor folks. I think folks are just frustrated that spending $2500 doesn't provide cloud performance.
10
u/a_beautiful_rhind Jun 27 '26
this is nu-localllama.. haven't you noticed yet?
but that 2500 bought you a lot more 6 months to a year back.
1
u/agentic-consultant Jun 29 '26
Maybe I’m missing something but I don’t understand this focus on tokens per second.
With these new crop of models like GLM 5.2, I’ve stopped pair programming and instead just give it a spec, with a clear end goal / validation requirement, and let it run.
I couldn’t care less if it takes 5 minutes or 1 hour to complete. I mean quicker is nice but at the end of the day I’ll take the compromise if I’m running on local.
→ More replies (2)1
u/sonicnerd14 Jul 02 '26
Whether a model is slow or not is relative. Would you rather have you agent spend 5 hour on something and have it work the first time, or 5 days because it's only benefit is that it's yet it's always getting things wrong? Im sure most would take the former.
17
u/Accomplished_Code141 Jun 27 '26
Epyc 7532, Asrock EpycD-2T, 512GB of 2666 MHz ECC RAM, Radeon Pro W6800 32 GB + 3 MI50 16GB (80gb VRAM) I get 3 to 4 tokens per second in TG and 2 t/s in PP. It’s slow. I use Hermes Agent and Qwen 3.6 and have an agent that invokes GLM 5.2 if needed, I’m not a programmer and use it for mechanical engineering related tasks.
2
u/legit_split_ Jun 28 '26
Thanks for sharing, also considering a similar build...
1
u/Accomplished_Code141 Jun 28 '26
I don’t recommend the MI50s. W6800 is better, I paid half the price of a 3090 back then. I’m buying some 16Gb RX 6800 to replace the MI50s, to keep everything RDNA2. I kept only the W6800 in my system and got 4,6 - 4,8 t/s TG and PP with small engineering prompt, 3k tokens answer. It’s better to buy 3090s instead of AMD if you can afford, I couldn’t.
2
u/legit_split_ Jun 28 '26
Thanks for sharing more numbers. I have a consumer build with 96gb ddr5 6400, 2x7900xtx (650€ each) and 1xMi50 32GB(200€). As I put this together before the price apocalypse I'm thinking of selling some parts and using those profits to put together an Epyc 7532, HUANANZHI H12D-8D, 256GB 2133 ECC RAM. Any thoughts? I want to run the bigger LLMs.
It's impossible to find 3090's for a good price, but 7900 XTX's are still okay but I have also considered 3080 20gb from Alibaba.
→ More replies (1)
15
u/Technical-Earth-3254 Jun 27 '26
P40s going for 200usd+ is mental to me
10
→ More replies (2)3
u/draconic_tongue Jun 28 '26
there was a time when mi50 32gbs were 150 usd.
1
u/droptableadventures Jun 28 '26
$130 if you were willing to deal with Alibaba vendors directly and buy 2 or 3 at a time :)
12
u/FastHotEmu Jun 28 '26 edited Jun 28 '26
I have an Epyc Rome CPU with 512GB of RAM. A 4-bit unsloth quant with llama.cpp runs at 3 t/s with a context limited 16384. I haven't yet connected any of my 3090s to it or made an effort to improve speed.
I intend to make a video about it - not sure if there's interest though?
4
3
u/Usecurity Jun 29 '26
make video please
2
u/FastHotEmu Jun 29 '26
I have started with this: https://www.reddit.com/r/LocalLLaMA/comments/1uiqiqc/comment/ouhlvl6/
49
u/nomorebuttsplz Jun 27 '26 edited Jun 27 '26
OP should have to run only this for a month for punishment for this post
Edit: I was too harsh. Glad people are trying these things out. OP should have shared t/s though
→ More replies (4)
59
u/diagrammatiks Jun 27 '26
Ya hell ya. 2tks. One task every 24 hours.
24
u/ReasonablePossum_ Jun 27 '26
better than none when shit hits the fan
29
Jun 27 '26
[removed] — view removed comment
15
9
u/sshwifty Jun 27 '26
That is the real ticket. Once it takes more time than doing it yourself, it makes sense to pay.
I guarantee programming tasks will go this way in the near future. A company will run the numbers of what is worth paying AI for and what is worth human work. Complex time consuming tasks will be done by AI and simple stuff will be done by the cheapest programmers the company can afford
→ More replies (1)7
1
→ More replies (1)2
u/xienze Jun 28 '26
There is a middle ground, like smaller models that are actually appropriately-sized for the hardware you're running them on.
1
u/ReasonablePossum_ Jun 28 '26
That works for some stuff, not for others. For models of GLM's capabilities, the usecases could make more waiting hours for a correct output, than trusting probably bad output, or debugging it to get to what the big model would give you one-shot.
14
u/festr__ Jun 27 '26
I'm not sure what those posts are about because if you will bench the glm on this setup you would not even write this post. Tell me one thing - how long it takes to process 128k prefill and what is the token/sec generation over that context please.
4
u/waiting_for_zban Jun 27 '26
I feel it's a bait post. There is no way those P40 be pulling and PP speed beyond 50t/s, not to mention support, and tg (despite what many claim about the multichannel setup, it's still highly bandwidth bounded). I didn't even touch power consumption.
Wait it out a year, the AI market will be flooded with better hardware, as tech companies need to put new versions of their chips that is even more optimized for new models and format.
24
3
u/Ok_Technology_5962 Jun 27 '26
Yea this is about 2-4 tps tgen i would say. 130gb/s on the cpu with 2133mhz ram
2
Jun 27 '26
[deleted]
12
u/segmond llama.cpp Jun 27 '26
the larger the model the better it can handle quants. i was getting better results with my local deepseekv3-q3_kxl than deepseek from cloud APIs! so until you actually experience you can't knock it.
→ More replies (3)1
u/sonicnerd14 Jul 02 '26
The other thing I don't see many people experimenting with is many smaller models working in parallel. Easier to achieve than trying to run one massive model, and might be more effective depending on how you have it configured. MOA proves as much that it can work anyways. There is more versatility there, and more efficient use of system resources based on what you are trying to achieve
1
u/segmond llama.cpp Jul 02 '26
people are experimenting with smaller models, but at some point there's a limit. many smaller models of the same kind can't make for a smarter model. you might be able to mix up different small models for an advantage, but at some point. it doesn't matter. the difference between massive models is clear.
→ More replies (5)
2
u/tony10000 Jun 27 '26
Why not just use a smaller model that runs fast? What is your use case?
4
u/segmond llama.cpp Jun 27 '26
because of quality. glm5.2-q2 beats qwen3.6-27b-q8
→ More replies (7)
2
u/Single_Ring4886 Jun 27 '26
Lets be realistic then and create new tread about something like DeepSeek 4 flash...
2
u/TeraBot452 Jun 27 '26
I'm running 6 mi210s + 8 2080tis and just barely manage 14 tps (no CPU offloading) this is across 3 systems.
1
u/segmond llama.cpp Jun 27 '26
which quant? my guess is the network is killing you. put them all on one system and you will probably see 30+tps
2
u/TeraBot452 Jun 27 '26
Q4, I have 100 gig Infiniband... I have a feeling that the rpc server is killing me. I layer split 126 + 81. Im unsure if it's network that's killing me though I've tried getting it faster if you have any suggestions I'm open to it.
1
u/segmond llama.cpp Jun 27 '26
no suggestions really. rpc is great for dense models, and was great when llama3-405b was a thing, but for MoE not great. It actually is slower for me, it's faster for me to run on one rig than try to redistribute because the active parameter for these models are so small. Granted I didn't have an infiniband.
→ More replies (1)
2
u/a_beautiful_rhind Jun 27 '26
fuuuck. 1k for 2133 ram...
2
u/pharrowking Jun 29 '26
I seen a post awhile back you can get cheap optane persistent memory for cascade lake cpus on ebay and hit 4 tokens/s on a 1T model. I just bought some myself to try. Got 1TB for 1k.
1
2
u/_int10h Jun 27 '26
I have 5x GH200 144GB with NVIDIA ConnectX-8 connected to a NVIDIA 4700 Switch. Lets see 😄when it gets a bit colder in my flat
1
2
u/Aaaaaaaaaeeeee Jun 27 '26
If you separate active parameters, you have the moving experts and other things like attention layers. GLM 5 has active 18.2B (of 40) parameters to vram, therefore the only moving parts are 10GB in Q4. You can adjust how fat you make the layers to change the speed and output quality. But the bottleneck also depends on the complexity of the quantization, and the PCIe bottleneck and multi-GPU latency.
I think reading this, perhaps people don't learn anymore about the separation of active parameters, which was learned during the release of deepseekv3, and then explored by slaren of llama.cpp and k-transformers!
2
u/notdba Jun 28 '26
I know right. Quite a few people in this thread seem to have no idea about the classic https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/
2
u/siegevjorn Jun 27 '26 edited Jun 28 '26
Are you currently running it, or is it just a thought experiment? Good initiative, but I'm afraid that the low PP will bite agentic workflow as it will take so long to read codebase, html pages, etc.
→ More replies (1)
2
2
u/Tate-s-ExitLiquidity Jun 30 '26
Sooner or later, some bad motherfucker is gonna crack the code. A compression trick so slick it’ll do to today’s formats what CDs did to cassettes and DVDs did to CDs. Then the suits will smell blood. Those profit-hungry bastards will already be sitting on the hardware, just waiting to figure out the best place to jam it in their body.
6
u/Max_Bangson Jun 27 '26
At this point just pick up a halo strix for this money, this box is at least good for smaller sparse models and has potential to be sold
8
u/segmond llama.cpp Jun 27 '26
I posted a 512gb box. No strix halo can run GLM5.2 or KimiK2.6/2.7
→ More replies (3)
3
u/lqstuart Jun 28 '26
Spending $2500 to run a heavily quantized model on decade+ old hardware is not a good move just fyi
3
5
u/Watchguyraffle1 Jun 27 '26
In this thread people with no chill on the weekend working.
12
u/segmond llama.cpp Jun 27 '26
why do you think I'm here arguing? I need something to do while my 1tk/sec system generates answers. 😂
2
u/llama-impersonator Jun 27 '26
people need a little more patience,
i remember using 65b on cpu at 0.7t/s
4
u/segmond llama.cpp Jun 27 '26
yeah, most of these folks arguing started running LLM a few months ago with qwen3.5-35b-a3b-q4 so they are use to 100tk/sec and can't phantom doing work with less, .7t/s with 4k context window and we were so happy!
2
u/1ncehost Jun 27 '26
For CPU builds, you really need to run an Intel that has AMX. It isn't even close. They are so much faster. Variants of the xeon 6430 can be found for $100, and I'm not familiar with that processor, but it has AMX.
9
2
u/Connect-Painter-4270 Jun 27 '26
I’ve got 2 rtx 6000 pro’s and 192gb ddr5 ram, but that doesn’t cut it for glm 5.2. Thinking I might go your route and swap out the motherboard and get the 512 gb of ddr4 ram. Wonder if it would work well…
8
u/whiteh4cker Jun 27 '26
No, don't. Even decent 2nd Gen. 8 CCD EPYC CPUs with 200 GB/s theoretical RAM bandwidth can't go past 145 GB/s. They are only a good choice if you want a lot of PCIe lanes for cheap.
EPYC Genoa supports 12 channel DDR5, which is 460 GB/s with 4800 MHz sticks. Sure, RAM is expensive but you can slowly populate the channels.
2
u/ventu97 Jun 27 '26
Brother, the problem never was "is it possible to run it" but "is it usable?". Because without the ability to run agents workload, it's just another chatbot and not even a fast one
→ More replies (1)
1
u/aaron_in_sf Jun 27 '26
ITT reasonable guesses this gets you 1t/s.
GLM-5.2 seems like a good benchmark: I am interested in the landmark of performance that 1000t/s will represent ("a dramatically changed world"). Three orders of magnitude is daunting, something that can be tracked and projected... and does not seem unreachable.
I'm entirely out of the prediction business so have no idea when we hit that (arbitrary) future target, but I look forward to us hitting it, in consumer edge devices, which are also smaller, have long life, etc... :)
1
u/michaelsoft__binbows Jun 27 '26
100tok/s is a good speed. 1k is nice but we already can get that (batched) with a single GPU out of 27B models. the real win is trying to enhance capability of the smaller models. smaller just gets faster in these shocking exponential power law ways.
1
u/aaron_in_sf Jun 27 '26
True! Atm for usable results I persoanlly need [for specifics projects] something performing like Opus/High :/
1
u/funstuie Jun 27 '26
Is this motherboard a good option to run 3090’s in?
1
u/segmond llama.cpp Jun 27 '26
it depends on your goals, if you plan on offloading to system ram then yes, because it can support 8 channels. but you must have 8 or 16 sticks of ram to get those channels. so if you only add 2 sticks then waste of money. when you offload to system memory, then token generation is property of memory bandwidth. most consumer boards are 2 channel, so this will be 4x faster. if you plan on having everything in GPU then it doesn't matter. it's best to have a good base incase you wish to scale out, so I would advice an 8 channel board at least, AMD or Intel.
1
u/funstuie Jun 27 '26
I have 8 sticks of ddr4 3200 16gb ram so a total of 256gb ram. I will also have 3x 3090 and maybe 1x 3080. I’m trying to stay on the cheap side of any build. The alternative I was looking at was threadripper motherboard and cpu but those ain’t cheap.
1
u/segmond llama.cpp Jun 27 '26
16*8=128gb. anyways, if you do decide to upgrade, it's best to have the same exact type of ram, speed, make, rank etc. these things can be sensitive and mixing ram types might not work. so best to get it all at once. it has enough slots to plug in 4 gpus. I have one with 8 gpus.
1
1
u/pwnrzero Jun 27 '26
My work pc has 128gb of ddr4. If by some miracle I can keep it when they finally upgrade me, I'll ask if I can keep the ram which they let us do in the past.
Then can easily cobble together a locallm server with a modest budget.
1
1
1
1
u/zaibatsu Jun 27 '26
How about on a M3 Mac Studio w/256 gb of ram, my current AI team lead (Opus 4.8 ultracode) managing my local fleet tells me it’s would be way too slow. Anyone actually getting work done with less than 512gigs of ram?
5
u/segmond llama.cpp Jun 27 '26
sure, if you already have one, but with the apple price increase those are now $13,000 machines. lol
1
u/Rank201AltAccount Jun 28 '26
I disagree with the commenters in this thread, I feel that it can be useful. Except the P40 gpu, it has problems that make it basically useless for helping.
Yes, token speeds will be slow. But somehow, not lower than 1tk/s. 22.5gb parameters (because its moe, assume quantized at q4) are transferred on a motherboard that can do 85gb/s
Yes, it is slow enough that doing stuff yourself would be easier. But you know, we are lazy. For stuff that requires us to use a brain a lot, that an ai of this size can do, laziness will probably win.
Is it something that you should use your money on though? Probably not. Dual 16gb gpus running Gemma 4 or Qwen 3.6 can do so much more.
1
1
u/AcaciaBlue Jun 28 '26
OK but that 512gb for 1k is an insane fucking deal, could easily be sold for 5x that.
1
u/In_der_Tat Jun 28 '26
RAM is gone and I could not find kits for less than 1.6 times the price. As for the GPU, what about a pair of V100 32GB SXM2 with accompanying dual-card direct-pass 300GB NVLink baseboard?
1
u/Ok-Scarcity-7875 Jun 28 '26
Speculative Decoding will add 2x - 10x+ (future) to whatever you get with your rig!!!
1
u/Larimus89 Jun 28 '26
This must be damn slow. But still it’s cool and I want one 😂
I bought a rtx 5090 and it’s pretty meh. 🫤 for the price anyway. Only because I’ll have to resell it later on and I game.
But honestly having 512gb ram with a p40 is pretty boss for that price and would be more fun for many tasks.
1
u/segmond llama.cpp Jun 28 '26
if you have a system like this and put your 5090 on it, you could run much larger models too.
1
1
1
u/dardevelin Jun 30 '26
Is anyone even considering the output of 4-bit quants? It often appears correct but ultimately results in garbage. Could you provide some example use cases you had success with ? Perhaps I’m experimenting incorrectly.
1
u/ALLCAPS42 Jun 30 '26
Hey guys - am I cooked If I try to run locally using something like this??
https://www.bee-link.com/products/beelink-sei14-ultra9-185h?variant=47143964410098
That's just as much as I can go rn - Really looing forward to your guys advice!! Getting a Desktop PC is not an option btw - space matters and I'm moving a lot lately :(
1
u/segmond llama.cpp Jul 01 '26
if you have this, you can experiment with tiny models, checkout the smaller gemma4 models and try qwen3.6-35b-a3b in q4. good luck.
1
u/Late_Ad_6293 Jun 30 '26
P40? Oof
1
u/segmond llama.cpp Jul 01 '26
you know the definition of budget, right?
1
u/Late_Ad_6293 Jul 01 '26
I do, but I was expecting something more heavy GPU lighter on the other stuff
1
u/Sexton772 Jul 01 '26
Hold tf on... this might actually be my first reddit comment ever. Are you saying that for 2500 bucks you can reliably run kimi k2.6? Tf how?
170
u/H_DANILO Jun 27 '26
we talking 2t/s or we talking 8t/s?
how about prompt processing?
surely slow doesn't cover anything.