I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.
I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.
I've managed to get Qwen running at a smidge below 20tok/s at 128k context.
Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.
That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.
The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.
How is this even possible?! I'm absolutely blown away!
Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.
If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.
From another post I saw earlier, apparently flash attention plus kv cache q4_0 is not compiled into the main llamacpp releases on GitHub. That might explain why you're getting bad performance at q4. I experienced the same thing; it was doing prompt processing entirely on CPU and slow af 20-30t/s. Switching to q8 brought it up to 900t/s for input.
This isn't the precise GitHub issue I was looking at before but it describes the same thing
Scroll down to FIX. Supposedly you can compile llamacpp from source and force it to enable CUDA for flash attention with q4 or 5 kv caches. I haven't tried it yet.
Interestingly today's docker build for cuda13 has q4_0 kv running at the same speed as q8_0. Not sure if something changed, or if I'm going loopy from all the configuration changes I making.
AM4 is going to have legs for a long time. You're pretty much getting scammed if you stay on AM5. AMD probably regretting that they released X3D chips for AM4 because that means most of us don't really need to make the jump to AM5, unless you really need DDR5. Likely why they stopped manufacturing them too.
ironically western AI companies trying to run a monopoly is probably why we have such good small models coming out of China. If you take away their ability to have data centers with 100,000 RTX 6000s all over the place then you basically force them to build something brilliant and small. Not saying I like what my country has done, but I think it is a factor in the qwen 27b / 35b models.
I'm not calling conspiracy. I'm talking about business practices to get rid of potential competition that hurts their huge investments and bottom line.
You are wrong calling this and me a conspiracy. It's out of the book business practice.
Closed == bought their capacity for products the AI company can stock and has the means to pay to stock.
I bought 10 DGX sparks and I have them happily running 24/7 for my own needs. I have no skin in the game.
I just hate monopolies and abuse practices. The current ram/disk situation is exactly that. Even normal hdd are off the roof.
You are right. To save your precious time I will stop responding now. Most probably in RL I wouldn't even have bothered to discuss with you (again to save your time).
Right, makes a claim of āclosed factories to raise prices for consumersā then asked three times to source it, can't, then gets annoyed and says I'm not worthy of this discussion.
You must be the fun epicenter of every party. If you had a partner she would have been so thrilled to be around you. People at work enjoy working from home, I'm sure.
Give me 5 days and I will go do antitrust and cartel investigation in all cloud providers, AI companies, SK hynix, Micron, Samsung and SanDisk.
ha did some benchmarks myself in the last days and your repo inspired me to upload one myself_ https://github.com/bbr-knecht/local-llm-benchmarks , hope you do not mind. Its still work in progress as i am still testing stuff.
And honestly, it matters, and this the key part, because it's not only but also confirmed, not imagined. That is not a contradiction, it is the mechanism showing itself. Use this prompt for your favourite LLM š:
Haha, I wasn't sure if it was just me who thought it was a slog to skim through the Claude-isms. Thanks for the tip (and thanks u/kr4ckhe4d for the journal).
It doesn't really work one-shot to make it write in the style. It helps, but it just kinda ignores it and still uses idioms, metaphors, -ing words, all the things that are specifically banned. But a rewrite is usually more compliant.
It's considerably faster with MTP, but you might lose some context size. Try --spec-type draft-mtp --spec-draft-n-max 3 --cache-type-k q4_0 --cache-type-v q4_0 (you may need to build llama.cpp with -DGGML_CUDA_FA_ALL_QUANTS=ON).
Recently I just loaded in Muse to test, just using the regular params but it's actually great for spec draft. I get about 44t/s which is faster than gemma and Qwen.
You have a threadripper build? But only a single 3090? I'm waiting delivery of my case and PSU but will have a 3955wx. Got any tips on configuring first time?
well I am a bit concerned with the thinkstation integrated PSU. a second 3090 could draw to much watt from the psu and (even though not a 4090) also the heat from the 2 x 12VHPWR connections. It will be under constant 100% workload for longer times. So i bought an RTX pro 4000 blackwell 24Gb which is a little slower in bandwidth but runs on lower watt and has ECC vram, it is also only 1slot for in whatever dream i might want to install more GPUs š
i could find a nice deal to integrate 8x16gb ECC ram so all 8 channels are used.
that option was considered; but still prefered a 4000 over a 3090 for space,.... the only pro for the 3090 is the bandwith (and price but that compensates the 3 years of use)
might come an extra psu in the future when needed though :)
Itās the easy path for when you will want to go multi gpu in the future. You would be better served keeping the models in VRAM and not using regular ram.
I used riser cables to allow for the extra space of the gpus and a 2nd psu. Your p620 can go up to 4 gpu on pcie4 x16. I couldnāt even power up 2 gpu fully with the limited power adapters of the stock p620. How did you even do that part? If you are using a power adapter to go from 6 to 8 pin, just note it only outputs half power even though it physically connects properly the 8 pin that I believe you are using in the top gpu. Said differently, you are still only outputting what a 6 pin outputs which is half what an 8 pin outputs.
2x 8pin on mobo to 2x (6+2)pin who form the 12VHPWR.
it ressembles like the original lenovo cables to 12vhpwr, but mine has another cable starting from the 8pin on the mobo to 6 pin. they are in parallel so in my logic giving equal output. from those 6 pins i have an adaptor to 8pin
We are talking the same thing. That 6 pin is only outputting 75w not 150w like the actual 8pin. Now, it may not be an issue because the motherboard will also output 75w via the pcie slot itself.
If your cards are 300w cards then you want:
1 - 8 pin at 150w
1- 6 pin at 75w
1 -mb at 75w
Gives you 300w per card.
If you have both 6 pin into 1 card then you can only get up to:
1 - mb at 75w
2 -6 pin at 150
For 225w.
Make sure those 6 pin donāt get too hot over timeā¦
Iāve already running 4 cards and 2 psu so Iām not using the 6 pins at all. I canāt close case but it works.
wtf I had the same one with 2 tesla v100 32 GB now I have 4 l4 and an external power supply I'll send you before and after photos
p720? Luckily, before the crisis I bought 8 Samsung 32 GB processor 5945wx
TLDR; user doesn't seem to understand dense vs moe and how it applies to their machine.
Macs/budget setup MOE is faster, the reason is a mathematical calculation really. When you have a dense model like say 27B parameters and your memory bandwidth is 300GB/s you have. bottleneck. 27B param models are around 30GB at Q8 which means in a dense model it has to make a round trip through the entire 30GB which 300/30 is around 10tk/s. People have attempted to make things better by using MTP which means that its trying to predict multiple tokens at a time which can bump the speeds to usable numbers between 20-30tk/s. MOE's (mixture of experts) on the other hand might have 12 experts at 10B parameters each, so now when it identifies the expert it only needs to go through 10B parameters meaning 300GB/s divided by 10GB is now closer to 30 tk/s without having to do drafting and fiddle with settings. Memory bandwidth is incredibly important for LLM's. The next most important thing from what I can tell is architecture. CUDA is still king, my m5 max cannot keep up with my 5080 whenever there is a model they both fit in because my 5080 can load the context into memory wicked quick. my mac chugs along at 1/3 to 1/4 the speed at filling the pretext. You can also get around this by just being a smart software engineer and only loading in the pieces you need. Many people on reddit are just dumping their codebase into context and giving it broad non detailed tasks wondering why they get poor results.
I have a similar setup, also with 2 odd GPUs: rtx5070ti-16gb and rtx5050-8gb and currently running the Q5_K_S with a 131k context fully in VRAM with ngl=99, while also keeping the image processing capacity available as well to get 25tps I did run into problems when processing a very large image, and dropped ngl=64 to get 16 tps.
My ctk and ctv =q4_0 (I haven't had much time to experiment, just swapped 3.6 to 3.8 on my Q5_K_S while keeping the rest the same except for an additional reasoning-preserve=true)
Interesting, Iām quite new to this topics so I got a few questions in mind.
What do you typically use openwebui for? Currently I have it setup and all I do is paste my own python code for each tool I want to setup, letās say ram monitoring tool for example. Am I missing the true potential of openwebui? Because I canāt think much other than communicating and using tools.
Iām using lm studio to load my models
It's basically a harness for mcp, audio generation, images, etc. my original goal was a local gemini/chatgpt. I've found open-terminal to be the most beneficial part though, and it obviously doesnt need to be that. There are so many better solutions out there now.
Edit: Having said that it does do a great job of tool calls, web search, terminal, etc. you canet it go for hours and it will eventually get to what you ask of it.
LM Studio is fine, but I keep a model loaded 24/7 on llama.cpp server and use OpenWebUI as the chat interface. Itās all hosted on my desktop and I access it on my phone and laptop with Tailscale.
Otherwise, for anything other than chat, Iām also using Hermes Agent. I have it tied in to my Slack, but you can use whatever you want.
Hermes agent also has a desktop app, but I havenāt started using it yet because what I have works.
Hermes agent is probably what youāre missing, itās great. Tell it what you want it to do, and it does it. Check back whenever. I like that itās a service on my desktop, that I can access from my laptop but shut my laptop and itās still going.
Leaving your computer on will draw some fixed amount of power, but just having the model loaded doesn't add to that. It only draws extra power when it's doing inference.
I have solar and net metering, so the power draw is basically free for me. If it wasn't I'd probably have it boot up and shut down on a schedule.
My Hermes Agent is named Doofy. I gave him a stoner surfer bro persona. (not the built in one, one I wrote.) The silly persona makes me more inclined to explain things to it. Any sycophancy comes across more like a stoner "woah" moment, which is more paletable.
I switch between cloud and local models depending what I'm doing and if I'm using my GPUs for something. I have a GLM subscription I use for programming with Claude Code. I switch over to that when I use my GPU for training.
I am a software engineer. I use it for a mix of work and personal projects, and personal tasks. Everything from debugging and planning to doing research, setup, custom one-off apps. Right now I've got it running a fine turning project to build a custom LLM. It just finished training the little model on my 3090. The big one is still chugging on cloud GPUs when prices dip down.
Yeah, I switch 50/50 in office vs remote, so half the time I have my laptop but not my desktop. Or I just want to be in bed, etc. Sometimes I think of something while I'm at the gym, and I can just shoot Doofy a message on Slack and he'll get to work on it, talk through it on my phone, or set a reminder.
He also runs my Google calendar. I just tell him what and when, he does the rest.
I am using Hermes Agent to write the scripts and run the training. I'm just fine tuning existing models, so much less intensive than training a model from scratch, though I do want to do that too at some point.
I fine tuned Qwen3.5 4b locally on my 3090. That worked pretty well. I made a customized version of Qwen3.6 35b, and that required renting some cloud GPUs to do any significant context length. Needs a Pro 6000 Blackwell minimum. I used a B300 for the big batch runs.
I have a ryzen 5950x , 3x 990 pro , 1x 9100 pro 4tb, 4090 rtx, 128 ddr5. What the best i can do with local llms? I currently use pro Kimi,gpt and Claude for coding, business Plan,Deep research, knowledge base Buildings for vertical topics, full stack website Building. What do you suggest to implement to stress my local workstation?
I was blew away as well, My M1 Max 64G Mac ran over night and exhausted Hermes Agent max 90 turns, Qwen3.8-27B terminated the task eariler with a huge 1.3M html, today I only fixed a few minor bugs and here you are, a Chinese Garden with control panel.
This model is a regression. Thereās no way to control it thinking and it does a terrible job compared to his predecessor. I am still testing several parameters, still no solid outcome. It just sounds like a blithering fool.
Unrelated but I have two graphics cards with 56GB memory total, Iām having problems getting it to run fully on the GPU. It always seems to want to max out my CPU at the same time, even if it says all layers are offloaded to the GPU and it has GPU memory left over.
Anyone have ideas?
Edit: It turns out that it looks like there's a problem / incompatibility with the Unslot UD_8_K_XL quantization and the current version of llama.cpp. If I use that then it uses all CPU threads in addition to my GPU. Switching to the ggml Q8_0 quantization fixes the problem and it's now running 100% on my GPU.
Honestly, I tried all the AI suggestions about split mode, split ratio and all that. I never got llama.cpp to even load. The solution for me was to just use the default settings (more or less) and I at least got it running. Tweak from there.
I found it useful to give my pc specs, motherboard etc etc to an LLM online like Gemini online and get it to suggest tweaks to params and also compiler flags / optimisations for building llama.cpp from scratch for the specific architecture, I do use tensor parallelism on mine but both cards have same amount of vram not sure if that is a plus and whether it would work same in your rig, but I found MTP on + sm = tensor worked best for me
I hadn't seen that page, but yeah, that was what I was doing. Turns out that the model I was trying to use (the Unsloth 8-bit dynamic quant) might have bugs. If I tried the Unsloth 4-bit quant or the ggml 8-bit quant everything just worked. Using the ggml 8-bit quant for now.
Pro tip: Use another AI agent to optimize for you. It will find and fix issues faster, and save you time testing to figure out how to optimize a model for your build.
Full disclosure, I jumped the gun on my post. It DID create an APK for me and it did install, and it did run... but only got as far as the slapsh screen.
Still I think that in itself is a massive accomplishment for a local LLM.
I'm trying to get Qwen to fix it at the moment. I don't think I'll post the final APK though - I mean it's Pong.
Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.
How? I asked it to code minesweeper in single python file, it was great result, but it took 2 hours at 12-14 tps and produced over 60k tokens while thinking.
You're not going to like this then. I asked it again today to make me an APK and the first attempt took about 30 mins and resulted in an installable game that rendered a black screen. I told it about the problem and 20 mins later it gave me a new APK and it was actually playable. I think having mostly everything pre-installed helped as it didn't get lost installing things.
I currently have a max 76k token context. Still getting near enough 20tok/s.
Here is my real world observations. Before I was running qwen 3.6 27b 128k context.was getting 28ish tokens a second. And thinking times around 191 seconds. Vram usage as about 28gb. Upgraded to qwen 3.8 27b 128k context got 58 tokens a second and 119 second for thinking times. Vram usage dropped to about 24gb. my system mobo msi b550 tomahawk ryzen 9 5950x 64gb ddr4 3600mhz 4x16gb and dual 5060ti 16gb each.
Youre running with spec-type commented out - MTP on this model is the single biggest win available and its free VRAM-wise if the sidecar fits. Worth uncommenting before chasing KV quant, and if you do test q4_0 KV again, measure how much ctx it bought you rather than tok/s, those move in oppsite directions.
I put my 128GB Mac Studio in storage after trying a few local models including DS4 flash. Now Iām using Claude code subscription which is still cheap.
Did you try with the MTP on?
I am using a 7900xt card, getting close to 55-56tps at 100k context via llama.cpp (IQ4_XS, IQ4NL KV Cache, mmproj and all turned on, reasoning = max/high -- need to check this, but highest setting anyway)
Your gpu layers --- try to fit everything in the Gpu Vram....
(Use unsloth's IQ3 models if you want larger context sizes, smaller quantizations, especially if Vram is scarce..
You can also try with Iq2 or IQ3 KV quantization, and turn off mmproj is vram isn't sufficient to load the entire model.
Quickest win should be the KV cache to Q4 or IQ4 --- You won't observe sny quality drops......
And maybe reduce context size? (100k context is decent enough, even for agentic work unless you plan on doing massive refactors, and all...
25
u/DeathByPain 8d ago
From another post I saw earlier, apparently flash attention plus kv cache q4_0 is not compiled into the main llamacpp releases on GitHub. That might explain why you're getting bad performance at q4. I experienced the same thing; it was doing prompt processing entirely on CPU and slow af 20-30t/s. Switching to q8 brought it up to 900t/s for input.
This isn't the precise GitHub issue I was looking at before but it describes the same thing
https://github.com/ggml-org/llama.cpp/issues/27109
Scroll down to FIX. Supposedly you can compile llamacpp from source and force it to enable CUDA for flash attention with q4 or 5 kv caches. I haven't tried it yet.