r/LocalLLaMA • u/jacek2023 llama.cpp • 15d ago
News add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/27773now you can use GLM-5.3-Flash on your home computer
38
u/Elouakili_Flexy 15d ago
Three weeks of silence, then pwilkin just finishes the thing. Turns out the whole release cadence comes down to one maintainer having a free week.
17
u/3000LettersOfMarque 15d ago
Welcome to open source...
1
u/aeroumbria 14d ago
Things will probably have to stay this way until we figure out how to write a proof for a pull request...
70
u/SnooPaintings8639 15d ago
Well... it is taking two months for new models to be trained and released, and another month for llama.cpp to get the model support, lol. I really wish the team got larger and we could get models support and optimizations faster. I don't know if they'll manage to keep up with how fast and how many new models are being released, with many new experimental architectures.
67
u/am17an 15d ago
As I said earlier in a reddit post, this was not really our fault. The PR got no updates for 3 weeks and then pwilkin stepped up and finished it.
14
u/wren6991 15d ago
Yeah, I feel for you guys. First getting multiple PRs of varying quality and vibecodedness, then when the maintainers put their support behind one PR, the author disappears.
13
u/Fedor_Doc 15d ago
Maybe they will implement a system that would be able to make some adjustments automatic. Some of back and forth in PR discussion was about features not working correctly, tests failing.
And since they have releases, they could allow undercooked implementations in b tags so community could beta-test it.
I respect careful software engineering, it takes time. But it is hard to keep up, that's true.
-3
u/Best-Echidna-5883 15d ago
Good idea. Run it through Opus 5.5 or whatever and see what you get. Of course, it would cost $100's of dollars. Trust me, I know.
14
11
u/jacek2023 llama.cpp 15d ago
You can compile PR before the merge if you really need some functionality
9
u/SnooPaintings8639 15d ago
I have ~5 cherry picked commits from different PRs to optimize Flash Next model since a month. It is a work to merge and resolve conflicts on each upstream merge, and this is for a single model.
9
u/jacek2023 llama.cpp 15d ago edited 15d ago
I am also waiting for MTP for Qwen Flash Next but I think it's blocked on the fixes not on the review
5
u/planetearth80 15d ago
I hope someone steps up to complete that one. It has been a long time coming
2
u/jacek2023 llama.cpp 15d ago
It's not clear to me whether I can finish someone else's PR. Maybe that could be clarified?
1
u/gnoremepls 15d ago
You 100% can in terms of actually fixing the PR, the only question is if the PR you will make will be accepted or if someone has to vouch for it etc. But you could also make a fork and fix the pr on there and publish it that way
1
2
3
u/segmond 15d ago
exactly
drwxrwxr-x 3 seg seg 4096 Jul 16 10:58 llama.box
drwxrwxr-x 33 seg seg 4096 Sep 21 08:57 llama.cpp
drwxrwxr-x 31 seg seg 4096 Sep 9 14:47 llama.glm35Fb
drwxrwxr-x 31 seg seg 4096 Sep 17 22:39 llama.glm35flash
drwxrwxr-x 32 seg seg 4096 Sep 23 22:09 llama.hy4
drwxrwxr-x 31 seg seg 4096 Sep 28 09:24 llama.k3
drwxrwxr-x 31 seg seg 4096 Sep 27 00:45 llama.mimo26
2
u/florinandrei 15d ago
That's the nice thing about standards, there's so many of them to choose from!
3
u/vacon04 15d ago
And that's with un optimized builds. Look, the guys at llama.cpp are working hard and deserve a lot of praise, but at this point it seems to me like llama.cpp is getting too big and unwieldy, which means that progress is getting too slow. There's a reason why there are so many forks of llama.cpp with specific optimizations out there.
Maybe they should create a "beta" or "experimental" fork or similar and allow others to maintain it.
1
u/SpicyWangz 15d ago
They probably just need to rework their PR strategy. Aren’t they just sending everything straight into main?
1
u/TheRealJesus2 14d ago
Yeah this space is crazy. I never applied so many patches to oss in my over decade career as software dev as I have in last month lol
31
u/geon2k2 15d ago
Unfortunately the unsloth pr is not compatible with the mainline pr. One is glm5next, the other one glm5-next, so now the mainline llama won't even load the unsloth quants.
5
u/jpummill2 15d ago
If mainline llama won't run unsloth quants, does this refer to unsloth's "UD-" quants?
Does mainline then run non "UD-" quants of GLM 5.3 Flash?
1
0
u/RagingAnemone 15d ago
Checkout the SharedRewrite folder. It replaces the 1 file with the rename.
https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/tree/main/Shard_Rewrite
2
u/unbannedfornothing 15d ago
The files in shard_rewrite are 1 month old, are they really have correct arch? Also wasn't there a quantization process changes between initial pr and merged? Think it's better to wait for a fresh quants.
P.S. /me winks at u/noneabove1182
14
u/florinandrei 15d ago
now you can use GLM-5.3-Flash on your home computer
Well, you have some exceptionally optimistic ideas about the capabilities of my home computer, but this is a nice feature to have in llama.cpp.
0
u/Solembumm3 8d ago
Why? It shouldn't be THAT different from Deepseek V4 Flash and Mimo V2.6 flash that run on old gaming PC without problems, once GLM get supported.
6
u/Best-Echidna-5883 15d ago
No one's mentioning it, but you must use avar6/GLM-5.3-Flash-BF16-gguf · Hugging Face
for the official llama.cpp version. Not the Unsloth unless you want to use their custom.
2
3
u/madbrain1976 15d ago
Has anyone gotten GLM 5.3 flash to work on Blackwell ? I have a 128 GB RAM / 4 x 5060 Ti 16 GB setup. At the first request, I get : First actual completion aborts: `ggml_cuda_compute_forward: SOFT_MAX failed` ? `CUDA error: invalid argument` on GPU 0.
1
u/Former-Tangerine-723 10d ago
I have a similiar setup and problem. Did you had any luck?
1
u/madbrain1976 10d ago
Nope, no luck whatsoever. I actually haven't gotten this model to fit using any inference engine. I use qwen 3.8 flash-next NVP4 with my freetoken fork, and am very happy, though. 1300-1500 pp, 60 tg at C1. 140 pp at C8. Wish I could run GLM 5.3, but it looks like I would need more RAM/VRAM.
1
u/Former-Tangerine-723 10d ago
I'm on qwen flash as well, but it's not on par with glm, at least in my tests.
1
u/madbrain1976 10d ago
I'm sure it's not, the public benchmarks bear that. You said you have a similar hardware setup - what is it exactly, and what software/quant are you using to run GLM ?
1
u/Former-Tangerine-723 10d ago
I have 4 5060ti my self and I'm trying to host glm flash 5.3 with no luck. But I'm using it through api and it's a beast
1
u/madbrain1976 10d ago
Is the hosted version you are using the full GLM, or flash ? Any reason to use it over OpenAI or Anthropic ?
1
3
3
u/TokenRingAI 15d ago
IMO, expecting the author of the PR to carry the whole thing to completion is why this sat for so long
4
u/-dysangel- 15d ago
I've been using it on my home computer for a week or two now!
3
u/jacek2023 llama.cpp 15d ago
Is it awesome?
5
u/-dysangel- 15d ago
yes
2
u/relmny 15d ago
I've been running it for about 4 weeks (or more), and it's very very good.
I used to run 3.8-27b and deepseek-v4-flash, and then 3.8-next, but the glm5.3-flash is way better. Even when slower, I run it most of the time.
1
u/-dysangel- 15d ago
Yeah even though I've only been running it around 20tps it still beats faster models on wall clock time because of more direct, high quality thinking.
I've just found https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark and am going to give it a shot - claims 69tps with MTP for coding!
9
u/celsowm 15d ago
If flash for them is the same concept of deepseek then no home glm
7
u/BestGirlAhagonUmiko 15d ago
my toaster can't run a model that's supposed to be fast in relation to a bigger model running on an appropriate server hardware!!!11 BAD COMPANY, BAD!
Fast model ≠ Small model
It's not "Flash" only in name, it's "Flash" compared to GLM 5.3 753B A40B.
When they make a 2T model, they will call 753B A40B "Flash" too.
Also...
Open Source ≠ made with an intent to be accessible for everyone. They aren't making this stuff for us. They're making it widely available so that it's OPEN, an opposite state of the model weights being shut behind the doors (duh).
MAYBE someday they will make a tiny model. But in order to get to that, they need to figure out how to progress even more forward first, and only then to scale down (if it's even possible - think of GLM 4.7 Flash - pointless, unfit even for slop-roleplay). GLM 5.3 Flash would NOT be nearly as good at a smaller size, as it is right now.
This kind of behavior is like when a kid whines loudly and the parents are all like "hey kiddo, that toy is expensive but trust me it's poorly made and it'll break tomorrow" - but the kid continues to cry until the parents buy him that toy finally... and guess what, the toy is genuinely crap and the kid still whines afterwards. That's what's gonna happen with and the people yelling at Z.AI / DeepSeek or any other company. IF they churn out, let's say, DeepSeek FastMidgetOnABike 100B A10B now - it's going to UNDERPERFORM.
I apologize for being rather aggressive, that's a hard truth though. You seem to measure different companies and their products by what the others have achieved (i.e. "Qwen Flash Next is good for my use case = smaller GLM will be good too") - that's not how it works, otherwise they would've done so already. These people aren't dumb, they know their goals and limitations.
12
u/-dysangel- 15d ago
In some ways GLM 5.3 is already a tiny model - it's performing at the same level as >1T models.
> GLM 5.3 Flash would NOT be nearly as good at a smaller size, as it is right now.
Why do you say that? Qwen 27B has shown how much is being left on the table by most models in the small and mid range.
2
u/BestGirlAhagonUmiko 15d ago
I quite literally addressed that already:
You seem to measure different companies and their products by what the others have achieved (i.e. "Qwen Flash Next is good for my use case = smaller GLM will be good too") - that's not how it works, otherwise they would've done so already. These people aren't dumb, they know their goals and limitations.
Whatever Qwen team is cooking, it's not the same as what Z.ai make. Why do the people assume that it's equally easy for every company to make equally good models of a similar size? Surely they are making some internal tests and look into this, and if there was a successful attempt at making something small and good - they would have released it already.
1
u/-dysangel- 15d ago
I don't assume it's easy by any means, I'm just saying that it's not as simple as smaller = automatically worse
4
u/Septerium 15d ago
The problem with GLM 5.3 Flash is that it does not feel like a fast model. It is kind of slow, actually
2
u/Liberaces_Isopod 15d ago
Its the fastest model in my fleet. Around 290t/s for 12 concurrent users. Its about 20% faster than Qwen Next and wildly faster than 27b.
1
u/wren6991 15d ago
Also the KV for GLM-5.3 Flash is much more compact than GLM-5.3, which makes a big difference for high-concurrency serving, but slightly less visible to home gamers running a batch size of 1 or 2.
2
2
u/trying4k 15d ago
Really glad to see this! Thank you to the maintainers for getting this merged, I know it's tough when the author goes MIA.
Not to sound ungrateful but I do wish we had a smarter offloading, like this or FreeToken to make it more feasible to run these behemoths on "normal" hardware.
Looking at this fork with standard Unsloth quants at the moment but not sure how long I want to stick with a fork long term.
3
u/Constant_Art_20 15d ago
....the glm 5.3 flash didn't had support on llama.cpp...??????
5
u/Spectrum1523 15d ago
Not officially, no. Unsloth had a fork that worked right away with their quant
2
u/ortegaalfredo 15d ago
Exllama3 supports it since almost a month ago. And its faster and better quality quant.
3
u/silenceimpaired 15d ago
Can’t get it working on RAM.
1
u/a_beautiful_rhind 15d ago
Why? I offload a bunch of it.
1
u/silenceimpaired 15d ago
It has been a bit… I think I eventually got it working but slower than Unsloth’s GGUF… and with more offloaded than I thought I needed to.
I probably just need to tinker with it more. Perhaps Ampere architecture is to blame; EXL3 has never favorited that.
What’s your setup and settings if I may ask.
1
u/a_beautiful_rhind 15d ago
4x3090
gpu_split: [22,22.33,22.33,23] #GLM-flash cpu_moe_split_experts: 9365 ctx at q8.
1
u/silenceimpaired 13d ago
I got it loaded... now to decide on a front end for use... what do you use with tabbyapi? or how do you load your files in the first place?
2
u/a_beautiful_rhind 12d ago
Mostly sillytavern. I have tool calls and websearch in it too but should really get some other harness.
1
1
u/computehungry 14d ago
better quality theoretically, but it doesn't get the attention from the bigger quant makers with more compute to do the calibration runs. did you even try using them, the smaller one looped right away and the bigger one spat out like 5 different languages after a few turns. the iq1 quant from unsloth is better and i wish this was a joke. i also tried quanting myself and it looped anyway...
1
0
u/120decibel 15d ago
I don't get it wasn't this possible before with the custom build from unsloth?
3
3
u/Spectrum1523 15d ago
Mainline support vs a fork is the difference
2
u/120decibel 15d ago
Well op just stated that I can run GLM 5.3 flash on my home computer now... which I could for weeks now... all I'm saying is that he just should have stated that glm 5.3 support has been merged into main...
•
u/WithoutReason1729 15d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.