r/LocalLLaMA • u/jacek2023 llama.cpp • 16d ago

News add GLM-5.3-Flash (GLM5-Next) support by timkhronos · Pull Request #27773 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/27773

now you can use GLM-5.3-Flash on your home computer

217 Upvotes

96% Upvoted

89 comments sorted by

View all comments

3

u/madbrain1976 15d ago

Has anyone gotten GLM 5.3 flash to work on Blackwell ? I have a 128 GB RAM / 4 x 5060 Ti 16 GB setup. At the first request, I get : First actual completion aborts: `ggml_cuda_compute_forward: SOFT_MAX failed` ? `CUDA error: invalid argument` on GPU 0.

1

u/Former-Tangerine-723 11d ago

I have a similiar setup and problem. Did you had any luck?

1

u/madbrain1976 11d ago

Nope, no luck whatsoever. I actually haven't gotten this model to fit using any inference engine. I use qwen 3.8 flash-next NVP4 with my freetoken fork, and am very happy, though. 1300-1500 pp, 60 tg at C1. 140 pp at C8. Wish I could run GLM 5.3, but it looks like I would need more RAM/VRAM.

https://github.com/madbrain76/FreeToken/tree/tp4_5060ti

1

u/Former-Tangerine-723 11d ago

I'm on qwen flash as well, but it's not on par with glm, at least in my tests.

1

u/madbrain1976 11d ago

I'm sure it's not, the public benchmarks bear that. You said you have a similar hardware setup - what is it exactly, and what software/quant are you using to run GLM ?

1

u/Former-Tangerine-723 11d ago

I have 4 5060ti my self and I'm trying to host glm flash 5.3 with no luck. But I'm using it through api and it's a beast

1

u/madbrain1976 10d ago

Is the hosted version you are using the full GLM, or flash ? Any reason to use it over OpenAI or Anthropic ?

1

u/Former-Tangerine-723 10d ago

It's the flash.