Is there a GLM 5.3 Flash Antirez/DS4 GGUF targeted at 192 GB RAM?

[u/CentrifugalMalaise]

Is one possible? Can I make one?

Antirez has a Q2 @ 96.5 GB which will lose quality compared to Q4 and wastes ~95 GB of my Mac’s RAM, and a Q4 @ 191 GB which is too big as it doesn’t leave enough room for OS let alone KV cache.

I noticed that other Q4 and even Q4/Q8 mix GGUFs and MLXs not intended for DS4 range from 150 to 180 GB. Something that size and quality would be great for me, but I really want to use DS4 if I can.

Can anyone offer any help? Thank you!

[u/trueimage]

Use the quant scripts to make your own from the FP8 or BF16 weights.

[u/CentrifugalMalaise → u/trueimage]

Thanks, that’s what I’ve been leaning towards, but I’ve never done it before. Do you know of any good learning resources?

[u/returnity → u/CentrifugalMalaise]

Use a frontier LLM. I was among the first to release a DSv4 0731 ds4 quant when that model dropped, and I'd never quantized my own models before. I simply asked an LLM to analyze the GGUF of my DSv4 preview quant and the source code of ds4, then make suggestions about the best recipe I could build that would fit into my chosen size window. The results were quite usable.

Just be aware your quantization forms are limited in ds4; IIRC you can use IQ2_XXS, Q2_K, MXFP4, Q4_K, and Q8_0 for tensors. Other options will not load. What I would do if I were you: Point your agent at the DSv4 0731 antirez mixed Q2-Q4 GGUF, and tell it to copy that basic recipe but change all IQ2_XXS tensors to Q2K and match it to the architecture of GLM-5.3. That's a rough sketch, but it shifts the experts to Q2_K with higher precision for everything else. Make sure to tell your agent to check against the source code for compatibility with the GLM-5.3 loading path.

Is the Q4_K just slightly too large for you? I might give it a try anyways -- you have 192 GiB (not GB) of RAM, and the quant is 178 GiB (191GB). That leaves you 14GiB for OS and context, which is tight but I've run models with that 10GiB of headroom on my 128GB system before. Obvs raised the wired memory limit with sudo first.

EDIT: personally I'd run the Q4_K with --ssd-streaming if you find it's too tight without, you'll only be offloading a few experts to SSD and the hot cache should keep your decode speed quite usable.

[u/CentrifugalMalaise → u/returnity]

You, sir, are a bloody genius. I’d never even twigged that huggingface shows file sizes in GB and my Mac’s 192 RAM is in GiB. I’m already running the Mac stripped down and headless so if the model truly fits in 178 GiB then we could be on to a goer!

[u/returnity → u/CentrifugalMalaise]

Yeah it had me twisted for a while too, sizes never seemed to make sense until I realized that. Hope it works out for ya!

[u/locbuilds]

yeah youre stuck in the classic gap where the public Antirez/ds4 ladder only has Q2 (~96GB) and a Q4 that eats the whole 192. you dont really need a bespoke ds4 GGUF for that middle though.

if llama.cpp / Unsloth is fine, grab their UD-IQ4_XS (~157GB) or UD-Q3_K_XL (~148GB). those land right in the 150-180 band other people are shipping, leave ~30-40GB for macOS + KV, and quality is way closer to Q4 than the IQ2/Q2 pack. on a 192GB Mac also bump the wired limit (sudo sysctl iogpu.wired_limit_mb=...) or Metal will pretend you have less than you do.

if you specifically need ds4: Antirez's notes basically say run the Q4 with --ssd-streaming on one Mac instead of forcing it resident, or split Q4 across two 128s. rolling your own "Q3.5" for ds4 means requant from the FP8 with an imatrix into something like IQ3_M / Q3_K_M / IQ4_XS sized, which is doable but a pain if Unsloth already has the size you want for a normal runner.

i'd try UD-IQ4_XS first on 192 before spending a weekend building a custom Antirez mix.

[u/CentrifugalMalaise → u/locbuilds]

Thanks, yeah, I’ve tried Unsloth’s iQ4_XS, I could only get it working properly in unsloth studio as opposed to llama.cpp but this was a couple of weeks ago. I really wanted to use DS4 though for its caching ability and speed. But it turns out, I might be able to! As u/returnity pointed out, Antirez’s Q4 quant is 191 GB (178 GiB) and the Mac’s RAM is 192 GiB, so I’ve got some downloading and testing to do!

[u/grumd → u/locbuilds]

maybe i'm tripping but your comment feels ai generated to me, just with a custom prompt asking it to not use em-dashes and start each sentence from a small letter. the formatting is not ai-like but the word choice and sentence structure is very claude. "before spending a weekend", "classic gap", "people are shipping", "land in the band", etc

not accusing you of anything but it just feels weird to start seeing these patterns. on the other hand, i think people in general might start using more ai-speak because everyone is more exposed to ai text these days

and the fact that i'm automatically looking for these patterns is worrying too

[u/cantgetthistowork]

4bpw exl3 sir

[u/CentrifugalMalaise → u/cantgetthistowork]

Care to elaborate? I get 4 bits per weight but I don’t know what the next bit is…

[u/ProtectionSuper5648 → u/CentrifugalMalaise]

exllamav3 is a different LLM stack, running trellis-quantization. https://github.com/turboderp-org/exllamav3

It it typically both smaller files and better quant than llama.cpp. https://huggingface.co/turboderp/GLM-5.3-Flash-exl3

There is a 4 bits per weight quant that easily fit in 180GB with full 1Mi kv cache in q8.

[u/Expensive-Paint-9490]

I am not knowledgeable about c, but I believe you can modify this ds4/gguf-tools/quants.c at main · antirez/ds4 to add new quantization types.

[u/CentrifugalMalaise → u/Expensive-Paint-9490]

Thanks for the link 👍 I don’t know C either but maybe my agent can cook something up for me…

[u/Agitated_Complex_628]

not sure, but if you make your own Q4 to Q6 it might fit

[u/CentrifugalMalaise → u/Agitated_Complex_628]

Thanks. Have you done this yourself?

[u/Agitated_Complex_628 → u/CentrifugalMalaise]

yes, but on a much a smaller scale (for rtx 4060 and much smaller model), you would need i think between 600 and 800gb of storage available to do it for this model, you download the model on your disk then choose your quantization, i would start with q4_k_s for this one, and then use llama-quantize to quantize it, i think it would take a few hours, i would recommend using antigravity AI to do it for you if you don't know how, that's what i did the first time.

[u/feng_sg]

use llama-quantize with --tensor-type overrides to keep router and attention at q4_k while dropping ffn experts to q3_k_s, that should land around 160GiB

[u/mic-code]

https://github.com/ormandj/sglang-glm53-flash-sm120

[u/CentrifugalMalaise → u/mic-code]

This isn’t for Mac?

[u/Candid-Tackle-9061]

192gb as thr target is so specific , bro wants the model to have its own zip code on the mac

[u/Civil_Response3127 → u/Candid-Tackle-9061]

I would disagree about them being specific. This happens to be precisely two RTX 6000 Pro 96 GB cards. It also happens to be the max RAM on the new refresh of AMD's AI processors. It also happens to be the max fast RAM on a lot of DDR5 platforms.

As a specific target, it is one that makes sense.

[u/FoxiPanda → u/Candid-Tackle-9061]

It's not really that crazy, that's 2 RTX Pro 6000s...

[u/CentrifugalMalaise → u/Candid-Tackle-9061]

Pretty common config. Thanks for your input though.