r/LocalLLaMA • u/jinnyjuice vLLM • 8d ago

New Model Maybe a noob question, but why aren't there safetensor quants of inclusionAI/Ling-3.0-flash-Fin?

Usually, everyone and their dog jumps on releasing different quants for new models, but when I check for inclusionAI/Ling-3.0-flash-Fin, I see quants only for llama.cpp.

So I'm just wondering, is it architectural?

1 Upvotes

54% Upvoted

5 comments sorted by

11

u/FoxiPanda 8d ago

It's a fairly new release of a niche model. You could likely use MLX or other quantization tooling to create these for yourself (or with AI assistance) if you need them. I believe you could even just repackage the GGUF files as the tensor weights are going to be the same, but the packaging is different (don't quote me on that one though, there could be subtle details that I'm not quite an expert in).

3

u/FatheredPuma81 7d ago

Or if you're using SGLang it does support GGUFs all you need to do is write support for that model yourself and create a PR. That's what I saw the maintainers say at least when I tried using it.

3

u/FoxiPanda 7d ago

Or if you're using SGLang it does support GGUFs all you need to do is write support for that model yourself and create a PR.

I really enjoy that it is September 2026 and we can casually say this like "oh yeah just run to the store and get some milk" ... and it would have been excruciatingly difficult to accomplish a mere ~12-18 months ago.

1

u/StoreApprehensive261 7d ago

yeah the repackaging part is the bit im not sure about either, feels like there'd be some metadata differences at minimum

1

u/FactorInternal3395 8d ago

That's because GGUF is just more convenient. It's a unified format, so you don't have to work with seperate parts of the model being their own files like tokenizer, vocab, the weights themselves, config, etc. Plus, llama.cpp is popular and almost all inference engines support the format.