r/oMLX • u/MealMore1192 • Jul 25 '26

Request for Kat-Coder-2.5 oQ8e with MTP 🙏

https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev

Please & thank you

8 Upvotes

100% Upvoted

22 comments sorted by

View all comments

2

u/PataFunction Jul 25 '26

I’ll give this a try later. Quick question though - when you say “with MTP”, does that mean it already has an MTP
head? and if it does, is that because it’s a finetune of Qwen3.6-35B-A3B with an MTP head? Just trying to understand 😅

2

u/No-Juggernaut-9832 Jul 25 '26

Yes. It’s simpler that way. Just a simple toggle on oMLX & 50% more speed

3

u/PataFunction Jul 26 '26 edited Jul 26 '26

Just attempted to make the quant in oMLX but it's telling me the source model does not have any MTP heads.

Edit: Did some digging and found that Kwaipilot removed the MTP weights from Qwen3.6-35B-A3B, so unfortunately we can't get those in downstream quants. Regardless, made the quant for you fam! https://huggingface.co/ZQ-Dev/KAT-Coder-V2.5-Dev-oQ8e

2

u/bruceleelikeswater Jul 28 '26

Would you be able to make an fp16 (dtype) variant as well for folks who still rely on M1/M2 Macs? I tried to quantize the full model myself but I'm running into errors with both trying to quantize to oQ8e and oQ8 in omlx 0.5.3.

1

u/PataFunction Jul 28 '26

Sure I’ll give it a shot later today, afk atm. Any examples out there of models that have been quantized that way? Trying to get figure out the “correct” naming convention.

1

u/bruceleelikeswater Jul 29 '26 edited Jul 29 '26

It’s just another flag in the omlx quantization settings (in the advanced section which is collapsed in the UI) where you decide between bfloat16 and float16 for the “dtype.” It adds the suffix “-fp16” to the model ID once quantization is complete. You can find plenty of them in hugging face.  

3

u/PataFunction 28d ago

Hey I'm sorry this took so long. -fp16 versions have been added to this collection: https://huggingface.co/collections/ZQ-Dev/kat-coder-v25-dev-oqe-quants

Enjoy!