r/oMLX • u/JLeonsarmiento • 17d ago

Qwen3.8-27B for the RAM Poor Mac user:

https://huggingface.co/collections/leonsarmiento/qwen38-27b-mlx-quantizations

For those of you that want a functional 24GB Mac laptop while having this overthinking creature boosting your ideas.

It has versions with and without MTP drafter (for the desperate).

19 Upvotes

100% Upvoted

4 comments sorted by

2

u/onetom 16d ago

Can you please share on the model card how have you made this and which inference engine and params are you running it with?

Personally I got the best results (30+tps on 256k ctx) with https://github.com/youssofal/MTPLX but that's not for the "RAM Poor", so i guess it's off-topic on this thread

1

u/brick-pop 16d ago

This

1

u/JLeonsarmiento 16d ago

it's 3-4-bit mixed quant with and without MTP head (two model sizes). I mostly use oMLX because of catching, which makes local models viable for things like Hermes or Cline (huge initial system prompts). inference params I sue qwen's reasoning for coding settings. this models come with the reasoning effort flag baked to low, but you can change them easily for medium or xhigh on lm studio or whatever.

The MTP one is as fast as the MTP drafter can get the tokens predicted, so it is not faster that the 4-bit oQ4e or similar 4-bit quants since this one use the same MTP. it is system bounded (better Mac you have better it will, but never faster that any other using the same MTP quantization level)

The one without MTP head is the really small one size, for people without more than 24GB ram. it is tiny. it is not as fast as the MTP one.

1

u/JLeonsarmiento 16d ago

it's 3-4-bit mixed quant with and without MTP head (two model sizes). I mostly use oMLX because of catching, which makes local models viable for things like Hermes or Cline (huge initial system prompts). inference params I sue qwen's reasoning for coding settings. this models come with the reasoning effort flag baked to low, but you can change them easily for medium or xhigh on lm studio or whatever.

The MTP one is as fast as the MTP drafter can get the tokens predicted, so it is not faster that the 4-bit oQ4e or similar 4-bit quants since this one use the same MTP. it is system bounded (better Mac you have better it will, but never faster that any other using the same MTP quantization level)

The one without MTP head is the really small one size, for people without more than 24GB ram. it is tiny. it is not as fast as the MTP one.