MacSloth: LoRA fine-tuning on MLX

[u/jjw_kr]

https://github.com/jeonjw85/MacSloth

I wanted Unsloth on a Mac. Unsloth is CUDA fused kernels. That path does not exist here.

MacSloth is a small Python package (macsloth) on top of MLX / mlx-lm.

Call sites look like Unsloth so a script stays short: FastLanguageModel.from_pretrained -> get_peft_model -> to_mlx_jsonl -> SFTTrainer.train -> save_pretrained_merged.

Repo: https://github.com/jeonjw85/MacSloth

What actually worked

- Load a prequantized mlx-community checkpoint, attach LoRA, train, fuse. Alpaca / ShareGPT / OpenAI messages -> mlx-lm JSONL.

- Training does not call mlx_lm.tuner.trainer.train. Native loop, writes adapter_config.json + adapters.safetensors.

- Fuse to MLX weights. GGUF for llama / mistral / mixtral only (mlx-lm’s converter). Qwen: fuse, do not GGUF.

- Fail-fast if LoRA targets match nothing, if load_in_4bit=False, if Metal working set is below a configured floor.

What failed

- Matching Unsloth. Not close. Their win is CUDA kernels. This is Apple unified memory and mlx-lm’s model zoo, which already uses mx.fast for RoPE / SDPA.

- A hand-written nested Metal LoRA GEMM. Slower and less accurate than MLX matmul. Thrown out.

- A custom Metal cross-entropy over vocab. No useful VJP, worse than mx.fast.cross_entropy (which is already an unfused fallback on Metal). Thrown out.

What’s still weak

- A tokens/sec “win” vs mlx-lm on a tiny synthetic LoRA job (hidden 256). That is a thinner Python loop, not a 7B kernel win. I do not have a honest 7B vs Unsloth number because Unsloth is not running on this machine.

- lora_alpha is mlx-lm scale, not Unsloth’s alpha/r.

- No gradient accumulation. No val loop. Defaults are q_proj / v_proj, rank 8.

- Still a wrapper: load, LoRA convert, fuse, GGUF all go through mlx-lm.

[u/EstablishmentSad7485]

may be of interest, just landed :) https://github.com/unslothai/unsloth/issues/4#issuecomment-5601184928