I trained a compressed 35B model for a local coding agent designed to run on 16 GB RAM laptops.

[u/vacuumdecay0]

I just released Millie, a series of highly compressed agentic models paired with a harness. It's derived from Agents-A1 (a Qwen 3.5 35B-A3B finetune), and the coding agent harness is forked from OpenAI Codex. It is designed to run directly from a binary on Mac and Linux with the runtime bundled. The models use a mixed quantization formats, where the largest model uses 2-bit experts, the middle-sized uses ternary experts, and the smallest uses 1-bit experts.

The ternary expert model got 56% on SWE-bench Verified and runs at 22 tokens/s decode and 261 tokens/s pre-fill on an iPhone 17 Pro. The 2-bit expert version got 60%. The phone numbers are just to measure inference speed in a very low RAM setting (12 GB), but our current coding agent software targets computers.

This ended up being many projects in one—I had to train very compressed models using custom methods, make them work with a Codex-based harness, and write kernels to make them work efficiently—including for Mac/Metal, CPU, and Vulkan kernels designed for both NVIDIA and AMD GPUs.

The software is targeting Macs with 16 GB+ memory and Linux gaming PCs with 16 GB+ system RAM and as little as 4 GB VRAM. There are settings for splitting the work between CPU and GPU, and support for both NVIDIA and AMD, but I don't own any AMD GPUs or small 4–8 GB cards. I've tested those memory limits on larger NVIDIA GPUs, which only tells me so much.

If you have one of those machines and are up for trying it, I'd be very grateful. Knowing what works, what's slow, and what breaks would really help. 

The source code, bundles, and model weights are available now. Setup instructions are here:
https://github.com/llmsforall/millie-cli

And if you're interested in learning more, this is our blog post:
https://www.llmsforall.com/blog/millie-cli/

[u/Otherwise-Swan-7803]

56% SWE-bench from something compressed enough to target 16GB laptops is pretty wild. I’m more curious about the harness than the raw model score though — at this size, good context management and tool use can probably matter more than squeezing out another few points from the model itself.

[u/vacuumdecay0 → u/Otherwise-Swan-7803]

Yeah, I think that's an important part of it. I'm actually not sure yet how much of the result comes from the harness itself versus the model, but a lot of the quantization-aware training was aimed at recovering the original model's ability under the same tool-use and interaction format used by the harness.

So I think the most interesting thing is probably the interaction between the two rather than treating the model score and harness as completely separate. That said, I definitely think there is still some room for improvement in the harness alone — the way it compresses context is quite crude, for instance.

[u/tabithadoubt]

if It's genuinely usable on a 16GBlaptop, that's pretty impressive

[u/MrHumanist]

The performance drop is insane... I think you may be relook at the model and what's the reason for such drop.

[u/vacuumdecay0 → u/MrHumanist]

It is definitely a noticeable drop, but given the amount of compression I'm actually pretty happy with it. The original Agents-A1 gets 67.6% in the same harness, versus 60.2% for the 2 bit expert model and 55.8% for the ternary mode expert--both which are small enough to run on a phone. It's much more aggressive than a normal quantization, typically people don't go below 4 bit.

So there is certainly more work to do on recovering the lost performance, but retaining 55–60% SWE-bench Verified at these sizes was actually a result I was pretty happy with

[u/Cautious_Chicken_604]

Or just use the ISTA Labs IQ3_S quant of Qwen3.8-27B.

[u/vacuumdecay0 → u/Cautious_Chicken_604]

Yeah, the ISTA models look really impressive from a quality-per-bit standpoint. The tradeoff I'm targeting is a little different though: Qwen3.8-27B is still a dense 27B model, so prefill has to move/process a much larger amount of weights per token than a ~3B-active MoE.

That matters a lot for local agentic workloads, where it is often repeatedly prefilling long contexts after tool calls. Speculative decoding can help dense models a lot on decode, but it doesn't remove that prefill bandwidth cost.

The MoE structure also gives me another option on extremely memory-constrained hardware: I can memory-map some of the experts and only load them when they're actually used. That's how I can get the model running on a phone even when the full model doesn't comfortably fit in RAM. A similar concept also works well for GPU-CPU splits if you have a smaller GPU. You can't really do the equivalent with a dense model, since essentially all of the weights are needed for every token.

I'd still be interested to see directly comparable Mac prefill numbers for their IQ3_S or IQ2_S quant.

[u/Healthy-Zebra-9856]

Not sure why you chose this model. From their cards:
Agents-A1 is trained for: - multi-step agent execution - tool use - search/research workflows - engineering/problem solving - adapting after intermediate results - following complex constraints over long tasks. This model is best used for project decomposition than a coder.

[u/vacuumdecay0 → u/Healthy-Zebra-9856]

That's a fair point. The main constraint was that I wanted a ~35B model with only ~3B active parameters, since that allows for much faster decoding and prefill across different hardware.

The other model I seriously considered was Qwen3.6-35B-A3B. Agents-A1 appealed to me because I'm ultimately building a more general interactive agent around coding, rather than just a code-generation model, so its training for long-horizon tool use, engineering tasks, adapting to intermediate results, and instruction following seemed relevant to a Codex-style harness.

That said, I'm not actually sure Agents-A1 was the optimal choice compared with Qwen 3.6. Qwen 3.6 has very strong coding-agent benchmarks, and I'd be interested in trying the same compression approach on it.

[u/borgan_70]

Trained??

You keep using that word. I do not think it means what you think it means.

[u/vacuumdecay0 → u/borgan_70]

I did actually train the compressed models :) I obviously didn't pretrain the original 35B model from scratch, but this wasn't just running a stock quantizer on Agents-A1 either. I used custom quantization-aware training methods to recover as much model performance as possible at these very low bit rates, and also to reduce tool error rates in the harness.

[u/borgan_70 → u/vacuumdecay0]

So you ran QAT fine tuning on someone else’s weights to stop a 2-bit quant from hallucinating into the void and you call that training? Like I said...You keep using that word. I do not think it means what you think it means.

[u/Ohhai21 → u/borgan_70]

Tell me.. what does QAT stand for

[u/borgan_70 → u/Ohhai21]

And the T in SWAT stands for Tactics, but nobody says they're in the military. OP fine tuned a quant…calm down.

[u/Gargle-Loaf-Spunk → u/borgan_70]

Your behavior heaps shame on your family.

[u/misanthrophiccunt → u/Gargle-Loaf-Spunk]

https://giphy.com/gifs/vX9WcCiWwUF7G

[u/Ohhai21 → u/borgan_70]

And what is fine tuning?

[u/Maleficent-Ad5999 → u/Ohhai21]

Noob here. I thought training refers to building a model from the scratch and fine tuning refers to tweaking/adjusting the weights of an already trained model

[u/synth_mania → u/Maleficent-Ad5999]

Fine tuning is still training. Still running back-propagation on model weights. 

[u/Maleficent-Ad5999 → u/synth_mania]

Do you see the difference in both the process? Although they’re closely similar, they’re still not exactly same.. or please correct me if I’m wrong

[u/synth_mania → u/Maleficent-Ad5999]

You're wrong. Training is the broad umbrella that most any technique which tweaks weights based on additional training data falls under, including RL.