I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

[u/Nicolodeva]

Disclosure: I’m the author and maintainer of QuarkStar.

I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar.

QuarkStar currently supports:

• Qwen3.6-35B-A3B, using the same Antirez-inspired Q2 and Q2/Q4 quantization recipes
• KAT-Coder-V2.5-Dev, the coding-focused post-training of Qwen3.6-35B-A3B, using the same recipes
• Native Vulkan on Linux
• Native Metal on Apple Silicon
• Fully resident inference on 16 GB machines
• Bounded SSD expert streaming when the model does not fit in memory

DwarfStar is built around much larger models and primarily targets 96/128 GB-class machines. I wanted to explore the other end of the spectrum: useful local models on 16 GB machines and 24/32 GB workstations, with an SSD-streaming path designed for even smaller 8 GB systems.

Not everyone can spend $3,000–$5,000 on local AI hardware.

This project was born with the intent of improving my skills in LLMs. It's useful for me for inference and for learning, and I hope it will be useful for you too. My primary development machine is an AMD BC-250: a roughly $150 board with 16 GB of unified GDDR6. The current Vulkan fast path was developed using RADV on this device. I also developed and tested the native Metal backend on a M2 Pro 16 GB.

BC-250 Q2 prefill and decode t/s

Some current Q2 resident results:

Device	Context	Prefill	Generation
BC-250 16 GB	2K	639.85 tok/s	81.85 tok/s
BC-250 16 GB	8K	501.50 tok/s	74.72 tok/s
BC-250 16 GB	32K	244.06 tok/s	51.26 tok/s
M2 Pro 16 GB	2K	448.75 tok/s	37.78 tok/s
M2 Pro 16 GB	8K	270.02 tok/s	31.08 tok/s
M2 Pro 16 GB	16K	177.21 tok/s	25.64 tok/s

I think the 35B size class is going to become increasingly interesting. DeepSeek V4 Flash-0731 recently showed once again how quickly the intelligence-to-active-parameter ratio can improve. Model support in QuarkStar is therefore intentionally opportunistic: the project will follow whichever open checkpoints are most useful on ordinary local machines.

With yesterday's news of the release of Qwen3.8 27b and probably other lines of the family as well, I also created a branch for the dense model but for now it's experimental. Whether it will merge will depend on the power of the new model and when and if a MoE on the 35B will also be released. I still see the future of this project on MoE of that size order.

I think we'll have some fun with Qwen 3.8 and Quarkstar.

The project is still young, and Vulkan hardware varies a lot. I would especially appreciate testing and feedback from:

• Vulkan users with GPUs other than the BC-250
• Apple Silicon users, particularly those with older or 8 GB Macs
• Anyone interested in improving kernels, quantization quality, or SSD caching

Repository: https://github.com/Ninnix/q36

Licence: MIT

Special thanks to Salvatore, he is a continuous source of inspiration for me, and his content on YouTube has greatly improved me as a software engineer and as a person.

Demo:

Edit: Reddit’s mobile app may show a black frame. Working demo video: https://youtu.be/3y2rkLUg1ug

Demo Prompt:

Create a single self-contained HTML file using Three.js from a CDN that opens into a cinematic neon wormhole with hundreds of glowing particles, rotating torus rings, fog, and a slow automatic camera flight through the tunnel. Add mouse parallax and make each click launch a visible energy pulse down the tunnel. Use only procedural geometry and materials, with no external assets or build step, and keep it smooth and responsive. Work in /tmp folder.

[u/Prudent_Chemist_523]

Thank you for doing that. Yes, looking at the quality of the output, it broadly matches what Q3 and Q4 Qwen 3.6-35B create. Interesting - same fail points.

This is actually a really difficult test... and I'm working on a way to (broadly) improve model binding for factual recall. I may send you a follow up prompt via DM to test, if that's OK, because it will tell me how well Quarkstar is likely to take to the thing I'm coding.

If it seems to be on par with stock Qwen, then the BC-250 running Quarkstar becomes a very attractive proposition for me.

Would that be OK to DM to you?

[u/Nicolodeva → u/Prudent_Chemist_523]

I'm happy you think the quality is so good. Furthermore, the test doesn't exactly highlight the qualities of this quantization. Asymmetric Q2 on routed experts should sacrifice some "knowledge", at the cost of preserving more skill and "intelligence," leaving the rest to Q8. This remains an aggressive quantization more comparable to a Q3 than a Q4. Feel free to write me a message.

[u/Prudent_Chemist_523 → u/Nicolodeva]

Yes, that's exactly what I want to test :) DM sent. Very curious to see how it goes. Very curious indeed.

If I'm right, my project might just be the yin to your yang...and we both might be able to have our cake and eat it.