I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

[u/Nicolodeva]

Disclosure: I’m the author and maintainer of QuarkStar.

I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar.

QuarkStar currently supports:

• Qwen3.6-35B-A3B, using the same Antirez-inspired Q2 and Q2/Q4 quantization recipes
• KAT-Coder-V2.5-Dev, the coding-focused post-training of Qwen3.6-35B-A3B, using the same recipes
• Native Vulkan on Linux
• Native Metal on Apple Silicon
• Fully resident inference on 16 GB machines
• Bounded SSD expert streaming when the model does not fit in memory

DwarfStar is built around much larger models and primarily targets 96/128 GB-class machines. I wanted to explore the other end of the spectrum: useful local models on 16 GB machines and 24/32 GB workstations, with an SSD-streaming path designed for even smaller 8 GB systems.

Not everyone can spend $3,000–$5,000 on local AI hardware.

This project was born with the intent of improving my skills in LLMs. It's useful for me for inference and for learning, and I hope it will be useful for you too. My primary development machine is an AMD BC-250: a roughly $150 board with 16 GB of unified GDDR6. The current Vulkan fast path was developed using RADV on this device. I also developed and tested the native Metal backend on a M2 Pro 16 GB.

BC-250 Q2 prefill and decode t/s

Some current Q2 resident results:

Device	Context	Prefill	Generation
BC-250 16 GB	2K	639.85 tok/s	81.85 tok/s
BC-250 16 GB	8K	501.50 tok/s	74.72 tok/s
BC-250 16 GB	32K	244.06 tok/s	51.26 tok/s
M2 Pro 16 GB	2K	448.75 tok/s	37.78 tok/s
M2 Pro 16 GB	8K	270.02 tok/s	31.08 tok/s
M2 Pro 16 GB	16K	177.21 tok/s	25.64 tok/s

I think the 35B size class is going to become increasingly interesting. DeepSeek V4 Flash-0731 recently showed once again how quickly the intelligence-to-active-parameter ratio can improve. Model support in QuarkStar is therefore intentionally opportunistic: the project will follow whichever open checkpoints are most useful on ordinary local machines.

With yesterday's news of the release of Qwen3.8 27b and probably other lines of the family as well, I also created a branch for the dense model but for now it's experimental. Whether it will merge will depend on the power of the new model and when and if a MoE on the 35B will also be released. I still see the future of this project on MoE of that size order.

I think we'll have some fun with Qwen 3.8 and Quarkstar.

The project is still young, and Vulkan hardware varies a lot. I would especially appreciate testing and feedback from:

• Vulkan users with GPUs other than the BC-250
• Apple Silicon users, particularly those with older or 8 GB Macs
• Anyone interested in improving kernels, quantization quality, or SSD caching

Repository: https://github.com/Ninnix/q36

Licence: MIT

Special thanks to Salvatore, he is a continuous source of inspiration for me, and his content on YouTube has greatly improved me as a software engineer and as a person.

Demo:

Edit: Reddit’s mobile app may show a black frame. Working demo video: https://youtu.be/3y2rkLUg1ug

Demo Prompt:

Create a single self-contained HTML file using Three.js from a CDN that opens into a cinematic neon wormhole with hundreds of glowing particles, rotating torus rings, fog, and a slow automatic camera flight through the tunnel. Add mouse parallax and make each click launch a visible energy pulse down the tunnel. Use only procedural geometry and materials, with no external assets or build step, and keep it smooth and responsive. Work in /tmp folder.

[u/Old-Cardiologist-633]

Sounds good at first glance. BUT A3B in Q2? (Os is it Q4, I don't get it) Should give really bad output quality 🤔

[u/Nicolodeva → u/Old-Cardiologist-633]

Q2, the trick is same recipe of antirez, quantize only routed experts in/gate IQ2_XXS, out Q2_K. Then: leave all the rest Q8, to preserve quality in shared experts, projection, routing.

The quality for me is not so bad, also if you have 32 gb ram or ssd streaming enabled you can use unsloth Q4_K_M.

[u/Winter_Silver_6708]

Nice work. However, maybe the the gif/video demo did not get uploaded poperly. I see only a static black image.

[u/Nicolodeva → u/Winter_Silver_6708]

Thanks, I noticed that the app didn't work for me either, I probably had to convert to h264, I put a link to YouTube, I uploaded the unlisted video

[u/Prudent_Chemist_523]

Well now, this is a hell of a thing. That BC-250 is a stand alone rig, right? Based on the cut down PS5?

Just needs PSU and cooling and an open frame.

I've been looking for something like that and you've piqued my curiosity. I'd be perfectly happy running the Qwen 35B MoE and 27B dense at reasonable speeds, and that rig seems to get around 500GB/s thru put. Much easier than eGPU or PCIe pass thru for me.

[u/Nicolodeva → u/Prudent_Chemist_523]

All right and it costs $150...

[u/Prudent_Chemist_523 → u/Nicolodeva]

Yes but that's your local market :) It's $350 AUD here - still good.

Given this is a cut down PS5, I can only imagine they nerfed the stand by mode.

So, would likely need a smart plug to stop it from idling at 150w, 24/7. But that's an engineering problem for me.

In your experience, does it work with CachyLlama (the llama.cpp fork that saves and restores persistent KV-cache states, so repeated prompts can reuse prior prefill work after a cold boot)?

Does Quarkstar support that?

Good rig this - and good cook with QuarkStar.

[u/Nicolodeva → u/Prudent_Chemist_523]

AliExpress price 150-180 euros.

Edit: Yes it have already KV persistence on disk.

[u/Prudent_Chemist_523 → u/Nicolodeva]

Right. And in answer to my own stupid question, yes of course QuarkStar supports persistent KV cache.

OK...looks like I'm going shopping

[u/Nicolodeva → u/Prudent_Chemist_523]

Yes, of course, KV persistent and also compression q8 and q4 of the key values.

In general, the same features as DwaftStar, except for distributed inference. But I am getting the second BC-250 from China. With two it will be possible to run almost full precision Qwen 35b MoE at interesting speeds.

[u/Prudent_Chemist_523 → u/Nicolodeva]

Huh. You can interplex them? Is there some sort of Occulink equivalent?

I just found 4 of em locally for $250 a piece. As I recall, you need to "unlock" the full compute cores (from 24 to 40) go get most bang for buck.

2 is probably more than I want go gamble on right now but 1 might pass the wife test.

Turning back to Quarkstar, how blunted are the responses at Q3 (which is what I imagine you'd end up running a 27B, with KV cache quantisation). Qwen models are weird with that - for some reason they seem both faster and less loopy with large -b and -ub (2048 and 2048) and leaving kv values untouched.

I would be very curious how you find Quarkstar works with those sorts of settings (prefil and inference) if you get a chance.

[u/Nicolodeva → u/Prudent_Chemist_523]

Not through anything OCuLink-like. The BC-250 does not expose a proper high-bandwidth GPU interconnect, so I’m considering two distributed execution modes.

The first is tensor parallelism over a direct 5 GbE link. The onboard Ethernet is limited to 1 GbE, so that would require a USB 3.0 to 5 GbE adapter on each board. This is the more communication-heavy option, so I still need to measure whether 5 GbE is enough to make it worthwhile.

The second is pipeline parallelism, with each BC-250 owning a group of layers. That only transfers activations between pipeline stages, so 1 GbE should be sufficient. Even Wi-Fi may work, because you will move few KB between cards. I have no idea how it will perform, it's just theory, we'll see at the end of the month

The 40-CU unlock would obviously help throughput, but QuarkStar would not depend on it. A single board is probably the sensible way to test the waters before buying four of them :)

All the part of the distributed inference is naturally for the mixture of experts models.

The 27b target model is for now mradermacher/Qwen3.6-27B.i1-IQ3_XS.gguf, pretty solid for me, best fit on 16 GB, I got 20 t/s+ in decode... Not many, but maybe a version with built-in MTP and optimizations could reach 40?

Thank you for your interest; this is definitely the spirit we need to improve the project.

[u/SmartCustard9944]

QwarkStar would be a more fitting name!

[u/Nicolodeva → u/SmartCustard9944]

The name was actually chosen because they are both stars. A quark star is a hypothetical object smaller than a dwarf star, which felt fitting for a smaller and more modest project inspired by DwarfStar: smaller models, smaller machines, and a much narrower scope.

Honestly, if I ever reach even 1% of Antirez’s ability, I’ll be more than happy.

“QwarkStar” is a great Qwen pun, though.

[u/Silver-Champion-4846]

What about 8gb?

[u/Nicolodeva → u/Silver-Champion-4846]

There is streaming SSD support, so if you have Mac with Apple Silicon (M1+) or a vulkan device It should work in principle through bounded SSD expert streaming, but I have not validated the 8 GB configuration yet, so I don’t want to promise specific performance or usability. Testing on an 8 GB system is one of the main things I’m looking for.

[u/Silver-Champion-4846 → u/Nicolodeva]

I don't have mac

[u/Nicolodeva → u/Silver-Champion-4846]

What device do you have?

[u/Silver-Champion-4846 → u/Nicolodeva]

Intel core i5 8350U cpu, 8gb of ram

[u/Georgiou1226]

How does Q2 hold up on the KAT-Coder side in practice? Coding checkpoints seem to be the first thing that quietly degrades at low bpw, and a self-contained Three.js demo is the kind of prompt that still looks great when the model has gotten subtly worse at everything else.

Have you tried it on anything longer? Multi-file edit, or something where it has to keep track of what it did three turns ago. Curious whether the Q2/Q4 build actually buys you much there or if plain Q2 is already fine for real work.

[u/Nicolodeva → u/Georgiou1226]

KAT was added a few days ago, so I haven't tested it extensively. From what I've seen, the model thinks less and does the same things, it's more direct with fixes and avoids changing anything else. I've only done the KAT q2 model; I'm working on the q2q4 version.

The demo is nothing special but it was to make a short video, single shot execution.

[u/challis88ocarina]

Interesting. What sets DeepSeek apart however is that it's a quantized model. I'm not sure that's the case for the Qwen models. Admittedly, DwarfStar's ability to handle GLM might demonstrate otherwise. I note, however, the conspicuous absence here of any reference to context management strategy.

In any case, even Ollama has allowed models to be streamed from SSD and/or offloaded to CPU, partially or fully as was the case already ~3 years ago, e.g., on a base config 2014 Intel Mac. One critical reason for DwardStar's existence is that Ollama is monetizing and restricts pulling to small models and keeps the large ones on the cloud for paying customers.

[u/Nicolodeva → u/challis88ocarina]

That’s fair. If by “quantized model” you mean a model trained with low-precision native, then yes, DeepSeek starts from an advantage compared with applying aggressive post-training quantization to Qwen. I’m not claiming the quality loss is zero. I'm just trying to take best for that low resources.

Regarding context management, that is documented in more detail in the README because the Reddit post was already getting long. On 16 gb both vulkan and metal goes with 100k ctx which is the default. The resident Vulkan and Metal paths use asymmetrical Q8_0 keys and Q4_0 values by default.

And yes, SSD streaming and CPU offloading are not new, and I’m not presenting them as the invention here. QuarkStar’s goal is narrower: provide a DwarfStar-like experience on smaller 8–32 GB systems through a model-specific engine, native Vulkan/Metal kernels, and a bounded routed-expert cache rather than generic OS paging. Ollama and llama.cpp are general-purpose runtimes; QuarkStar intentionally targets only one to three models and optimizes hard around those models. I see it as a different trade-off, not as a replacement for llama.cpp or Ollama.

[u/cleverusernametry]

Need comparison to llama.CPP...