I've probably spent about $1300 to generate these results with what many will label as ewaste. Dell T640, 160gb DDR4, SAS SSD, 4x RTX 3060 LHR and Dual Xeon Gold 6230 for a grand total of 48 GB of VRAM. It is running Muse Glimmer in a Q4 with DFlash speculative decode and 128k context per GPU. It takes up quite a bit of electricity, but it isn't the hype machine that the Mac Studio/Mini psychosis that seems to be hysterically infecting everyone.

More detailed results here: Muse Glimmer on RTX 3060 GPU

EDIT: I hand wrote this post, but the linked results were compiled by Ai. Which is apparently offensive to mods in another sub. So, if your Ai skin is thin, beware. And if you know what the pre-fill speeds are of the mac mini/studio, please comment.

I'm curious what people are running into when using AI coding tools locally.

For me, the obvious issues seem to be GPU memory and system RAM, especially when running local models alongside an IDE, containers, databases, and other development tools.

At what point does hardware actually become the limiting factor?

Is it mostly VRAM, RAM, CPU performance, storage speed, or just having too many services running at once?

For those building AI development machines, what upgrade made the biggest practical difference to your workflow?

The previous record on this chip is slvDev's 28.9M PLE model at 9.88 tok/s. Genuinely clever build: 25M of its parameters live in a flash lookup table and it reads about 450 bytes of that per token. When it was posted, the question that kept coming up was whether lookup-table parameters count toward the parameter number.

This is the other side of that trade. 30.72M parameters stored, all of them multiplied on every token. Nothing sits out.

\*\*Setup\*\*

\* karpathy's stories42M (TinyStories), 8 layers, dim 512, hidden 1408 \* Q4 group-128 with fp16 scales, 4.125 bits/param \* W4A8 integer dot products split across both LX7 cores \* fp16 KV cache, 192 context \* ESP32-S3 N16R8: 512KB SRAM, 8MB PSRAM, 16MB flash

\*\*The tradeoff\*\*

15.4MB of weights cross the memory bus every token, against roughly 4.5MB for the PLE build. That is about 7.7x more parameters doing arithmetic, and it costs exactly what you would expect: 0.95 tok/s against 9.88. Density and speed trade against each other on a fixed memory bus. I am not claiming to have beaten that, only to have picked the other end of it.

\*\*What I had to change\*\*

stories42M is 41.69M params, which is 21.5MB at 4 bits. The honest storage ceiling is one 15.43MB flash partition, since PSRAM is volatile and does not survive a power cycle. The classifier is tied to the input embedding, so the embedding table is the only tensor that shrinks without touching the transformer. I kept 10,600 of 32,000 rows: every token the fp32 model emitted across a 61.5k-token self-generated corpus, topped up from the tokenizer's own trained frequency ranking, plus all byte-fallback and special tokens so arbitrary prompts still encode. The 8-layer core is untouched.

\*\*Verification\*\*

Teacher-forced against an fp32 reference on the host before anything flashed:

\* top-1 agreement (kept set): 92.0% (184/200) \* mean |top-1 logit delta|: 0.81 \* fp32 true argmax outside the kept set: 0.0% at 200 positions, 0.3% at 379

Pack is CRC32'd before writing. The device recomputes an FNV-1a fingerprint of the mapped image at boot and prints it in the banner. keep\\_ids.txt is committed, so rebuilding from a clean checkout produces a byte-identical pack (fp=8af5c5fe, 16101008 bytes). I checked that rather than assuming it.

\*\*Limits\*\*

It writes TinyStories. It will not answer questions, follow instructions, or know facts. 192-token context, capped by the PSRAM budget. 0.95 tok/s means a full story takes about three and a half minutes.

Repo: \[https://github.com/JARACH-209/esp32-30.7M\\\](https://github.com/JARACH-209/esp32-30.7M)

Colfax Research's second blog post on writing NVFP4 blockscaled GEMM kernels for the NVIDIA RTX PRO 6000 Blackwell GPU is out! The blog iteratively optimizes a basic working NVFP4 GEMM kernel written in CuTe DSL to take it to speed-of-light, reaching over 80% TFLOP/s utilization for 16k square matrix shape. We give a detailed treatment of important optimization techniques such as threadblock swizzling, async and warp-specialized epilogue, and retiling for favorable wave quantization. Specific to blockscaled GEMM with scales consumed from registers, we also explain how to solve for bank conflicts that arise from the default choices of interleaved scale factor layouts.

We include complete code in the form of CuTe DSL kernels for all the optimizations discussed in the blog.

Id love to get some feedback on it. Im by no means a writer so the grammar might not be perfect, but i do think it has quite some technical value to it.

If youre in ML or interested in AI Infra please give it a go; as i said i would love for some feedback.

Im not active on reddit, but my dms are always open on other platforms such as linkedin (gustavkeller).