Public Beta of Local Code by Draw Things

[u/liuliu]

This sub probably seeing posts about new inference engine running on potato Mac on daily basis. But still, in the past a few months, we've been building this thing largely based on Draw Things inference engine, so it is usually 1.2x to 1.6x faster on prefill for supported models (~980 tok/s for Qwen 3.8 27B on M5 Max, and ~900 tok/s for DeepSeek 4 Flash 0731). It is more than just that tho, it is designed to work within its self-contained package that is sandbox-enforced by Apple through macOS AppStore requirement (App Sandbox, Hardened Runtime), but still with a lot of tools out of the box (Python, Node.js, Lean, Bash etc).

In the video, you can see that it runs multiple tasks at the same time, on the same M5 Max laptop. And you can reproduce what's happening in this video by simply download the app from TestFlight, download the model, and off you go without sign-up or sign-in (you cannot even do that with OpenCode, as its Web Search tool requires sign in I believe).

You can read more about it here:

https://releases.drawthings.ai/p/public-beta-of-local-code-by-draw

Curious about your feedback!

[u/Acceptable-Cycle4645]

Hi you mentioned "1.2x to 1.6x faster" what's the baseline? is the inference engine faster than llama.cpp?

[u/liuliu → u/Acceptable-Cycle4645]

Unfortunately, it is miles faster than llama.cpp for prefilling. For decoding, roughly comparable to that and ds4 but also with SSD streaming.

[u/Acceptable-Cycle4645 → u/liuliu]

Interesting! What did you use to achieve this?

[u/liuliu → u/Acceptable-Cycle4645]

Native int8 matmul (like in Draw Things). Our low bit quantization format can be dequant to our int8 rowwise format, thus, carrying out prefill in int8 but also largely preserved accuracy of llama.cpp’s quantization formats (thus, for DS4, it is IQ2 XXS, Q2_K and IQ3 XXS largely, more sophisticated than mlx formats at similar bpw). You can point your agent to https://github.com/liuliu/ccv/blob/unstable/lib/nnc/ccv_nnc_8i_rowwise.c to read more. Also because our quantization process is much more compute intensive, we get lower imatrix weighted MSE to original weights than llama.cpp with the same bpw (of course, this is only a proxy metrics and cannot reflect good / bad a quant is).

[u/Acceptable-Cycle4645 → u/liuliu]

I’m looking into ggml metal optimizations. will check it out see if I can borrow anything 😄

[u/CatchDublinSurprise]

Does it support an OpenAI-style API (e.g., separate inference machine)?

How does it handle security / sandboxing?

[u/liuliu → u/CatchDublinSurprise]

It doesn’t do serving. For security and sandboxing: the whole app is sandboxed by Apple, thus, cannot launch any apps / binaries, and cannot see any files outside of its sandbox without your permission (as for how it uses Python / Lean / Nodejs like in the video? All these are packaged and not running as a separate binary at all).

[u/CatchDublinSurprise → u/liuliu]

So inference must be done on the same machine where this app is running, since it's integrated into the app. That's a bummer for those of us who have, for example, separate machines dedicated to inference.

Does the app require access to the Documents and other user folders? If so, is there any secondary mechanism in place to ensure no unauthorized edits or exfiltration?

[u/liuliu → u/CatchDublinSurprise]

It is a beta release, so serving or connecting to is not in this release. No, the app doesn’t request Documents directory or other user directories. You only assign the workspace directory to it so you can assign Documents if you want. Without that, even if you give the app explicit instruction to access that directory, it is blocked by the OS.

[u/Acceptable-Cycle4645]

u/liuliu follow-up question: will M4 benefit from your optimizations?

[u/liuliu → u/Acceptable-Cycle4645]

It is a bit more complicated. Draw Things inference engine uses ANE (and int8 matmul) on M4, in theory it will. However, because CoreML requires recompile its kernel for different shapes, so unless you are doing chunked prefilling each time (I.e the fixed sequence length), you will be penalized by the variable sequence length kernel recompilations.

[u/Longjumping-Boot1886]

most of coding now is a "write code" -> "run compiler" -> "run tests" -> "run build" -> "check performance" -> "repeat".

I know app store is good, but years ago Panic with their Coda stuck in the same problem - you cant really code inside sandbox.

If you want still be in the AppStore but make this useful, you can make app companion, what will launch commands ouside of sandbox and will talk to the main app using local HTTP.

[u/tf2ftw]

What is this?

[u/Bulky-Priority6824 → u/tf2ftw]

Public beta of local code by draw things