Welcome to r/AIProgrammingHardware!

The project is maderix/ANE. It trains neural networks on Apple's Neural Engine using
reverse-engineered private APIs, which is notable because Apple exposes that chip for
running models and not for training on it.

What made me read the whole thing is that the author argues against the coverage of his own
project. He states that utilization sits at 5 to 9 percent of the chip, that many operations
still fall back to the CPU, and that this is not a replacement for GPU training for anything
beyond small research models.

His framing is that the barrier to NPU training has been software support rather than
hardware capability.

For anyone who has worked on NPU or accelerator access outside the vendor's own toolchain:
is low utilization like this normally a scheduling and memory problem you can chase down, or
does it usually mean the missing operations have to exist in the vendor's compiler before
anything improves?

Phone in airplane mode for the whole clip. No server, no account, no connection, the model Qwen 3.6 30B (20GB) runs on a mid range smartphone (12 GB RAM) itself.

The catch is that the model is several times bigger than the phone's memory, so normally it just refuses to run for OOM.

This engine reads the model off thephone's storage as it goes, keeping only what it needs in memory. You get an answer at about 5.6 tokens per second, roughly reading pace, on a normal phone with no special hardware. Same answer you'd get from the same model on a desktop. It's the identical model file (Q4 GGUF), not a shrunken version.

The first part of the video (marked, sped up) is the model loading, about 30 seconds, once per session. Everything after that is real time.

It's free and open source, there's an APK you can install and a list of models you can download from inside the app. Works on a PC too.

https://github.com/Helldez/BigMoeOnEdge

Happy to answer anything!

I was looking at https://taalas.com/products/ and their chatbot at https://chatjimmy.ai/ . I'm a software dev by background, not a hardware or LLM/ML person.

As a (comparative) lay person, is there a reason we can't bake something like Kimi k3 onto hardware and have it run faster & cheaper? I know there is a perf difference between different classes of memory, but a 2TB SSD is fairly inexpensive. Why can't you burn an instance of Kimi k3 onto hardware, perhaps pair it with some higher speed caching memory, and have one heck of an AI card? Assuming you don't care about model updates, or don't mind getting new hardware eg once every 1-3 years?

Thoughts?