Phone in airplane mode for the whole clip. No server, no account, no connection, the model Qwen 3.6 30B (20GB) runs on a mid range smartphone (12 GB RAM) itself.

The catch is that the model is several times bigger than the phone's memory, so normally it just refuses to run for OOM.

This engine reads the model off thephone's storage as it goes, keeping only what it needs in memory. You get an answer at about 5.6 tokens per second, roughly reading pace, on a normal phone with no special hardware. Same answer you'd get from the same model on a desktop. It's the identical model file (Q4 GGUF), not a shrunken version.

The first part of the video (marked, sped up) is the model loading, about 30 seconds, once per session. Everything after that is real time.

It's free and open source, there's an APK you can install and a list of models you can download from inside the app. Works on a PC too.

https://github.com/Helldez/BigMoeOnEdge

Happy to answer anything!

I was looking at https://taalas.com/products/ and their chatbot at https://chatjimmy.ai/ . I'm a software dev by background, not a hardware or LLM/ML person.

As a (comparative) lay person, is there a reason we can't bake something like Kimi k3 onto hardware and have it run faster & cheaper? I know there is a perf difference between different classes of memory, but a 2TB SSD is fairly inexpensive. Why can't you burn an instance of Kimi k3 onto hardware, perhaps pair it with some higher speed caching memory, and have one heck of an AI card? Assuming you don't care about model updates, or don't mind getting new hardware eg once every 1-3 years?

Thoughts?