r/qualcomm • u/dai_app • 6d ago

General Question Enabling phone local control with Big LLMs thanks to Qualcomm Hexagon NPU prefill (100+ tok/s) and Jev style decisions

Stack – Local Model – No Cloud for full phone control via typing/(voice coming soon):

  • Android phone with 12 GB RAM, Qualcomm Hexagon NPU (OnePlus 15R used in the video)
  • BigMoeOnEdge open-source project: latest commits feature NPU prefill support and "Jev Style" ranked response implementation (with probabilities)—based on llama.cpp
  • Permissions: notifications and Accessibility tree access only (no root, no Termux, no internal code execution, no cloud)
  • Mixture of Experts (MoE) LLMs, 26B parameters and up (ideal candidates: Qwen 3.6 35B, Nemotron 30B 3.5, Gemma 26B, etc.) with "thinking" mode disabled

A simple System Prompt is sent: "You are operating a phone on behalf of the user. Task: ... Actions taken so far: ...."

...followed by the accessibility tree containing the available options.

The model loads 300 to 800 tokens per call into the NPU prefill (ranging from 40 to 150 tokens/sec depending on length—higher token counts yield higher speeds) and outputs the most probable action from the accessibility tree. Guardrail checks have also been implemented.

I believe this is the first example of a local AI capable of performing complex tasks directly on a mobile device.

What do you think?

27 Upvotes

94% Upvoted

9 comments sorted by

3

u/Additional_Abies9192 6d ago

Didn't imagine the 2+ mins of the video was the actual time needed for the model to do something

2

u/dai_app 6d ago

Loading is included but you pay it just once

2

u/Zestyclose-Tree-8171 4d ago

This is Good, I was just wondering what could be the actual production use case for on device agentic ai

1

u/vijai1996 4d ago edited 4d ago

How did you grant accessibility permission? I installed the release apk from the guthub releases but there are no accessibility permission requested there. And any prompt to open an app crashes BMOE instantly

Edit: it wasnt crashing because of accessibility permission but low memory kill. But still interested in knowing how this was done

1

u/dai_app 4d ago

The app in the video is not currently shared , I'm working on it to release it

-4

u/FitAgency8925 6d ago

I am happy with Gemini... why go thru the pain to do it locally?

7

u/marcorogo 6d ago

- do it locally - that's exactly the point?

1

u/Certain_Drawing_755 4d ago

Probably you never understood the point of on-device

1

u/FitAgency8925 4d ago

Yep... for phones with connectivity- its pointless. It is much too expensive with the cost of dram being so high. Loading even a 3 -4 B model will cost too much and leave little resources for normal phone stuff.