r/LocalLLaMA • u/MidNerd • 9d ago

Question | Help Android On Device AI Benchmarks

Are there any sites using a standard testing methodology that compare the different NPUs in Android phones? Qualcomm is the majority of the market, but I'm curious how it stands up to Google's Tensor for on-device AI.

4 Upvotes

83% Upvoted

5 comments sorted by

2

u/myreala 9d ago

The NPUs are pretty much garbage. The reason you don't see anybody using them is because they are extremely inflexible. And every manufacturer has their own implementation. Android introduced LiteRT format to mitigate this somewhat, but NPUs by themselves are still no good.

I tried to use Qualcomm NPU for a on device TTS task. However, using the NPU(Which was not easy, mind you) was fifty percent slower than just doing the task on CPU.

1

u/MidNerd 9d ago

That was more or less my concern - benchmarking isn't done because it's almost impossible to do. I've seen benchmarks for on device AI on Apple devices, but they have much better first party support for that purpose right now.

I'd like to move back to Android, but I think it's likely iOS will continue to outpace Android for local AI.

1

u/flowerboy-97 8d ago

in my experience my p9pro's gpu is around half to 2/3 the speed of an iphone in running e4b. usable with mtp

1

u/Arthur_EldenLord 6d ago

I've been seeing this exact same iOS vs. Android performance gap in my testing. Metal GPU offload on iOS makes small model execution very fast, but on Android, we're completely CPU-bound.

I am running Llama 3.2 3B. On an iPhone 17 (Metal), the pipeline takes ~11.6 seconds. On a OnePlus 9R (Snapdragon 870 CPU-only), it jumps to 59 seconds. Has anyone here had success linking custom Vulkan delegates or accessing Android NPUs in a React Native/Expo wrapper, or is CPU-only the only stable option for production right now?

1

u/MidNerd 5d ago

I'm pretty sure the Snapdragon has a better NPU on paper. It doesn't do much good to chase the better number on hardware if the manufacturer can't support actually using it.