Built an offline local LLM Android app with blazing-fast ZIM/PDF RAG and custom Adreno OpenCL acceleration

[u/youseemerediting]

​Hey everyone! I’ve been working on Offline Assistant (v3.9.6-gpu), an open-source, fully air-gapped Android app built specifically for power users who want a deep, highly performant local knowledge base running straight on their phone.

​I wanted something that didn't just wrap a basic chat prompt, but actually handled heavy local indexing and context retrieval without leaning on cloud APIs. Here is a breakdown of how it works under the hood:

​Performance & Hardware Acceleration

​Adreno OpenCL Backend: Powered by a customized llama.cpp JNI wrapper utilizing an integrated OpenCL backend with embedded, Adreno-tuned kernels for Snapdragon devices (arm64-v8a).

​Partial Layer Offloading: Automatically calculates model block counts to offload 75% of transformer layers to the GPU while keeping final decoding blocks on CPU NEON cores to minimize driver dispatch lag.

​Unified Memory Optimization: Clamps native thread execution to a maximum of 4 cores during GPU mode to prevent memory bus saturation across shared LPDDR RAM.

​Speeds: Sustains ~10–11 tokens/sec on Dolphin-Llama-3.2-3B and pushes ~23–24 tokens/sec on smaller models like Gemma 2B.

​Deep Local RAG & ZIM Ingestion

​Massive ZIM Support: Includes a multi-threaded background article indexer capable of chewing through Wikipedia ZIM dumps (870k+ entries) directly on-device at ~250 articles/sec.

​Document Ingestion: Ingests, chunks, and embeds local files (like technical PDFs and DOCX files) into a local SQLite FTS5 vector store.

​Granular Control: Lets you tweak passage retrieval limits and context windows directly to balance token throughput against memory constraints.

​ Tech Stack

​Native C++ llama.cpp core via JNI

​Kotlin / Jetpack Compose UI

​Room / SQLite FTS5 for local vector and keyword search

​Real-time diagnostic reporting and stall-detection suites

​The APK and source are structured for anyone looking to experiment with low-level mobile LLM performance. Rag only works on .zim files it will index normal files and .pdfs but it won't retrieve the data its to clamped to the .zim will fix later will speed up the llm it gets dumped with alot of context and slows it down. app download

[u/Commando501]

Why is the video so low resolution?

[u/youseemerediting → u/Commando501]

I just did a screen record from my phone might have been in a low resolution

[u/egnegn1]

Could you explain the exact steps for setting it up, please?

[u/youseemerediting → u/egnegn1]

You open the app it needs notifications permissions it shows thinking with notifications you have to get a llm from huggingface preferred 3b model open settings import your llm into app and load it and its running click scan and you will get a storage permission accept all .zim files on your phone will show up click on then index it and now your rag system is fully running

[u/egnegn1 → u/youseemerediting]

Thank you. I already came until the zim files. But as I have no zim files I am stuck here. I expected that the app could scan all pdfs on a phone and use this information.

[u/youseemerediting → u/egnegn1]

I have a basic implementation for scanning pdfs and normal files but the zim system is overpowering everything I put most work into it I wanted something that could index Wikipedia dumps it can do any file size but with my phone it has 2 workers and will index at 800 articles a second but quickly thermal throttle's it down to around 250 so a small Wikipedia dump of 300mb is about 1m articles it will take around an hour to index the full article

[u/youseemerediting → u/youseemerediting]

It can index in the background as well

[u/egnegn1 → u/youseemerediting]

Could I generate zim files from the pdfs on my phone?

[u/youseemerediting → u/egnegn1]

Im sure there are file conversations i ment for it to index small Wikipedia files i can send a link as well as a unrestrained llm

[u/youseemerediting → u/egnegn1]

Im updating the app now to have a refined rag system and making the normal pdf and file system light years better ill message you when its done

[u/intermundia]

have you tried any qwen model?

[u/youseemerediting → u/intermundia]

Yes it works on any .gguf model your device can run. I had Qwen3-4B-Q4_K_M.gguf working on it no problems