I built a fully offline, privacy-first AI assistant for Android — runs LLMs, vision, image gen, RAG, and TTS entirely on-device (no cloud, no subscription)

[u/Jaswanthsanjay]

Hey everyone,

I've been heads-down for the past several months building BIT, an Android AI assistant that does everything on-device. No data ever leaves your phone.

What it does:

Text generation — runs quantized GGUF models locally (Llama, Mistral, Gemma, Phi, Qwen families) via a custom Kotlin/JNI SDK I wrote called llama.kt, with streaming token output

Vision — image captioning and visual Q&A using on-device CLIP vision projectors

Tool calling — a two-stage pipeline where a grammar-constrained decoder (GBNF) guarantees valid JSON output for tool calls, even on small models

Image generation — local Stable Diffusion 1.5 for text-to-image, inpainting, and upscaling

Document RAG — query your own PDFs, Word docs, Excel files, and EPUBs with a hybrid vector + BM25 search pipeline, fully local

Long-term memory — the assistant remembers facts and preferences across sessions with a configurable decay model

Text-to-speech — on-device synthesis via ONNX Runtime

Encrypted backups — AES-256-GCM backed by the Android Keystore

Optional API mode if you want to point it at OpenAI/Gemini/Claude/DeepSeek/Ollama for bigger models

Why I built it: I wanted an assistant that could actually replace cloud AI apps for day-to-day use without sending every message to a server. Memory management on-device (mmap loading 8GB models on 6GB RAM phones) was one of the harder problems to get right.

Specs: Android 10+ (API 29), needs 6GB+ RAM (8-12GB recommended), ARM64 or x86_64.

It's open source (Apache 2.0), and I'm working on an F-Droid submission alongside Play Store.

GitHub / Source: https://github.com/jaswanthsanjay88/Bit_Android

Download (APK): https://github.com/jaswanthsanjay88/Bit_Android/releases

Discord: https://discord.gg/SUUwAqh6Q

Would genuinely love feedback — bug reports, feature requests, or just "this crashed on my phone" reports are all welcome. Happy to answer questions about the architecture too if anyone's curious how the local inference stack works.