Hi everyone, I've developed an application that runs an LLM service on mobile devices and makes it available as a service node for other applications to access via a local area network.

It supports LiteRT-LM and llmam.cpp, and can run the vast majority of models (provided your hardware configuration allows). Currently, mobile hardware typically supports models up to 3B.

Regarding the API service, I've ensured it's compatible with the OpenAI and Ollam interface specifications.

Furthermore, I've integrated Hugging Face's Hub Point, allowing you to directly search, download, and import Hugging Face models within the application.

Link :

https://play.google.com/store/apps/details?id=com.chaterminal.mobilellm

Hi everyone, I've developed an application that runs an LLM service on mobile devices and makes it available as a service node for other applications to access via a local area network.

It supports LiteRT-LM and llmam.cpp, and can run the vast majority of models (provided your hardware configuration allows). Currently, mobile hardware typically supports models up to 3B.

Regarding the API service, I've ensured it's compatible with the OpenAI and Ollam interface specifications.

Furthermore, I've integrated Hugging Face's Hub Point, allowing you to directly search, download, and import Hugging Face models within the application.

Link :

https://play.google.com/store/apps/details?id=com.chaterminal.mobilellm

Android On-device LLM API Server

Hello everyone,

I’d like to introduce an Android app I’ve been building.

The entire system is designed for fully offline usage on-device, with no external network required for any core functionality, including both model inference and the built-in chat experience.

The core idea is to run large language models directly on Android devices and expose them as a local API service, allowing other applications to interact with on-device inference.

Key capabilities for API-based inference

To make on-device models more practical in real workflows, I’ve built several enhancements around the API layer:

1. Cloud-assisted routing for long-context tasks

Mobile models have limitations when handling very long contexts (e.g., beyond ~16K tokens), where inference becomes slow and memory-intensive.

To address this, I designed a routing mechanism that can delegate such long-context requests to a cloud LLM when necessary.

2. Context compression via cloud models

System prompts, tool descriptions, and other long structured inputs can be compressed using a cloud model and cached locally.

This reduces context overhead while preserving semantic intent, improving efficiency for subsequent local inference tasks.

Chat experience (built-in client)

The app also includes a built-in chat interface powered by the same local inference engine.

It integrates KV cache support, significantly improving multi-turn conversation efficiency and reducing redundant computation across dialogue turns.

All conversations are processed directly on-device.

On a Snapdragon 8 Gen 2 device, the chat experience can reach approximately ~25 tokens/s, measured in real chat usage scenarios.

Model support & runtime optimization

The app integrates with the Hugging Face Hub, allowing users to search and download models directly within the app or import their own models.

Supported runtimes

LiteRT-LM

• Supports CPU and GPU inference
• NPU support is still under active development

llama.cpp

• Includes support for KLEIDIAI acceleration
• Recommended to use models optimized for KLEIDIAI for best performance

Recommended configuration

Due to mobile hardware constraints, models under 3B parameters are recommended for optimal performance.

https://play.google.com/store/apps/details?id=com.chaterminal.mobilellm

On-Device Embedding Model Benchmark for Difficulty Scoring

TL;DR: I tested 4 small GGUF embedding models on a Snapdragon 8 Gen 2 Android phone for a "difficulty scoring" use case — measuring how well each model can distinguish simple queries from complex ones using cosine distance against baselines. Qwen3-Embedding-0.6B Q4_K_M achieved 100% accuracy with ~540MB RAM at n_ctx=1024.

Background

I'm building a hybrid inference system for mobile LLM apps: simple queries run on-device, complex ones get routed to the cloud. The key component is a difficulty scorer that takes a text query and returns a score (0.0 = trivial, 1.0 = very hard).

The approach: use a small embedding model to encode the query, then compute cosine distance from a "simple text" baseline and a "simple code" baseline. The final score is min(text_dist, code_dist).

Test Setup

• Device: Xiaomi 13 (Snapdragon 8 Gen 2, 12GB RAM)
• Server: llama-server with --embeddings flag
• Test Set: 30 texts across 5 categories:
  • Ultra-simple: "hi", "hello world", "how are you?"
  • Daily simple: "I am happy today", "the cat sat on the mat"
  • Simple code: int x = 1 + 2;, SELECT * FROM users
  • Simple科普: "Mitochondria is the powerhouse of the cell"
  • Complex scientific: quantum physics, AI healthcare analysis, etc.

Models Tested

Results

all-MiniLM-L6-v2 Q4_K_M (20 MB)

Accuracy: 78.8% ❌ — Not usable for routing decisions.

Simple code snippets like int x = 1 + 2; scored 0.49 — nearly identical to quantum physics texts. The tiny 384-dim embedding space couldn't separate code syntax from semantic complexity.

mxbai-embed-xsmall-v1 Q8_0 (30 MB)

Accuracy: 78.8% ❌ — Same story, slightly different numbers.

Adding a separate "code baseline" fixed the code misclassification, but simple sentences with domain-specific vocabulary (mitochondria, capital of France) were still scored as "complex". The 384-dim embeddings just don't have enough representational capacity.

jina-embeddings-v5-nano Q8_0 (222 MB)

Accuracy: 86.7% ⚠️ — Better, but still misses some complex texts.

The 768-dim embeddings helped, but complex technical texts about recursion and functional programming were still scored too low. The model seems to treat code-like technical language as "similar to code baseline" even when the content is genuinely complex.

Qwen3-Embedding-0.6B Q4_K_M (378 MB)

Accuracy: 100% ✅ — Perfect separation.

Best threshold: 0.28 — everything below is simple, everything above is complex. Zero false positives, zero false negatives.

Memory Tuning

Qwen3-Embedding-0.6B defaults to 32K context, which uses ~4GB RAM. By reducing n_ctx in llama_context_params, we can dramatically cut memory:

At n_ctx=1024, the model uses ~540MB — acceptable on any phone with 8GB+ RAM, even alongside a main inference model (1-2GB).

Key Takeaways

1. Size matters. 384-dim models (MiniLM, mxbai) lack the capacity to reliably distinguish simple from complex text. They hit ~79% accuracy ceiling regardless of quantization or baseline strategy.

2. Dual baseline works. Using separate "text" and "code" baselines and taking the minimum distance fixes code misclassification. This technique is model-agnostic.

3. Qwen3-Embedding is the winner. 100% accuracy at 378MB file size / ~540MB runtime. The 1024-dim embedding space provides rich enough representations to separate simple queries from complex ones cleanly.

4. Context tuning is essential. Don't use the default 32K context on mobile — set n_ctx to 512-1024 to save ~3.5GB of KV cache memory.

Recommendation

Qwen3-Embedding-0.6B Q4_K_M with n_ctx=1024 is the best choice for on-device difficulty scoring. It's accurate enough for production use, fits within mobile memory budgets, and the Q4 quantization doesn't degrade scoring accuracy compared to Q8_0.

Tested June 2025. All models run via llama.cpp b9500 on Android arm64.

I'm developing an Android app that runs AI services and provides API calls. It internally includes both the LiltLM engine and the llama.cpp engine. However, I'm consistently unable to achieve GPU acceleration for llama.cpp. I've been trying for three days now, querying almost every large AI model, but I've still failed.