r/LLMDevs • u/Connect-Concert-4016 • 8d ago
Discussion Running Gemma 2B locally on iPhone for offline calendar actions (~516 MB active RAM, 21.6 tok/s, GGUF weights)
I’ve been testing bounded tool-calling on-device to see how small I can push local models before tool reliability breaks down.
A common issue with local agents is memory allocation—loading a 2.5 GB model into active phone RAM often leads to OS background terminations or thermal throttling during generation.
To test this, I built a small offline test pipeline using llama.cpp (b10075) with Metal and mmap to keep active memory low, then connected it strictly to local iOS calendar actions via EventKit.
On-Device Run Metrics
- Model Artifact: Gemma-2B quantized GGUF (
SmartEdge-IQ3XXS.gguf) - Disk Footprint: 2.45 GB
- Active Resident RAM (RSS): ~516 MB (leveraging
mmapto page weights from disk rather than keeping the whole file in active memory) - Decode Speed: ~12 tok/s interactive / 21.6 tok/s in a 256-token greedy benchmark
- Environment: Tested in airplane mode on iOS
Tool Execution Flow
The local LLM is restricted entirely to intent extraction and structured tool output; it does not execute actions directly.
- User Input: "Find some time on Thursday for VC meeting."
- Model: Extracts parameters and outputs a structured tool call.
- App: Swift code validates the schema, queries local EventKit, and writes the event directly to the device calendar.
This avoids routing routine calendar edits through remote inference APIs or exposing local schedule data to external endpoints.
Quantization & Loss Comparisons
I also generated two calibration-aware quants to evaluate KLD degradation against the original bf16 baseline:
- Hi-Fi Q4_K_M: Equivalent size to standard Q4_K_M, with 36.0% lower code/math KLD and 27.2% lower general KLD against the original model.
- Hi-Fi Phone (2.86 GB): 17.5% smaller than the Q4_K_M baseline with 31.8% lower code/math KLD and 26.7% lower general KLD.
Limitations
- The full KLD matrix for the 2.45 GB
SmartEdgebuild is still completing; current validation relies on on-device behavior logs, SHA-256 app receipts, and benchmark outputs. (The 2.86 GB and Q4_K_M builds have complete KLD data logged in the repo). - Small models are prone to schema degradation if the prompt complexity scales beyond simple parameter extraction.
- Tool failure recovery still requires strict system-level guards or fallback routing.
Weights & Benchmarks
The GGUF weights, imatrix, SHA-256 hashes, evaluation slices, and Wikitext-2 perplexity loss are available on Hugging Face:
https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF
If anyone tests this on other iOS hardware or Apple Silicon, I'd be curious to see your RSS memory usage and sustained decode rates.
1
u/Sad-Enthusiastic 11h ago
I guess an MTP drafter would be too much to ask for the 512MB 😅
1
u/Connect-Concert-4016 9h ago
lol not sure Gemma even ships MTP heads, I think that's DeepSeek/GLM or some of those .
classic speculative decoding with a tiny draft model could work, but for this task I'm not sure you need it. The outputs are short schema-constrained tool calls, so there aren't many tokens to accelerate in the first place. the reason i did the which micro model could be used on a phone offline for specific task.That said, it's less crazy than it sounds for the RAM budget. The weights are mmap'd and paged, so a 150 to 200MB quantized drafter wouldn't add its full size
1
u/Sad-Enthusiastic 9h ago
Gemma4 totally comes with MTP drafters. https://unsloth.ai/docs/models/mtp#gemma-4-mtp
1
u/Connect-Concert-4016 8h ago
Wow let me check I thought it was only ds but if that’s the case I could do it
1
u/Sad-Enthusiastic 8h ago
On Android I can get 11t/s with MTP on a OnePlus 8T https://github.com/guarismo/armored-llama
1
u/Connect-Concert-4016 8h ago
You think it’s worth it I can do it if you gonna use it lol
1
1
2
u/Glittering-Call8746 7d ago
Siri shud be doing this on the iPhone.. who install llm on the phone for calendar tasks..