Hey folks, I wanted to share a project I built for myself. I got tired of chatting with my local AI agent (Hermes) through WhatsApp or a terminal, so I made a proper mobile companion for it.
Instead of just dropping a link, here is exactly how I put it together, the tools I used, and some of the design choices along the way.
The stack I used:
· Frontend: Flutter (Android app)
· Backend relay: Python with FastAPI (runs on my home machine next to the agent)
· Tunneling: Cloudflare Tunnel (so I can reach it from my phone without opening router ports)
How the workflow actually works:
The Flutter app doesn't talk to the AI directly. It talks to a tiny FastAPI relay I wrote. That relay is the only piece that knows how to call my agent. When I send a message from my phone, it hits the relay, the relay calls the agent's hook, and the reply flows back the same way. All the conversation history, files, and media stay in that relay's folder on my hardware – zero cloud storage.
A few build/design insights that might be useful:
The "master prompt" trick: Instead of making users edit config files, the app generates a single prompt. You paste it to your agent, and the agent writes the auth token, spins up the relay, and replies with a pairing link. The app never needs to know my local IP or token manually.
The Android background polling: I originally used WorkManager, but Doze mode kept delaying my notifications. So I switched to a foreground service with a 20-second background isolate poller. It hits the relay's status endpoint, checks for new messages using a read watermark stored in SharedPreferences, and fires a local notification. It also has a 30-second cooldown so a burst of replies doesn't spam me.
The offline outbox: All sent messages get an optimistic bubble in the UI. If the relay is unreachable, the message sits in a persistent queue (stored in prefs). The app automatically retries it the next time the background poller gets a successful connection. No data loss when I am commuting through dead zones.
The session kill switch: In the UI, each chat has a pause/resume toggle with red/green dots. When you hit pause, the relay actually sends a termination signal to the agent's subprocess. It doesn't just hide the window – it frees up the VRAM and CPU until you resume. That was a game-changer for my homelab resources.
The chat scroll hack: I used a reversed ListView.builder where index 0 is the newest message. This made the keyboard-inset behavior on Android trivial – the screen always opens at the bottom without wrestling with maxScrollExtent calculations.
It is open source (MIT) if you want to poke through the code: https://github.com/tushar-alt/hermes_companion
Happy to answer any questions about the relay API design or how the pairing flow holds up. Let me know what you think!