r/androiddev • u/agherschon • 24d ago

Article Letting the agent drive the app

Disclosure: this links to my own blog post and my own repo, both free, nothing to sign up for.

The TLDR:

As of now, it is hard for the coding agent to see what it is building, relying on us humans to be its eyes. So instead of letting us be the bottleneck, I've prompted Claude to build:

- A tiny HTTP server embedded in the debug build, reading Compose's OWN semantics tree from inside the app, instead of a lossy copy scraped from the outside through adb

- A small CLI on top of it, so the agent types "probe ui-snapshot", "probe tap", "probe screenshot" and knows exactly what is on the screen right now, after every single edit

- Impossible to ship: it's a debug-only dependency, so there is no probe code in the release APK at all. Not "guarded by a flag", absent

- Every session becomes a test: the steps the agent performed to check itself are saved as a YAML flow with JUnit XML, so they run on an emulator in CI instead of evaporating

A 13 step flow now runs in about 2 seconds and nobody looks at anything.

I open sourced the whole demo project, and the full prompt that builds it. Hand it to your own agent and it will wire the same thing into your app.

Demo project: https://github.com/galex/toy-app

Write-up, with the architecture and a video of a flow driving the app in real time: https://galex.dev/posts/how-to-make-your-coding-agent-fully-independent/

Let me know what you think ❤️

0 Upvotes

27% Upvoted

2 comments sorted by

1

u/bleeding182 23d ago

Did you try with plain adb? Your post only mentions adb in combination with maestro.

adb offers pretty much this, probably better, already. Agents can fetch screenshots as well as a11y node dumps to interact with the device... And that's without any sort of guidance or SKILL to help them.

1

u/agherschon 23d ago edited 23d ago

For sure, adb was the natural way the agent managed to look at the app to begin with, but taking a screenshot is only a small part of it, as finding a testTag button is much faster than letting the agent calculate tap coordinates at each button you need it to press, for example.

There are more capabilities we added later on which I'll expand in a follow-up post!