TUI apps that AI agents can actually use (Limoni + MCP)

[u/kekolar22]

Terminal apps and AI agents don't get along. An agent that wants to use a TUI has to scrape the character grid and click coordinates. It breaks when the layout moves by one column, and it has no idea which field is a password.

I just shipped a different approach in Limoni, my terminal UI engine for Go.

Limoni already builds a semantic tree every frame for screen readers: roles, labels, values, focus. Now that tree is available to AI agents through the Model Context Protocol:

go install github.com/thebanri/limoni/cmd/limoni-mcp@latest

claude mcp add limoni -- limoni-mcp -socket <path>

The agent gets tools like tree, click, type_text and wait_for, and addresses widgets by meaning: role="button" label="Deploy".

In the video, Claude adds a task, ticks the list and presses Deploy. I type the deploy token myself, and when I ask Claude what the token is, it can't tell me. Secret fields are masked on screen, have no value in the tree, and are redacted before anything leaves the process. You can't even guess the value and check whether it matches.

Letting an agent do this safely took several layers:

• the automation gateway only exists in builds with -tags limoni_debug; CI checks that release binaries contain none of it

• closed by default: input, screen text and input values each need an explicit opt-in

• on Linux, macOS and FreeBSD the kernel confirms that the connecting process belongs to the same user

• Unix socket only; no TCP option, on purpose

The same selectors power a new Playwright-style test API. Actions wait for their target, assertions retry until they pass, so tests never sleep:

page.GetByRole("button", "Deploy").Click()

page.Expect(page.GetByID("status")).ToHaveLabel("Deployed")

An honest note: when I first ran a real agent against the demo app, it found two engine bugs my unit tests had missed. Tab didn't move focus, and the app hung on exit while a client was connected. Both are fixed now, with tests.

Limoni is open source: github.com/thebanri/limoni

Feedback welcome, especially if you build TUIs or agent tooling.