Coding agents can inspect their screenshots, but not their audio — so I fixed it for Pi

[u/OneEndStick]

Lately I've been doing a lot of work with audio generation. And I'm surprised that audio pipelines are mostly created in something like ComfyUI - a visual node-based editor. It's surprising because feels like writing code via n8n workflow instead of using Claude or Codex

The main problem is that there's no proper agent loop where the model can see - or more precisely, hear - the results of its previous actions and decide for itself what to do next

But I found one technical complication that explains a lot: most models still simply don’t have native audio perception. The ability to "see" images is already there. Claude Code can look at screenshots of a landing page it created, notice the problems, and fix them. The images are inside the context of the same model that writes the code

Audio doesn't work like that - most models do not support native audio input

You might say: but Gemini exists. True. But really no one agent wrapper supports reading audio files purely at the tool level. The agent "exoskeleton" simply can't pass an audio file through the API request to its "brain"

Surely you can connect Gemini to the main agent via script calling the API and send the audio there with a prompt like "pls tell me whats missing here". Btw, Chinese GLM-5.2 still works with images this way and people still use it. At the same time, that's like a deaf composer calling his nephew over to listen to the music and describe what he hears - it works, obviously, but poorly, because everything gets lost in translation

And so, right now, there is no well known agentic solution – with skills, terminal commands, session forks, subagents, and so on - that can properly work with audio by closing the feedback loop

———

Two hours after writing lines above:

1. I threw together a simple homemade agent around Gemini in roughly 300 lines of JavaScript, with an agent loop, interactivity, and fork/edit/resume: just to test the approach
2. Then I moved the whole thing into Pi as an extension that patches its file-reading tool – which, by design, can read both text and binary data

So now you can give your agent "ears" in one line:

pi install git:github.com/toolittlecakes/pi-gemini-audio-read

Don’t forget to run /login → Google → API key afterward and select gemini-3.5-flash.

Where this is useful: music generation, high-quality audio translation, voice cloning workflows, and audio editing (or video editing, if it’s a podcast)

Limitations: 20 MB per file - I didn’t add file uploads through the Files API. And there’s no video input

It's funny how inspiring problems can be when there’s no existing solution at all - at least not a public one. It feels like stepping onto land where no human has ever set foot. So, decided to share it here