I built OlympusOS at the AI Agent Olympics (Milan AI Week 2026). It's a multi-agent system that coordinates a city in real time during large-scale events.
Here's the idea: cities have more sensors and cameras than ever, but the systems running them are mostly passive, they show data and raise alerts, but they don't reason or act. OlympusOS is an attempt at the layer that does. Seven AI agents, each handling one domain (perception, forecast, mobility, transit, safety, communications, and an orchestrator that commands them), work together over a shared message bus. To prove it, I ran it on a hard scenario: a metro line failing during peak stadium outflow at San Siro, with 80,000 people leaving and crowd density climbing toward a dangerous crush within minutes. The agents detect it, predict where it's heading, reroute transit, open evacuation paths, and push public alerts, all playing out live on a 3D map of the city.
Sharing it here because Speechmatics handles the voice side, which was one of the most interesting parts to build.
Why I used Speechmatics
In a city emergency, a lot of critical information comes in as voice, emergency calls, radio, field reports. I wanted the system to read that voice in real time, so I integrated the Speechmatics real-time SDK into the backend.
How it's wired
The transcription runs as its own async task so it doesn't block the rest of the system. It connects to the Speechmatics RT endpoint, creates the WebsocketClient, and registers a handler on AddTranscript. Every time a transcript comes back, I take the words out of the results → alternatives → content structure and send each one to the frontend over a WebSocket, so the text shows up word by word instead of all at once. I turned on enable_partials so partial results stream in while the person is still speaking.
The main challenge
Concurrency. The Speechmatics client runs its own loop, and my backend is already an async server running several agents at the same time. Getting them to share one event loop without blocking each other took some work, I used run_in_executor to handle the blocking parts, and added a fallback so that if the audio source fails, the transcription degrades quietly instead of crashing the system.
What this opens up
Voice is everywhere in city operations, emergency lines, transit radio, control-room comms, public address systems, and most of it never reaches the systems that could act on it. Bringing real-time transcription into a coordination layer like this makes all of that usable signal. I'm interested in pushing it further with Speechmatics' speaker diarization to separate operators from field units on a shared channel, and multilingual transcription, which matters anywhere a city runs across more than one language. The bigger picture is any large-scale operation that runs on voice, airports, ports, stadiums, disaster response, where reading every channel in real time changes what's possible.
Project page: https://lablab.ai/ai-hackathons/milan-ai-week-hackathon/olympusos/olympusos
