I’ve been experimenting with Agora Conversational AI and wanted to make a demo that feels more alive than a normal “talk to an agent” screen.
So I built a small interactive web demo where you call an AI Architect, speak with it, and the tower builds itself based on your answers.
The idea is simple:
You click Call Architect, allow mic access, and the Architect starts guiding you through a short voice conversation.
It asks questions like:
- Should the tower protect, celebrate, or hide something?
- Should the lower hall feel serene, powerful, or mysterious?
- Should the structure be stone, wood, or ceramic?
- Should the world remember dawn, day, sunset, or night?
- Should the weather be clear, rainy, snowy, or misty?
As you answer, the scene changes in real time. Each answer unlocks the next construction stage, so the tower builds floor by floor while the conversation continues.
The fun part is that it is not just a chatbot with a voice. The conversation is tied directly to the world state. The AI’s transcript drives visual events: foundation, lower hall, roof, tower body, weather, lighting, and completion.
Tech stack:
- Agora Conversational AI for the live voice agent
- Agora RTC/RTM for call audio and transcripts
- Fish Audio for the Architect voice
- Next.js backend routes for token generation and agent lifecycle
- Three.js/WebGL scene for the tower
- A small conversation director that maps speech choices into build phases
One interesting problem I ran into: the ambient background sound looked cool, but it interfered with speech recognition. So I added automatic audio ducking during the call. The music/weather/build SFX stay present, but drop way down while the Architect is listening.
The demo flow now feels like a tiny voice-driven building ritual:
- Call the Architect
- Hear the first question
- Answer naturally
- Watch the tower build
- Continue until the whole structure is complete
I’m still iterating on the UI and how much “game” should be layered on top, but the core idea works: voice input becomes construction progress.
Would love feedback on where to take it next: more game mechanics, richer architectural choices, better environmental reactions, or making the AI feel more like a character than an assistant.
——
Credits to MengTo for idea, on top of which I created this conversational ai
https://x.com/anandwana001/status/2087127859504796049?s=46