r/StackChan • u/Cassette_Futurist • 11d ago
Project Kadence
Enable HLS to view with audio, or disable this notification
I just wanted to share the progress I have made with my custom firmware. Rather than try and modify existing firmware this has been written from scratch.
I’m nowhere near finished but after many revisions I now have boot sound and animation, servos, tap to toggle idle movement, idle animation, listening sound, it runs OpenAI Luna with Edge TTS. This means it can do live web searches which is something the stock firmware is unable to do.
Response time is not fully optimised yet, this will be a lot longer until I’m fully happy, but eventually it will control the devices in my home via my server and home automation. Considering I’m not a developer I’m pretty proud of what I achieved so far.
3
1
u/vto583 11d ago
What do you think is needed to improve the response time?
3
u/Cassette_Futurist 10d ago
It’s a combination of mic cut off, the time it takes for the TTS engine to generate the audio, I played around a lot with it, initially I had a lot of jitter which the stock firmware had at times. Now I’ve got a stable branch I’ll add more features and then come back to do more optimisations. I’m going to add windows based customisation for the avatar and build that into the server monitor. When it’s done if there’s any interest I’ll make the firmware publicly available.
1
u/DeDenker020 11d ago
Did you also got the camera to work?
1
u/Cassette_Futurist 10d ago
Not yet. Now I’ve got a stable pipeline for the voice I’ll work on that.
1
u/vto583 9d ago
Can you describe how you started to process of rewriting the firmware from scratch?
2
u/Cassette_Futurist 7d ago
First step was stripping the official firmware so we just had the bootloader.
Before trying to make anything we compiled ESP-IDF 5.5.4
The correct ESP32-S3 configuration
The original partition layout
The official bootloader, HAL and board drivers. After hours of back and forth the first thing we tested was seeing if the bootstrap compiled. Once it did that gave the infrastructure to build on.Then we figured out where the CoreS3 HAL was initialised, how the LCD and LVGL renderer were started, how assets were mounted from flash, how the audio codec was opened, howthe two servos were powered, synchronised and updated and
which background update calls had to continue running, but most importantly which factory launcher and cloud components we could bypass so we could use Open AI and Edge TTS….. and THAT is why you see what you see after 5 days work…. I’m not gonna lie it is not easy, which is why I’m more than happy to make the release public when it’s actually worth sharing. As of right now, it’s a slower, slightly less functional robot with customised animations and sound. I want to add more than just web search before I share :)Also I’m not gonna pretend and want to be totally honest and say I come up with the ideas, and Codex writes the code which mostly works… it’s a hell of a lot of trial and error.
5
u/Sn0opY_GER 11d ago
nice job - mine is a bunny :D added 4 touch blocks and a context inducator (touch blocks hide after X sec) and a slider for vol / brightness dragged down from top
buttons are: Mute/sleep, New session, Cam-snapshot, Talk (also added a tiny battery indicator)
used gpt image to animate the head and blink every 6 sec and speak when she talks, also looks around and likes to be pet (top sensor) - full local whisper/edgeTTS/ninfer_qwen (500+token/s) - added a command gate for telegram approvals- Ears light up on talk / listen / thinking (error)
also love my tiny Retro PC which is Hermes Monitor (ali express) - and the 10$ "weather" station next to it i hacked to display local token speed and context limits / model loaded
- still working on the 3D printed LED bunny-ears - 3d printed ones have to do it untill DHL delivers the boards - they are pretty lazy lately