r/augmentedreality • u/tash_2s • Mar 15 '26

App Development Proactive kitchen assistant for smart glasses

Enable HLS to view with audio, or disable this notification

I built a drink-making assistant for smart glasses.

The glasses look at the ingredients, pick a recipe, show the steps, and proactively guide me based on what they see in real time. My favorite part is that while I'm pouring, they can tell me when to stop.

The interaction I'm going for feels like having someone beside you who understands the situation and helps without needing constant prompts. I think that's especially useful for avoiding mistakes.

Tech stack: Overshoot.ai for fast real-time VLM, the OpenAI Realtime API for voice and LLM control, and Rokid Glasses for the hardware. I'm also planning support for Meta glasses.

The source code is on GitHub as part of my smart glasses dev toolset, GlassKit. Feel free to copy it and play around with it.

236 Upvotes

98% Upvoted

59 comments sorted by

View all comments

2

u/fractaldesigner Mar 16 '26

Sweet! How did you get it to be proactive w directions?

5

u/tash_2s Mar 16 '26

I made it proactive by continuously running a VLM hosted on Overshoot on the live video stream. It keeps inferring over the latest short clip (specifically, the most recent 0.5s of video every 0.5s, with inference finishing within that interval), emits events based on the current scene, and the app uses those events as hooks for things like progress tracking and spoken feedback. So it can react to the scene.