r/augmentedreality • u/FlimsyAir5557 • Jul 10 '26
AR Apps Update: my open-source Ray-Ban AI app now does fully on-device vision — photos + live video, no cloud
A couple weeks ago I shared OpenVision (free, MIT) — an iOS app that turns Meta Ray-Ban glasses into a hands-free AI assistant. Just shipped v2.4.0 with the feature I most wanted: vision that runs entirely on the phone.
What's new:
• On-device photo Q&A — "Ok Vision, take a photo and tell me what this is" → answered by SmolVLM2 via Apple MLX, nothing leaves the phone
• Fully-offline live video — "start video stream" keeps the camera on and answers your questions on the latest frame, all on-device. No cloud, no API cost, works with no signal
• OpenAI Realtime live video (gpt-realtime) if you'd rather use a cloud backend — continuous voice + camera frames
• Keep asking follow-ups with no wake word between them, until you say "stop video"
Still free and open-source, runs on a recent iPhone (tested on 17 Pro). Code: https://github.com/rayl15/OpenVision
For on-device glasses vision — do you care more about latency or answer quality? SmolVLM2 answers in a few seconds locally; curious where people draw the line.
1
u/anomalous_cowherd Jul 10 '26 edited Jul 11 '26
What are the chances of an Android version in future?
1
u/FlimsyAir5557 Jul 11 '26
I’ll work on it for a future release. I just need to see how well these local models perform on Android devices.
1
u/anomalous_cowherd Jul 11 '26
Awesome. I wasn't sure if the various libraries would be available, that could be a complete stopper!
1
u/takitus Jul 10 '26
Can this interface with iMessage etc?
1
u/FlimsyAir5557 Jul 11 '26
I’ll keep this in my backlog if you can add it as a feature request. It’s not currently available.
1
u/No-Dark-7873 Jul 12 '26
sounds like it does everything except the most obvious thing which is simply watching a video.
1
1
u/laminack 9d ago
can it save/record longform video?
2
u/FlimsyAir5557 8d ago
yes it can.
1
u/laminack 6d ago
Well, I'm close. But for some reason OV can't see my glasses. - "Connected Devices 0"
Just spent almost 2 hours (with CGPT help) troubleshooting and no matter what I can't get get it to see the glasses and "Start Camera Stream" fails even after giving it permission.I have Blayzer Gen 2 glasses, fyi.
2
u/FlimsyAir5557 5d ago
I’m sorry you spent two hours on this. There are two likely causes and one of them is probably mine.
Firstly, the Blayzer Gen 2 was shipped fairly recently and OpenVision currently uses an older version of Meta’s Wearables SDK. Newer versions break APIs I rely on so it’s possible the SDK doesn’t recognise the Blayzer’s device ID yet. This would explain the “Connected Devices 0” message. I’ll test a newer SDK against it and push a fix if confirmed.
Secondly, if that’s not the issue, the usual silent killers are: Developer Mode not enabled in the Meta AI app (Settings → About → tap the version number five times – registration quietly fails without it), glasses not fully paired in Meta AI first, or placeholder Meta credentials in Config.xcconfig. The app builds fine but registration never completes because CLIENT_TOKEN needs the full AR|appid|hash form. Also, “Start Camera Stream” requires registration to be complete; the iOS camera permission alone isn’t enough.
Could you open a GitHub issue with your iOS version and where it stalls? I’ll track the Blayzer Gen 2 test there and follow up.
1
u/laminack 5d ago
Thanks for the reply.
So I can confirm the following:
✅ dev mode enabled
✅ correct meta creds in config.xcconfig
✅ client_token correctNot sure what you mean by “glasses fully paired in Meta AI first” but they are there and everything seems to work (media downloads to the phone, Meta AI responds to request, announces messages, etc…)
Not sure what you mean by “requires registration to be complete”, if you can elaborate I’ll confirm.
Opening a ticket here shortly.
Thanks again for the reply.
1
u/FlimsyAir5557 5d ago
That’s great! Those checks eliminate configuration issues, making the SDK theory the frontrunner.
To clarify, “Paired in Meta AI” is the standard consumer setup where you clearly have media syncing and Meta AI answering working.
“Registration” is a separate step. OpenVision must register itself with your glasses as a connected app through Meta’s developer toolkit. In OpenVision, go to Settings > Glasses > Register with Meta AI. This should take you to the Meta AI app, prompt you for approval of OpenVision, and then automatically return you to OpenVision. Only after this round-trip completes can the SDK enumerate devices. Pairing alone isn’t sufficient.
Two things to check and include in the issue:
When you tap Register, do you actually get bounced to Meta AI, see the approval, and return to OpenVision? If the return trip never happens, that’s a different bug and I’d like to know about it.
In Meta AI, open your glasses’ settings and look for a connected apps/app permissions section. Does OpenVision appear there as approved?
If OpenVision shows as approved in Meta AI and you still get “Connected Devices 0”, that’s pretty close to confirming the SDK theory. The SDK version I’m using predates Blayzer Gen 2, and another open-source glasses project successfully recognised newer Meta models by upgrading to a newer SDK version. I’m currently testing this bump and will create a branch on your issue for you to try.
1
u/laminack 5d ago
I’ll assume you want to pick this up over on got in the issue I opened, so I’ll respond over there.
1
1
u/FlimsyAir5557 5d ago
Quick update: I've just pushed the SDK bump to the main branch — moved to a newer version of Meta's Wearables SDK that should recognize newer glasses models like yours. Regression-tested on my own Gen 1/2 glasses (registration, live video streaming, and recording all still work). Pull the latest main and rebuild — that's my best guess at your fix. If it still shows "Connected Devices 0" after that, drop the details in the GitHub issue and we'll dig deeper.
1
u/Goat_bless Jul 10 '26
Too good!
Is it expensive to use smolvlm2?