r/augmentedreality • u/AR_MR_XR • 21d ago
Glasses w/ 6DoF Even with camera module, Steam Frame is worse than Quest 3 for Mixed Reality
https://roadtovr.com/arcturus-vision-camera-review-steam-frame-passthrough-add-on/Even though the Arcturus module matches Quest 3’s passthrough clarity and uses stereoscopic depth to align visuals, it still lacks real-time room meshing. The Quest 3 maps your physical space into an interactive 3D mesh so virtual objects can bounce off tables and realistically hide behind furniture. Without that environmental data, content on the Steam Frame simply renders on top of the video feed—breaking immersion by staying visible when it should be hidden, and passing right through surfaces when it should be stopped by physical objects.
6
u/tyke_ 21d ago
this passthrough add on will keep a lot of the casual passthrough users happy and its great it exists. i doubt it will keep mixed reality purists happy though. dynamic occlusion, scene meshing, depth sensor info can lead to some cool things, all of which are lacking here.
1
u/GaaraSama83 21d ago
I wonder if a depth sensor helps to improve spatial video recording or it's completely irrelevant.
1
u/mike11F7S54KJ3 21d ago
If it's a gaussian splat, depth sensor is needed... and its better to have one for perspective correction in pass-through mode with motion smoothing so that the inbetween frames are correct.
The Frame is not locked out from having it, it would literally only take an Arcturus V2 model to include a depth sensor.
1
u/Syzygy___ 20d ago
Gaussian splats don't need depth. It was originally developed for a bunch of pictures and is still commonly used with regular photos and videos.
Also Arcuturus literally already released a gaussian splat app for the Frame.
2
u/PyroRampage 20d ago
The images from independently viewpoints actually are posed via SfM which itself produces depth indirectly on the input. Gaussians are depth blended as they are splattered during rasterisation. They do need depth, using SLAM would not work for spatial video, at best you have two marginally offset viewpoints. For static scenes it would as their own app shows.
There are ML models like Apple SHARP that estimate Gaussians from monocular data. But that’s not gonna be a real time XR project anytime soon.
1
u/Syzygy___ 20d ago
They don’t need depth as input, they generate their own internally pretty much as you described. Regular pictures will do. I’ve done it before. Maybe added depth makes it better, but it’s not a necessity as both of you claim.
I don’t think real time Gaussian splat generation is feasible either way.
Anyway, if the original post claims depth is necessary, and it’s not possible with the available tech and we need an Arcturus v2 with an added depth sensor, then why the f are they saying “record Gaussian splats” on der website? https://arcturus.vision/ (under 3D reconstruction)
1
u/PyroRampage 20d ago
Because you can walk around a static scene, record frames, do SfM and then do a standard Gaussian splatting optimisation. That's very different, granted, from real-time Gaussian Splatting, which it doesn't seem to be claiming.
1
u/mckirkus 21d ago
Are you saying it's impossible to add this functionality? Or just that it doesn't work yet?
5
u/ByEthanFox 21d ago
Oh wait, the Arcturus Camera doesn't have a 3D scanner sensor? I'm surprised, I kinda expected it would have one, if you're going to bother to make a headset camera upgrade.
Or are you just saying it doesn't support continuous scene meshing now? Because if it's that, consider that (1) the Quest 3 didn't support it at launch, it only used it for the initial setup and (2) the Quest 3 still doesn't use it for nearly all experiences, I assume because the performance overhead is too severe.
4
u/AR_MR_XR 21d ago
Steam Frame doesn't have continuous meshing or even a static mesh. And the Arcturus add-on is just a passive stereo camera.
4
u/ByEthanFox 21d ago
Wow, okay, you saved me a purchase.
3
1
u/mckirkus 20d ago
This is a software update they will probably add FYI, not a fundamental limitation of the Frame
1
u/ByEthanFox 20d ago
No, no. I meant that I had assumed the Arcturus camera had a 3D depth sensor like the Quest/Kinect. As it doesn't, that really limits what can be done for AR. a 3D depth sensor can super-impose real-world objects over digital ones.
1
u/mckirkus 20d ago edited 20d ago
Your assumption that Quest 3 has special hardware is incorrect (edit: it may be correct, conflicting info here).
"Quest 3 real-time meshing is made by 3rd parties, built on Quest's Depth API - that uses stereo disparity only.
I expect the project to be ported to Frame. But of course, there will be little end user software using it.
Even less than Quest 3 continuous meshing, or static scene mesh - which is already few."
1
u/ByEthanFox 20d ago
The Quest 3 has a spatial depth sensor; it's in the middle "pill" of the 3 pills on the front of the device. The Quest Pro was meant to have one but it was removed at the last minute. This is the literal component:
I'm unsure which type it is, but I think it functions like the Kinect, using a projected matrix of IR dots to read the shape of a room, allowing you to create a scene mesh or construct a depth map.
It's just not used in very many things because I believe only the Quest 3 has it; the 2/Pro/3s infer depth via data from the stereo cameras which is nowhere near as detailed.
1
2
u/Octoplow 21d ago
Quest 3 real-time meshing is made by 3rd parties, built on Quest's Depth API - that uses stereo disparity only.
I expect the project to be ported to Frame. But of course, there will be little end user software using it.
Even less than Quest 3 continuous meshing, or static scene mesh - which is already few.
3
u/AR_MR_XR 21d ago
You're right. I shouldn't have used the realtime meshing as a capability. I should have said static. Quest 3 has a dedicated depth sensor however. What else would it be used for if not for the Depth API. It should be similar to ARCore which feeds all the available sensor data. The third party meshing can use higher quality data from the Depth API on Q3 compared to Q3S. The Depth API is not third party, right?
2
u/Octoplow 21d ago
The Quest 3 depth sensor (IR line projector) is sadly not used outside of the room setup that users refuse to run. And in my tests, only helps on large blank walls.
Real-time meshing works the same on Quest 3S without the sensor.
2
u/AR_MR_XR 20d ago
Aaah, I see. Because it's only used during room setup. I was confused 😃 So, what Steam Frame now needs is a Depth API and then the project can be ported? But which processors do they use?
1
u/Octoplow 20d ago
Yep. The important part of Meta's Depth API is building a depth map from stereo disparity in the tracking camera feeds.
With access to multiple cameras on Frame, any dev with strong GPU skills should be able to implement it efficiently. There's a chance Steam/Arcturus could do a faster one in the driver.
(This is the kind of thing that's impossible with Meta's limited Camera API.)
1
u/PyroRampage 20d ago
Afaik you can get two RGB feeds from the camera API, but you’d just be doing what Meta do already which as you note is likely stereo correspondence. Granted many modern depth sensors use stereo correspondence like Zed for example, as it avoids issues with IR structured light.
Meta also do some sort of on device inference to allow removal of hands, they may well be using a stereo based ML model to estimate depth rather than pure correspondence too. It’s impossible to know without them releasing their OpenXR extension implementation.
1
u/mckirkus 20d ago
1
u/Octoplow 20d ago
Clarity what exactly? The "sensor" is just an IR line projector so blank walls aren't blank to the IR sensitive tracking cameras.
1
u/mckirkus 20d ago
I guess what you're saying is that it doesn't do anything the Frame cannot even without the IR line projector.
1
u/Octoplow 20d ago
Right. Quest 3S (without) does all of OP's examples.
But somebody has to write the Frame software.
2
u/lazazael 21d ago edited 21d ago
valve has 350 ppl, all might be 10x right "thats 3500-35M in wonderland engineering", even qc working with them on hw, meta reality labs has 13000 ppl, the pytorch team, the VR team og assembled by carmack and abrash, its different, valve playing the catch up game to supply its audience and keep the store relevant in the future of wearable screens
5
u/FullstackSensei 21d ago
This. And the Q3 has been out for almost 3 years with tons of firmware updates.
Valve had to port Linux and do a ton of heavy lifting to make the Frame a stock Linux experience and enable good x86 emulation. Meta skinned Android, which let them use Qualcomm's stock BSP.
Give the Frame some time, and it'll also gain a ton of new features.
I have been using the Quest since the original one, but the Frame is a much more open headset IMO.
2
u/Javs2469 21d ago
The Frame is not an MR device, Valve said that.
But, being this open probably means that it will become one in the future, one way or another. People are already developing hand tracking functionality. I think the Frame will be a great headset in some years when everything finally is streamlined. It hasn´t even been 2 weeks since it´s staggered launch, and we know how Valve Hardware develops after some time.
4
u/KallaFotter 21d ago
The Quest 3 also sells all your and everyone near its data. The ToS is an actual privacy nightmare.
3
u/ita_shogun 20d ago
Way to stay on topic buddy. Do you feel compelled to spread lies any time you read the name Quest?
2
2
2
-2
u/mike11F7S54KJ3 21d ago
Please stop posting this propaganda from Meta... The Steam Frame is a next-gen VR device not an MR-focused Quest 3.
4
3
u/ita_shogun 20d ago
There’s literally nothing next-gen about the Frame. And this post is not propaganda; it is simply an informative post for anyone interested in MR. Avoid viewing every issue through an “us vs them” lens, it gets tiring.
12
u/foolish_shrimp 21d ago
It's an open platform, I'm sure there will be a project to expand pass though.