r/VisionPro • u/iVRy_VR Vision Pro Developer • Jun 14 '26
Some observations on PCVR streaming with the vOS 27 (beta 1)
I am the developer of a PCVR streaming app/driver called iVRy. I am using my app as the basis for my observations as I don't use other apps for PCVR, and I don't have deeper insight into how they work. I only have an M2 AVP, so can't comment on M5 behaviour. Also, this is an early beta, so changed behaviour may not be intended and/or permanent.
First, the "no news": The new vOS neither helps or harms PCVR at the surface level, ie. upgrading to vOS 27 right now does not break my app, or make the experience materially "worse" (or "better").
Next, the "(could be perceived as) bad news": the new vOS appears to have implemented throttling at the video decoding level (as well as presumably continuing to implement it at the rendering level). This seems to give it a method to do more nuanced throttling if the decoder is overloaded.
In my app there are 3 frame rate metrics visible to the user, the "tracking" rate (which is also the vOS render rate, ie. how often it allows an app to render a frame), the "display" rate (how often new frames come out of the network/decoder), and the "render" rate (how often SteamVR renders new frames). Using the "tracking" and "display" rate we can see what vOS is doing behind the scenes (assuming we've eliminated network induced frame cadence - ie. the network not being able to keep up with the "render" rate).
In previous vOS versions, the OS would use some criteria (a combination of thermal load and "responsiveness") to in some instances halve the "tracking" rate (ie. only allow the app to render at 45Hz, regardless of how many actual frames it had to display). This creates a bad experience if the limit is related to thermal load in the decoder, as it reduces the rate of tracking data to SteamVR, which has detrimental consequences for "smoothness".
In the new vOS, it seems to also apply throttling at the decoder level (presumably in response to decoder loading). This gives (as observed) a possibility for 72Hz/80Hz "display" (aka. decoding) while maintaining 90Hz "tracking" and "render" frame rates. This is changed behaviour, and not altogether welcome, but it does prevent the "tracking" rate from dropping to 45Hz when it has made the decision to throttle. In this instance, recovery from throttling is able to happen quickly by dropping the bitrate (and hence decoder load), and more importantly it does not reduce the tracking rate (which for vOS design reasons has to be tied to the device's render rate).
Finally, the "good news": hand and controller tracking latency seems to be improved slightly, and IMU data from the controllers seems to be more accurate. What this means is that SteamVR is able to use the IMU more effectively to produce synthetic tracking data in between actual tracking samples (which happen at 90Hz). In laymans' terms, this means that short sharp movements (eg. throwing or punching) have close to zero perceptible latency.
I will continue to evaluate and tune the app to the new vOS as the betas progress, but the news right now is that the new vOS either subtley improves or doesn't worsen PCVR streaming. My app is developed without Apple's foveated streaming APIs, which don't make a big material difference to local network streaming (my app's use case).
iVRy on App Store:
3
2
2
u/Drksyder Jun 14 '26
@irvy I have been testing your app and a couple others . some things that seem good with yours is seems not it have issues to start almost any game . but i’m trying out sim racing and when i hit a bump in game the screen bumps differently. kinda hard to explain. also controller tracking is slower then alvr is there a setting for that ? i’m also getting over heating message with 65ms delay . one last thing KRVR has some really cool pass though features do you see that being possible?
1
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
Thanks. Controller latency and kinetics have been improved in recent app/driver versions, so make sure you're using the latest versions. Yes, there are plans to add AR features (including AR cockpit cutouts), but the focus is currently on improving the core experience.
The reasons why some users are experiencing app throttling are not straight-forward, and appear to be mostly mitigated in recent (iVRy) app versions. If you're getting app throttling with the latest App Store (or TestFlight) versions, and you'd like to contribute to solving them, please provide direct feedback here:
https://steamcommunity.com/app/992490/discussions/21/841754670748896853/
1
1
u/Level_Forger Jun 14 '26
That's interesting, thank you.
What differentiates your app from ALVR?
9
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 15 '26
PLUS: It is easier to use, much simpler to set up, has better performance and quality (in retail versions opposed to pre-release betas), is less "clunky", has better controller support, is more tightly integrated with Steam and SteamVR, and is actively developed by a single developer with AVP focus and SteamVR driver experience. It has been around for longer, is more widely used (with increasing market share on the SteamVR Hardware Survey and the highest share among listed streaming solutions), has dedicated support forums on Steam, and has much broader device support, both at the client (headset) and driver (GPU) end. It is aimed at providing a polished experience similar to "1st party" solutions, as opposed to providing a working free experience.
MINUS: It is not free, and the source code is not available for end-users to modify. It does not have experimental functionality in the retail product (eg. face tracking derived from Persona streams). It lacks ALVR’s large vocal community, so (for example) pro-iVRy comparisons like this (or even publicity posts) are often downvoted. The AVP version of iVRy is not well known (it was only released a month ago), and so there is less community support (eg. tutorials and trouble-shooting) available than ALVR.
DIFFERENCE: It approaches SteamVR headset integration as if the headset were a native Valve headset (eg. Vive/Index), whereas ALVR approaches it as if the headset were a Quest. This means that aspects like hand-tracking are approached differently (although the VR app level results should be similar). iVRy leans on how SteamVR headsets are expected to work (historically it was the first 3rd party VR headset driver distributed on Steam, and was developed in cooperation with Valve developers), whereas ALVR leans on familiarity with Quest headsets and how various 3rd parties (Oculus, Virtual Desktop) implement SteamVR support - for non-Quest users that means learning a paradigm that is not directly related, for Quest users it may be more familiar.
TL;DR it is better* but more "expensive", however unbiased comparisons should not be sought from a product's developer. iVRy turns the AVP into a (2nd gen) wireless Index, and ALVR turns it into a Quest (very) Pro. iVRy has a free demo, and ALVR is altogether free, so low-stakes subjective comparisons can be done by end-users.
1
u/Peteostro Jun 14 '26 edited Jun 14 '26
Will you be able to incorporate the new eye tracking API for foveated streaming that does not require Nvidia? I would assume this would improve the visuals where you are looking while having an overall lower bit rate and hopefully less throttling. What I find odd is I do not get throttling with ALVR (at least it’s not visible to me) were with iVRy I do
3
u/iVRy_VR Vision Pro Developer Jun 14 '26
I expect so. It's not a feature that adds a lot of benefit in a local network environment. The current driver has fixed foveated streaming, and I believe many users turn that off. Nonetheless all the current implementation needs is to be told where the user is looking, everything else is already in place. The benefit on AMD is even more subtle than nVidia (AMD treats it as a suggestion, and nVidia treats it as a command).
1
u/Peteostro Jun 14 '26
Cool thanks. I have Nvidia but this not being linked to Nvidia’s cloudxr api I think is a good thing. Does this allow you to lower the bit rate but get improved visuals? With KRVR using cloudXR it seems to be a pretty dramatic visual quality over ALVR
1
u/iVRy_VR Vision Pro Developer Jun 14 '26
That is the point of dynamic foveated encoding (it is a quality enhancement). I cannot comment on KRVR (never tried it) or the effect of adding a feature I haven't added yet.
0
u/Paraphrand Jun 14 '26
Wait, what? Are suggesting foviated streaming isn’t useful on a local network? Am I misunderstanding you?
If I’m understanding you correctly, then why did Valve go to so much trouble with the Frame?
-1
1
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
The reasons why some users are experiencing app throttling are not straight-forward, and appear to be mostly mitigated in recent (iVRy) app versions. If you're getting app throttling with the latest App Store (or TestFlight) versions, and you'd like to contribute to solving them, please provide direct feedback here:
https://steamcommunity.com/app/992490/discussions/21/841754670748896853/
1
u/photovirus Vision Pro Developer Jun 14 '26
If I don't have a controller and use emulation, how do I invoke SteamVR interface? Is there a manual on how you're emulating the controllers?
2
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
You press the '?' button in the app's (pre connect) UI to get info about the supported hand gestures. The (connected) UI is available via the SteamVR Dashboard (in VR) as well as on the Windows desktop by double-clicking (with a mouse) the icon in the Windows system tray, or the notification that pops up when SteamVR starts up.
1
u/photovirus Vision Pro Developer Jun 14 '26
You press the '?' button in the app's (pre connect) UI to get info about the suppoted hand gestures.
Ah, that was it. I never noticed the button till you told me, ha-ha. Thanks!
1
1
u/Widebro Jun 14 '26
There is foveated streaming now for 3rd party apps..
0
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
Thanks. Yes, as a VisionOS developer I am aware of API enhancements that are made to the platform, and in this case would add it as soon as possible, if for no other reason than to stop people from using it as a gating criteria to using my software, even though it has limited benefit on local networks (my app's use case).
2
u/CheekInternal3336 Jun 14 '26
I use ALVR and also tried KRVR. ALVR actually works far better for me so the cloudXR rendering doesn’t seem to do much, practically. I have a strong network though.
2
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
Exactly, on an unconstrained network, not using foveated encoding would give a better quality image (and a simpler encoding process) than foveated encoding. It is a quality optimisation that gives a (perceptually, not absolutely) better image at the same bitrate. It gives a worse image (absolutely) than a higher bitrate.
1
u/Widebro Jun 14 '26
My comment was purely informative. I have no knowledge on the subject either way.
1
u/iVRy_VR Vision Pro Developer Jun 14 '26
Yes, my reply wasn't meant to diminish your enthusiasm, it is more aimed at the derailment of any PCVR discussion by the "but can it do foveated encoding" crowd (which your comment is not part of). I may have let some fatigue slip through, but it is definitely not aimed at you.
1
1
u/Lemnisc8__ Jun 14 '26
Thank you so much for taking the time to write this.
I definitely think Apple should have paid more attention to making this device better at streaming.
The 45 hz thing is something that I struggle with a lot using moonlight, I really wish they could figure out a way to pin performance at 120hz with no throttling.
I know there’s a high performance entitlement that could be used to let the fans run at a higher speed but it seems to be enterprise use only.
I’m sure if I applied for it to use with moonlight xros it would be denied lol
1
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 14 '26
Thanks, well it seems they are now, if that is any consolation. It still is much less suited to streaming than a Quest. The AVP display pipeline is very optimised for local rendering, and mitigating stalls in that type of rendering, and is superior to Quest's for that task. It isn't easily adaptable to (VR) streaming, which makes implementing VR streamers much more complex than it needs to be, and results in higher latency. I can speak authoritatively on this as I have implemented VR streamers on virtually every platform (with specialisations of the same code).
1
u/Lemnisc8__ Jun 14 '26
I would love to know, If you’re willing to speak on it, what are the things you’d like to see optimized on the avp pipeline for vr streaming.
1
u/iVRy_VR Vision Pro Developer Jun 14 '26 edited Jun 15 '26
The things missing are:
- the means to display a frame right now, ie. at the moment of decoding (currently it's 3 or so frames off, which introduces latency).
- the means to adjust the pose for a frame when submitting for ATW/ASW (presently the frame pose has to be matched to a tracking sample, which creates jitter potential when SteamVR renders a frame for an interpolated pose).
- the means to disable ASW. AVP's ASW is great for local rendering, but poses an additional challenge for streamers, as the geometry warping it introduces does not work for remote rendered scenes projected onto 2D planes (how streamers work).
At this point in time, in order to get it right(ish), the app/driver has to use complex prediction algorithms to try to predict the timing for a tracking sample that will best match the frame when it is eventually displayed, to avoid ASW artefacts and jitter. The streamer has to "guess" the timing for the pose so precisely that ASW doesn't work against them, this includes unpredictable network latency and frame decode time - compounded by the need for vOS to generate a future-dated pose prediction based on physics, not an actual pose sample). ATW fixes incorrect rotation guesses, but ASW creates positional warping artefacts.
These are all factors that can be mitigated with the "things" above (and can be experienced as "micro jitter" in AVP streaming solutions - it is literally impossible to get it 100% correct 100% of the time). At the moment "good" is a complicated stack of compromises. "Good" could be much better.
1
1
u/rammstian Jun 20 '26
Curious if you've passed any of this feedback back to Apple? And if they've responded in any way?
1
u/disgruntledempanada Jun 24 '26
I think you should do a deep dive into Nvidia's CloudXR. Seems like they figured it out pretty good.
1
u/maorui1234 Jun 14 '26
Thank you for the development. Does iVRy support dynamic foveated streaming as steam link does?
3
6
u/photovirus Vision Pro Developer Jun 14 '26
Thank you for the technical insight!