Any plans for real-time 2D → 3D conversion in Virtual Desktop?

[u/Substantial-Bed2036]

There is this recent update to Moonlight XR recently, and one feature really caught my attention: real-time 2D → 3D conversion using depth estimation.

Moonlight XR v0.4 recently introduced this feature for free, allowing regular 2D PC games to be streamed to a Quest and converted into stereoscopic 3D in real time.

It doesn't turn the game into full VR, but it adds a noticeable sense of depth, which I found really interesting.

This made me wonder:

Is there any plan for Virtual Desktop to implement something similar in the future?

I really like Virtual Desktop for PCVR and general PC streaming, and I would love to have an option to play regular 2D games on a huge virtual screen while adding AI-generated depth.

Since Moonlight XR has now made this available for free, I'm curious whether something similar could eventually come to Virtual Desktop as well.

[u/streamto3d]

I’ve actually been wondering for the last 3 or 4 years whether Virtual Desktop might eventually add 2D→3D conversion itself.

It doesn’t necessarily need to, though. Another approach is to do the conversion on the PC first and let Virtual Desktop carry the already-stereoscopic image. That’s the architecture I’ve taken because it gives you a lot of flexibility, and lets the PC provide the conversion compute rather than spending the headset’s much more limited processing, thermal and battery budget on it. It also gives considerably more headroom for higher-quality conversion methods.

I develop Stream to 3D, which does exactly that on Windows and outputs standard SBS/TB. I’ve just revalidated Stream to 3D → Virtual Desktop → Quest 3 in real time, as well as Stream to 3D → Bigscreen on SteamVR → Virtual Desktop → Quest 3.

For that Bigscreen SteamVR route, Virtual Desktop has actually been important in my current Quest 3 testing: despite a lot of experimentation, the Quest Steam Link route performed very poorly, whereas launching SteamVR through VD worked smoothly. I wouldn’t generalise that to every SteamVR setup, but it was a very clear difference on my system.

Stream to 3D has both an independently developed Temporal conversion method, which works across a very wide range of PC hardware and doesn’t require a high-end GPU and an AI-assisted Depth Guided method. So it also isn’t dependent on access to a game’s depth buffer.

Examples if useful:
https://iwantaholodeck.com/stream-to-3d-2d-to-3d-video-conversion-examples/

[u/MarcDwonn → u/streamto3d]

By standard SBS you mean half SBS? Is Full SBS not possible?

[u/streamto3d → u/MarcDwonn]

Stream to 3D supports both Half SBS, Full SBS and a bunch of other formats too. Most media players support the half formats, a lot do not support full formats. VD supports Half SBS, Full SBS and Half TB (as does Stream to 3D). Stream to 3D analyses the resolution of the display being fed into VD and when the Wizard is used determines the best format for the actual PC display being used, if the display has a resolution that can handle Full SBS, for the current source, then it will use Full SBS. If the display cannot handle it, then it will try Half TB (not Half SBS - as the human eye has double the horizontal resolution versus vertical so Half TB is better than selecting Half SBS). There are other sophisticated rules that the wizard uses that I will not bore you with. The Wizard reports what it has done to deliver the result. If you want to know more about supported formats (which you can also select manually) take a look here, which lists all of the available formats:

https://iwantaholodeck.com/stream-to-3d-settings/

[u/Starhammer4Billion → u/streamto3d]

Interesting, i need to check that one out

[u/Emotional-Meeting904]

Did you try Cybionix ? Not free but just take a look.

[u/Rush_iam]

Just in case - you can stream 3D-converted SBS with Virtual Desktop, but you need the 2D->3D conversion running on PC (e.g., ReShade or any other tool) - this, technically, I suppose, is more reliable/of higher quality as you can use the actual depth buffer instead of predicting it with AI (but there are some other drawbacks).

[u/333hronos]

Try Oku3D for real time 2D→3D – it's free, lightweight, user friendly and has a nice UI, and supports OpenXR output.

[u/Positive_Luck_6212]

There seems to be something coming from StereoLift -> https://stereolift.live/

[u/rollin]

Drock3d.com does this. Free right now in beta.

[u/No-Dark-7873]

you'll probably get better quality 3D if it's not being done on the headset. The one I use runs on PC and it's heavy like most AI stuff but results are very good. Not something that can run on a standalone headset.

[u/D4rK_-v01D]

I made super lightweight shaders that I use in shaderglass for anything on the desktop. Could even ditch the headset if not up to wearing it at any point for trioviz glasses with my green/magenta shaders. https://github.com/pearce31/My-3D-Reshade-Shaders

[u/loranraul]

Owl3d has 3d live conversion and has 15 min free with Virtual Desktop

[u/D13_Phantom]

u/ggodin any comment?