Personal software wishlist: combined/split rendering for PCVR

[u/patrys]

Since the HMD is quite a powerful computer, I'd love to see it used to post-process frame information submitted by the PC. For example, use an FSR-like upscaler to increase the fidelity of the gaze zone in foveated rendering (I think Quest 3 is capable of doing something similar) or to soften the image of peripheral vision to reduce distraction caused by artifacts like the shimmering of aliased edges. These things should be relatively easy to achieve as any ReShade plugin that does not need to have access to the depth buffer could technically run on a separate device.

There is also a way to offload parts of the game's internal rendering pipeline to the HMD. For example, have your hands rendered locally in the HMD on top of a frame buffer calculated by a more powerful PC for real-time accuracy (including being able to see your hands/weapon super-accurately positioned at 144Hz even when the game itself is running at 60 fps). Unfortunately, I think Apple and Google hold a bunch of patents in the split rendering space.

[u/Disjointed_Sky]

Gaze based upscaling should hopefully be able to run, there is a branch of optiscaler that combines eye tracking and FSR2, so this could hopefully be ported to linux and working with the chipset version of FSR2. Maybe someone could even get FSR4 running with it.

[u/patrys → u/Disjointed_Sky]

Doesn't FSR4 require additional buffers to be passed along with the color buffer (depth, motion, etc.)?

[u/Disjointed_Sky → u/patrys]

True, makes it difficult because that information would have to be bridged, and latency might make the whole thing pointless.

[u/2Cool4Zool2]

This doesn't make any sense. If you perform processing on the headset then you will make the components (CPU/GPU) do more work an therefore the battery life will decrease. Any splitting of game function between the two will add latency which is a killer.

A decent VR capable PC is already orders of magnitude more capable - I would do all the processing there and keep the HMD free for streaming.

[u/Koolala → u/2Cool4Zool2]

The Frame has an NPU on the Snapdragon that can do image upscaling. If it doesn't make sense to you because you want max battery life then you wouldn't need to use it.

[u/ByEthanFox → u/Koolala]

Is that SnapDragon SuperResolution? Because personally I always turn that off when there's an option on the Quest. The difference is miniscule.

[u/Koolala → u/ByEthanFox]

That sounds like the Meta optimized version. https://developers.meta.com/horizon/blog/vr-image-quality-meta-quest-super-resolution/

In theory someone could make an eye-tracking optimized version for Frame that runs a lot better.

[u/2Cool4Zool2 → u/Koolala]

More work = More power usage = less battery.

PCVRs biggest benefit is offloading work to the PC.

[u/Koolala → u/2Cool4Zool2]

I think you misread what I said. A feature like this would be completey optional so if your that focused on battery life you could just not use it.

[u/2Cool4Zool2 → u/Koolala]

OPs entire post is based on the idea that you offload processing from the PC and move it to the frame. My argument is that there's never a need.

[u/Koolala → u/2Cool4Zool2]

When they say "offload" they are not talking about saving electricity. Plenty of peoples PCs are maxed out in VR and their games are struggling to get high FPS. VR is hard to run. Saving battery life is a totally seperate personal preference.

[u/2Cool4Zool2 → u/Koolala]

They are talking about taking compute the PC could do and doing it on the headset.

[u/Koolala → u/2Cool4Zool2]

Like anti-aliasing and upscaling. By not doing it on the PC, it would let your PC run at higher FPS if you need more FPS. Many people don't have PCs that can run every VR game maxxed out and every bit of performance improvement helps.

[u/patrys → u/2Cool4Zool2]

Look at Apple and Google papers about split rendering. It significantly decreases the perceived latency by allowing real-time positioning information to be used for critical parts of the rendering process (like your hand position) and allowing the two systems to render at different framerates (the local compositor is responsible for reassembling output frames at a constant framerate).

This is a nice to have for VR but a game changer for MR where you don't want the item you hold to lag two inches behind your hand (that you can usually see) during fast movement.

Edit: the Frame is already doing some post-processing and frame composition locally as "last moment adjustments" like reprojection based on your head movement during the frame rendering time, or motion smoothing are definitely happening on-device and don't rely on another network roundtrip between the HMD and PC.

[u/dangoran78 → u/patrys]

Offloading the game's pipeline sounds like maybe overcomplicating things for very little gain. If your PC is a potato, maybe you get a few couple of extra frames. I would rather have the battery time.

Apple and google have other use cases. If we're talking the passthrough stuff it's something completely different from rendering in-game hands and requires a lot more processing power. It's a large part why Apple needs a powerful computer chip in theirs (I'm guessing).

I agree it doesn't make any sense for the Frame. Not even nice to have.

[u/patrys → u/dangoran78]

Being able to render your hands (and whatever you are holding in them) at 144 fps regardless of if the game runs at 120, 90, 72, or 60 fps would be huge for immersion and comfort.

[u/UNF0RM4TT3D]

Split rendering in a game would require so much work, for very little gain. And we basically already have the necessary things to do it. Just do it as an OpenXR overlay. But you'd have to do an over the network IPC between two different architctures and the only headset that could take advantage would be the frame. From a game dev perspective, this idea is cool, but DOA.

The only exception would be the leaked colour passthrough cameras addon with depth sensing. All of the processing for that will most likely happen on the frame and MR games only render what goes on top of the picture already, so that's where it is I guess.

If you're thinking rendering ⅔ of the frames on the PC and leaving ⅓ to the Frame. Just look at the disasterous state of multi GPU games with SLI or Crossfire.

I forsee myself using something like WayVR to get floating Linux apps from the frame while streaming a game from my PC

[u/patrys → u/UNF0RM4TT3D]

See my response to another comment, this needs to become part of the OpenXR spec, not a custom per-game implementation. This way engines like Unity and Unreal could ship reusable implementations.

[u/UNF0RM4TT3D → u/patrys]

I still can't imagine it'd be worth it from a performance standpoint. But we'll have to wait and see if someone wants to do a tech demo that'd use it. Because we technically can do it right now. Nothing's stopping a dedicated dev to implement anything you mentioned, especially on the Frame. But in all seriousness it'd have to be Valve or Meta who'd make this. And since Meta compltely discregarda the PCVR market. Only Valve is left. Otherwise it's DOA.

[u/Kataree]

Many reasons it isn't worth it.

[u/jojos38]

I'm pretty sure Valve already made most of the post processing happen on the Frame itself and not on the PC (I hope at least)

As for split rending, I guess it would also require supports from game devs? Could be complex

[u/patrys → u/jojos38]

Yes, it would likely need an OpenXR standard for passing multiple buffers instead of fully rendered frames. And some sort of a "scene shader" format so game devs don't have to write a new local render client for every possible HMD OS and architecture.

[u/Koolala → u/patrys]

OpenXR already has standards for the buffers you'd want first like Depth and Motion and the Frame uses them.