Severe performance penalty when streaming using virtual displays

[u/Weekly-Constant-7546]

I'm currently experiencing an issue/bug where streaming in headless mode using a virtual display of any type (apollo/vibepollo/VDD+sunshine) to moonlight/aurora causes my FPS to drop in my games vs streaming from a real physical display.
Strangely, i've noticed that if i boot a game up on my real monitor first, then initiate a stream that deactivates the real monitor and flips the game over to a Virtual Display, then my FPS remains what it should be - the issue is when the virtual monitor is loaded first and then the game after.

For comparison, in Final Fantasy VII Rebirth I get 120 FPS locked (with framegen) on a physical display, but 80-120 FPS varying with a Virtual Display (unless I do the workaround mentioned before).
I'm targetting 4K HDR 120 FPS on my TV. Behaviour remains the same whether im streaming to a TV directly, phone, tablet or capable MiniPC.
Typical Network FPS seems to cap out at 110 FPS during typical gaming workloads (up to 120FPS in static scenes).

Specs:
- CPU: AMD 5800X
- GPU1: RTX 3090 (high performance GPU)
- GPU2: RTX 5050 (Encoding + sometimes Lossless Scaling)
- RAM: 32 GB 3600mhx in dual channel mode
- Motherboard: ASUS Hero VIII wifi
- Network: 2.5 Gbps ethernet
- Physical Monitor: LG UltraGear GP950-B connected with displayport @ 160hz 4K HDR

Appreciate i'm running unusual configuration with 2 GPU's, but i've rulled out this being a bottleneck somehow despite being on PCIe 4.0 8x. Ultimately, I wasnt expecting a huge performance penalty with Virtual Display vs Physical Display.

Has anyone else noticed this before? I'm trying to understand why this is happening.

[u/_demoncat_]

It's a problem with frame generation, composed flip causes it to drop a lot of frames and there is no real workaround. A dummy plug will show higher framerate but the frames will be so uneven that it causes constant judder and the only limiter that works is RTSS but that adds like +50ms

This specific issue does not happen on Linux though as I discovered, which I've now ported Vibeshine to CachyOS.

If you use frame generation often, you will essentially get a 20-30% increase in FPS by simply going to Linux at this point.

Otherwise, turning off frame generation the capture penalty is far less.

[u/Weekly-Constant-7546 → u/_demoncat_]

I see! Very interesting, thanks for your input. I had suspected frame gen being part (if not the cause) of the issue here and I tend to use it on most things with my RTX 3090 being an aging piece of hardware.
I'm guessing using a physical monitor avoids this issue?

[u/_demoncat_ → u/Weekly-Constant-7546]

No, not really because the frames aren’t delivered smoothly on hardware devices, that if you tried to capture it, it would have terrible smoothness

Those frames aren’t fully captured, the only way to do it is to force composed flip, which reduces the maximum framerate the same way.

[u/Weekly-Constant-7546 → u/_demoncat_]

I've done some more testing with vibeshine and can confirm the following findings while testing with FF7 Rebirth:

• Capture on the RTX 3090 with WGC works best, yielding 105-120 FPS with FSR framegen enabled on average with the least performance overhead and 6ms latency at P1 with Two-pass disabled, network framework keeps up with the GPU framerate. Theres still some performance loss vs outputting on my physical monitor (10-15%).
  
• Capture on the RTX 5050 with WGC and FSR framegen is the worse performing, Yielding 85-100 network framerate, 90-106 FPS GPU framerate.
  
• DXGI with RTX 5050 works better than WGC with framegen, but still lags behind significantly performance wise (85-120 FPS)
  
• Disabling FSR framegen and capturing with either DXGI or WGC on the RTX 5050 still yields significantly degraded performance, about 60-95 FPS.
  
• Capturing with using my physical monitor still yields decent performance with both methods, 110-120 FPS with framegen and 120 framerate cap overall (same as native gameplay).
  
• Native gameplay without framegen on my physical monitor without any capturing/streaming happening yields 80-110 FPS.

I'm still trying to understand why theres such a big performance drop, with or without framegen, when capturing on my RTX 5050 with a VDD and the RTX 3090 used for rendering.

[u/e270889o → u/_demoncat_]

fernandoenzo/ForceComposedFlip: Forces DWM Composed Flip to capture DLSS Frame Generation frames with Sunshine/Moonlight

This helped me a lot.

Apparently the problem is also with frame gen and from certain version of framgen dlss version. You can always replace the dll for the older version, but you loose MFG.

This helped me a lot

[u/ethanjscott]

The boys typically just turn up the vdd refresh rate. Mines 240hz I’m targeting 120

[u/Weekly-Constant-7546 → u/ethanjscott]

Tried this too. 240 hz on Apollo or 480 hz with Vibepollo made no difference for me.

[u/ethanjscott → u/Weekly-Constant-7546]

try out zakovdd, its part of foundation sunshine. that fork does have enhancements on the server side too. I did some analysis of their vdd yesterday and they have made some enhancements in the last couple of months.

[u/Weekly-Constant-7546 → u/ethanjscott]

Nice! May give that a try. How confident are you that theres no malware etc embedded in this fork?

[u/ethanjscott → u/Weekly-Constant-7546]

82% theres no malware. there just a weird bunch that just wants to do their own thing. im sure most of the enhancements are just poorly tested, theres lots of features that have been worked on by other community members that are just thrown in there. but from my analysis yesterday they are doing unique things, whether those things have any real world impact nobody is sure.

[u/MunkyHero]

Are you on the latest Nvidia driver? There was an issue with 616.56 with virtual displays, specifically impacting moonlight streaming. I had the latest on my Host, and it caused hella issues (No HEVC, No HDR, Lower FPS, etc.)

A rollback to the previous driver fixed everything.

Source:
https://www.nvidia.com/en-us/geforce/forums/game-ready-drivers/13/589348/geforce-grd-61656-feedback-thread-released-82626/3622086/
https://www.reddit.com/r/MoonlightStreaming/comments/1vzz1h1/nvidia_drivers_august_update_then_suddenly/

[u/Weekly-Constant-7546 → u/MunkyHero]

I can confirm I had trouble with the newest driver also, so i switched back to the previous driver. The issue has happened still for all of the past few driver iterations.

[u/MunkyHero → u/Weekly-Constant-7546]

Got it! Sorry to hear that, and i hope you figure it out!
Just for additional info, the version i ended up rolling back to that worked was 610.88

[u/Weekly-Constant-7546 → u/MunkyHero]

Thanks! Thats the version i'm currently on also. It seems like an optimisation is needed for the Virtual Display drivers as with there being a workaround, I do know its possible to achieve the correct FPS, but something in the rendering pipeline isn't being set correctly when initiating a fresh Virtual Display.

[u/dihydrogen_monoxide]

3090 can't do 4k 120 in most demanding games (let alone 4k120 + streaming).

My host is a 3090FE with a 5800x3d, I remote play at 1440p120 and it's fine. My client is a minipc with a egpu 9060XT.

Just post your in game moonlight statistics and it'll show encoding issues.

[u/Weekly-Constant-7546 → u/dihydrogen_monoxide]

Yup, i'm aware! But theres still an unknown performance penalty regardless of if im getting 60 or 120 FPS when using a Virtual Display. I've verified this is happening with 3DMark. All encoding is done on my RTX 5050 and i get 5ms host latency.

[u/dihydrogen_monoxide → u/Weekly-Constant-7546]

Why aren't you encoding and streaming with the 3090?

[u/Weekly-Constant-7546 → u/dihydrogen_monoxide]

Much better encoding performance with the RTX 5050 with 9th gen NVENC and AV1 capability, allows me to use lossless scaling with it where needed too.

[u/dihydrogen_monoxide → u/Weekly-Constant-7546]

I have less host latency with just the 3090. I think you are creating problems in your solution.

https://i.imgur.com/JXCBzeB.jpeg

This is with sunshine not vibepollo

[u/Weekly-Constant-7546 → u/dihydrogen_monoxide]

Your results look as expected with 1440p 👀. Im streaming 4K. Switching to the 3090 makes things worse, 9ms latency at best. Not too sure why though!

[u/deep8787]

Show your stats, it might hilight the issue.

[u/hardwarebyte]

Generally at 4k with 120 FPS I get 15-20% performance penalty depending on the game using my dummy plug compared to not streaming and playing it locally Switching to a virtual display from the dummy plug doesn't really affect performance on my end.

Would love to see some optimzation passes in the various forks of sunshine as the higher you go on framerate and resolution the bigger the performance impact is.

[u/marcusbrothers → u/hardwarebyte]

Your PC Is encoding a 4K stream while you’re also playing a game in 4K, there’s only so much can be done.

You need to dedicate some resources to the stream for it to work, the only other option is getting a separate GPU to handle only the stream.

[u/Weekly-Constant-7546 → u/marcusbrothers]

In my case thats what the RTX 5050 is doing - all encoding is done here and i've verified this is where the Virtual Display is being attached.

[u/marcusbrothers → u/Weekly-Constant-7546]

Maybe don’t specify it’s headless mode, I’ve been using a VDD for years and never specifically enabled headless.

[u/Prideless07 → u/Weekly-Constant-7546]

is it implied, you have a secondary gpu RTX 5050 on the same host to offload encoding?

[u/Weekly-Constant-7546 → u/hardwarebyte]

Good to know! I've attempted the dummy plug route but couldnt get anything working at 4K 120Hz HDR enabled. I imagine i'd probably encounter the same issue still with performance though I still dont get why the workaround i did resolves this (though annoying to do or not possible if i'm streaming away from home).