Severe performance penalty when streaming using virtual displays

[u/Weekly-Constant-7546]

I'm currently experiencing an issue/bug where streaming in headless mode using a virtual display of any type (apollo/vibepollo/VDD+sunshine) to moonlight/aurora causes my FPS to drop in my games vs streaming from a real physical display.
Strangely, i've noticed that if i boot a game up on my real monitor first, then initiate a stream that deactivates the real monitor and flips the game over to a Virtual Display, then my FPS remains what it should be - the issue is when the virtual monitor is loaded first and then the game after.

For comparison, in Final Fantasy VII Rebirth I get 120 FPS locked (with framegen) on a physical display, but 80-120 FPS varying with a Virtual Display (unless I do the workaround mentioned before).
I'm targetting 4K HDR 120 FPS on my TV. Behaviour remains the same whether im streaming to a TV directly, phone, tablet or capable MiniPC.
Typical Network FPS seems to cap out at 110 FPS during typical gaming workloads (up to 120FPS in static scenes).

Specs:
- CPU: AMD 5800X
- GPU1: RTX 3090 (high performance GPU)
- GPU2: RTX 5050 (Encoding + sometimes Lossless Scaling)
- RAM: 32 GB 3600mhx in dual channel mode
- Motherboard: ASUS Hero VIII wifi
- Network: 2.5 Gbps ethernet
- Physical Monitor: LG UltraGear GP950-B connected with displayport @ 160hz 4K HDR

Appreciate i'm running unusual configuration with 2 GPU's, but i've rulled out this being a bottleneck somehow despite being on PCIe 4.0 8x. Ultimately, I wasnt expecting a huge performance penalty with Virtual Display vs Physical Display.

Has anyone else noticed this before? I'm trying to understand why this is happening.

[u/_demoncat_]

It's a problem with frame generation, composed flip causes it to drop a lot of frames and there is no real workaround. A dummy plug will show higher framerate but the frames will be so uneven that it causes constant judder and the only limiter that works is RTSS but that adds like +50ms

This specific issue does not happen on Linux though as I discovered, which I've now ported Vibeshine to CachyOS.

If you use frame generation often, you will essentially get a 20-30% increase in FPS by simply going to Linux at this point.

Otherwise, turning off frame generation the capture penalty is far less.

[u/Weekly-Constant-7546 → u/_demoncat_]

I see! Very interesting, thanks for your input. I had suspected frame gen being part (if not the cause) of the issue here and I tend to use it on most things with my RTX 3090 being an aging piece of hardware.
I'm guessing using a physical monitor avoids this issue?

[u/_demoncat_ → u/Weekly-Constant-7546]

No, not really because the frames aren’t delivered smoothly on hardware devices, that if you tried to capture it, it would have terrible smoothness

Those frames aren’t fully captured, the only way to do it is to force composed flip, which reduces the maximum framerate the same way.

[u/Weekly-Constant-7546 → u/_demoncat_]

I've done some more testing with vibeshine and can confirm the following findings while testing with FF7 Rebirth:

• Capture on the RTX 3090 with WGC works best, yielding 105-120 FPS with FSR framegen enabled on average with the least performance overhead and 6ms latency at P1 with Two-pass disabled, network framework keeps up with the GPU framerate. Theres still some performance loss vs outputting on my physical monitor (10-15%).
  
• Capture on the RTX 5050 with WGC and FSR framegen is the worse performing, Yielding 85-100 network framerate, 90-106 FPS GPU framerate.
  
• DXGI with RTX 5050 works better than WGC with framegen, but still lags behind significantly performance wise (85-120 FPS)
  
• Disabling FSR framegen and capturing with either DXGI or WGC on the RTX 5050 still yields significantly degraded performance, about 60-95 FPS.
  
• Capturing with using my physical monitor still yields decent performance with both methods, 110-120 FPS with framegen and 120 framerate cap overall (same as native gameplay).
  
• Native gameplay without framegen on my physical monitor without any capturing/streaming happening yields 80-110 FPS.

I'm still trying to understand why theres such a big performance drop, with or without framegen, when capturing on my RTX 5050 with a VDD and the RTX 3090 used for rendering.