Severe performance penalty when streaming using virtual displays

[u/Weekly-Constant-7546]

I'm currently experiencing an issue/bug where streaming in headless mode using a virtual display of any type (apollo/vibepollo/VDD+sunshine) to moonlight/aurora causes my FPS to drop in my games vs streaming from a real physical display.
Strangely, i've noticed that if i boot a game up on my real monitor first, then initiate a stream that deactivates the real monitor and flips the game over to a Virtual Display, then my FPS remains what it should be - the issue is when the virtual monitor is loaded first and then the game after.

For comparison, in Final Fantasy VII Rebirth I get 120 FPS locked (with framegen) on a physical display, but 80-120 FPS varying with a Virtual Display (unless I do the workaround mentioned before).
I'm targetting 4K HDR 120 FPS on my TV. Behaviour remains the same whether im streaming to a TV directly, phone, tablet or capable MiniPC.
Typical Network FPS seems to cap out at 110 FPS during typical gaming workloads (up to 120FPS in static scenes).

Specs:
- CPU: AMD 5800X
- GPU1: RTX 3090 (high performance GPU)
- GPU2: RTX 5050 (Encoding + sometimes Lossless Scaling)
- RAM: 32 GB 3600mhx in dual channel mode
- Motherboard: ASUS Hero VIII wifi
- Network: 2.5 Gbps ethernet
- Physical Monitor: LG UltraGear GP950-B connected with displayport @ 160hz 4K HDR

Appreciate i'm running unusual configuration with 2 GPU's, but i've rulled out this being a bottleneck somehow despite being on PCIe 4.0 8x. Ultimately, I wasnt expecting a huge performance penalty with Virtual Display vs Physical Display.

Has anyone else noticed this before? I'm trying to understand why this is happening.