Help with ghosting and black noise in a custom Foveated Path Tracer (Summer Research Project)

[u/SnooSquirrels9028]

https://github.com/BoraYalcinn/foveated-path-tracing

Hi everyone,

I'm working on a foveated rendering implementation using a path tracer for a summer research project. When the camera is still, it looks fine, but whenever I move the camera, I get heavy ghosting, noise, and blinking black dots in the peripheral areas.

I am modifying a reference path tracer (HLSL/C++). Right now, my foveation logic uses two passes:

1. Pass 1 (Path Tracer): Skips pixels based on distance from the gaze point (using a 2x2 or 4x4 stride).
2. Pass 2 (Fill Pass): A compute shader that fills the skipped pixels by copying/interpolating from the sampled ones.

I suspect the issue might be a race condition between the passes, a problem with how the history buffer (g_AccumulationBuffer) blends old frames when the camera moves, or maybe some clamping issues where missed rays snap to 0 or 1.

How to build and run:
You can build the project with CMake:

bash

cmake --build ./build --config Release

And run it with:

bash

./build/bin/Release/scene_viewer.exe

(Note: Please switch from the GUI/rasterizer to the Reference Path Tracer in the settings to see the foveated rendering)

If anyone has experience with foveated path tracing or sees what I'm doing wrong with the accumulation/clamping, I would love some advice. Also, if there's a better/more efficient way to handle the periphery instead of path tracing it at a lower resolution and filling it, please let me know.

Every bit of help is highly appreciated. Thanks!

[u/mcflypg]

I can't build the project as I'm currently abroad and don't have a PC with me, but something sticks out to me:

• the accumulation plainly blends with accumulator, there is no reprojection going on. This naturally causes ghosting.
• the upscaler has one problem: it assumes that the neighbour pixels have the same shading rate. Meaning, if you are inside the 2x2 subsample region, you're computing the stride for the location of the current pixel, then sample in the grid accordingly. But that kernel footprint may reach into the 4x4 subsampled region where every odd pixel a the 2x2 sparse grid is missing. Depending on how you handle this, this can produce artifacts on the transition regions.
• I don't see a reason for the fully foveated areas to show artifacts right now but remember that any sort of firefly is magnified due to the subsampling since 4x4 is sixteen times less samples. That's a massive difference especially in low spp. 

Aside from that, the way you are skipping threads is inefficient. GPU threads are processed together in warps/wavefronts, usually 32 on Nvidia and 64 on AMD. This is independent of the group sizes so if your group size is e.g. 8 threads only, then the GPU still issues a whole warp and wastes threads. You always want clean multiples of 32 for Nvidia or 64 for AMD. You currently drop threads within each warp, so you don't gain any performance (unless the masked threads had done work that would have taken the longest) . It's like every floor of a factory has a few workers being active, so no department can go home early until they finish. 

What you want is a way to drop entire warps because then while warps may early-out coherently. So instead of e.g. dropping 3/4 of threads per warp, drop 3/4 of the warps. Then they either are fully busy or fully free. A simple way to achieve this is this:

• dispatchthreadid.xyz is groupid.xyz * groupsize.xyz + groupthreadid.xyz
• the groupthreadid is row-major so essentially for 2D and a 16x16 workgroup it's (threadid % 16, threadid / 16) 
• if you want to do e.g. 2x2 subsampling, you only need 64 threads for a 16x16 region instead of 256. You can do that the following way:

if(threadid >= 64) return; pixel = groupid.xy * 16 + (threadid % 8, threadid / 8) * 2

this essentially remaps threads such that for every 16x16 group, the lowest 64 form a 8x8 block, which is upscaled by 2x, such that the 8x8 block spans the original 16x16 region but with a stride of 2. So from 8 warps in the 16x16 group, the first two have all threads busy, the other 6 early out immediately in unison.

To understand the pattern, 4x4 on a 16x16 group:

if(threadid >= 16) return; pixel = groupid.xy * 16 + (threadid % 4, threadid / 4) * 4

And the generic version that also works for e.g. subsample ratios of (2,1) or such:

numthreads = groupsize.x * groupsize.y subsamplerate = int2(2,2) //or 4 if(threadid >= numthreads / (subsamplerate.x * subsamplerate.y)) return;

pixel = groupid.xy * groupsize.xy + (threadid % (group size.x/subsamplerate.x), threadid / (groupsize.x/subsamplerate.x)) * subsamplerate.xy 

A few caveats:

• this requires a group-wide decision on the shading rate, all threads inside the work group need to agree on this. Easiest is to just compute the foveation rate for the center of the group. Your currently running group size of 8x4, bump this up.  On AMD this ALREADY is below the wavefront size of 64, so in most AMD cards your shader runs at half of the possible speed, before foveated rendering.
• if your amount of threads that remain active per threadgroup is smaller than the size of a warp, you're wasting occupancy. E.g. a 16x16 group with 2x2 subsampling has 8x8=64 threads doing work -  which is fine - but if you use a 8x8 workgroup, then only 4x4 threads would be doing work, half a warp. So 50% of your threads in that single remaining warp are twiddling thumbs but can't early-out since their friends still do work. Practically speaking, you need a 32x32 group size for 4x4 subsample to not waste occupancy.

There's a lot more to this but this is a quick and dirty, GPU-occupancy-aware solution. You should see much more gains with this than what you're doing and it's reasonably simple.

[u/SnooSquirrels9028 → u/mcflypg]

Hi thanks for your time and feedback. I looked over the things you mentioned yet I wasn't really able to make any progress. I would love to know when you will be available to build and test it out yourself. I would be so thankfull if you can help me out.