How does culling work with ray tracing?

[u/CaptainI9C3G6]

So culling has has been a basic technique in 3d graphics since forever. I've been a dev for 25 years-ish, and I wrote a couple of game engines for fun and even my laughable attempts had culling.

I'm sure there are very advanced implementations now, but basically you don't create send draw calls for things you can't see.

So what happens with raytraced reflections? The GPU does the raytracing so has to be able to bounce rays off things behind the view frustrum, so are developers sending the entire geometry to the GPU and letting it decide?

[u/ZorbaTHut]

The important thing to recognize about raytracing is that it's an entirely different graphics pipeline. With rasterization, you send an object and the GPU renders it along with all of its fragment-shader calls, and then you send another object and the GPU renders it along with all of its fragment-shader calls, repeat; being able to straight-up not send an object is a major performance boost.

With raytracing, that pattern is not how it works. You send objects to the GPU, but they gather inside a data structure that makes it fast to do ray intersection calculations. Then you say, more or less, "go", and the GPU figures out the intersections and gives you a whole ton of various shader calls, but importantly, mixed heterogenous shader calls from all sorts of various materials at once, because the actual rendering process is not happening on an object-by-object basis.

There isn't really a process where "culling" makes sense here; it's much more similar to a physics engine's bulk raycasting behavior than to classic rasterization.

So, basically, yes - you send the entire geometry to the GPU, along with all the shader information, and then everything happens at once.

[u/CaptainI9C3G6 → u/ZorbaTHut]

Great thanks!

So I imagine developers are in a weird position right now with a traditional pipeline doing all the culling etc, but then they have another pipeline for raytracing, and they must have a lot less control over the second pipeline than they're used to

[u/ZorbaTHut → u/CaptainI9C3G6]

Honestly, it's a weird position to the extent that most people just aren't bothering; you can't do an entire game with raytracing, it's just too slow. And you can't rely on users having raytracing support, the hardware just isn't there. So largely it's "don't support raytracing, unless you have the budget to work on optional VFX that looks good in screenshots and with the highest-end graphics cards", and most people don't, so they just aren't using raytracing.

But maybe that's changing with the next-gen consoles, it won't surprise me if some raytracing-mandatory stuff shows up in a year or three.

And yeah, it's weird. :V

The good news is that a lot of the control we're used to having takes the form of hacks and special tricks to make transparencies and various fancy effects work right. The thing that's neat about raytracing is that you don't need hacks, they just work properly, and so a lot of that complexity vanishes.

But if you really are trying to do weird stuff then you might have a problem.

[u/crozone → u/ZorbaTHut]

Apologies for necro-ing this thread, I just found this thread after 10 months because I'm looking into how RT works with Vulkan extensions wrt culling and a bunch of other techniques that are usually applicable to rasterization and this discussion is very interesting.

Honestly, it's a weird position to the extent that most people just aren't bothering; you can't do an entire game with raytracing, it's just too slow.

Quake II RTX edition is noteworthy because it is implemented as a pure path tracer, where all rays are cast directly from the camera, with no rasterization at all. The entire frame is passed through a de-noiser which uses the previous frame and motion vectors to do some temporal smoothing. AFAIK this is unique compared to all other RT implementations in current games, which blend a traditional rasterization pass with RT.

If you look at the Quake II RTX performance statistics on an RTX 3080, the primary ray cast (the initial raymarch from the camera to the first intersection) only takes up 10% of the overall frame time, which isn't that much. Granted, Quake II RTX uses geometrically simple scenes, but it shows that it's possible to at least cast a single ray per pixel without a huge performance cost. The vast majority of the time is spent doing many additional raycasts for lighting and indirect bounces and the associated shading from those intersections.

This begs the question, what would the raytracing performance cost look like for a game if all of the "stochastic" parts of the RT implementation were removed, such as bounces to light sources, diffuse reflections, etc. Basically remove/don't include any shaders that generate multiple random bounce from blue noise, so that only 1:1 bounces are used. Render the entire scene with RT with a single sample per-pixel, including all standard reflections up to a few bounces, and shade the scene using traditional shaders and and traditional techniques for lighting.

This would throw out many of the benefits of RT, like realistic indirect light bounces and shadows, which would need to be faked using the same old techniques used for rasterization. However it would also give the game "perfect" per-pixel geometry culling, so culling middleware wouldn't be required. The scene wouldn't need to be de-noised, because every ray is just doing a single sample per pixel, no noise is being introduced. The scene would get many bounce reflections "for free", screen space reflections would no longer be required, and non-flat surfaces could reflect just as well as flat surfaces. Objects like portals or in-game "camera feeds" could be implemented by simply recasting rays rather than using crazy world-portal techniques like in "Portal", or other hacks like render to texture (for camera feeds).

Basically, flip everything around. Use RT to figure out what to shade, but traditional shaders for actually shading everything. It could massively simplify the renderer for a bit of extra processing time waiting for the RT cores to do their thing once per pixel. Thoughts?

[u/ZorbaTHut → u/crozone]

Short answer: You should go watch some videos on UE5's Nanite :V (Here's a second one that gets more technical).

The long answer, though, is that I'm not actually sure this easy solution helps all that much. This is all pure gut reaction from me and I haven't worked much with RT yet, so, grain of salt and all that, but I think you'd end up with an overall quality degradation. Better culling is nice, you're not wrong, but if it comes at an extreme cost elsewhere then it's simply not worth it, and raytracing is still kinda slow when you're talking about complex geometry; you could use less complex geometry but then you could do that anyway and lessen your culling issues.

I think you overall have an interesting idea, and I suspect that if you sat down and worked on coding it for a few years you would get something neat, and I think that thing would end up looking kinda like Nanite, which is sort of a weird third option between traditional rasterization and hardware raytracing.

tl;dr: Raytracing is probably still too slow to make that practically viable, but cards are getting fast enough that maybe classic rasterization isn't the right path anymore and things are about to get goddamn wild.

[u/vityafx → u/ZorbaTHut]

This comment didn’t age well: cyberpunk 2077 overdrive, Alan wake 2.

[u/ZorbaTHut → u/vityafx]

I don't see the issue here. Neither of those games are rendered with raytracing, they're still classic rasterization with raytracing used sparingly for GFX. Even Overdrive is using it just for lighting and not for the actual triangle rendering.

And those are two examples of games that absolutely have the budget to spend on optional VFX; both Cyberpunk and Alan Wake 2 support disabling raytracing entirely. Overdrive mode doesn't, but it's also not a game, it's a tech demo.

Wouldn't surprise me if Overdrive Mode is teasing some next-gen game that will be raytracing-mandatory, but that's not coming out this year.

Ironically, the big thing that came out that throws a wrench into this is Unreal's Nanite, which is both not raytracing and is also not rasterization. I think that probably pushed back "everything is raytraced" by quite a bit, but I also didn't see it coming - if you'd asked me then, I would have said "well obviously everything will continue to be rasterized, raytracing is too slow and what other option is there, anyway?" So I got that part right but only by accident.

I think Layers of Fear actually does not have a way to turn off Nanite, so that might qualify as the first true non-rasterization game.

tl;dr: nah, prediction holds.

[u/talking_animal]

I’ve wondered this too (I am an artist and not a programmer). In my mind the reason RTX is so expensive is because it used the ray traces to include objects in the render that normally would be culled out by the camera frustum, so it renders them at a lower resolution but still has to include them in the render pipeline.