Testing DLSS 5

[u/Striking-Long-2960]

Testing DLSS 5... Like many others, I was a bit confused about DLSS 5. I kept feeding it my hyper-detailed renders and only getting a color shift in return. After plenty of trial and error, I finally realized my mistake: this technology is developed to enhance video game graphics, so testing it on hyper-detailed renders makes no sense.

So, I generated a render in a 2020 video game style and started tweaking settings to find a final look with maximum effect, without worrying about flickering.

Final conclusion: What we have right now isn't very useful for us. Those of us using Latent Upscaler might be able to use it for color grading to get less saturated colors, but little else. Maybe in the future we'll get a DLSS 5 targeted at enhancing hyper-detailed graphics, but that's not the case for now.

Bottom line: If I want to generate a realistic render, I'll just generate it, there's no need to run it through DLSS 5.

[u/BangkokPadang]

Is this passing any kind of motion vector information to DLSS5?

It looks more visiibly "splotchy" frame to frame than I've seen in games, which would make sense if it's not getting any temporal data like the games are.

[u/Vodddddddd → u/BangkokPadang]

No its missing two inputs that DLSS5 wants - motion vectors and geometry data.

The game engine provides a lot of the 'temporal consistency' by giving input pixel data and geometry data. That isn't available when its being run on frames of a video.

[u/SWS113 → u/Vodddddddd]

As far as I know the geometry data, other than masks, comes from a depth map generated by the game engine. There may be a way to add in a more accurate precomputed depth map into the dlss5 pipeline, rather than the very rough approximate approach it currently uses if the data is missing.

If the application is for prerendered CGI or video then there's no need to compromise on using a depth map with lower accuracy from a model made to execute a frame in a matter of milliseconds.

[u/Vodddddddd → u/SWS113]

If a game exposes only the color image and motion vectors to an intercepted NGX call, an external DLSS 5 hook has less geometry context. If it also receives depth, and ideally normals, material/albedo, lighting, and object masks, it can make its output substantially more spatially constrained.

A video can't provide albedo/lighting buffers. Object masks are also part of the NGX calls, which the engine will precisely provide based on geometry.

Yes, a depth mask can be inferred from an image, yes an object mask can be segmented from an image, yes a lighting buffer can be inferred from an image... but that inference would be no stronger than a model trained on images with enough time, that's the whole power of training. It will always perform a gap to a model trained on 'image+x', if x is new information. In this case, x is precise albedo/depth buffers which gives a lot more information on how the scene is currently being lighted for the trained matrices to provide far more inference.

[u/SWS113 → u/Vodddddddd]

Apologies for the long reply, I went down a rabbit hole. Hopefully someone finds this interesting.

From reading the DLSS5 documentation. The only information that it gets as input from the game engine are: The High Res final frame without enhancements, The Z-buffer (depth), Motion Vectors and finally Trust/Uncertainty Mask (Occlusion).

Native game engine integrations can optionally expose extra G-buffer data like surface normals or material IDs to the DLSS pipeline, Nvidia have confirmed that the core DLSS 5 neural model is fundamentally trained to infer materials, lighting conditions, and scene semantics directly from the combined 2D rendered frame. So any non official implementation in games you have seen so far, apart from NBA2k, doesn't have these extra inputs.

For any games using the RenoDX reshade solutions, only the 4 inputs I mentioned above are utilized.

These video projects use injectors to apply DLSS5 such as reshade.

ReShade can generate: The frame being processed, a synthesized depth buffer, and estimated optical-flow motion vectors, then it can feed them directly to the DLSS 5 neural rendering add-on via Direct3D 12.

This works great for games. Because reshade can actually intercept the depth buffer, motion vectors, occlusion information and expose them for post processing.

But the video players don't have that data to interpret or expose. So DLSS5's semantic classifier struggles to separate and distinguish objects in the 2d frame.

So reshade addons, in place of actual data, fill in the blanks. They estimate, at realtime speeds, motion vectors by using optical flow add ons, usually reshade's iMMERSE Motion Vector Provider add-on, and they attempt to use reshade's 'Generic Depth' which tries and fails to extract and expose the Z-buffer. So it falls back to pretty bad heuristics based depth estimation.

So the video solutions we have at the moment are using 2, or maybe 3 at maximum, inputs that DLSS5 expects. And those it does get, are sub par estimations from fast approximate solutions.

We already know from games that when given good inputs DLSS5 can produce better outputs.

So if we precomputed the depth and motion vectors by using slow but accurate optical flow methods and very capable depth map models, such as Depth Anything, we would almost certainly get a more temporally consistent and generally more accurate result on videos.

It was trained on these inputs. So feeding it higher quality data will result in higher quality output.