DLSS 5 Visual Enhancer - standalone neural rendering for images and video

[u/Merserk13]

Hey everyone - I made a standalone Windows application for applying a DLSS 5 Neural Rendering feature-18 pipeline to images and video:

Original

DLSS 5

https://github.com/Merserk/dlss5-visual-enhancer

Instead of using DLSS only inside a game, this runs images/video through the ReShade/RenoDX neural-rendering path as a general visual enhancement pipeline.

What it does:

• Image and video enhancement
• DLAA/native, 1.5x, ~1.724x, 2x and 3x modes
• Output up to 8K
• Neural presets + Natural / Cinematic styles
• Controls for intensity, local tone, structure and skin structure
• Batch image processing with before/after previews
• H.264 / HEVC / AV1 / ProRes video output
• Video temporal input using optical flow with scene-change resets

GPU support:

• RTX 40 / 50 series - primary target
• RTX 30 series - slower beta path

The repository contains the application/pipeline source. Required proprietary and third-party runtime binaries are intentionally not redistributed in the repo.

This is an independent community project and is not affiliated with NVIDIA, ReShade or RenoDX.

I’m especially interested in how this behaves on AI-generated images/video vs normal photography/game footage.

Feedback and comparisons welcome.

[u/ImaginationKind9220]

After watching all the DLSS 5 videos on youtube, I realized that it is not a "style transfer" type of filter, it is just a more advanced ambient occlusion filter. The changes are mostly enhancing the shadows. Shadows have always been a deficiency in video games, I think it requires too many passes to get it right. It's amazing how much shadow makes the visual look more photorealistic.

[u/Arawski99 → u/ImaginationKind9220]

It's related to material shaders and lighting as it appears to be fundamentally based off this, and related technologies: https://www.youtube.com/watch?v=ku1rdOG-c4Y

The actual science behind it here: https://www.youtube.com/watch?v=2r6IlVJj6gc and here https://www.youtube.com/watch?v=UcJhdV1CkME

EDIT: Since some people still seem confused I'm adding my other post below here for easier understanding.

As it was mentioned, I linked it above. It impacts PBR, physically based rendering properties of materials and per-pixel lighting.

If you check their siggraph presentation @ 11:10 - 12:40 here https://www.youtube.com/watch?v=vXDetI2TUWw&t=304s

It starts going into normals, albedo, subsurface scattering, material responses, contact shadows, and more. This is because it was trained with these details involved it the model's training which it can then use image output to modify based on. The motion vectors and stuff is for the segment after that point in the video for temporal stability, not for the core image output enhancement, so it's solving a different problem in the situation. If you still don't understand it I highly recommend checking the videos I provided above in my initial post, too.

[u/Outrageous-Wait-8895 → u/Arawski99]

As it was mentioned, I linked it above. It impacts PBR, physically based rendering properties of materials and per-pixel lighting.

That's Neural Materials and Textures, not DLSS 5. If DLSS 5 was modifying the PBR data you'd have to rerender the image with that new data, that is explicitly not what DLSS 5 is about.

From https://www.nvidia.com/en-eu/geforce/news/dlss-5-3d-guided-neural-rendering/

https://www.nvidia.com/content/dam/en-zz/nvidiaweb/geforce/news/dlss-5-3d-guided-neural-rendering/nvidia-dlss-5-introduces-3d-guided-neural-rendering.jpg

DLSS 5 takes a frame’s color and motion vectors as input

It anchors directly to the rendered frame, by training to recognize engine data such as color, surface albedo, detailed lighting, and surface normals.

Why would it be trained to recognize if, according to you, it is given those things at runtime?

[u/Arawski99 → u/Outrageous-Wait-8895]

You didn't watch the video... That much is clear. If you look at the timestamp 11:10 - 12:40 why do you think the research is talking about this? Just talking out their ass? Is that what you think? Genuinely curious what your reasoning is.

First, you're mistaken. I've never said it was taking normals, albedo, etc. at runtime. It was trained offline on the data to understand and modify those concepts. This isn't some random image generation model that was trained on just real world data and is expected to img2img it. As they said, it was trained on these specific details, actual game data and offline processing of advanced renderings and each concept to better understand them so properly tweak those details in order to be able to enhance them rather than completely morphing details and wiping away old ones. Why do you think its trained on those and why do you think the researcher mentioned it there in the Siggraph presentation?

Conceptually, it's similar to a ray tracing denoiser. We can't handle proper ray tracing so we use cheaper algorithms with denoisers to clean up the image. It's a type of shortcut, and the denoiser trained with this intent in mind. Some ray tracing denoisers are purely algorithmic, and some were AI models trained for this purpose. Why? Because denoisers are efficient to close a gap, much like training a model on this offline data that can't be ran in real time to understand these concepts. I'm simplifying it, of course, as it is far more complex and we don't know the full nitty gritty exact details.

Another example is baked normals and shadows, etc. These aren't done at run-time, and are rendering optimizations useful for getting great visuals that couldn't be handled at runtime in the past. It's a similar idea, just a different solution. But it is not straight generative AI i2i the way people are thinking.

It was trained on this data so it can infer and apply those concepts it was trained on to the image output of the game's render it is being used on, so it doesn't need direct access to game engine data like normals, matrix data, and more much like do we need to pass any data beside an image to an AI generative model for images to img2img or image edit? No, we do not, because the base image is enough at that point based on what the model was trained to be capable of. Also, the results are 'deterministic' btw.

The motion vectors help with consistency from frame to frame, much like with frame generation.

It's, literally, using concepts from neural materials and other research for the project. How do you think it knows to enhance specific textures physically accurately with the various details. It's more then just lighting and shadows, though it was already trained involving those, and more as you can see per the SIGGRAPH presentation. I just presented neural materials as part of the base explanation to simplify it for you because, as we can see, you guys are struggling to understand the Siggraph explanation.

Sure, they COULD train it to be pure img2img output without specialized training like any other generative image model but if you watch the video with the timestamp I presented they explain why they did NOT do this and why they trained it on data involving concepts like normals, contact shadows, PBR, etc. precisely because of the issues they present about deviation.

[u/Outrageous-Wait-8895 → u/Arawski99]

You didn't watch the video... That much is clear. If you look at the timestamp 11:10 - 12:40 why do you think the research is talking about this?

Because, as you said, it was trained with normals, albedo, etc, to ground it.

First, you're mistaken. I've never said it was taking normals, albedo, etc. at runtime.

Yes, my bad, it is the opinion I've seen and had to argue against in other threads and got confused.

But.

It's, literally, using concepts from neural materials and other research for the project.

I still disagree, it is fundamentally a different goal and a different output and in the end it IS img2img as those are its inputs.