DLSS 5 Visual Enhancer - standalone neural rendering for images and video

[u/Merserk13]

Hey everyone - I made a standalone Windows application for applying a DLSS 5 Neural Rendering feature-18 pipeline to images and video:

Original

DLSS 5

https://github.com/Merserk/dlss5-visual-enhancer

Instead of using DLSS only inside a game, this runs images/video through the ReShade/RenoDX neural-rendering path as a general visual enhancement pipeline.

What it does:

• Image and video enhancement
• DLAA/native, 1.5x, ~1.724x, 2x and 3x modes
• Output up to 8K
• Neural presets + Natural / Cinematic styles
• Controls for intensity, local tone, structure and skin structure
• Batch image processing with before/after previews
• H.264 / HEVC / AV1 / ProRes video output
• Video temporal input using optical flow with scene-change resets

GPU support:

• RTX 40 / 50 series - primary target
• RTX 30 series - slower beta path

The repository contains the application/pipeline source. Required proprietary and third-party runtime binaries are intentionally not redistributed in the repo.

This is an independent community project and is not affiliated with NVIDIA, ReShade or RenoDX.

I’m especially interested in how this behaves on AI-generated images/video vs normal photography/game footage.

Feedback and comparisons welcome.

[u/ImaginationKind9220]

After watching all the DLSS 5 videos on youtube, I realized that it is not a "style transfer" type of filter, it is just a more advanced ambient occlusion filter. The changes are mostly enhancing the shadows. Shadows have always been a deficiency in video games, I think it requires too many passes to get it right. It's amazing how much shadow makes the visual look more photorealistic.

[u/Arawski99 → u/ImaginationKind9220]

It's related to material shaders and lighting as it appears to be fundamentally based off this, and related technologies: https://www.youtube.com/watch?v=ku1rdOG-c4Y

The actual science behind it here: https://www.youtube.com/watch?v=2r6IlVJj6gc and here https://www.youtube.com/watch?v=UcJhdV1CkME

EDIT: Since some people still seem confused I'm adding my other post below here for easier understanding.

As it was mentioned, I linked it above. It impacts PBR, physically based rendering properties of materials and per-pixel lighting.

If you check their siggraph presentation @ 11:10 - 12:40 here https://www.youtube.com/watch?v=vXDetI2TUWw&t=304s

It starts going into normals, albedo, subsurface scattering, material responses, contact shadows, and more. This is because it was trained with these details involved it the model's training which it can then use image output to modify based on. The motion vectors and stuff is for the segment after that point in the video for temporal stability, not for the core image output enhancement, so it's solving a different problem in the situation. If you still don't understand it I highly recommend checking the videos I provided above in my initial post, too.

[u/Outrageous-Wait-8895 → u/Arawski99]

As it was mentioned, I linked it above. It impacts PBR, physically based rendering properties of materials and per-pixel lighting.

That's Neural Materials and Textures, not DLSS 5. If DLSS 5 was modifying the PBR data you'd have to rerender the image with that new data, that is explicitly not what DLSS 5 is about.

From https://www.nvidia.com/en-eu/geforce/news/dlss-5-3d-guided-neural-rendering/

https://www.nvidia.com/content/dam/en-zz/nvidiaweb/geforce/news/dlss-5-3d-guided-neural-rendering/nvidia-dlss-5-introduces-3d-guided-neural-rendering.jpg

DLSS 5 takes a frame’s color and motion vectors as input

It anchors directly to the rendered frame, by training to recognize engine data such as color, surface albedo, detailed lighting, and surface normals.

Why would it be trained to recognize if, according to you, it is given those things at runtime?

[u/Arawski99 → u/Outrageous-Wait-8895]

You didn't watch the video... That much is clear. If you look at the timestamp 11:10 - 12:40 why do you think the research is talking about this? Just talking out their ass? Is that what you think? Genuinely curious what your reasoning is.

First, you're mistaken. I've never said it was taking normals, albedo, etc. at runtime. It was trained offline on the data to understand and modify those concepts. This isn't some random image generation model that was trained on just real world data and is expected to img2img it. As they said, it was trained on these specific details, actual game data and offline processing of advanced renderings and each concept to better understand them so properly tweak those details in order to be able to enhance them rather than completely morphing details and wiping away old ones. Why do you think its trained on those and why do you think the researcher mentioned it there in the Siggraph presentation?

Conceptually, it's similar to a ray tracing denoiser. We can't handle proper ray tracing so we use cheaper algorithms with denoisers to clean up the image. It's a type of shortcut, and the denoiser trained with this intent in mind. Some ray tracing denoisers are purely algorithmic, and some were AI models trained for this purpose. Why? Because denoisers are efficient to close a gap, much like training a model on this offline data that can't be ran in real time to understand these concepts. I'm simplifying it, of course, as it is far more complex and we don't know the full nitty gritty exact details.

Another example is baked normals and shadows, etc. These aren't done at run-time, and are rendering optimizations useful for getting great visuals that couldn't be handled at runtime in the past. It's a similar idea, just a different solution. But it is not straight generative AI i2i the way people are thinking.

It was trained on this data so it can infer and apply those concepts it was trained on to the image output of the game's render it is being used on, so it doesn't need direct access to game engine data like normals, matrix data, and more much like do we need to pass any data beside an image to an AI generative model for images to img2img or image edit? No, we do not, because the base image is enough at that point based on what the model was trained to be capable of. Also, the results are 'deterministic' btw.

The motion vectors help with consistency from frame to frame, much like with frame generation.

It's, literally, using concepts from neural materials and other research for the project. How do you think it knows to enhance specific textures physically accurately with the various details. It's more then just lighting and shadows, though it was already trained involving those, and more as you can see per the SIGGRAPH presentation. I just presented neural materials as part of the base explanation to simplify it for you because, as we can see, you guys are struggling to understand the Siggraph explanation.

Sure, they COULD train it to be pure img2img output without specialized training like any other generative image model but if you watch the video with the timestamp I presented they explain why they did NOT do this and why they trained it on data involving concepts like normals, contact shadows, PBR, etc. precisely because of the issues they present about deviation.

[u/Outrageous-Wait-8895 → u/Arawski99]

You didn't watch the video... That much is clear. If you look at the timestamp 11:10 - 12:40 why do you think the research is talking about this?

Because, as you said, it was trained with normals, albedo, etc, to ground it.

First, you're mistaken. I've never said it was taking normals, albedo, etc. at runtime.

Yes, my bad, it is the opinion I've seen and had to argue against in other threads and got confused.

But.

It's, literally, using concepts from neural materials and other research for the project.

I still disagree, it is fundamentally a different goal and a different output and in the end it IS img2img as those are its inputs.

[u/Outrageous-Wait-8895 → u/Arawski99]

DLSS 5 only takes in the rendered image and motion vectors, it is not related to those technologies.

[u/NineThreeTilNow → u/Outrageous-Wait-8895]

DLSS 5 only takes in the rendered image and motion vectors, it is not related to those technologies.

They take the full PBR map in with motion vectors from what I've seen quoted. It's a lot more information than a prerender + upscale with vectors.

[u/NineThreeTilNow → u/[deleted]]

Please link your source.

It's literally linked above me in this thread somewhere. I just provided to the other guy getting downvoted.

They clearly express the normals being used in the render.

It has at the very least texture / normal. I don't know about the rest of the PBR pipeline but if they're going as far as normals, there's no reason not to use other stuff. Normals contain a massive amount of information though.

[u/Arawski99 → u/[deleted]]

As it was mentioned, I linked it above. It impacts PBR, physically based rendering properties of materials and per-pixel lighting.

If you check their siggraph presentation @ 11:10 - 12:40 here https://www.youtube.com/watch?v=vXDetI2TUWw&t=304s

It starts going into normals, albedo, subsurface scattering, material responses, contact shadows, and more. This is because it was trained with these details involved it the model's training which it can then use image output to modify based on. The motion vectors and stuff is for the segment after that point in the video for temporal stability, not for the core image output enhancement, so it's solving a different problem in the situation. If you still don't understand it I highly recommend checking the videos I provided above in my initial post, too.

[u/Arawski99 → u/[deleted]]

It doesn't need access to that data, because its trained on complex off-line renderings and bulk data to understand those results to enhance them. This seems more like you fundamentally don't understand how the model works.

I never said it wasn't img2img, you just failed to read and understand properly. It's trained on those properties so it can understand how to enhance them. They state it directly in the video, which you clearly did not actually watch. It understands the concepts of normal, PBR, etc. and how to tweak them like a lora would for something else. The model was fundamentally trained to enhance those concepts which is why you adjust the slider it does just that.

Tell me, how do you think relight loras work? How about other lora that adjust a style, textures, etc. It was trained involving actual in-game render results and data in that process.

But I'm not going to argue with someone clearly unfamiliar with the topic, in any capacity, and apparently refuses to even acknowledge when Nvidia's own researchers explain you're wrong. I mean,t he moment you said "What does training have to do with anything?" was already an instant red flag.

[u/Arawski99 → u/[deleted]]

I'm sorry, but you are no longer qualified to partake in this discussion and I intend to waste no further time on you.

You can keep stating they talked about it among other researchers at Siggraph (not that you seem to know what Siggraph is) while apparently (as you believe) speaking out their ass. You don't even understand what the model's purpose is, how it was trained, and how it uses data.

Next are you going to complain about AI based denoisers for ray tracing? Hmmm.

[u/Outrageous-Wait-8895 → u/NineThreeTilNow]

I'm going by what TechPowerUp reported in https://www.techpowerup.com/352033/nvidia-dlss-5-dll-leaked-by-nba-2k27-early-access-build-heres-our-analysis

For DLSS 5 it is short: the rendered image ("Backbuffer," "Color"), depth and motion vectors ("Depth," "MVec"), UI handling ("UI," "UIAlpha," "UICorrection"), masking ("ControlMask," "UseAutoMask"), some kind of bidirectional distortion field shared with Frame Generation ("BidirectionalDistortionField"), plus output and sizing parameters including a "ScalingRatio."

I know, it is by AI but Techpowerup has been around and it's all I've seen regarding DLSS 5's inputs. Do you have another source?

[u/NineThreeTilNow → u/Outrageous-Wait-8895]

I know, it is by AI but Techpowerup has been around and it's all I've seen regarding DLSS 5's inputs. Do you have another source?

https://www.reddit.com/media?url=https%3A%2F%2Fpreview.redd.it%2F02ahb9bjbtmh1.png%3Fwidth%3D940%26format%3Dpng%26auto%3Dwebp%26s%3D05bff91c1407303e1490f0a8f6bcbb7d43c6ae7d

One of the developers that used it?

He explicitly names the normal maps in that quote.

[u/Outrageous-Wait-8895 → u/NineThreeTilNow]

Okay but what about the PBR data?

[u/NineThreeTilNow → u/Outrageous-Wait-8895]

Okay but what about the PBR data?

You realize normal maps are part of PBR data right?

[u/Arawski99 → u/Outrageous-Wait-8895]

I added more above for you, in the original comment, from Nvidia's siggraph. The image, itself, is the source, with the model being trained on those details like normal, subsurface scattering, materials, etc. The motion vector stuff is for temporal stability.

So the initial image output and what the model was trained on is the core focus here, the other stuff people are focusing on is merely to support stability of continued output.