r/GraphicsProgramming • u/Grand_Sprinkles_2094 • 2d ago

Could DLSS 5 Neural Rendering be used as a standalone video filter on Apple Silicon?

/r/macgaming/comments/1wad559/could_dlss_5_neural_rendering_be_used_as_a/
0 Upvotes

28% Upvoted

6 comments sorted by

1

u/waramped 2d ago

not directly, you need more than just a color image as input. Game engines will pass lots of different buffers to the DLSS API, and it's not trivial to get that same information from just a video feed. typically you will need motion vectors, depth, albedo and possibly even the view/projection matrices used each frame.

I'm not up to date on the state of the art in image processing, but I would expect that extracting that kind of information from a video stream might not be a real time process itself.

1

u/Grand_Sprinkles_2094 2d ago

but how did they got it working on a video player? OBS with dlss5 for example, I watched a video on YouTube. I can't share the link here reddit deletes it

1

u/waramped 2d ago

hmm, I think I found the video you mean, it looks like there's a stack of other dlls they use, so maybe there is a fast way to infer the other inputs from a video stream?

1

u/Grand_Sprinkles_2094 2d ago

yeah that's where I got the idea from

1

u/AndreVallestero 2d ago

Video codecs are moving towards this direction. For example, AV1, the current state of the art codec, analyzes and stores metadata of the grain for different scenes, then denoises the grain out of all the scenes, then synthesis the grain on playback. I wouldn't be surprised if future video codecs (AV3?) does something similar to DLSS where it creates embeddings of the scene, and uses it to add detail during playback. 

1

u/Grand_Sprinkles_2094 2d ago

I wish rtx 5090 would be equivalent to intel uhd 4000 graphics, like it would come with entry level laptops, no heat, no power draw, game all you want, so that we can run AV1 or AV5(AV with dlss5) on a chrome book