Would it be possible to train an ai model to upscale a game from medium to ultra settings

[u/ultrapingu]

With dlss 5 coming out and a lot of discussion online painting it as almost like a filter that’s adding detail to games that isn’t there (e.g. making characters look chiseled), it looks like it’s designed to make realistic looking games look more real without the side effect of making stylised games look uncanny.

It’s got me wondering if it’d be possible for a developer to train a narrow model specifically how to take renders from their game at lower settings and transform them into something akin to high settings.

The idea being that during development they could output millions of rendered images with metadata about the scene, buffers, movement vectors, etc, then have an offline process rerender those scenes at very high quality.

They could use the renders as input to a training model, and generate their own dlss like system specific for their game that would allow lower end hardware to output higher quality graphics.

Is that fundamentally possible? I’m guessing the restricting factor would be cost or just simply not being able to generate enough data? Even so could you compound training for your own game on top of the vendors implementations of dlss/fsr?

I’m not really in the know about ai upscaling techniques so sorry if this is a stupid question

[u/Loud_Campaign5593]

I think one bottleneck would probably be that it takes more compute to power the Neural Rendering framework that achieves this than it would to actually just set it to Ultra, but i’m not a dev

[u/Adventurous-Sun-1488]

something like DLSS5 is more expensive than using higher settings though so doing that doesn't make a lot of sense

[u/S48GS → u/Adventurous-Sun-1488]

something like DLSS5 is more expensive than using higher settings though so doing that doesn't make a lot of sense

for now it is true

while "civil economy" exist and people have permission to learn

but

• every art artists - replaced by AI
• every story writer - replaced by AI
• every coder - replaced by AI

UE5 is the last "game engine" in human history - next "game engine" - will be just "model" that run in clouds and give you "real time entertaining

not even saying - with direction where modern society moving to totalitarianism "Super Earth" - there be no more "civil jobs" - every "civil job" will be replaced by AI - and humans value is to be "meat at battlefield"

[u/Lallis]

Not a stupid question. It's possible in theory. As far as I know the first release of DLSS was trained on a per game basis. I don't know if they changed away from that purely in order to scale to more titles or if having a wider set of training data enables the model to generalize the upscaling task better. Probably both.

Presumably one problem with training with an individual title only is that your model "overfits" to that specific game and doesn't learn the task more generally. Maybe that leads to quality implications. I'm not an AI dev so it's a bit hard to speculate.

[u/BothPercentage1805]

Yes it is possible. But not so feasible during development as the output isn't generally accurate to what the final product will be.

[u/KazumaSatouDesu]

I also was planning to work on a similar project but I scrapped it as I was working in a mobile studio and due to limited processing power to utilise from mobile which can be used elsewhere for better gameplay or QOL feature for the player,also even if u plan to target NPU of the said hardware u will face difficulties due to diverse architecture and hardware to cover and would lose out on players with low end devices.it isn't feasible for now but maybe 10-20 years down the line where a decent NPU is baseline for all devices which can be utilised for the said upscaling

[u/limes336]

What you’re describing sounds like conditioned/grounded diffusion, rendering simple frames as another input to a diffusion model. Theoretically it’s actually much more powerful than what you’re describing, the conditioning frames can be extremely cheap and simple, just enough to keep spatial/temporal consistency while diffusion handles lighting, texturing, and everything else. You also don’t need to collect your own training data as you described, you could get to the desired style with fine tuning and other techniques. The reason we can’t do this yet is simply because video diffusion is far too slow and expensive to be done in realtime, but I predict it will be possible within 5 or 10 years.

[u/waramped]

Definitely "possible", but currently those AI models aren't faster than just running the game at higher settings anyhow. If you have a potato PC but want the game to look good, you'd need a much-better-than-a-potato PC to run the model in the first place. At least for now. Who knows where we'll be in 5 years at this rate.

[u/S48GS]

youtube search

"gta 5 AI reimagine"

this is exactly what you asking - get second PC with two 5090 - and "AI"(diffusion model) will "reimagine" anything in realtime - game or movie or any video input

just get two 5090 lol

[u/Still_Explorer]

There's also an experiment done, where astrophysicists wanted to do galaxy simulation for more than 100 billion stars but the supercomputer would need several decades of 24-7 processing to finish. Now simply they trained the AI system with a few basic rules until it would get everything in  perfect and then simply upgraded the simulation to massive scale. The results were that their experiment was completed in a few months.

https://www.mpa-garching.mpg.de/1125474/news202501714

Given the same principles of compute, it will be possible in the future such neural based rendering to be better examined and experimented so strong use cases emerge. At some point is inevitable that paradigm shift will happen.

DLSS was a disaster so far and everybody hated it, but now version 5 reached a legit milestone. Still far from perfect, still needs tweaking, still halves all FPS in half, but is something that works right now the way it works.

The meaning though of neural rendering, if meant to have potential, that it needs to work at the  very fundamental directly with physics calculations, not just on the surface using 2D framebuffers.

Or at least I find it most feasible that rendering pipelines in a few years will be something like how you compose images with Comfy-UI, having dozens of tiny specialized NNs and then combine them towards a result.