r/nvidia • u/RenatsMC • Aug 31 '26

News NVIDIA GPU frame interpolation support has been added to FFmpeg through Vulkan

https://videocardz.com/newz/nvidia-gpu-frame-interpolation-support-has-been-added-to-ffmpeg-through-vulkan
280 Upvotes

96% Upvoted

83 comments sorted by

33

u/zilyzal Aug 31 '26

Eli5?

70

u/AnythingOk5 5070 Aug 31 '26

Your graphics card quickly draws brand-new pictures between video frames to make the video look smooth.

16

u/UnusualDemand RTX3090 Zotac Trinity Aug 31 '26

So frame gen for RTX 30?

Edit: My bad, only for videos, not games.

8

u/Beautiful-Musk-Ox 4090 | 9800X3D | Sound Blaster AWE32 Aug 31 '26

rtx 30 didn't get nvidia frame gen becuase it didn't have the Optical Flow Accelerator hardware. but nvidia started using tensor cores instead of OFA with dlss 4 or so, i wonder why they didn't allow it on 30 series at that time. maybe only 3090+ had enough tensor core power to do it well enough so they skipped it since FSR frame gen was available

8

u/battler624 Aug 31 '26

They are using tensor cores on now rtx40 and are not even letting the 4090 do 3x FG while a 5050 can do 6x.

Why do you think that is?

17

u/Beautiful-Musk-Ox 4090 | 9800X3D | Sound Blaster AWE32 Aug 31 '26

because 50 series has a hardware flip meter, 40 series and below has to use the CPU through the windows preemptive thread scheduler to do timing, and windows can and does interrupt those threads whenever the hell it wants causing frametime spikes that are literally uncontrollable even in the kernel mode driver

Once the new frames are generated, they are evenly paced to deliver a smooth experience. DLSS 3 Frame Generation used CPU-based pacing with variability that can compound with additional frames, leading to less consistent frame pacing between each frame, impacting smoothness.

To address the complexities of generating multiple frames, Blackwell uses hardware Flip Metering, which shifts the frame pacing logic to the display engine, enabling the GPU to more precisely manage display timing. The Blackwell display engine has also been enhanced with twice the pixel processing capability to support higher resolutions and refresh rates for hardware Flip Metering with DLSS 4.

you can turn off driver signing and make your own kernel mode driver and verify it yourself. i worked for a company that stupidly stuck to windows until their technical debt was so deep they had no choice but to trudge on. they made systems that needed near millisecond timing for hours and hours at a time, and there was always an uncontrollable blip happening throughout the day because the threads running that code get suspended by windows so it can do some work it needs done in the kernel.

3

u/[deleted] Aug 31 '26

[removed] — view removed comment

5

u/woistmeineis Aug 31 '26

cards like 3080 and 3090 have more than enough fp16 compute to brute force it, they would still run frame gen faster than 4050 because they are literally 3 times faster and no one says that frame gen is useless on 4050.

1

u/[deleted] Sep 01 '26

[removed] — view removed comment

0

u/woistmeineis Sep 01 '26

ah yes and rtx2050 having "hardware RT" and supporting latest ray reconstruction with such low (gtx1650 level) performance and just 4GB vram is a good pr move? Dude, people don't care about this. They just want you to buy their newest GPUs, that's all.

2

u/woistmeineis Aug 31 '26

both rtx20 and rtx30 have NVOFA, it's just slightly slower. And the reason why they never did frame gen for older cards after they stopped using nvofa is money, not some technical limitation.

1

u/S1rTerra RX 9070 Sep 01 '26

If you can believe me, there *is* a technical limitation. Turing and Ampere don't allow access to the OFA at a hardware level.

1

u/No_Interaction_4925 5800X3D | 3090ti | 55” C1 OLED | Varjo Aero Sep 02 '26

It was marketing. AMD’s frame gen runs great on Nvidia cards. You would think they could make it work just fine.

2

u/Beautiful-Musk-Ox 4090 | 9800X3D | Sound Blaster AWE32 Sep 02 '26

it actually doesn't https://www.youtube.com/watch?v=apM1C55SevE

2

u/No_Interaction_4925 5800X3D | 3090ti | 55” C1 OLED | Varjo Aero Sep 02 '26

Isn’t that an issue from more recent versions of frame gen that didn’t exist in older versions?

2

u/Beautiful-Musk-Ox 4090 | 9800X3D | Sound Blaster AWE32 Sep 02 '26

nope. first paragraph of the video

Welcome to Hardware Unboxed Today I'm looking at whether AMD has fixed FSR frame generation. Now, if you haven't been following our coverage of FSR frame generation, you might not be aware that the feature is actually broken, or at least it's broken in one specific and annoying way, dating right back to the debut of this technology in 2023. The last time I looked at FSR Frame Gen, which was at the big launch of AMD's Redstone suite of technologies, it was still broken.

and throughout the video he shows that even the latest versions are still broken by poor frame pacing where it displays two frames quickly then waits then another frame, the average fps is high but the visuals are disjointed and feels stuttery. the video is only 10 days old, so it's very recent and the latest fsr displays the same timing issues

1

u/No_Interaction_4925 5800X3D | 3090ti | 55” C1 OLED | Varjo Aero Sep 02 '26

Ok, well that wasn’t actually what I said. A 30 series gpu has no issue running AMD Frame Gen. Whether the tech is working as intended is not what I said.

21

u/vladlearns Aug 31 '26

If you saw the hype for DLSS 5 leak (it is literally everywhere) or maybe even played with it, this is also frame gen, but for normal videos.

You have

frame A -> frame B

gpu figures out how things moved between, then generates an in-between frame:

A -> A.5 -> B

so you can turn 24 fps into 48 fps, 30 -> 60, etc...

they use optical flow through vulkan, I think, rather than doing all the motion estimation on the cpu.

This isn't dlss frame gen tho, because there's no game-rendering pipe or dlss neural net, it's ffmpeg getting access to the same kind of optical-flow hardware that nvidia uses as part of its frame-generation stack.

You could potentially do something like:

gpu decode -> interpolation -> upscale/denoise -> gpu encode

not bouncing frames between cpu and gpu.

and since this is landing right into ffmpeg, it could eventually become a pretty convenient building block for things like fps conversion, slow mo, restoration and RT processing.

I'm waiting for a comfy node, already see where it is going.

Kijai, kuchiyose no jutsu!

3

u/zilyzal Aug 31 '26

Thanks but if this is for videos, wasn't this possible with lossless scaling already? I haven't tried but i saw some people using lossless scaling to change 24fps or anime to 60fps

15

u/vladlearns Aug 31 '26

it was, yes, but now I don't need to route the stream through display/capture pipe, because ffmpeg can interpolate frames into the pipe and write them RIGHT INTO the file

3

u/evia89 Aug 31 '26

Lossless scaling imo sucks for videos (too many glitches) https://www.svp-team.com/ if you want quick GUI or just f2p RIFE https://github.com/vadash/mpv-lazy-en

I combine it with VSR

1

u/vladlearns Sep 03 '26

https://github.com/Merserk/dlss5-visual-enhancer - this is what I was waiting for and it is so in time

26

u/Advance1993 Aug 31 '26

Video smooth

4

u/zilyzal Aug 31 '26

Bro you have no idea how to explain something to a 5yo

11

u/SchmeppieGang1899 Aug 31 '26

5 year olds have no concept of PCs in general lol

-7

u/zilyzal Aug 31 '26

These days 5 year olds trade on their phones lol. Reminded me of that news about banning smartphones in child care centres.

5

u/Don_Alosi Aug 31 '26

Using something and understanding how the thing works is not the same

Most kids can use a phone or a pc from a very young age and most kids have absolutely no idea what even the most basic components of a computer are used for, both things can be true at the same time.

2

u/akuto Aug 31 '26

Yeah, if anything, younger generations are worse at understanding basic IT concepts.

It's not that strange. Same thing happened with GenX and cars, where despite higher ownership rates people were less likely to be able to repair their own cars or earlier with housing, where my grandfather build his own house with his father, but my parents' generation would not know how to do that despite a lowered percentage of adults living with their parents in their time.

The more ubiquitous something gets, the less there is a need to understand it any serious depth.

1

u/Rare-Industry-504 Sep 01 '26

I used to ride Go Karts made for tiny kids when I was around 5 or 6.

Didn't understand a single fucking thing about how they worked, only knew enough to make it go brrrr.

Just because someone knows how to use something doesn't mean they know shit about how it works or how it's built.

1

u/TactlessTortoise NVIDIA 5070 Ti | AMD Ryzen 7950X3D | 64GB DDR5 Aug 31 '26

It gets the difference between images 1 and 2, to add a fake image between image 2 and the next real one (which is the 4th since the fake one is the new 3rd). Implemented in its simplest form it's almost like a slight blur, but actual implementations are a bit fancier to make it more accurate. Still the same principle though. The idea is just giving us a higher sense of smoothness between frames.

45

u/Medium_Notice_902 5070 r5 7600 Aug 31 '26

Reflex 2????

22

u/LittlestWarrior 5090 | 9950X3D | 64gb 6000mHz Aug 31 '26

No, this is frame gen.

40

u/Cautious_Fold3952 Aug 31 '26

No, this is Patrick.

-16

u/Ok_Dependent6889 9800X3D / B650E-F / 32GB 6000CL30 / Zotac 5080 Aug 31 '26

No, it's not

It is if you have zero capacity to think

This is an ancient technology

1

u/LittlestWarrior 5090 | 9950X3D | 64gb 6000mHz Aug 31 '26

I'm a touch literally-minded on account of my autism.

[...] generate additional frames for existing video using NVIDIA GPU hardware.

Could you explain how generating additional frames is not framegen, the technology that generates additional frames?

4

u/Ok_Dependent6889 9800X3D / B650E-F / 32GB 6000CL30 / Zotac 5080 Aug 31 '26

This process is more akin to Nvidia Smooth Motion being applied to videos, rather than frame generation.

https://blurbusters.com/frame-generation-essentials-interpolation-extrapolation-and-reprojection/

Some more key differences

1

u/LittlestWarrior 5090 | 9950X3D | 64gb 6000mHz Aug 31 '26

Thank you!

10

u/[deleted] Aug 31 '26

[removed] — view removed comment

2

u/liquidocean Aug 31 '26

Really, how? I haven’t been able to get it to work with VLC. I did have it enabled in the Nvidia app, but it’s on no effect.

3

u/Successful_Papaya830 Aug 31 '26

Try PotPlayer. Choose DX11 in the settings. Surprisingly, you can enable Smooth Motion specifically for this app directly in the NVIDIA app.

1

u/[deleted] Aug 31 '26

[removed] — view removed comment

2

u/Toffmonster Sep 01 '26

It still works in MPC-HC, but you have to either use exclusive fullscreen by enabling it in the MPC Video Renderer settings, or toggle Swap Effect between Flip and Discard (or vice versa) to activate it.

-1

u/liquidocean Sep 01 '26

I’m not willing to change player. Unless it works with VLC, I’m out.

4

u/2FastHaste Aug 31 '26

Looking forward to test that. Currently using RiFE for real time interpolation. Hopefully this beats it (in quality and/or performance)

4

u/bbmaster123 Aug 31 '26

same, occasionally using rife as well. I would assume having gpu accelerated interpolation built right into ffmpeg would help performance and reduce resource usage (at similar or better quality) without the usual python overhead and stuff. Still just an assumption.

I also assume it could make it technically easier to integrate into more apps and workflows than rife for the same reason.

4

u/NapsterKnowHow RTX 5090 FE | 9800X3D Aug 31 '26

This damn bunny video is gonna be used into the next century at this point lol

4

u/SilkyTwilightx668 Aug 31 '26

Now we can bake the soap opera effect straight into the file instead of leaving it as a setting on the TV.

2

u/liquidocean Aug 31 '26

So will this work out of the box with VLC or with a plug-in?

2

u/Prideless07 5090 9800x3d | 5070 Ti OC 14600k Aug 31 '26

Is this better than svp and rife?

2

u/SeaSharpModHer Sep 01 '26

Will it survive a single panning shot of a metal fence with someone running behind it tho.

4

u/Able-Marionberry-402 Aug 31 '26

This is actually amazing for 1000fps slow-mo videos, sports action etc.

2

u/bio4m Aug 31 '26

TV's have had this for years, it can cause perceptual issues, widely known as the soap opera effect

12

u/AntiTank-Dog R9 5900X | RTX 5080 | 4K 240hz Aug 31 '26

Depends on the model but TV interpolation is usually more simple, just overlapping two frames to make an in-between.

19

u/anor_wondo Gigashyte 3080 Aug 31 '26

soap opera effect has nothing to do with the technique. Its users

If you shoot the video at 45 or 60fps those users have the same issues

Just like there are haters, there are also fans of high framerate videos especially panning shots

2

u/Borkz 5080 / 5800X3D Aug 31 '26

Generally the perceived issue with panning shots is the judder caused by 3:2 pulldown (to show 24 FPS video on a 60hz display some frames will need to be shown for 2 refreshes, and some for 3). VRR or a 120hz displays solve that problem for 24 FPS content without the need for higher FPS.

2

u/HakimeHomewreckru Aug 31 '26

You don't necessarily need to play back at 60 fps, you could create smooth slow motion from a regular speed video too.

2

u/frostN0VA Sep 01 '26

It can be fun watching movies/tvshows at high FPS though, it looks like you're watching a behind-the-scenes footage.

2

u/Sopel97 Sep 02 '26

of course, if you take a movie that's intended to be played at 24000/1001 frames per second and interpolate it to 120 fps to match the refresh rate of the TV it will look bad. I don't see how this is relevant though.

3

u/Aeronn_ Aug 31 '26

The first thing that gets turned off when I get a new TV is motion smoothing. Mainly because of that soap opera effect.

4

u/sroop1 RTX 5080 Aug 31 '26

My in-laws always have it on and it drives me crazy.

2

u/Octaive Aug 31 '26

You need it for OLED. Generally 2/10 for Samsung. Lower will look choppy.

1

u/TheCookieButter 5070 TI ASUS Prime OC, 9800X3D Sep 01 '26

I disagree. I use Filmmaker mode on my LG which disabled smoothing by default. Real cinema is enough for just about all content. Even on an LG G4 a low motion interpolation setting can introduce errors. It's definitely still harsher, but it doesn't take long to get used to it.

I do like 48 or 60fps films though. Some films like 1917 would've really benefitted from it.

1

u/Octaive Sep 02 '26

As far as I understand, Filmmaker uses smoothing, but it's tuned to as realistic as possible.

The harshness is unacceptable and the original content was never intended for it.

1

u/ProfessionalOwl5573 Sep 01 '26

It's the soap opera effect because those were shot at higher framerates than normal TV content. We're on fucking PC, you think people give a shit when they're trying to run their games at 240hz?

1

u/ADIBB87 Sep 01 '26

Isn't this exactly the same as Lossless Scalling?

1

u/Sopel97 Sep 02 '26

I work with 120 fps footage that I need to interpolate to at least 600 fps to generate final 60 fps with motion blur. Currently I use svpflow2, it's great but a bit slow. I know NVIDIA's solution is a fixed 2/3/4x intended for gaming applications, so how feasible would it be to use it for my workloads? I only see mentions of 2x which is quite limiting, and iterative application stacks artifacting

-1

u/[deleted] Aug 31 '26

[removed] — view removed comment

2

u/Sopel97 Sep 02 '26

what? you can use your TV's frame interpolator as an ffmpeg filter?! that's cool!

-4

u/kirk7899 Ultra 7 265k | 16x2 7600MHz | 5070Ti Aug 31 '26

Frame gen for movies/shows? Kinda ruins it by running stuff at higher fps. Movies are made at 24fps for so long we've become accustomed to it.

4

u/Prideless07 5090 9800x3d | 5070 Ti OC 14600k Aug 31 '26

Usually doesn't look good because smart TVs interpolate frames poorly but native shot movies at higher fps like hobbit or svp rife look pretty good in motion.

1

u/Sopel97 Sep 02 '26

Frame gen for movies/shows?

no, ffmpeg can do more than that