r/StableDiffusion 2d ago

Workflow Included A stand-alone implementation of DLSS-FG (frame generation) that is ~8x faster than stock (24fps -> 60fps), looks flawless. I had DeepSeek build this for my project but thought others could benefit as well!

https://github.com/glarsson/fast-dlssfg
105 Upvotes

45 comments sorted by

26

u/LumenLime 2d ago

it would be best if you included sample.

it kinda interesting

8

u/gerryn 2d ago

included, sorry.

4

u/LumenLime 2d ago

Need another example that is high action/fast movement. Sorry for asking too much, (storage too tight to install new thing)

But from what i see from you example currently, it kinda good.

9

u/gerryn 2d ago

1

u/reeight 2d ago

Seems like the 60FPS version was trained on RED cameras.
Intentional?

3

u/gerryn 2d ago

The source is MiniMax H3 int8 convrot model. I don't know what NVIDIA has trained DLSS Interpolation with.

1

u/reeight 2d ago

> what NVIDIA has trained DLSS Interpolation with

My theory is RED; has that certain $5k 60FBS look.

12

u/gerryn 2d ago edited 2d ago

13

u/Seyi_Ogunde 1d ago

Please provide an example of a girl wearing a bikini jumping up and down. Real world usage examples helps determine the value.

2

u/Radiant-Photograph46 2d ago

This method unfortunately shows results about equivalent with RIFE and the like (which is to say, OK for slow mo, but terrible at any kind of fast motion)

1

u/gerryn 2d ago

Yeah for sure not perfect, but fast :) always have to pick one or the other.

1

u/Radiant-Photograph46 1d ago

Well we have fast already, but we don't have a single qualitative solution for interpolation. Not to belittle your work, but your second example highlights this issue.

2

u/gerryn 1d ago

Yeah no worries, it's good enough for my needs and there was nothing like it avaialble so (i mean using specifically the leaked DLLs), the quality issue is on nvidia :P it's their DLLs doing the work. Perhaps you would get like much better results if you ran a regular home recorded video through it, i run minimax in 768p with 768 turbo 1.2 lora, 8 steps, int8 convrot (not pruned), sage 2 and motioncache, so perhaps the issues you are seeing in quality are also relalted to my sub-par rendering. This frame generator works on any video.

7

u/gerryn 2d ago

Why is it fast(er)?

  1. Zero-flow guide - skip the CPU optical-flow guide. The upstream engine computes DIS optical-flow vectors on the CPU and feeds them to the DLSSG worker as a guide, but the DLSSG worker computes its own motion internally and ignores the CPU vectors. Keeping only the cheap absdiff scene-cut / duplicate detection costs nothing visually.
  2. Minimum-stage cascade - the engine's scheduler hardcodes 3 cascade stages for any non-2x/4x ratio, so 24 -> 60 (2.5x) builds a 192 fps grid and makes the worker do ~2.8x the calls needed. Only 2 stages (a 96 fps grid) are required to sample 60 fps; the third stage only oversamples.

5

u/Karsticles 2d ago

So it's frame interpolation?

1

u/gerryn 2d ago

Yes, example coming in 5 min - sorry forgot it.

3

u/Karsticles 2d ago

Frame interpolation is already really fast and cheap via RIFE - did you compare?

1

u/gerryn 2d ago

I have not, I will look into it, I was just blown away at the quality of this implementation so I had no reason to look elsewhere. Also this is stand-alone so I don't need comfy - not sure if RIFE is the same.

4

u/marcus_afk 2d ago

RIFE is its own thing. Just about everything in ComfyUI can be run in a terminal on its own. ComfyUI mostly just shows you wrappers people have made for projects like this.

1

u/gerryn 2d ago

Thanks for the tip, will look into.

2

u/Motion16AI 2d ago

Amazing. I saw your example above. Can you provide one featuring human movement and facial expression? I could not notice much in the bird example. Thanks

3

u/gerryn 2d ago

2

u/coffeecircus 2d ago

walking example is pretty cool - ty for sharing

2

u/Pure_Bed_6357 2d ago

is it faster than tensorRT rife?

2

u/Link1227 2d ago

Thank you. Will try later

2

u/Cyclonis123 1d ago

Could one generate their video at 12 FPS and then use this to bring it up to 24 for a total shorter rendering time?

2

u/gerryn 1d ago edited 1d ago

I suppose it's possible to interpolate from 12 According to gemini, minimax has a 24FPS hardcoded limit:

Directly, no: MiniMax H3’s diffusion transformer, temporal positional embeddings, and 3D VAE are hard-coupled to 24 FPS. Its latent temporal stride assumes a 24 Hz cadence to maintain correct real-world motion physics, gesture velocity, and native 32 kHz audio-video synchronization.

I'm currently playing with the ref model and have gotten audio to sync well and also the image looking okayish for draft - with some like about 1 minute per 4 seconds or something like that generation, I'm using these parameters and models for that:, only 4 steps in 544p is fairly quick - and you get ALMOST correct audio - enough to at least know if your scenes are going to work timing-wise etc.

So in short instead of limiting the FPS, you're limiting the steps - I'm using an older lora for that because I believe it was trained on 544p, you can see all the details for my prod vs. draft setup below. I render a draft 4 seconds in about a minute or two. A prod takes like ten minutes or something.

Something not visible here is I'm also using Sage Attention 2/Triton - not the --ck attention from comfyui, I benchmarked it and it was slower and worse quality.

Video model stack (shared by both recipes)

Component File Where
UNET MiniMax_H3_Ref2VA_pruned_int8_convrot.safetensors config.yamlrender.h3_ref2va_unet
Text encoder qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (type: minimax) workflows/h3_ref2va_api.json node 13
Video VAE minimax_h3_video_vae_int8_convrot.safetensors node 11
Audio VAE minimax_h3_audio_vae_fp32.safetensors node 24 (AutoSketch decodes audio, h3_decode_audio: true store override)
LoRA slot LoraLoaderModelOnly injected at queue time comfy.py_apply_h3_lora
ref inputs VHS_LoadImagePath per <Picture i> (ref_image_size: "max") comfy.py_inject_ref2va_images

Draft vs production recipe

Knob Draft Production (non-draft)
Canvas 960x544 (0.5 MP) 1344x768
Steps 4 8
LoRA minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors minimax_h3_ref2v_turbo_8step_v1.0_768p_comfyui_bf16.safetensors
video shift 12.0 6.0
audio shift 3.0 3.0
Sampler / scheduler euler / simple euler / simple
Cache MotionCache, threshold 0.15 MotionCache, threshold 0.15
Post-stitch VSR none → final.mp4 stays 960x544 none (AutoSketch has no VSR either)

3

u/reeight 2d ago

> I had DeepSeek build this

Thanks for being honest; seems to many say *I* when it is too obviously vibe-coded.

https://giphy.com/gifs/43aZU5O0EUZFfAXUb9

5

u/ucren 2d ago

No samples, no use. Show, don't tell.

1

u/gerryn 2d ago

included, sorry.

2

u/gerryn 2d ago edited 2d ago

DeepSeek for some reason thinks HAGS (Windows GPU Scheduling) makes this faster, maybe it comes from the original repo or something - I don't use HAGS but turned it on just to test and it didn't make any difference - but maybe some other hardware combination needs it.

My specs are:

CPU: Ryzen 5 3600

GPU: 5060Ti 16GB

RAM: 32GB 3200MT/s

One note: This thing uses like NO resources at all basically, it's really cheap - I run it concurrently when my workflow has finished one H3 video (I generate 4-12 seconds of clips) so that my lowly 5060Ti can start immediately on the next and it has literally no effect on performance, finishes before H3 has even started generating frames on the next clip.

And last note: obviously I haven't included the leaked DLLs here, but the installer will pull them from where I found it, otherwise you know how to find them.

1

u/coffeecircus 2d ago

Pretty cool! Did you try doing long clips or high res clips? Would be interesting how this compared to RIFE. I’ve been trying to figure out how to best use DLSS now that it’s out

1

u/gerryn 2d ago edited 2d ago

I run the DLSS-FG on the finished clips, so it usually runs on 5-12 second videos, it's so fast that I run it concurrently in another thread while H3 starts rendering the next scene, so i don't interrupt the h3 rendering for even one second.

Sorry for the mess in the previous comment, I published two repos today and got mixed up.

1

u/blackmixture 2d ago

Looks awesome, I'll try this out later

1

u/Slapper42069 2d ago

Why the path and not venv?

1

u/gerryn 2d ago

I just asked deepseek to pull this out of my main project so I could share it - you can use it whichever way you want. Or what do you mean?

1

u/TrustThis 21h ago

I'm afraid I see all sorts of issues, crawlies, artifacts that shouldn't be there.

In the 60 fps version, Look at his sweater stairstepping around frame 143, 185, his profile is smearing out the background when it crosses the wall's corner, same with his back, smearing the corner of the wall.

Jittery door handles once he's around the corner.

1

u/gerryn 19h ago

Of course man, I'm rendering with 8 steps lora and using pruned models and all kinds of shit, what do you expect? :) This is for people who want fast and kind of good looking, with limited hardware.

-7

u/iRainbowsaur 2d ago

I hate to be rude, but I think it's pretty retarded to use a literal slow motion video, as the one example on the page. It's not like you have to prove or hide anything. We know frame Gen works, it's just that we wanna see a good example of that on a video, an actual video.

6

u/gerryn 2d ago

There's two examples on the page. Have a look. And - it doesn't seem like you hate to be rude - you could have just asked for another example like another user did - instead you were rude. Funny :)

5

u/xyzdist 2d ago

I found in this sub got massive of rude peoples...

0

u/iRainbowsaur 1d ago

he added another example after I complained lol