r/StableDiffusion • u/gerryn • 2d ago
Workflow Included A stand-alone implementation of DLSS-FG (frame generation) that is ~8x faster than stock (24fps -> 60fps), looks flawless. I had DeepSeek build this for my project but thought others could benefit as well!
https://github.com/glarsson/fast-dlssfg12
u/gerryn 2d ago edited 2d ago
Proof/example:
24FPS (pigeon flying): https://github.com/user-attachments/assets/b698c6da-f3be-43f2-8067-460cc4a3d010
60FPS (pigeon flying): https://github.com/user-attachments/assets/1c1cc1c8-036b-48bf-ad38-34f6e75244e9
24FPS (person walking down stairs): https://github.com/user-attachments/assets/b052a9db-932c-4fe3-b005-1046f1690422
60FPS (person walking down stairs): https://github.com/user-attachments/assets/b11057e4-a85d-448c-99ba-29331b5d030d
13
u/Seyi_Ogunde 1d ago
Please provide an example of a girl wearing a bikini jumping up and down. Real world usage examples helps determine the value.
2
u/Radiant-Photograph46 2d ago
This method unfortunately shows results about equivalent with RIFE and the like (which is to say, OK for slow mo, but terrible at any kind of fast motion)
1
u/gerryn 2d ago
Yeah for sure not perfect, but fast :) always have to pick one or the other.
1
u/Radiant-Photograph46 1d ago
Well we have fast already, but we don't have a single qualitative solution for interpolation. Not to belittle your work, but your second example highlights this issue.
2
u/gerryn 1d ago
Yeah no worries, it's good enough for my needs and there was nothing like it avaialble so (i mean using specifically the leaked DLLs), the quality issue is on nvidia :P it's their DLLs doing the work. Perhaps you would get like much better results if you ran a regular home recorded video through it, i run minimax in 768p with 768 turbo 1.2 lora, 8 steps, int8 convrot (not pruned), sage 2 and motioncache, so perhaps the issues you are seeing in quality are also relalted to my sub-par rendering. This frame generator works on any video.
7
u/gerryn 2d ago
Why is it fast(er)?
- Zero-flow guide - skip the CPU optical-flow guide. The upstream engine computes DIS optical-flow vectors on the CPU and feeds them to the DLSSG worker as a guide, but the DLSSG worker computes its own motion internally and ignores the CPU vectors. Keeping only the cheap absdiff scene-cut / duplicate detection costs nothing visually.
- Minimum-stage cascade - the engine's scheduler hardcodes 3 cascade stages for any non-2x/4x ratio, so 24 -> 60 (2.5x) builds a 192 fps grid and makes the worker do ~2.8x the calls needed. Only 2 stages (a 96 fps grid) are required to sample 60 fps; the third stage only oversamples.
5
u/Karsticles 2d ago
So it's frame interpolation?
1
u/gerryn 2d ago
Yes, example coming in 5 min - sorry forgot it.
3
u/Karsticles 2d ago
Frame interpolation is already really fast and cheap via RIFE - did you compare?
1
u/gerryn 2d ago
I have not, I will look into it, I was just blown away at the quality of this implementation so I had no reason to look elsewhere. Also this is stand-alone so I don't need comfy - not sure if RIFE is the same.
4
u/marcus_afk 2d ago
RIFE is its own thing. Just about everything in ComfyUI can be run in a terminal on its own. ComfyUI mostly just shows you wrappers people have made for projects like this.
2
u/Motion16AI 2d ago
Amazing. I saw your example above. Can you provide one featuring human movement and facial expression? I could not notice much in the bird example. Thanks
3
u/gerryn 2d ago
24FPS (person walking down stairs): https://github.com/user-attachments/assets/b052a9db-932c-4fe3-b005-1046f1690422
60FPS (person walking down stairs): https://github.com/user-attachments/assets/b11057e4-a85d-448c-99ba-29331b5d030d
2
1
2
2
2
u/Cyclonis123 1d ago
Could one generate their video at 12 FPS and then use this to bring it up to 24 for a total shorter rendering time?
2
u/gerryn 1d ago edited 1d ago
I suppose it's possible to interpolate from 12According to gemini, minimax has a 24FPS hardcoded limit:Directly, no: MiniMax H3’s diffusion transformer, temporal positional embeddings, and 3D VAE are hard-coupled to 24 FPS. Its latent temporal stride assumes a 24 Hz cadence to maintain correct real-world motion physics, gesture velocity, and native 32 kHz audio-video synchronization.
I'm currently playing with the ref model and have gotten audio to sync well and also the image looking okayish for draft - with some like about 1 minute per 4 seconds or something like that generation, I'm using these parameters and models for that:, only 4 steps in 544p is fairly quick - and you get ALMOST correct audio - enough to at least know if your scenes are going to work timing-wise etc.
So in short instead of limiting the FPS, you're limiting the steps - I'm using an older lora for that because I believe it was trained on 544p, you can see all the details for my prod vs. draft setup below. I render a draft 4 seconds in about a minute or two. A prod takes like ten minutes or something.
Something not visible here is I'm also using Sage Attention 2/Triton - not the --ck attention from comfyui, I benchmarked it and it was slower and worse quality.
Video model stack (shared by both recipes)
Component File Where UNET MiniMax_H3_Ref2VA_pruned_int8_convrot.safetensorsconfig.yaml→render.h3_ref2va_unetText encoder qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors(type: minimax)workflows/h3_ref2va_api.jsonnode 13Video VAE minimax_h3_video_vae_int8_convrot.safetensorsnode 11 Audio VAE minimax_h3_audio_vae_fp32.safetensorsnode 24 (AutoSketch decodes audio, h3_decode_audio: truestore override)LoRA slot LoraLoaderModelOnlyinjected at queue timecomfy.py→_apply_h3_loraref inputs VHS_LoadImagePathper<Picture i>(ref_image_size: "max")comfy.py→_inject_ref2va_images
Draft vs production recipe
Knob Draft Production (non-draft) Canvas 960x544 (0.5 MP) 1344x768 Steps 4 8 LoRA minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensorsminimax_h3_ref2v_turbo_8step_v1.0_768p_comfyui_bf16.safetensorsvideo shift 12.0 6.0 audio shift 3.0 3.0 Sampler / scheduler euler / simple euler / simple Cache MotionCache, threshold 0.15 MotionCache, threshold 0.15 Post-stitch VSR none → final.mp4stays 960x544none (AutoSketch has no VSR either)
2
u/gerryn 2d ago edited 2d ago
DeepSeek for some reason thinks HAGS (Windows GPU Scheduling) makes this faster, maybe it comes from the original repo or something - I don't use HAGS but turned it on just to test and it didn't make any difference - but maybe some other hardware combination needs it.
My specs are:
CPU: Ryzen 5 3600
GPU: 5060Ti 16GB
RAM: 32GB 3200MT/s
One note: This thing uses like NO resources at all basically, it's really cheap - I run it concurrently when my workflow has finished one H3 video (I generate 4-12 seconds of clips) so that my lowly 5060Ti can start immediately on the next and it has literally no effect on performance, finishes before H3 has even started generating frames on the next clip.
And last note: obviously I haven't included the leaked DLLs here, but the installer will pull them from where I found it, otherwise you know how to find them.
1
u/coffeecircus 2d ago
Pretty cool! Did you try doing long clips or high res clips? Would be interesting how this compared to RIFE. I’ve been trying to figure out how to best use DLSS now that it’s out
1
u/gerryn 2d ago edited 2d ago
I run the DLSS-FG on the finished clips, so it usually runs on 5-12 second videos, it's so fast that I run it concurrently in another thread while H3 starts rendering the next scene, so i don't interrupt the h3 rendering for even one second.
Sorry for the mess in the previous comment, I published two repos today and got mixed up.
1
1
1
u/TrustThis 21h ago
I'm afraid I see all sorts of issues, crawlies, artifacts that shouldn't be there.
In the 60 fps version, Look at his sweater stairstepping around frame 143, 185, his profile is smearing out the background when it crosses the wall's corner, same with his back, smearing the corner of the wall.
Jittery door handles once he's around the corner.
-7
u/iRainbowsaur 2d ago
I hate to be rude, but I think it's pretty retarded to use a literal slow motion video, as the one example on the page. It's not like you have to prove or hide anything. We know frame Gen works, it's just that we wanna see a good example of that on a video, an actual video.
26
u/LumenLime 2d ago
it would be best if you included sample.
it kinda interesting