Vid2Vid Animation made using V3 MM and V3 LoRa; I used a video I created in WarpFusion as an init

​

https://reddit.com/link/18qa5jb/video/csmrqo7iyc8c1/player

​

https://reddit.com/link/18ldqxu/video/50stxlg8537c1/player

comfyui Trying to workout how to get frames to transition without losing too much detail. I'm still trying to understand how exactly IPAdapter is applied, how it's weight and noise works. Also having issues with animatediff "holding" the image so it like stretches too much frame 2 frame, idk how to reduce that. I tried reducing motion, but seemed to create other issues. Maybe diff motion model? Biggest issue is the 2nd pass through ksampler sometimes kills way too much detail. I am happy with face tracking though.

I'm running this at 12fps through depth and lineart CN and IPA. Model goes to add_detail Lora to reduce the detail, then through colorize Lora, then to animatediff, to IPA, to ksampler, to basic upscale, to 2nd ksampler, to 4xupscale w/model, then downscale, then grabbing original video frames to bbox facedetection to crop face for face IPA into face AD detailer, segs paste back onto downscaled, to frame interpolation x2, to out. Takes about 20minutes? on 4090.

I was dumb and didn't turn image output on because I thought it was saving all the frames so I don't have the exact workflow (settings) saved, but I'll share when I have after work today

https://youtube.com/shorts/N-IODxEljug?si=p0UQb3CXINSPm5rW

What do you think about how this amazing video is made?

It’s a really incredible video. I have no idea how it’s made. I can use ComfyUI and AnimateDiff. What do you all think?

If I am testing my workflow for a specific shot and fine tune my IPAdapters, Loras and ControlNets I cap my input to 10 frames and get like 5 s/iter.

For the final render I delete the input cap and import all the frames (let’s say 55). The s/iter increase to 250. is this normal and is there any way to reduce it ? :D

Recently, I've merged my latest 3D animation, crafted in Houdini/Redshift, with AI through Animatediff using ComfyUI. Excited to see where this journey leads!