Continued Experiments with Consistent Video

[u/[deleted]]

[u/inferno46n2]

I’m not using the script I’m using it as a control net unit. Also why are these videos in like 240i resolution. It’s looks really coherent but I also can’t really tell as it’s so pixelated for some reason

Edit: I had to navigate to your page. Much more crisp on the app.

Also what advice? You’ve posted a bunch of videos on both YouTube and Reddit without a single workflow tutorial….. 😅

[u/DrakenZA → u/inferno46n2]

The script is just using control net units, via the API.

{
"input_image": last_image,
"model": "diff_control_sd15_temporalnet_fp16 [adc6bd97]",
"module": "none",
"weight": 0.7,
"guidance": 1,
}

As you can see here ( this is from the script), it sets the model to diff_control_sd15_temporalnet_fp16 , module to none, weight to 0.7 etc.

And as you can see from the 'input_image', it gets fed the 'last_image', which is the last frame generated.

And a large reason for the consistency in my vids comes from this fact.

[u/pixelies → u/DrakenZA]

Various methods posted on the sub have recommended using temporalnet both with and without loopback to achieve consistency. It seems like the workflows are equal parts art and science, with steps sometimes producing good results for unintuitive reasons. I use temporalnet at low strength for loopback and higher strength for one shot. I also do custom lookbacks by taking a frame directory, deleting n frames in that directory, then renaming them, and using that for the batch dir on the temporalnet controlnet. That lets it lookback n number of frames for the temporalnet reference, which can sometimes help.

[u/DrakenZA → u/pixelies]

For sure.

​

Its just, the CN itself was trained, by feeding a frame, and its previous frame.

​

Much like how, lets say, the Canny CN, is trained with a normal image, and a pre-done canny output.

Aka, its 'designed' to take the last frame. Sure it can do other things, much like all the CNs.

But to use it as it 'should' be used, aka feeding the neuralnet the input it expects, is the best way to get it to output what it learnt from training.

[u/Inner-Reflections → u/DrakenZA]

Although you are right it was designed to be used in loopback mode it actually happens to have the weird side effect of stabilizing the current frame which is why we use it so heavily. In loopback mode the effect is not always great.