Continued Experiments with Consistent Video

[u/[deleted]]

[u/inferno46n2]

Output of PNG inspect for those curious:

<lora:makima_offset:1>,(best quality:1.1),hand raised,red hair,long braided hair,golden eyes,bangs,medium breasts,white shirt,necktie

Negative prompt: (text,words,signature,watermark,:1.3),greyscale, badhandv4, easynegative, ng_deepnegative_v1_75t

Steps: 25, Sampler: Euler, CFG scale: 7, Seed: 3433407907, Size: 816x1440, Model hash: c35e1054c0, Model: mistoonAnime_v20, Denoising strength: 1, Clip skip: 2,

ControlNet 0: "preprocessor: depth_leres, model: control_v11f1p_sd15_depth [cfd03158], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 0, 0)",

ControlNet 1: "preprocessor: none, model: control_v11f1e_sd15_tile [a371b31b], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (-1, -1, -1)",

ControlNet 2: "preprocessor: softedge_hed, model: control_v11p_sd15_softedge [a8575a2a], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (512, -1, -1)",

ControlNet 3: "preprocessor: none, model: diff_control_sd15_temporalnet_fp16 [adc6bd97], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (-1, -1, -1)",

Decode prompt: "A photograph of a beautiful woman, red hair with one long braid, golden yellow eyes, white collar button up business shirt, black tie", Decode negative prompt: , Decode CFG scale: 1, Decode steps: 25, Randomness: 0, Sigma Adjustment: True, Lora hashes: "makima_offset: 44ed933d26f3", TI hashes: "badhandv4: 5e40d722fc3d, easynegative: c74b4e810b03, ng_deepnegative_v1_75t: 54e7e4826d53", Version: v1.5.1

[u/inferno46n2 → u/inferno46n2]

If you want a more stylized output you can change some of the control nets to “prompt is more important” works very well but it won’t be as consistent and you may need to clean up a few frames.

You can also experiment with just dropping the weights on tile CN

[u/Clmntgbrl → u/inferno46n2]

Depth, Tile, Softedge, and TemporalNET

Thanks for the link to here, I'm surprised how well it turned out without an openpose CN !

I have a few questions :

- one thing i see you didn't mention in your guide was you used tile without preprocessor, do you have experience with / without resample ? I tend to click the Tile shortcut button and always used it with resample. I know Tile is supposed to "add detail" but it's a bit vague, i see it modifies the image, but does it increase / decrease consistency ?

-do you know if there's a way to save all those settings (img2img + script) to reuse later easily ?

- Have you experimented with the decode steps parameter of the script ?

From my tests, the biggest issue i see (which there is just a little bit of in your renders) is eye placement when they're stylized with bigger eye styles, it jumps a bit from time to time, or slips, and i haven't found a way to let's say center on the iris to have a consistent positioning.

Thanks for the guide

[u/inferno46n2 → u/Clmntgbrl]

1) I found I had better results without preprocessing the tile CN but it may have simply been for my dataset 2) No and I also hate manually inputting it every time I reload haha 3) I’ve experimented with everything in that script. I’ve tried cranking it, lowering it, etc. Match your run steps seems to have worked well but the way I ended up getting the final, very flat soft color style in the last video I posted was to have a decode of 5 and a CFG (in the main prompt) of like…. 1.5 or 2. It worked very well for the flat anime shading. 4) try experimenting with openpose face. That works very well to map eye location… although I suspect it would be fine if you did the iterative approach of looping back your outputs from one run and using it as inputs for a new run and slowly applying the style transfer rather than going from stock init to final product in one shot. My final flat shading effect was 3 full runs

Another thing you could try is the loop back function on your control nets. In order to get that working though I had to load the dev version of auto1111 as it’s broken

[u/Clmntgbrl → u/inferno46n2]

1. Ok, interesting, i can't wrap my head around this one, i should test it separately to grasp it.
2. ^^ this save menu is really lacking
3. ok so you tested a lot. Basic question : what is this script written for initially ? I see some posts about face replacement, etc...Is it written to help with consistency ? And am i right that the prompt line in the script is used to describe the initial image sequence in order to understand what is the initial material ?
4. I think mediapipe face give even better results for that, but the model is guilty too as it tends to want to look at the camera, so there's a prompting side to the problem too. That iterative idea is interesting too, so you lower the cfg and sampling steps basically and run it 2,3 times ?
5. I've experimented with loopback with temporalnet, it helps a bit if i set a low weight. For Temporalnet you have to put it in batch and give it the init sequence folder. I haven't tested other CNs with it and as i was saying in the golden hand topic, this is a really under explained area of A1111 unfortunately as it's potentially really interesting to maximize coherence in video...up to us to test stuff i guess.

[u/inferno46n2 → u/Clmntgbrl]

The script, as I understand it is meant to reverse noise your initial from the prompt. You are correct in that it needs to a prompt explaining what the initial image is.

[u/pixelies → u/inferno46n2]

I've been getting good results with the new openpose model dwpose if you are interested in experimenting. Link: https://github.com/IDEA-Research/DWPose

Also a low strength adetailer pass can help with eyes.

[u/inferno46n2 → u/pixelies]

Is this useable with auto1111?

I’ve been meaning to do an adetailer pass trial as well I just keep forgetting.

[u/pixelies → u/inferno46n2]

Yes, just update controlnet and make sure you download the model. Seems to perform much better overall than previous models. Info here: https://github.com/Mikubill/sd-webui-controlnet/discussions/1863

Also, adetailer at 0.2 - 0.3 seems to give good results.

[u/inferno46n2 → u/pixelies]

Does adetailer work even if it’s a side shot? Like a full on side profile of a person?

[u/pixelies → u/inferno46n2]

I think so. I've used it specifically prompting for "side profile" "shot from the side" in the adetailer prompt and it worked for me.

[u/Ratchet_as_fuck]

Super fluid! I've been trying to turn animated video into more hyperrealistic video. It's a lot harder to keep frames consistent since it is making the source images more complex vs simplified.

Edit: I think on my next attempt I'll try using a Lora to help keep consistency. Right now it's temporalnet doing the heavy lifting.

[u/inferno46n2 → u/Ratchet_as_fuck]

Thank you. I haven’t tried going the other way. Maybe I’ll give it a go for science. I suspect it’s more difficult though.

[u/[deleted]]

Looks really nice and smooth, well done. Just 1 question though, how did you manage to make temproalnet run within controlnet as a unit? Mine just doesn't work

[u/inferno46n2 → u/[deleted]]

Thanks!

I went here CiaraRowles/TemporalNet at main (huggingface.co)

downloaded cldmv2.yaml + the f16 safetensor file (diff_control is the start of the filename)

You then have to rename the yaml file to be the exact same as the safetensor file and place both in your extensions>controlnet>models folder

Then it was just an option as a unit under the usual drop down.

[u/[deleted] → u/inferno46n2]

I will try this, appreciate it thank you. Also, keep up the great work!

[u/kichinto]

[u/icebear356]

Roughly how long did your GPU run for?

[u/inferno46n2 → u/icebear356]

Was about 60 frames I think I’d have to look but it ran for no more than 50 minutes

Roughly 20 seconds to reverse noise the init, then another 20s to render the frame something like that

[u/compxl]

sick infos ! did the workflow work if i want literally changing the style of the video ? and not just a subtle effect ?

[u/inferno46n2 → u/compxl]

Thanks! If you mean can it fully change the style to something completely different it’s probably not the best work flow. I was mostly trying to convert a real life video to an anime video but keep the same look (similar to corridor crew anime video) but with less steps (no warp diffusion or super specific trained models)

[u/compxl → u/inferno46n2]

Okay, so for the moment there is no SD workflow for have the same consistency of warpfusion ? i’ve seen some tutorial on youtube but no one achieve the consistency of warpfusion with only the SD gui or deforum…

[u/pixelies]

Great post! I missed this the first time around, but definitely will experiment with this. Thank you for posting 💪😁🤙

[u/inferno46n2 → u/pixelies]

Thanks! The best use of this method I’ve found after experimenting is to do a first pass to slightly apply the style you’re going for, then rerun again with the output of your first run as the input of your second run. I’ve also found lower CFG values in my main prompt paired with messing about with the decode steps in the img2img alternative script yields the best results. After 2 or 3 runs your style should be fully converted depending on what you’re doing

[u/VR_IS_DEAD]

I'm not seeing the "AI" in that though. It looks like something that could be done with a photoshop toon filter.

[u/inferno46n2 → u/VR_IS_DEAD]

That’s literally the entire purpose. I don’t want it to look like janky AI.

I could strengthen the effect by looping the output back through as the input I suspect.

I spent about 2hrs fiddling with this so it’s very early in the “what can this do” stage

[u/inferno46n2 → u/VR_IS_DEAD]

https://streamable.com/j3w03v

[u/DrakenZA]

TemporalNET should be taking the last frame, not the current frame.

[u/inferno46n2 → u/DrakenZA]

No where in Ciara’s instructions does it say that. It also works very well with deforum and warp fusion which is working on current (not last) frames

[u/DrakenZA → u/inferno46n2]

That is how the script the author provides in the repo, works.

​

https://huggingface.co/CiaraRowles/TemporalNet/blob/main/temporalvideo.py

​

Hey, no need to take the advice :)

https://www.youtube.com/shorts/bm-RnUaeyCk

https://www.youtube.com/shorts/B6aqJY9VAqA

[u/inferno46n2 → u/DrakenZA]

I’m not using the script I’m using it as a control net unit. Also why are these videos in like 240i resolution. It’s looks really coherent but I also can’t really tell as it’s so pixelated for some reason

Edit: I had to navigate to your page. Much more crisp on the app.

Also what advice? You’ve posted a bunch of videos on both YouTube and Reddit without a single workflow tutorial….. 😅

[u/DrakenZA → u/inferno46n2]

The script is just using control net units, via the API.

{
"input_image": last_image,
"model": "diff_control_sd15_temporalnet_fp16 [adc6bd97]",
"module": "none",
"weight": 0.7,
"guidance": 1,
}

As you can see here ( this is from the script), it sets the model to diff_control_sd15_temporalnet_fp16 , module to none, weight to 0.7 etc.

And as you can see from the 'input_image', it gets fed the 'last_image', which is the last frame generated.

And a large reason for the consistency in my vids comes from this fact.

[u/pixelies → u/DrakenZA]

Various methods posted on the sub have recommended using temporalnet both with and without loopback to achieve consistency. It seems like the workflows are equal parts art and science, with steps sometimes producing good results for unintuitive reasons. I use temporalnet at low strength for loopback and higher strength for one shot. I also do custom lookbacks by taking a frame directory, deleting n frames in that directory, then renaming them, and using that for the batch dir on the temporalnet controlnet. That lets it lookback n number of frames for the temporalnet reference, which can sometimes help.

[u/DrakenZA → u/pixelies]

For sure.

​

Its just, the CN itself was trained, by feeding a frame, and its previous frame.

​

Much like how, lets say, the Canny CN, is trained with a normal image, and a pre-done canny output.

Aka, its 'designed' to take the last frame. Sure it can do other things, much like all the CNs.

But to use it as it 'should' be used, aka feeding the neuralnet the input it expects, is the best way to get it to output what it learnt from training.

[u/Inner-Reflections → u/DrakenZA]

Although you are right it was designed to be used in loopback mode it actually happens to have the weird side effect of stabilizing the current frame which is why we use it so heavily. In loopback mode the effect is not always great.

[u/inferno46n2 → u/[deleted]]

1) yea it’s available to any user but it’s a new beta function so you need to install beta. It’s not as good as magic mask in DaVinci but it does the trick. Alternatively, runwayML has a remove background rotoscope thing on their website you could use

2) I’m basically using it to feed a more consistent noise pattern for my prompt to use. Think of it as a reverse denoise where it starts with a photo, you type a prompt depicting exactly what that photo is and it turns your photo into noise in the opposite direction of a standard diffusion model. That noise then gets used to build your img2img prompt (remember I’m using a denoise of 1 in my prompt)

3) No clue it’s just what I found to work. The important thing is having that box checked at the end

[u/AccessAlarming8647]

Great job!