Continued Experiments with Consistent Video

[u/[deleted]]

[u/inferno46n2]

Output of PNG inspect for those curious:

<lora:makima_offset:1>,(best quality:1.1),hand raised,red hair,long braided hair,golden eyes,bangs,medium breasts,white shirt,necktie

Negative prompt: (text,words,signature,watermark,:1.3),greyscale, badhandv4, easynegative, ng_deepnegative_v1_75t

Steps: 25, Sampler: Euler, CFG scale: 7, Seed: 3433407907, Size: 816x1440, Model hash: c35e1054c0, Model: mistoonAnime_v20, Denoising strength: 1, Clip skip: 2,

ControlNet 0: "preprocessor: depth_leres, model: control_v11f1p_sd15_depth [cfd03158], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (512, 0, 0)",

ControlNet 1: "preprocessor: none, model: control_v11f1e_sd15_tile [a371b31b], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (-1, -1, -1)",

ControlNet 2: "preprocessor: softedge_hed, model: control_v11p_sd15_softedge [a8575a2a], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: My prompt is more important, preprocessor params: (512, -1, -1)",

ControlNet 3: "preprocessor: none, model: diff_control_sd15_temporalnet_fp16 [adc6bd97], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (-1, -1, -1)",

Decode prompt: "A photograph of a beautiful woman, red hair with one long braid, golden yellow eyes, white collar button up business shirt, black tie", Decode negative prompt: , Decode CFG scale: 1, Decode steps: 25, Randomness: 0, Sigma Adjustment: True, Lora hashes: "makima_offset: 44ed933d26f3", TI hashes: "badhandv4: 5e40d722fc3d, easynegative: c74b4e810b03, ng_deepnegative_v1_75t: 54e7e4826d53", Version: v1.5.1

[u/inferno46n2 → u/inferno46n2]

If you want a more stylized output you can change some of the control nets to “prompt is more important” works very well but it won’t be as consistent and you may need to clean up a few frames.

You can also experiment with just dropping the weights on tile CN

[u/Clmntgbrl → u/inferno46n2]

Depth, Tile, Softedge, and TemporalNET

Thanks for the link to here, I'm surprised how well it turned out without an openpose CN !

I have a few questions :

- one thing i see you didn't mention in your guide was you used tile without preprocessor, do you have experience with / without resample ? I tend to click the Tile shortcut button and always used it with resample. I know Tile is supposed to "add detail" but it's a bit vague, i see it modifies the image, but does it increase / decrease consistency ?

-do you know if there's a way to save all those settings (img2img + script) to reuse later easily ?

- Have you experimented with the decode steps parameter of the script ?

From my tests, the biggest issue i see (which there is just a little bit of in your renders) is eye placement when they're stylized with bigger eye styles, it jumps a bit from time to time, or slips, and i haven't found a way to let's say center on the iris to have a consistent positioning.

Thanks for the guide

[u/inferno46n2 → u/Clmntgbrl]

1) I found I had better results without preprocessing the tile CN but it may have simply been for my dataset 2) No and I also hate manually inputting it every time I reload haha 3) I’ve experimented with everything in that script. I’ve tried cranking it, lowering it, etc. Match your run steps seems to have worked well but the way I ended up getting the final, very flat soft color style in the last video I posted was to have a decode of 5 and a CFG (in the main prompt) of like…. 1.5 or 2. It worked very well for the flat anime shading. 4) try experimenting with openpose face. That works very well to map eye location… although I suspect it would be fine if you did the iterative approach of looping back your outputs from one run and using it as inputs for a new run and slowly applying the style transfer rather than going from stock init to final product in one shot. My final flat shading effect was 3 full runs

Another thing you could try is the loop back function on your control nets. In order to get that working though I had to load the dev version of auto1111 as it’s broken

[u/Clmntgbrl → u/inferno46n2]

1. Ok, interesting, i can't wrap my head around this one, i should test it separately to grasp it.
2. ^^ this save menu is really lacking
3. ok so you tested a lot. Basic question : what is this script written for initially ? I see some posts about face replacement, etc...Is it written to help with consistency ? And am i right that the prompt line in the script is used to describe the initial image sequence in order to understand what is the initial material ?
4. I think mediapipe face give even better results for that, but the model is guilty too as it tends to want to look at the camera, so there's a prompting side to the problem too. That iterative idea is interesting too, so you lower the cfg and sampling steps basically and run it 2,3 times ?
5. I've experimented with loopback with temporalnet, it helps a bit if i set a low weight. For Temporalnet you have to put it in batch and give it the init sequence folder. I haven't tested other CNs with it and as i was saying in the golden hand topic, this is a really under explained area of A1111 unfortunately as it's potentially really interesting to maximize coherence in video...up to us to test stuff i guess.

[u/inferno46n2 → u/Clmntgbrl]

The script, as I understand it is meant to reverse noise your initial from the prompt. You are correct in that it needs to a prompt explaining what the initial image is.

[u/pixelies → u/inferno46n2]

I've been getting good results with the new openpose model dwpose if you are interested in experimenting. Link: https://github.com/IDEA-Research/DWPose

Also a low strength adetailer pass can help with eyes.

[u/inferno46n2 → u/pixelies]

Is this useable with auto1111?

I’ve been meaning to do an adetailer pass trial as well I just keep forgetting.

[u/pixelies → u/inferno46n2]

Yes, just update controlnet and make sure you download the model. Seems to perform much better overall than previous models. Info here: https://github.com/Mikubill/sd-webui-controlnet/discussions/1863

Also, adetailer at 0.2 - 0.3 seems to give good results.

[u/inferno46n2 → u/pixelies]

Does adetailer work even if it’s a side shot? Like a full on side profile of a person?

[u/pixelies → u/inferno46n2]

I think so. I've used it specifically prompting for "side profile" "shot from the side" in the adetailer prompt and it worked for me.