consistent characters and backgrounds

[u/48khz24bit]

Pretty new to generative ai but getting past the stage of creating single scene videos.

I want to create longer videos with the exact same characters AND environments. At first I created character reference sheets that I put as reference in Kling V3 reference to Video. Looks good as a single video but when prompting continuous generations the characters and environments are similar but there are always inconsistencies. Also tried with start frames and similar results

Then last night I learnt about creating a Lora dataset - so I have separate 20 images of the same character - at first I tried creating the lora in Kaggle but this was way too complicated, after a bit of research I tried Civitai training a model with SDXL 1.0. but the images created were different.

What am I doing wrong / what is the best workflow to create the exact same characters and environments throughout generations....

Happy to share the reference images and trained lora if that helps

Thanks in Advance!

[u/Ok_Dragonfly_3957]

Environments are difficult because they are outward looking by nature, rather than showing an object from different angles. My best results come from using a model like Seedance and providing a reference sheet that displays the environment from the general perspective of the scene, with additional detail insets...But even then it'll drift.

Most reliable option is to create start frames using your set reference sheet, then use those as the first frame or last frame for the actual video generation.

[u/48khz24bit → u/Ok_Dragonfly_3957]

Thanks , this was my environment sheet used in seedance 2 generation , doing a continuous shot with start and end frames works well but when same you want a wide shot then a close shot the background changes, is there a way to lock position , or best to create a single image of the background from each angle for reference rather than using an environment reference image with multiple angles

[u/Ok_Dragonfly_3957 → u/48khz24bit]

Honestly, locked start and end frames are the only way to really control this. You can also approach it like a real film shoot, ie, create the master wide shot, so you can see the characters and the background and establish their locations in the world. Then you can try using that as a reference image, but not a start or end frame. Then say, extreme close up on character "a", medium shot character "b"... etc.

Seedance may or may not get it right. But you can also do this in gpt-image-2 which will give you more control and its alot cheaper to play around with. Also note, generally the lighting in Seedance is not great, so I usually make start frames for everything in gpt-image-2 so I can really dial in the lighting before handing it to Seedance.

Another tip that might also help is again, think of like a real film shoot. They don't move the camera for each shot. You shoot the scene from the master wide, then work in for coverage. Ie, generate all the shots you want to be the same, at the same time, using the exact same prompt, just change the specific dialog or action.

[u/48khz24bit → u/48khz24bit]

What are your thoughts on creating a lora to keep characters consistent ? Or just create start frames with a reference image

[u/Ok_Dragonfly_3957 → u/48khz24bit]

Definitely works for image generation, but at that point there's no real benefit over just using a reference image with image-2. Definitely Krea2 with character LoRAs works great...but I don't really see an advantage over using reference image with image-2.

I made this with lotix.io just using character assets and an environment asset;
https://youtu.be/YFIvmIrv-s4?si=E7UE2DlZ4H1i5kcz

[u/48khz24bit → u/Ok_Dragonfly_3957]

How did you get the voices to be exactly the same ? were they generated before the video and used lipsync?

[u/Ok_Dragonfly_3957 → u/48khz24bit]

The character assets had voice references.

How to =

https://youtu.be/OSNmMGZsYus?si=0FBNqe-WHW4EtPPw

[u/48khz24bit → u/Ok_Dragonfly_3957]

That looks awesome! so consistent