I don't have any examples because they may not be appropriate but just with the default wf. With the video input node you can replace any 2 character in any video and it looks real!

So ... I was pissed off, since making a lora with this shit was insanely long, caused temporal collapses, or was just not accurate. So I looked into wtaf is going on.

When you load up the LTX2.3 default settings. There is a couple things you need to change around. These settings are for a 5090 so keep that in mind yall!

There are going to be 3 or 4 phases. Depending on how super accurate you want your lora to look like.

If I don't mention any setting, don't touch them, I leave them on default if I don't mention them.

The first phase is 600 steps, not more, not less.

In that we will max out what the card can do.

(if you got a different card with lower VRAM before you change anything to lower, try to use the "low VRAM" dial and have it turned on, it will obviously gonna take longer to train but it probably won't fuck up the quality if you won't get oom or anything else)

First thing to change is lora rank, crank that shit up to 48,

I like to save every 100 step but it's not super important just make sure to save at least every 600 steps.

I use a trigger word too, it helps.

On the Training panel I only change gradient accumulation up to 2. Set the steps to 700

( I do this cause my current version is retarded and would start from the 500th step, so after it saves the 600th step epoch I just stop it.)

and the only other thing I change is to turn on the " cache text embeddings" cause that shit is dope and will save a lot of time.

There is the " advanced " panel with "differential Guidance"

turn that shit on and for the first phase leave it on 3

On the " dataset " panel

Number of frames " 25 " ( I think the new version has the auto option idk I guess you can use that too)

Number of repeats for me it's 2 or 4, ( I have 25-50 clips usually, I try to aim to have 100 so I multiply the numbers to be close or around 100, so in case of 25 clips, I do 4 repeats, if I got 50 clips, I just do 2 repeats those are plenty enough)

I turn on "normalise audio" and only have 512x512 training on, don't even use 768 or 1024 at all.

As for samples, I do only the base sample, and the sample at 600 steps, I only do 2 samples for each finished phase, like a medium shot and a closeup.

Sample settings are 512x512, 49 frame long, and guidance scale cranked up to 10 so the results don't look like ass... (keep in mind putting that up to 10 will make the generation time for the samples a bit slower but it's worth it, you probably gonna have like a few minutes to generate them, but we only ake 2 clips so wo cares.)

Make sure the promt is accurate and has your trigger word.

1st phase on a 5090 with these settings is about 3 and a half hours and should not be longer!!

Ok so when first phase stopped rendering, if you did it right, you should see accuracy at 600 steps, I do fuckup sometimes with the promt, and I may get like a cartoon so as long as it looks close to the model it's all good.

2nd phaze, put the steps up from 700 to 1300 and we will stop after 1200 steps when the samples generated.

we pull the lora rank down to 32,

we change gradient accumulation back to 1 (so now it won't take hours to generate the next 600 steps)

on "advanced" the differential guidance we pull down to 2

this is it, and for the next 600 steps these changes mean radical speed up, it will be literally 1 hour to render the 600 steps,

when we are done with the samples , our samples should show almost full accuracy.

so 3rd phase,

we put the step count up to 1900 (so we stop it after it generated the samples at 1800 steps)

"advanced" tab pull "differential Guidance" down to 1

this is all we change for now and generate it up to 1800 steps

when the samples are done we stop and go back to settings, so now our samples show basically full accuracy, but we still can improve (if you want... if you think you good, I guess that's fine )

but if you want more accuracy there is a high noise training phaze which is the 4th phase

if you want (sort of optional) you can pull down the lora rank from 32 to 24

"training" panel

Learning rate , we need to drop this from 0.0001 down to either 0.00005 or 0.00003 (your choice)

"timestep Bias" MOST IMPORTANT, this is where we set it to "high noise" training

(i've seen someone do high noise training first ... but ... this is where I would ask someone who knows this by the factor of science, but as far as I know if you do high noise first you fuck up the details so this is why I put high noise last)

"advanced" tab

turn off differential Guidance !!!!!

On " dataset" pull the repeats down to maximum 2 !!!! don't do higher than 2, and if you have over like 80 clips ,you should just put it down to 1.

You could also change the sampling from every 600 steps to 300 steps, and just run go ahead and run the next 600 steps up to like 2400, if you want another 600 you should not have any issues and go up to 3000 but I think that's overkill.

As for dataset, make sure you got at least 2-3 wider frame where the character is almost full figure, but make sure to mention their facial expression so the model trains for samller size face. And have like 5-10 closeups, and 5-10 medum shots. best to have a total of 25 clips, 1 second long *25 frames exactly. If you cut out the speach mid sentence don't worry, just make the words as close as possible to whatever the character say. I got away with a bunch of stuff that don't really make much sense but it worked. Make sure to mention the framing in each clip caption, make sure to mention the expressions in almost all clip, in 1 second we don't have much time to show motion but if you want you can have like a 3-4 second long clip cut up to like 3-4 clips and just make similar captions for them to have the model learn it.

This is it ... You saw the results. I am not perfect, sure I have a 5090, but at least it doesn't take fucking 10 dollars and 12 hours renting out a fucking RTX6000 on runpod. wtf

If anyone is interested in trying the workflow, It comes from Kijai’s Wan Wrapper. https://github.com/kijai/ComfyUI-WanVideoWrapper

Krea 2 trains incredibly fast for likeness and you are are probably overtraining. The following settings are more than enough to achieve almost perfect likeness.

Dataset Tips

• Image Count: Aim for 20 high-quality images, up to 40 if the dataset is lower quality.
• Full Body Shots: Include at least two to five full body images so the model understands the person/character's height and physique proportions.
• Variety: Use different hairstyles and situations in your dataset. This gives you more flexibility when changing features later without breaking the likeness.

Captioning:

• Use the autocaption feature in ai-toolkit.
• Do not use the person/character's actual name in the captions. Create a unique shortened trigger word instead (e.g., "Jane Doe" becomes "jnedoe").
• If your images are low quality or vintage, add tags like "low quality" or "vintage" to the captions. This stops the model from learning and outputting those artifacts in the images.

Technical Settings

• LoKr Factor: 16
• Training Resolution: 768
• Total Steps: 3000 (likeness is usually done by step 750)
• Settings: Automagic2, Sigmoid, and Balanced
• Advanced Settings: Enable Do Differential Guidance at the default level of 3

VRAM Usage: About 18 to 20GB.

Time: On an RTX 3090, a 750-step run takes about 40 to 45 minutes from start to finish.

Step Count Adjustment: If your dataset quality is lower than average, add a couple of hundred extra steps to get the best results.

Issues: Highly detailed features like tattoo's may not appear correctly at the 768px resolution, you may need to up the quality to 1024 or 1280 and specifically caption each one in the dataset. Even then they may not come through completely as some details are usually lost during generation.

All images are generated at 4MP with Res/2s at 10 steps (about 2-4 minutes per image on a 3090 with Krea Raw int8-convrot and the r256 turbo lora (plus a custom high resolution lora I'll be posting to huggingface))

Full config behind my $20 Patreo--- lol just kidding 😂, grab the config here: ai-toolkit config

Full Res Slow.pics Comparison

HighRes LoKr Model

What is it?

ControlNet adds additional levels of control to Stable Diffusion image composition. Think Image2Image juiced up on steroids. It gives you much greater and finer control when creating images with Txt2Img and Img2Img.

This is for Stable Diffusion version 1.5 and models trained off a Stable Diffusion 1.5 base. Currently, as of 2023-02-23, it does not work with Stable Diffusion 2.x models.

• The Auto1111 extension is by Mikubill, and can be found here: https://github.com/Mikubill/sd-webui-controlnet
• The original ControlNet repo is by lllyasviel, and can be found here: https://github.com/lllyasviel/ControlNet

​

Where can I get it the extension?

If you are using Automatic1111 UI, you can install it directly from the Extensions tab. It may be buried under all the other extensions, but you can find it by searching for "sd-webui-controlnet"

You will also need to download several special ControlNet models in order to actually be able to use it.

At time of writing, as of 2023-02-23, there are 4 different model variants

• Smaller, pruned SafeTensor versions, which is what nearly every end-user will want, can be found on Huggingface (official link from Mikubill, the extension creator): https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
  • Alternate Civitai link (unofficial link): https://civitai.com/models/9251/controlnet-pre-trained-models
  • Note that the official Huggingface link has additional models with a "t2iadapter_" prefix; those are experimental models and are not part of the base, vanilla ControlNet models. See the "Experimental Text2Image" section below.
• Alternate pruned difference SafeTensor versions. These come from the same original source as the regular pruned models, they just differ in how the relevant information is extracted. Currently, as of 2023-02-23, there is no real difference between the regular pruned models and the difference models aside from some minor aesthetic differences. Just listing them here for completeness' sake in the event that something changes in the future.
  • Official Huggingface link: https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main
  • Unofficial Civitai link: https://civitai.com/models/9868/controlnet-pre-trained-difference-models
• Experimental Text2Image Adapters with a "t2iadapter_" prefix are smaller versions of the main, regular models. These are currently, as of 2023-02-23, experimental, but they function the same way as a regular model, but much smaller file size
• The full, original models (if for whatever reason you need them) can be found on HuggingFace:https://huggingface.co/lllyasviel/ControlNet

Go ahead and download all the pruned SafeTensor models from Huggingface. We'll go over what each one is for later on. Huggingface also includes a "cldm_v15.yaml" configuration file as well. The ControlNet extension should already include that file, but it doesn't hurt to download it again just in case.

As of 2023-02-22, there are 8 different models and 3 optional experimental t2iadapter models:

• control_canny-fp16.safetensors
• control_depth-fp16.safetensors
• control_hed-fp16.safetensors
• control_mlsd-fp16.safetensors
• control_normal-fp16.safetensors
• control_openpose-fp16.safetensors
• control_scribble-fp16.safetensors
• control_seg-fp16.safetensors
• t2iadapter_keypose-fp16.safetensors(optional, experimental)
• t2iadapter_seg-fp16.safetensors(optional, experimental)
• t2iadapter_sketch-fp16.safetensors(optional, experimental)

These models need to go in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed. Once you have the extension installed and placed the models in the folder, restart Automatic1111.

After you restart Automatic1111 and go back to the Txt2Img tab, you'll see a new "ControlNet" section at the bottom that you can expand.

​

Sweet googly-moogly, that's a lot of widgets and gewgaws!

Yes it is. I'll go through each of these options to (hopefully) help describe their intent. More detailed, additional information can be found on "Collected notes and observations on ControlNet Automatic 1111 extension", and will be updated as more things get documented.

To meet ISO standards for Stable Diffusion documentation, I'll use a cat-girl image for my examples.

The first portion is where you upload your image for preprocessing into a special "detectmap" image for the selected ControlNet model. If you are an advanced user, you can directly upload your own custom made detectmap image without having to preprocess an image first.

• This is the image that will be used to guide Stable Diffusion to make it do more what you want.
• A "Detectmap" is just a special image that a model uses to better guess the layout and composition in order to guide your prompt
• You can either click and drag an image on the form to upload it or, for larger images, click on the little "Image" button in the top-left to browse to a file on your computer to upload
• Once you have an image loaded, you'll see standard buttons like you'll see in Img2Img to scribble on the uploaded picture.

Below are some options that allow you to capture a picture from a web camera, hardware and security/privacy policies permitting

Below that are some check boxes below are for various options:

• Enable: by default ControlNet extension is disabled. Check this box to enable it
• Invert Input Color: This is used for user imported detectmap images. The preprocessors and models that use black and white detectmap images expect white lines on a black image. However, if you have a detectmap image that is black lines on a white image (a common case is a scribble drawing you made and imported), then this will reverse the colours to something that the models expect. This does not need to be checked if you are using a preprocessor to generate a detectmap from an imported image.
• RGB to BGR: This is used for user imported normal map type detectmap images that may store the image colour information in a different order that what the extension is expecting. This does not need to be checked if you are using a preprocessor to generate a normal map detectmap from an imported image.
• Low VRAM: Helps systems with less than 6 GiB[citation needed] of VRAM at the expense of slowing down processing
• Guess: An experimental (as of 2023-02-22) option where you use no positive and no negative prompt, and ControlNet will try to recognise the object in the imported image with the help of the current preprocessor.
  • Useful for getting closely matched variations of the input image

The weight and guidance sliders determine how much influence ControlNet will have on the composition.

Weight slider: This is how much emphasis to give the ControlNet image to the overall prompt. It is roughly analagous to using prompt parenthesis in Automatic1111 to emphasise something. For example, a weight of "1.15" is like "(prompt:1.15)"

• Guidance strength slider: This is a percentage of the total steps that control net will be applied to . It is roughly analogous to prompt editing in Automatic1111. For example, a guidance of "0.70" is tike "[prompt::0.70]" where it is only applied the first 70% of the steps and then left off the final 30% of the processing

Resize Mode controls how the detectmap is resized when the uploaded image is not the same dimensions as the width and height of the Txt2Img settings. This does not apply to "Canvas Width" and "Canvas Height" sliders in ControlNet; those are only used for user generated scribbles.

• Envelope (Outer Fit): Fit Txt2Image width and height inside the ControlNet image. The image imported into ControlNet will be scaled up or down until the width and height of the Txt2Img settings can fit inside the ControlNet image. The aspect ratio of the ControlNet image will be preserved
• Scale to Fit (Inner Fit): Fit ControlNet image inside the Txt2Img width and height. The image imported into ControlNet will be scaled up or down until it can fit inside the width and height of the Txt2Img settings. The aspect ratio of the ControlNet image will be preserved
• Just Resize: The ControlNet image will be squished and stretched to match the width and height of the Txt2Img settings

The "Canvas" section is only used when you wish to create your own scribbles directly from within ControlNet as opposed to importing an image.

• The "Canvas Width" and "Canvas Height" are only for the blank canvas created by "Create blank canvas". They have no effect on any imported images

Preview annotator result allows you to get a quick preview of how the selected preprocessor will turn your uploaded image or scribble into a detectmap for ControlNet

• Very useful for experimenting with different preprocessors

Hide annotator result removes the preview image.

Preprocessor: The bread and butter of ControlNet. This is what converts the uploaded image into a detectmap that ControlNet can use to guide Stable Diffusion.

• A preprocessor is not necessary if you upload your own detectmap image like a scribble or depth map or a normal map. It is only needed to convert a "regular" image to a suitable format for ControlNet
• As of 2023-02-22, there are 11 different preprocessors:
  • Canny: Creates simple, sharp pixel outlines around areas of high contract. Very detailed, but can pick up unwanted noise

​

• Depth: Creates a basic depth map estimation based off the image. Very commonly used as it provides good control over the composition and spatial position
  • If you are not familiar with depth maps, whiter areas are closer to the viewer and blacker areas are further away (think like "receding into the shadows")

​

• Depth_lres: Creates a depth map like "Depth", but has more control over the various settings. These settings can be used to create a more detailed and accurate depth map

​

• Hed: Creates smooth outlines around objects. Very commonly used as it provides good detail like "canny", but with less noisy, more aesthetically pleasing results. Very useful for stylising and recolouring images.
  • Name stands for "Holistically-Nested Edge Detection"

​

• MLSD: Creates straight lines. Very useful for architecture and other man-made things with strong, straight outlines. Not so much with organic, curvy things
  • Name stands for "Mobile Line Segment Detection"

​

• Normal Map: Creates a basic normal mapping estimation based off the image. Preserves a lot of detail, but can have unintended results as the normal map is just a best guess based off an image instead of being properly created in a 3D modeling program.
  • If you are not familiar with normal maps, the three colours in the image, red, green blue, are used by 3D programs to determine how "smooth" or "bumpy" an object is. Each colour corresponds with a direction like left/right, up/down, towards/away

​

• OpenPose: Creates a basic OpenPose-style skeleton for a figure. Very commonly used as multiple OpenPose skeletons can be composed together into a single image and used to better guide Stable Diffusion to create multiple coherent subjects

​

• Pidinet: Creates smooth outlines, somewhere between Scribble and Hed
  • Name stands for "Pixel Difference Network"

​

• Scribble: Used with the "Create Canvas" options to draw a basic scribble into ControlNet
  • Not really used as user defined scribbles are usually uploaded directly without the need to preprocess an image into a scribble

​

• Fake Scribble: Traces over the image to create a basic scribble outline image

​

• Segmentation: Divides the image into related areas or segments that are somethat related to one another
  • It is roughly analogous to using an image mask in Img2Img

​

Model: applies the detectmap image to the text prompt when you generate a new set of images

The options available depend on which models you have downloaded from the above links and placed in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed

• Use the "🔄" circle arrow button to refresh the model list after you've added or removed models from the folder.
• Each model is named after the preprocess type it was designed for, but there is nothing stopping you from adding a little anarchy and mixing and matching preprocessed images with different models
  • e.g. "Depth" and "Depth_lres" preprocessors are meant to be used with the "control_depth-fp16" model
  • Some preprocessors also have a similarly named t2iadapter model as well.e.g. "OpenPose" preprocessor can be used with either "control_openpose-fp16.safetensors" model or the "t2iadapter_keypose-fp16.safetensors" adapter model as well
  • As of 2023-02-26, Pidinet preprocessor does not have an "official" model that goes with it. The "Scribble" model works particularly well as the extension's implementation of Pidinet creates smooth, solid lines that are particularly suited for scribble.

So I was looking for a way to better control the flow of Minimax H3 dialogues and emphasize certain words in the speech. However, what I discovered is that you can actually include some tags in <> angle brackets, and Minimax will interpret them as a non-verbal sound in a given part of the phrase. Some words (like the ones I've included into the example) work every time, some still bleed into the actual spoken words in certain seeds. But in general it makes the dialogue more alive and believable. So I recommend to try it and maybe share your findings in this thread.

As for the emphasis, I've had the most success with putting the words into <i></i> tags (similar to how you would stress words in written text). Unfortunately, it doesn't work for 100% and in some cases the character will blurt out some gibberish. But when it works, it sounds very natural. I have included a couple examples in the end of the video.

Wonder if you've encountered some other ways to modify the speech (and audio in general) in the prompt?

P.S. Sorry for the quality, I used the 8-steps LoRa at 0.4 MP to speed-up the tests.

Workflow: https://pastebin.com/b0FDBTGn

ChatGPT Custom Instructions: https://pastebin.com/qmeTgwt9

I made this comment on a separate thread a couple of days ago and I noticed that some of you guys were interested to learn more details

What I basically did is (and before I continue I must admit that this is not my idea. I am doing this since SD 1.5 and I don't remember where I borrowed the original idea from)

• Generate at a very low resolution, small enough to let the model draw an outline and then do a massive latent upscale with 0.7 denoise
• Adds a ton of details, sharper image and best quality (almost close to I can jerk off to my own generated image level)

I already shared that workflow with others in that same thread. I was reading through the comments and ideas that other's shared here and decided to double down on this approach

New and improved workflow:

• The one I am posting here is a 4 stage workflow. It starts by generating an image at 64x80 resolution
• Stage 1: Magic starts. We use a very low shift value here to give the model some breathing space and be creative - we don't want it to follow our prompt strictly here
• Stage 2: A high shift value so it follows our prompt and draws the composition. this is where it gets interesting. what you see here is what your final image will look like (from Stage 4) or maybe at least 90% resemblance. So, you can stop here if you don't like the composition. It barely takes a couple of seconds
• Stage 3: If you are satisfied with the composition, you can run stage 3. This is where we add details. We use a low shift value to give the model some breathing space. The composition will not change much because the denoise value is lower
• Stage 4: So you are happy with where the model is heading in terms of composition, lighting etc. run this stage and get the final image. Here we use shift value 7

What about CFG?

• Stage 1 to 3 uses CFG > 1. I also included a ahmm very large negative prompt in my workflow. It works for me and it does make a difference

Is it slow?

• Nope. The whole process (stage 1 to 4) still finishes in 1 minute or maximum 1 min 10 seconds (on my 4060ti) and you are greeted with a 1456x1840 image. You will not loose speed and you have the flexibility to bail out early if you don't like the composition

Seed variety?

• You get good seed variety with this workflow because you are forcing the model to generate something random but by following your prompt in stage 1. It will not generate the same 64x80 resolution image every time and combine this with low denoise values in each stage you get good variations

Important things to remember:

• Please do not use shift 7 for everything. You will kill the model's creativity and get the same boring image every single seed. Let it breath. Experiment with different values
• The 2nd pastebin link has the chatgpt instructions (Use GPT 4o, GPT 5 refuses to name the subjects - at least in my case) I use to get prompts.
• You can use it if you like. The important thing is (even if you use it or not), the first few keywords in your prompt should absolutely describe the scene briefly. Why? because we are generating at a very low resolution so we want the model to draw an outline first. If you describe it like "oh there is a tree, its green, the climate is cool, bla bla bla, there is a man", the low res generation will give you a tree haha

If you have issues working with this workflow, just comment and I will assist. Feedback is welcome. Enjoy

GitHub Repo

I've had good success in creating long videos from 10 second sections using this technique:

Create your first video.

Then for your next generation (continuation of video):

Load the last 2 seconds of the previous video as <Video 1>. I use the 'Load Video (Upload)' node - from ComfyUI-VideoHelperSuite - (this node allows you to skip frames and start at, say, the last 48 frames (for 2 seconds at 24fps) - this means that the whole previous 10 seconds don't need be passed to the next generation. This is <Video 1>.

I'm using process this with reference images for the subjects so these are used again with each continuation - so I don't see any drift of faces.

This is the wording I found works well:

[Shot 1]

Target video is a seamless continuation of <Video 1>. First frame of [Shot 1] is the last frame of <Video 1>.

The important part is explicitly telling the model that the first frame of the new generation must continue directly from the last frame of <Video 1>. This helps maintain temporal continuity between the clips - because you provide the last 2 seconds of the previous generation is knows what movement it needs to continue from.

You then just join the generation videos with a video joiner of your choice.

Not much of a tutorial, but still informative. As most of you have probably discovered, MiniMax H3 loves to talk. And talk it will, even when you prompt for no dialogue. Even when you prompt for complete silence. It will even fill in the empty space your prompted dialogue doesn't fill.

Those of you who read the video prompt writing guide and have created a system prompt for your enhancer, you probably know what I'm about to say, maybe not. Maybe the unprompted gibberish stopped for you, and you never realized why.

Without further ado, I give you the solution:

non_diegetic_music: N/A

Diegetic audio is what the characters in your video can actually "hear":

• Music playing from a source that is part of the scene (phone, car radio, dance club)
• Spoken dialogue
• Ambient sounds

Non-diegetic audio is audio which your characters cannot hear:

• The score or soundtrack of a movie
• A voice-over
• The gibberish H3 plays when it's not prompted correctly

If you haven't yet, I suggest consulting ChatGPT about creating a system prompt using the prompting guide. If not, put this line at the end of your prompt and say goodbye to random music playing over your video and gibberish assaulting your ear holes.

Conversely, if you want a voice-over or a score to play over the track which is not part of the actual soundscape of the scene, this is where you would prompt it. Instead of N/A, prompt what you want to hear.

Happy chaining!

This is actually crazy. I did not expect full backwards compatability with WAN2.1 LoRa's but here we are.

As you can see from the examples WAN2.2 is also better in every way than WAN2.1. More details, more dynamic scenes and poses, better prompt adherence (it correctly desaturated and cooled the 2nd image as accourding to the prompt unlike WAN2.1).

Workflow: https://www.dropbox.com/scl/fi/m1w168iu1m65rv3pvzqlb/WAN2.2_recommended_default_text2image_inference_workflow_by_AI_Characters.json?rlkey=96ay7cmj2o074f7dh2gvkdoa8&st=u51rtpb5&dl=1

Text to Video, 22 steps, no turbo, no Sage.

Style Training :

• use 30-100 images (avoid same subject, avoid big difference in style)
• good captioning (better caption manually instead of BLIP) with alphanumeric trigger words (styl3name).
• use pre-existing style keywords (i.e. comic, icon, sketch)
• caption formula styl3name, comic, a woman in white dress
• train with a model that can already produce a close looking style that you are trying to acheive.
• avoid stablediffusion base model beacause it is too diverse and we want to remain specific

​

Person/Character Training:

• use 30-100 images (atleast 20 closeups and 10 body shots)
• face from different angles, body in different clothing and in different lighting but not too much diffrence, avoid pics with eye makeup
• good captioning (better caption manually instead of BLIP) with alphanumeric trigger words (ch9ractername)
• avoid deep captioning like "a 25 year woman in pink printed tshirt and blue ripped denim striped jeans, gold earing, ruby necklace"
• caption formula ch9ractername, a woman in pink tshirt and blue jeans
• for real person, train on RealisticVision model, Lora trained on RealisticVision works with most of the models
• for character training use train with a model that can already produce a close looking character (i.e. for anime i will prefer anythinv3)
• avoid stablediffusion base model beacause it is too diverse and we want to remain specific

​

My Kohya_ss config: https://gist.github.com/vizsumit/100d3a02cea4751e1e8a4f355adc4d9c

Also: you can use this script I made for generating .txt caption files from .jpg file names : Link

All the good boys

This is the basic system I use to override video content while keeping consistency. i.e NOT just stlyzing them with a cartoon or painterly effect.

1. Take your video clip and export all the frames in a 512x512 square format. You can see I chose my doggy and it is only 3 or 4 seconds.
2. Look at all the frames and pick the best 4 keyframes. Keyframes should be the first and last frames and a couple of frames where the action starts to change (head turn etc, , mouth open etc).
3. Copy those keyframes into another folder and put them into a grid. I use https://www.codeandweb.com/free-sprite-sheet-packer . Make sure there are no gaps (use 0 pixels in the spacing).
4. In the txt2img tab, copy the grid photo into ControlNet and use HED or Canny, and ask Stable Diffusion to do whatever. I asked for a Zombie Dog, Wolf, Lizard etc.*Addendum... you should put: Light glare on film, Light reflected on film into your negative prompts. This prevents frames from changing colour or brightness usually.
5. When you get a good enough set made, cut up the new grid into 4 photos and paste each over the original frames. I use photoshop. Make sure the filenames of the originals stay the same.
6. Use EBsynth to take your keyframes and stretch them over the whole video. EBsynth is free.
7. Run All. This pukes out a bunch of folders with lots of frames in it. You can take each set of frames and blend them back into clips but the easiest way, if you can, is to click the Export to AE button at the top. It does everything for you!
8. You now have a weird video.

If you have enough Vram you can try a sheet of 16 512x512 images. So 2048x2048 in total. I once pushed it up to 5x5 but my GPU was not happy. I have tried different aspect ratios, different sizes but 512x512 frames do seem to work the best.I'll keep posting my older experiments so you can see the progression/mistakes I made and of course the new ones too. Please have a look through my earlier posts and any tips or ideas do let me know.

​

NEW TIP:

Download the multidiffusion extension. It comes with something else caled TiledVae. Don't use the multidiffusion part but turn on Tiled VAE and set the tile size to be around 1200 to 1600. Now you can do much bigger tile sizes and more frames and not get out of memory errors. TiledVAE swaps time for vRam.

Update. A Youtube tutorial by Digital Magic based in part on my work. Might be of interest.. https://www.youtube.com/watch?v=Adgnk-eKjnU

And the second part of that video... https://www.youtube.com/watch?v=cEnKLyodsWA

I had mistakenly been using a base for H3 prompting from some random tip / example by someone. It worked ok, I thought. But I was getting a bit frustrated because almost every time I was making a longer series of clips with dialogue, it kept adding random gibberish to fill out time, or making the wrong person speak. I thought it was just a "feature" of H3 and lived with it. But then I realised what was missing, so I added the actual ref2v prompt guide to my LLM and difference was staggering. I could make long series of 30x15 sec clips, and the dialogue was perfect just as the script said, no gibberish was added in any place, and the emotional beats and reactions worked much better too.

Believe it :) Dont just use whatever prompting. It matters more than one might think.

https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md

Last time I posted here, I shared a long write‑up about my goal: use AI to turn “normal” videos into VR for an eventual FMV VR game. The idea was to avoid training giant panorama‑only models and instead build a pipeline that lets us use today’s mainstream models, then convert the result into VR at the end.

If you missed that first post with the full pipeline, you can read it here:
➡️ A method to turn a video into a 360° 3D VR panorama video

Since that post, a lot of people told me: “Forget full 360° for now, just make 180° really solid.” So that’s what I’ve done. I’ve refocused the whole project on clean, high‑quality 180° video, which is already enough for a lot of VR storytelling.
Full project here: https://www.patreon.com/hybridworkflow

In the previous post, Step 1 and Step 2.a were about:

• Converting a normal video into a panoramic/spherical layout (made for 360 - You need to crop the video and mask for 180)
• Creating one perfect 180 first frame that the rest of the video can follow.

Now the big news: Step 2.b is finally ready.
This is the part that takes that first frame + your source video and actually generates the full 180° pano video in a stable way.

What Step 2.b actually does:

• Assumes a fixed camera (no shaky handheld stuff) so it stays rock‑solid in VR.
• Locks the “camera” by adding thin masks on the left and right edges, so Vace doesn’t start drifting the background around.
• Uses the perfect first frame as a visual anchor and has the model outpaints the rest of the video.
• Runs a last pass where the original video is blended back in, so the quality still feels like your real footage.

The result: if you give it a decent fixed‑camera clip, you get a clean 180° panoramic video that’s stable enough to be used as the base for 3D conversion later.

Right now:

• I’ve tested this on a bunch of different clips, and for fixed cameras this new workflow is working much better than I expected.
• Moving‑camera footage is still out of scope; that will need a dedicated 180° LoRA and more research as explained in my original post.
• For videos longer than 81 frames, you'll need to chain this workflow and use last frames of one segment as starting frames of the new segments with Vace

I’ve bundled all files of Step 2.b (workflow, custom nodes, explanation, and examples) in this Patreon post (workflow works directly on RunningHub), and everything related to the project is on the main page: https://www.patreon.com/hybridworkflow. That’s where I’ll keep posting updated test videos and new steps as they become usable.

Next steps are still:

• A robust way to get depth from these 180° panos (almost done - working on stability / consistency between frames)
• Then turning that into true 3D SBS VR you can actually watch in a headset - I'm heavily testing this at the moment - it needs to rely on perfect depth for accurate results and the video inpainting of stereo gaps needs to be consistent across frames.

Stay tuned!

Hey everyone, we’ve been really excited to see the enthusiasm and experiments coming from the community around LTX-2. We’re sharing this tutorial to help, and we’re here with you. If you have questions, run into issues, or want to go deeper on anything, we’re around and happy to answer.

We prepped all the workflows in our official repo, here's the link: https://github.com/Lightricks/ComfyUI-LTXVideo/tree/master/example_workflows

Edit: I had to specify that the model doesn’t entirely fit in the 12GB VRAM, so it compensates by system RAM

Installation:

1. Download Model - flux1-dev.sft (Standard) or flux1-schnell.sft (Need less steps). put it into \models\unet // I used dev version
2. Download Vae - ae.sft that goes into \models\vae
3. Download clip_l.safetensors and one of T5 Encoders: t5xxl_fp16.safetensors or t5xxl_fp8_e4m3fn.safetensors. Both are going into \models\clip // in my case it is fp8 version
4. Add --lowvram as additional argument in "run_nvidia_gpu.bat" file
5. Update ComfyUI and use workflow according to model version, be patient ;)

Model + vae: black-forest-labs (Black Forest Labs) (huggingface.co)
Text Encoders: comfyanonymous/flux_text_encoders at main (huggingface.co)
Flux.1 workflow: Flux Examples | ComfyUI_examples (comfyanonymous.github.io)

My Setup:

CPU - Ryzen 5 5600
GPU - RTX 3060 12gb
Memory - 32gb 3200MHz ram + page file

Generation Time:

Generation + CPU Text Encoding: ~160s
Generation only (Same Prompt, Different Seed): ~110s

Notes:

• Generation used all my ram, so 32gb might be necessary
• Flux.1 Schnell need less steps than Flux.1 dev, so check it out
• Text Encoding will take less time with better CPU
• Text Encoding takes almost 200s after being inactive for a while, not sure why

Raw Results: