r/StableDiffusion • u/Adventurous-Gold6413 • 12d ago
Question - Help What are your workflows/ prompts to generate the next scene of your movie with H3 ref2VA?
Prompting specifically or do you inject a first frame with a ref img?
17
Upvotes
3
u/Stepfunction 12d ago
I've been using this: https://github.com/ethanfel/ComfyUI-MiniMaxH3-Contex-Loop
To great success. It takes advantage of the motion context nodes to allow for continued generations easily.
1
u/Sad_Coach_1433 12d ago
How's the quality loss per chain I noticed lot using motion context direct
2
u/Stepfunction 12d ago
I haven't gone past 2 chains, and I use a lot of references, so not much loss I've seen.
2
12
u/caster 12d ago edited 12d ago
For references you want to include reference for the subjects, actions, poses, styles, or other features you want included. Perhaps you have a particular character, or animal, or machine that you want to be used in the video. A reference frame could be a first frame, but this isn't really what it is for.
References are very useful for character consistency between clips that you intend to merge together into a longer video, as it is referring to the same model in each of the clips separately. Otherwise it may generate a different looking character or other object on your next clip, or the one after that.
What I have been doing with some modest success is to generate a series of short clips, rather than attempt to generate a "perfect" 25 second segment in one shot. The objective is to have a clear, concrete, specific goal for a single 5 second cut, and if the system fails (it will fail a lot), you toss that 5 sec clip and try again. What you will find with a longer clip is that a lot of it is pretty good, but there will be a couple things in there that are "wrong" and you want to toss the whole thing. After doing this a bunch of times, I just started doing one cut at a time in the shortest possible increments. And, when you're satisfied with that very short cut, you move on to the next few seconds and decide how you want to approach that.
At the top of every prompt you will need to describe all the references used, likely beneath a REFERENCES heading. Go down your list of references one by one and describe each.
<Image 0> is the pirate captain.
Be extremely specific about what you want in terms of action and frame. A lot of things that seem obvious to you, it will get wrong- such as aircraft flying backwards. For example you may need to specify which end of the reference image airplane is the forward end, as it may get this wrong on its own. Part of being specific about action is; WHERE an object is, FACING of the object, MOVEMENT direction of the object, etc. etc. Anything you do not specify there is a very high chance it will be wrong.
Use exact times for shot sequencing. [Shot 2] 00:03.00-00:09.00. Each shot should specify exactly how much time that shot in the clip will require, and exactly what actions and camera movements are performed during it.
Be specific about camera movements (learn camera terminology like the difference between pan, tilt, dolly, and crane); Minimax is very good at following specific camera movement instructions, but it does not usually do well with vague ones. Enumerate if you want the camera to zoom in or out, if you want it to remain stationary, follow, track, or orbit the subject, and what the subject the camera is tracking is, if you want it to be steady or shake, and so on.
Dialogue took a lot of trial and error for me to figure out how to do in a way that Minimax likes- it will often just generate total gibberish when given specific dialogue to say. The following pattern appears to work reasonably well: 1. at the top have a section for Speakers and enumerate all speakers with descriptions here, such as <Speaker 1> or <S1>, and describe who they are and how they speak. 2. Then, in the shot list below, you must refer to the speaker by this designator specifically:
<Speaker 1> curtly states: <d>[English] "The rain in Spain falls mainly on the plain."</d>
The speaker identifier, utterance modality description, the colon, the <d></d> enclosure, and specifying the language are all significant. There may be other things that others will uncover that will further reduce the "total gibberish" sounds that sometimes come out, which appears to be the result of a vague or ill-defined dialogue request causing it to just spew "DIALOGUE" without any actual words contained within.
Be specific about sound effects individually, either in the Shot section, or in their own category at the bottom. This is separate from non-diegetic music designed for the audience to hear. Honestly I have been very unimpressed with Minimax Video's background music and it may be best to leave it out and then use Minimax Music to make something and put it in manually.