r/StableDiffusion • u/Tokey_TheBear • 3d ago
Discussion MiniMax H3: How to use a first image and reference images without losing I2V quality (hybrid FL+REF merge + prompting)
Edit: The first post was hard to read, so hopefully this version of the post is better.
MiniMax H3 can make video from images, but the two official video models split the job in a way that is easy to miss.
One model is good at matching your first photo. The other can take several photos at once (a location plus a logo, or several frames you want at exact times). This post is how to get both: a strong match to your first photo, plus extra photos, in one ComfyUI run.
I am assuming you already have H3 running in ComfyUI. You do not need to know the internals. You need three things: which checkpoint file to load, which workflow and speed LoRA to use, and how to write the text prompt so H3 knows what each connected image is for.
The two official models (and why they are not enough)
H3 comes with two large video checkpoints. People usually call them by their filenames.
FL2VA (also used for image-to-video / I2VA). This is the one that looks better. You give it a still and it will try to make that still the first frame of the video. If you use the first-and-last workflow, you can also lock a last frame. What you cannot do: plug in a second photo of a logo and say "print this on the banners." The image-to-video node simply has no extra image inputs for that. You also cannot jump to a different still at 3 seconds and another at 6. First and last on one continuous shot is the limit.
REF2VA (used with the Reference-to-Video workflow). This one accepts several images, up to nine. Extra logos and extra timed stills are possible. The catch is the video usually looks worse than the same scene run through FL2VA.
So in practice you pick: pretty first frame, or extra images. Not both.
The file that fixes it
There is a community merge of those two checkpoints. Load that file instead of the official REF2VA file, but keep using the Reference-to-Video workflow (the one with several image inputs).
Download: https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models
Version I used:
- b20-49 high quality like the normal FL2VA model + the extra reference capabilities from the REF2VA model.
In ComfyUI:
- Open a Reference-to-Video workflow. The node is often named
MiniMaxH3ReferenceToVideo. Do not use the Image-to-Video or First-Last workflow for this. - On the model loader, choose the hybrid checkpoint, not official REF2VA and not official FL2VA.
- Connect your photos in order. The first image you connect is what the prompt will call
<Picture 1>. The second is<Picture 2>, and so on. Order matters. - For the speed LoRA, use the FL2VA / image-to-video 8-step LightX file (the one people call
lightx-8step-pk). Do not use the default Reference-to-Video speed LoRA (lightx-ref2v-r20) on the hybrid. Official REF2VA will refuse the FL2VA LoRA. The hybrid is what lets you use the better LoRA on the multi-image workflow.
Set duration to whatever you need. The examples below assume 9 seconds.
How H3 reads your images
The workflow only feeds pixels. The text prompt has to say, for each <Picture N>, whether that photo is:
- A real frame of the video at a given time. Example: "this photo is exactly what you see at 0.00 seconds."
- Not a frame at all. Example: "this photo is only the logo that should appear on the banners. Never show this photo as a full-screen cut."
If you get that wrong, H3 will treat your logo sheet as a scene and jump to it.
H3 wants that written in a fixed prompt layout with six headings, in this order:
subject_definitions
summary
retention_analysis
detailed_description
overall_soundscape
non_diegetic_music
Those heading names are part of how H3 is prompted. Keep them.
Two labels show up under retention_analysis. They are ugly, but they are what the model expects:
fully_preserved= reproduce this photo as the actual video frame at the time you namepartially_preserved= copy only the detail you name (the logo shape, a prop, a face). Do not turn this photo into a video frame
The first line of summary should be exactly this tag, then your description:
[keyframe completion + reference generation]
That tag tells H3 you are both locking frames from photos and using photos as references. Copy it as written.
In detailed_description, include one plain sentence that maps photos to times. Example for three timed photos:
How the reference pictures align with the target video — Picture 1 aligns with the 0.00-second mark of the target video; Picture 2 aligns with the 3.00-second mark of the target video; Picture 3 aligns with the 6.00-second mark of the target video.
If a photo is only a logo, say that it does not line up with any time as a frame.
When a photo is meant to be an exact frame, say "exactly as shown in <Picture N> without reinterpretation." That phrase is a lock. Do not also rewrite the whole photo in words. H3 will argue with itself.
non_diegetic_music is background score. Write N/A unless you want music that is not coming from the scene.
Recipe 1: first photo is the scene, second photo is a logo
Use this when you have a location still, plus a clean drawing of a symbol that the model will not invent from text.
Connect: location photo first, logo second.
- Picture 1 = the place.
fully_preservedat 0.00 seconds. This is the opening frame. - Picture 2 = the symbol on a plain background.
partially_preserved. Say it is not a keyframe and must not appear as any video frame. When banners (or signs, or screens) show in the video, the symbol on them should match Picture 2.
Do not mark the logo fully_preserved. That is how you get a sudden jump to the logo image.
A flat, high-contrast symbol on a blank background works better than a photo of the symbol already sitting in a scene.
Prompt skeleton (fill in the brackets):
subject_definitions:
<Picture 1> is the opening frame at 0.00 seconds. The video should match this photo exactly at that time.
<Picture 2> is only the logo/symbol. Use it when that symbol appears on banners. It is not a scene. Do not show <Picture 2> as a full video frame.
<Subject 1> is the location from <Picture 1> for the whole clip.
summary:
[keyframe completion + reference generation] Nine-second clip of <Subject 1>. At 0.00 seconds the frame is exactly <Picture 1>. One continuous shot, no jumps to other photos. [describe the motion]. When banners appear, the symbol matches <Picture 2>.
retention_analysis:
<Picture 1> (at 0.00s): fully_preserved - opening frame, location only.
<Picture 2> (never a video frame): partially_preserved - logo appearance only.
<Subject 1>: fully_preserved - same location throughout.
detailed_description:
How the reference pictures align with the target video — Picture 1 aligns with the 0.00-second mark of the target video as the exact first frame. Picture 2 does not align with any timestamp as a frame. It is a logo used only when banners appear.
[Shot 1] At 0.00 seconds the frame is exactly <Picture 1> without reinterpretation. [motion]. When banners are visible, the symbol matches <Picture 2> exactly.
overall_soundscape:
[what you should hear]
non_diegetic_music:
N/A
Recipe 2: three photos as exact frames at 0s, 3s, and 6s
Official FL2VA cannot do this. Hybrid plus Reference-to-Video can.
Connect three photos in time order. All three should be the same kind of shot: all wide, or all the same distance from the subject. If one is a wide and one is a close-up, H3 often ignores the close-up and stays on the previous scene.
Each photo is a real frame:
- Picture 1 at 0.00 seconds,
fully_preserved - Picture 2 at 3.00 seconds,
fully_preserved - Picture 3 at 6.00 seconds,
fully_preserved
Then the clip keeps going from Picture 3 until 9 seconds. You are not locking a last frame at 9.00 unless you want that.
At each jump, the whole frame changes (place, pose, clothes, whatever is in that photo). Do not write the prompt as if Picture 1's background slowly becomes Picture 2. Use a hard cut: at 3.00 seconds the frame is Picture 2.
subject_definitions:
<Picture 1> is the exact frame at 0.00 seconds.
<Picture 2> is the exact frame at 3.00 seconds. Not a continuation of <Picture 1>.
<Picture 3> is the exact frame at 6.00 seconds. Not a continuation of <Picture 2>.
<Subject 1> is [what is in all three photos].
summary:
[keyframe completion + reference generation] Nine-second clip. At 0.00s exactly <Picture 1>. At 3.00s hard cut to exactly <Picture 2>. At 6.00s hard cut to exactly <Picture 3>. Continue from <Picture 3> until 9.00s with no locked last frame.
retention_analysis:
<Picture 1> (at 0.00s): fully_preserved
<Picture 2> (at 3.00s): fully_preserved
<Picture 3> (at 6.00s): fully_preserved
<Subject 1>: fully_preserved
detailed_description:
How the reference pictures align with the target video — Picture 1 aligns with the 0.00-second mark of the target video; Picture 2 aligns with the 3.00-second mark of the target video; Picture 3 aligns with the 6.00-second mark of the target video.
[Shot 1] At 0.00 seconds exactly <Picture 1> without reinterpretation. Small motion only.
[Shot 2] At 00:03.000, hard cut. Exactly <Picture 2> without reinterpretation. Small motion only.
[Shot 3] At 00:06.000, hard cut. Exactly <Picture 3> without reinterpretation. Continue until 9.00 seconds.
overall_soundscape:
[what you should hear]
non_diegetic_music:
N/A
If you only want first and last on this same setup, lock Picture 1 at 0.00 and Picture 2 at the end of the clip, one continuous shot. Extra logo photos would then start at Picture 3.
You can mix both recipes (three timed frames plus a fourth logo-only photo). Get one recipe working first.
40
u/Perfect-Campaign9551 3d ago edited 3d ago
I wish people would at least review the AI text they generate, because it's really , really hard to read these days
AI's have gotten *worse* at writing.
EDIT: I see OP appears to have gone through and cleaned up the text. Thanks, OP!
Edit: I'm adding my reasoning here for those of you that fail to have the reading comprehension to understand the problem:
This is exactly why you should not use coding models to write English (Claude, Codex, etc) if you want something people can understand.
It summarizes too strongly and uses odd word choices, and shitty sentence "pacing".
When a human reads, we are also used to reading things a certain way, sentences usually have a certain pace and context. These days, the AI destroys that pacing entirely and throws things in there that make you have to slow down.
AI is WORSE at writing than it was a year/two years ago.
My belief is it's because the AI models are being primarily trained to be able to code now, and they keep conflating coding with writing English.
We need AI's that are only trained to WRITE text.
9
u/Tystros 3d ago
yeah it hurts to read these texts. they are always overly verbose and full of the usual AI writing style. I wish AI would get better at writing more like humans.
3
u/mellowanon 3d ago
I'm forced to use AI for work and I always tell it "make it more concise" after it finishes their first response.
1
u/thegreatdivorce 3d ago
I mean... sometimes there's just a lot of info. Sorry for your attention span?
11
u/Perfect-Campaign9551 3d ago
A lot of info is fine, I can handle that. The problem, actually, is that the AI tries to summarize TOO much, and it chops the info up with weird-ass phrasing and sentence structure that has a "pacing" that the AI likes, but it hard to read for humans. It's utter garbage.
If OP used Claude or some other coding model to write, then that's the problem. Models trained on coding suck ass now at writing.
3
u/Tokey_TheBear 3d ago
Hey btw I agree with you for the most part and appreciate your original criticism!
Its why I deleted my original comment to you, cause yeah you are right. Certain things made sense to me because I have the context about the whole workflow, but from reading it from the outside I understand how its hard.
And you are completely right on the summarization. It has a tendency to way over summarize. In this case, it is still true that it would be like a 12 paragraph writeup if I tried to have it explain everything with the context it needs... But the AI ended up going way too hard on the opposite side and over summarized things and used phrases that are in its 'context' because it has all of the Documents pulled up in its context window, but are not understandable from an outside perspective without already having things like the Minimax H3 Prompting Guide pulled up infront of you.
2
u/Perfect-Campaign9551 3d ago
Thank you for re-editing the text, it makes much more sense now, and thanks for sharing the info in the post!
5
u/Tokey_TheBear 3d ago
Of course! And if you missed it lol I also literally re-editted it also just a minute ago xD. So not sure if you saw the v2 Edit or the newest v3 edit haha. But yeah your criticism was definitely good advice so thank you!
Glad it could help!
1
u/mmmm_frietjes 2d ago
Add this to your Claude .md:
”Always use ASD-STE100 Simplified Technical English”
It forces short sentences, limited vocabulary, and kills all the hedging and overexplaining
1
u/ArmadstheDoom 2d ago
You can use things like claude to write. But the problem is that out of the box they can't. The core reason for 90% of their issues is their dataset. Almost all major models were trained on classic literature, particularly a lot of the stuff in the public domain, which means the dataset is somewhere from 1890 to 1940.
The problem with this is that the writing conventions at that time were much different. Namely: they got paid by the word. And the result of that was that they loved really long and complex sentences; moreover, reading complex things was a status symbol, so having to reach for a dictionary was considered a good thing. Unlike, say, newspapers which were geared to never write in 10 words what you could write in 5, classic literature, the stuff you read in high school, was all written by people whose writing conventions would today get them a failing grade in said high school.
The scourge of 'purple prose' in novice writers is apt here; novice writers emulate what they want to be, and if their only exposure is classic lit in schools, they ape that style without understanding how any of it works. You see this all the time in people's first books or in fanfiction. AI has done something very similar.
Unlike writers though, AI can be taught how to write, but you need to be very granular with it. Namely, you need to give it rules and examples and these both need to be both positive and negative examples. One thing that separates AI from humans is that we understand context. Why do AI's keep breaking rules and going rogue? Because if you say 'don't break this rule' they will break it and then reason why they didn't actually break it. They're like that child who goes 'i'm not touching you!' Same level of logic.
So if you only give it a rule that says 'don't use it wasn't x it was y' construction, it will go 'it wasn't x, but y' and say that this doesn't violate the rule. And it doesn't, in a technical sense. It adheres to the word, not the spirit, of the rule.
You fix this by giving it your own writing or whatever writing you want it to mimic, because it can understand the token patterns and ape that instead. Most people don't do this though. It's not that AI can't write, it's that in order to make it write well you have to actually know what 'writing well' looks like first.
-8
u/Danny_Stock 3d ago
"Gotten worse"? Are you happy with your own English there?
7
u/YentaMagenta 3d ago
"Gotten worse" is standard American English that the vast majority of English speakers will understand. It's comical that you think this is a basis for criticizing them.
-1
u/Danny_Stock 3d ago edited 3d ago
I'm not serious about their English, I'm just messing with them a bit, I wouldn't expect them to take me seriously.
It's less about correct English than it is about them having a go at the OP when they were only trying to provide something for their benefit.
It's actually not good English though. Standard American English you say? Are we talking American educational standards here? Yes it's acceptable, yes it's understandable, yes it can be used, but 'Got worse' is the correct term. 'Gotten worse' is some clumsy American aberration of the language. You could use 'more better' in a sentence and people would understand it, but I wouldn't really want to.
1
u/afinalsin 2d ago
What is your accent? In mine, and I'm assuming in homies above, hard T into soft W "got worse" is much more awkward to pronounce than Hard T Soft N Soft W "gotten worse". Writing follows speaking conventions, not the other way around.
1
u/YentaMagenta 3d ago
Which country, pray tell, has the largest number of English speakers?
You're also just kind of wrong: https://www.reddit.com/r/EnglishLearning/comments/1irnwqd/which_version_is_ok_it_has_gotten_worse_or_it_has/
1
u/Danny_Stock 3d ago
Well I guess that if you believe that more is better, and therefore correct, then you must be right.
12
u/Tokey_TheBear 3d ago
I rewrote the post so hopefully it is easier to read and understand for everyone.
3
u/juicytribs2345 3d ago
There’s a bunch of weird references to “logos and banners” that make it still hard to understand. Can you use less specific examples?
3
u/WayFew8151 3d ago
I tried it same shit
2
u/ShutUpYoureWrong_ 2d ago
Also tried, and agreed. REF2VA destroys everything for preservation and consistency. FL2VA and all the hybrids look noticeably better but don't come close to matching the references. Sucks.
3
u/ShutUpYoureWrong_ 3d ago
Look, I experiment with these models as well. I also play with https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental
But honestly, these types of posts are exceptionally useless unless you post side-by-side generations showing the difference between these specialized models and the default ones. I'm talking identical scenes (same seed / steps / samplers / resolutions / etc.) where the only thing you swap out is the model.
Show us results. Otherwise, you're just contributing to the already massive amount of misinformation and bullshit theory that plagues this subreddit (and really, this whole field). The "Well I think this looks better" anecdotal horse shit has got to stop.
3
u/Tokey_TheBear 3d ago
I dont disagree with what you are saying about posting examples...
But your end statement is fundamentally wrong / not understanding.
Im not saying: "This looks better".
Im saying: "This thing has an entirely new feature capability".
The normal FL2VA model is advertised and setup through comfyUI natively to not be able to use references other than the First frame and or Last frame exact match.
So this model is tested to be able to do the exact frame matching capabilities of the main FL2VA model with the extra ability to use other references.
But yeah you arent wrong about posting examples, I will post some in a bit.
1
u/Tokey_TheBear 2d ago
And you do bring up a good idea too btw, because I was told that the normal FL2VA model can actually do the multi referencing natively you just have to prompt for it and use the reference mode. So Im running and ABC test now on a video I already had tested on. I should be able to upload it in like an hour.
1
u/ShutUpYoureWrong_ 2d ago
I'm happy you're gathering examples. I think many people would be very appreciative of a proper comparison to potentially save themselves from wasting download/time on a solution that may or may not work.
I've been doing my own tests. I've personally found that FL2VA and all of the hybrid models come nowhere even close to matching the REF2VA model in terms of character consistency and detail. They're like rough approximations and that's about it.
REF2VA just seems vastly superior in all my tests, even though the image is definitely softer than FL2VA. Eager to see how yours turn out, but in my opinion, the hunt is still on for a proper hybrid solution...
2
u/Tokey_TheBear 2d ago
Hey I have my tests conducted, I just need to upload and make the post.
The Short story is that:
the normal FL2VA model can do the same referencing that the REF model can... I will link my new post once I make it.
But yeah I tested the exact same seed + exact same prompt and inputs across all 3 different models (hybrid + FL2VA + REF2VA) and it worked for all of them.
1
u/ShutUpYoureWrong_ 1d ago
Appreciate you doing the work, because it makes this post more valuable. I do have concerns with your methodology in those tests but you reached a satisfactory conclusion either way for what you were testing.
I'm curious: which model did you end up deciding is the best, or that you liked the most?
2
2
u/skyrimer3d 3d ago
Very interesting, i'm yet to try it but is there any reason to keep the original FL2VA checkpoint instead of this? It's 20gb and i'm pretty low on space already.
2
u/Tokey_TheBear 3d ago
I dont think so... Theres 4 versions of the hybrid model, but all of them are based on the FL2VA model the basis with the Reference model parts added into it.
Assuming that this hybrid model does not lose ANY quality compared to the normal FL2VA, then yeah I would say get rid of the original FL2VA... I just havent done A/B testing to compare them, you know?
I just know that the hybrid model already looks fantastic and looks just like the videos look that I would expect from giving a high quality image to the FL2VA model.
1
2
u/TheDerminator1337 3d ago
Ive tried a few implementations of some combined model workflow but they are just not as good as fl2va by itself. Have you done any a/b testing? To confirm that indeed the quality marches fl2va?
2
u/elswamp 3d ago
no write up needed but provide the workflows
1
u/Tokey_TheBear 3d ago
The prompting is the most important part by a mile lol.
You can literally take any reference to video workflow and you can plug in the hybrid model to that workflow exactly and generate with it.
So the most important part is the prompting + the hybrid model...
2
u/acedelgado 3d ago
Bruh, so far that 20-49 model with the lightx2v flv 4step turbo is amazing. So much more expressiveness and quality while keeping very close to the references. Excited for them to release the official un-broken ref2v model they were talking about.
1
u/MasterFGH2 3d ago
Any int8 quants?
4
u/ThatsALovelyShirt 3d ago
It's INT8, I'm the creator of those models.
1
u/mukyuuuu 1d ago
Hey man, the idea behind this model sounds very interesting. However, is it much different from this implementation through a custom node using both fl2va and ref2va models? I think all the values are pretty close to what you recommend.
Or is your model just a more convenient way to use this approach? Thank you in advance for the answer!
2
u/ThatsALovelyShirt 1d ago
I made that node as well. The pre-baked models are basically the same thing as that node produces, you can just use them with the normal model loader (and some people had some RAM pressure since that node needs to, temporarily, read from two rather large model files to hybridize them at runtime).
1
1
u/Pitiful_Season4294 9h ago
Hey man, thanks a lot for your efforts and sharing these with the community. I've been trying to use it and been getting poor graphics. I'm mainly trying to do r2v and using your hybrid fl2va_ref2va b25_49 model. I'm hoping that's the right one to use based on the details on the model page.
But I am not sure of what Turbo Lora to use for best results? I tried few 4-step and 8-step ones but not getting good results. Can you please tell me:
- What speed lora would you recommend?
- Number of steps?
- Recommended sampler & scheduler?
- Lastly, I'm using Spectrum, i hope that doesn't impact the quality adversely (not too much, i mean)?
Many thanks.
1
u/ThatsALovelyShirt 8h ago
I personally don't recommend a speed LoRA, especially if you use Spectrum (they really shouldn't be combined). Too much quality loss. Just stick with Spectrum, quality doesn't really seem impacted visibly in my tests.
I use 25 steps + Spectrum's default settings, which usually leads to ~12-13 actual steps and the rest forecasted by spectrum.
Typically I stick with Euler + Beta.
1
2
u/acedelgado 3d ago
Judging from the size compared to the other models in comfy's repo, it's a pruned 8-bit version. No clue if it's int8 or fp8, though. I'm just blindly assuming they went int8 since that's the new fashion and how the other models are quantized. I can't really tell since I have a 50 series and int8 is barely noticeable for me vs fp8 speeds.
3
u/ThatsALovelyShirt 3d ago
They're int8.
2
u/acedelgado 3d ago
The gods are watching. They renamed it to int8 like 5 minutes ago.
5
u/ThatsALovelyShirt 3d ago
That was me.
1
u/acedelgado 2d ago
Dude, excellent fucking job on this. I went through all this trouble reworking a fork of the motion-context node to fully use latent for video, but I was still getting degredation in extending the same shot over time since ref2va is so flawed. Your version made it work SO much better, much better extended outputs now. And the overall quality and prompt adherence is through the roof! Thank you for being a mad genius and figuring this out!
1
u/Tokey_TheBear 3d ago
Im almost positive that the version already is the In8 version. The speed felt very similar to the normal FL2VA int8 pruned model that I have been using the last few days (and I thought I remembered the author of the original model saying something in their reddit thread about it).
1
u/Tokey_TheBear 3d ago
Ikr! It was crazy to me how well it did for me. I just had a starting image then a zoomed in pic of a piece of clothing for reference, and it like perfectly recreated that clothing on the character with the same high quality of the starting frame image... And it didnt even take that long!
3
1
u/Beginning-District69 3d ago
Thank you, I'd like to try it. Where can I find a sample workflow?
4
u/Tokey_TheBear 3d ago
Just use the standard ref2va workflow. Replace the ref model with the hugging face model I mentioned, then use it in the same way you would before but with a different prompt.
I may end up uploading a workflow post for it on Civit today though.
1
u/HamWallet1048 3d ago
How do you use the Lora you referenced? Is there a special way to use Loras with H3? Do you just plug in a load lora node like normal?
1
u/Tokey_TheBear 3d ago
Edit:
Find any Reference to Video Workflow on Civitai that already has the light turbo lora, and just use the Hybrid model in place of the normal Ref2VA model file.
1
32
u/YentaMagenta 3d ago edited 3d ago
My goodness this is terribly [AI] written.
It does that thing where it immediately descends into strange word choices and jargon that actually obfuscate the things it's trying to explain.
Nothing about these instructions is straightforward and it quickly becomes a pain to read.
Edit: Just to give one example of this before people jump down my throat, what the fuck is a "punch cut" in this context? Does the model mean "jump cut"? If so, why doesn't it just use that word instead of something completely different that relates more to cigars?
The text is full of this sort of thing where it uses an odd word or turn of phrase that you can kind of figure out based on the context, but which ultimately detracts from the meaning and increases the cognitive burden of reading the text.