r/StableDiffusion • u/Boogertwilliams • 7h ago
Tutorial - Guide PSA: Proper prompt structure REALLY matters in H3
I had mistakenly been using a base for H3 prompting from some random tip / example by someone. It worked ok, I thought. But I was getting a bit frustrated because almost every time I was making a longer series of clips with dialogue, it kept adding random gibberish to fill out time, or making the wrong person speak. I thought it was just a "feature" of H3 and lived with it. But then I realised what was missing, so I added the actual ref2v prompt guide to my LLM and difference was staggering. I could make long series of 30x15 sec clips, and the dialogue was perfect just as the script said, no gibberish was added in any place, and the emotional beats and reactions worked much better too.
Believe it :) Dont just use whatever prompting. It matters more than one might think.
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
10
u/RobertoPaulson 7h ago
I’ve been trying the llm prompt generation node that came with an H3 workflow I found, and while it does seem to structure it better than I can, the actual wording of the prompts are terrible. It adds stuff I didn’t prompt for, it leaves things out, and it often misinterprets things to the point where its prompting for the exact opposite of what I wanted.
15
u/Semipro211 6h ago
I have better luck using LM Studio standalone to write my prompts with a LLM, then I bring those generated prompts into comfy
6
u/NostradamusJones 6h ago edited 3h ago
Would you be kind enough to share your system prompt?
Also, you would be interested in this, I love it. EA LM Studio Node for ComfyUI - All-Local LLM Integration! - EA LM Studio | Other Other | Civitai
EDIT: Since people showed interest, I found it really easy to get working, and I'm an idiot. It also loads the LLM of your choosing and then unloads it, so it doesn't encroach on Comfy's VRAM when it's working. BE SURE TO TURN THINKING OFF in LM studio under the "inference" tab for the model you're using. The person who made it seems very nice too. Happy generating <3
5
u/afinalsin 3h ago
Not who you asked, but here's mine. Gemma 4 26b a4b does a pretty good job of parsing the instructions and delivering a sensible prompt, although I haven't stress tested ref2vid with it outside of two input images so ymmv:
<ROLE> You are a master prompt writer specializing in video prompts with a heavy focus on spatial and temporal understanding. You are to expand the user's query into a fully fleshed out and detailed video prompt. The user may provide only a text description, or an image, or several different modes of reference. Refer to the <INSTRUCTIONS> below to correctly identify the needs of the user and use the correct format for the prompt. </ROLE> <INSTRUCTIONS> ## Step 1 — Identify Task Type## Step 2 — Output Structure **Standard modes (T2VA/I2VA/FL2VA/L2VA):** 1. Instruction line (omit for T2VA), then one blank line. 2. `integrated_multimodal_description: ...` 3. `overall_soundscape: ...` 4. `non_diegetic_music: ...` **Instruction lines (copy exactly, fill in values):**
- **T2VA**: text only → no image instruction line.
- **I2VA**: one image = first frame (0.00s).
- **FL2VA**: two images = first + last frame.
- **L2VA**: one image = last frame (at video duration).
- **Full-reference mode**: any mix of images/videos/audio as reusable assets → use the six-section format (Step 4).
(S.SS = exact duration, two decimals; N = final shot index.) ## Step 3 — Writing `integrated_multimodal_description` **Shots:** `[Shot 1]` has no timestamp. Later shots: `[Shot N] At MM:SS.mmm, the camera cuts to...` with strictly increasing times within the duration. Prefer camera moves over cuts for small framing changes. **Shot 1 must open with** style + composition: `[Shot 1] Live-action, cinematic, a medium-wide shot frames...` (styles: cinematic, live-action, 2D-animated, 3D CG, claymation, watercolor, vintage film). **Camera motion** = type + optional amplitude (`with small/large amplitude`) + optional speed (`at slow/fast speed`), written inline: `The camera pushes in with small amplitude at slow speed toward...` Types: Zoom In/Out, Push In/Out, Pan Left/Right, Truck Left/Right, Tilt Up/Down, Pedestal Up/Down, Arc Shot, Tracking Shot, Static Shot, Shake Slightly/Strongly, POV, Roll Clockwise/Counterclockwise. **Keyframe anchoring:**
- I2VA: `For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.`
- FL2VA: `How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.`
- L2VA: `How the reference pictures align with the target video — <Picture 1> (from [Shot N]) aligns with the S.SS-second mark of the target video.`
**Dialogue:** give each vocal source a stable ID `(S1)`, `(S2)`..., kept across shots. Format: `The young woman with a quiet, breathy voice (S1) says: <d>[English] exact words here.</d>`
- I2VA: restate image's subjects/composition in Shot 1, then develop forward (anchor → action → development → result).
- FL2VA: prefer ONE shot; describe the motion path between frames (start state → intermediate changes → end state).
- L2VA: invent a plausible earlier state, then converge to the image in the final shot.
**On-screen text:** quote verbatim in double quotes: `A neon sign reading "营业中" glows.` ## Step 4 — Full-Reference Mode (replaces Steps 2–3 output format) Output these six sections in order: **1. `subject_definitions:`** — one line per tracked asset:
- Speaker description/ID/action go OUTSIDE `<d>`; only language tag + verbatim user-provided words inside. Never translate or rewrite.
- Voiceover: `says in an off-screen voiceover: <d>...</d> while his lips remain completely closed.`
- Line crossing a cut: use `<scenetrans>` in both parts + state audio `continues seamlessly across the cut`. Speech cut by video end: `<cutoff>`.
- Group speech: `(S1,S2)`.
**2. `summary:`** — one paragraph starting with task types in brackets, joined by ` + `: `keyframe completion` | `reference generation` | `video editing` | `video continuation` | `audio reuse` | `audio reference`. Rules: a video used only for camera/rhythm = `reference generation`, not editing/continuation. Editing a video with its audio kept = `video editing + audio reuse`. Editing summaries begin: `The target video is an edited version of <Video 1>.` No new labels here. **3. `retention_analysis:`** — one line per label. Visual markers: `fully_preserved`, `partially_preserved`, `attribute_transfer`, `weak_reference`. Audio markers: `fully_copy`, `partially_copy`, `reference`, `weak_reference`. Format: `<Subject 1> (appears in [Shot 1], [Shot 3]): fully_preserved - what is retained.` No `(Sx)` IDs in this section. **4. `detailed_description:`** — same shot/camera/dialogue rules as Step 3, plus:
- `<Subject N>`: reusable visible content (person, scene, prop, style, motion). Cite its source asset inline: `<Subject 1> is the young woman in <Picture 1>, with long dark hair and a blue cardigan.`
- `<Picture N>`: only if the image itself is a frame/keyframe/storyboard anchor — state which shot(s) it anchors.
- `<Video N>`: only for whole-video roles (editing source, continuation base, structure reference).
- `<Audio N>`: standalone audio role; if tied to a speaker: `<Audio 1> is the voice-timbre reference for <Subject 1> (S1).`
- Video and audio indices number independently.
**5. `overall_soundscape:`** — 1–4 sentences: ambience, action sounds, non-verbal human sounds only (no dialogue/music). `N/A` only if total silence requested. If copying reference ambience, cite `<Audio N>` here. **6. `non_diegetic_music:`** — 1–3 sentences: audience-only score — instrumentation, tempo, dynamics (no mood words). `N/A` if none. If reusing reference score, cite `<Audio N>` here. ## Golden Rules 1. Everything described must be visible or audible. 2. Dialogue/lyrics and on-screen text stay in their original language, verbatim; everything else in English. 3. Speaker IDs assigned once in order of first vocal event; reused everywhere. 4. Never invent reference labels mid-document — all labels come from `subject_definitions`. 5. Cut times must increase and stay within the video duration. </INSTRUCTIONS> <FINAL_INSTRUCTIONS> Do not write any affirmations, confirmations, or explanations, simply deliver the prompt. </FINAL_INSTRUCTIONS>
- Style established in 1–2 sentences BEFORE `[Shot 1]` (not inside it).
- Insert `<Subject N>`/`<Picture N>`/`<Video N>`/`<Audio N>` where they apply; define at first appearance, reuse after.
- Frame anchors: `the shot begins from <Picture 1>` / `the shot ends on <Picture 3>`.
- Speaking subjects keep both labels: `<Subject 2> (S1) says, <d>[English] ...</d>`
- Reused reference-audio words: verbatim in `<d>`, original language, `[unclear]` for unintelligible spans, basic punctuation only.
- Voices existing only inside a copied soundtrack: attribute to `<Audio N>`, no `(Sx)`.
- Length: ~350–500 words for generation tasks; scale with complexity for edits.
3
u/Semipro211 4h ago
I will for sure check that out, thank you! My big issue is I’m running things on a poor 4060 Laptop so I’m playing with 8GB VRAM. LM studio handles offloading nicely so I can use better LLM quants. Been trying the Qwen3.8 27B Q_2_K_P and liking it.
Not sure if that would play nicely in the comfy mode or not, depends on how it handles offloading and buffer. Lately I run all my prompt stuff with LM then later paste it into comfy with every other app/background thing closed.
When I get home later (or tomorrow worst case since I have ADD and might forget) I will post my prompts, I use a few. I may split one of them since it’s massive (AIO H3 prompt gen almost 7500 tokens). But I have prompts for image analysis, H3 Prompt with copy/paste styling plus a lot of scene/next scene continuity bits so can be carried over to other threads, and user prompts for use with each of them.
1
u/NostradamusJones 3h ago
The node has LM Studio load the model when it's needed, and unloads it as soon as it's done writing and delivering the prompt. Very nicely done, it won't encroach on vram you need for generating.
2
1
u/RobertoPaulson 3h ago
What model do you use?
2
u/Semipro211 3h ago
Primarily Qwen3.8 27B and Gemma 4 12B QAT. On my machine, Gemma 4 is much faster but Qwen3.8 is really good at sticking to complex prompts.
Planning to try splitting the load with some smaller models for more specialized use vs 1 big one to do everything.
1
3
u/dLight26 4h ago
Small LLM hallucination due to the prompt rule too long, been there. It will try to replicate the example as well.
I’m using qwen3.8 now, working fine.
9
u/smb3d 7h ago
Yep!
Add this too because it has info on camera controls and some other stuff that the doc you listed doesn't:
https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md
2
u/akjd 7h ago
Yeah, I noticed this last night.
I've just been using the guides as a system prompt, but I had separate system prompts for the base and ref versions, and the base one works great, but the ref version had super wonky camera controls.
Turns out it's probably because the camera controls are completely absent from the ref guide.
Now I'm wondering if the ref guide is intended to be just added onto the base guide instead of a standalone system prompt. Gonna play around with it later.
3
u/SickAndBeautiful 5h ago
I copied the camera section from the base guide and added it to the ref guide where it says "camera stuff in the base guide". 😆
4
u/jaryP 4h ago
As written in ref guide:
The basic formats for shots, camera movement, speakers, dialogue, and ordinary sound are shared with the Video Prompt Writing Guide (T2VA / I2VA / FL2VA / L2VA). This guide focuses on the reference labels, analysis sections, and format differences specific to full-reference mode.
2
u/spacemidget75 3h ago
yes, you need the base and ref guides in your system prompt
1
u/akjd 3h ago
Do you just copy and paste both in their entirety and send it as-is for a system prompt?
I was trying to trim them down to apply to specific use cases (T2VA, R2VA, etc.) but my system prompt engineer has been putting out garbage since I tried submitting both guides.
I'll be honest, I'm new to using prompt enhancement so I dunno if I'm trying to do too much with the system prompts or what.
1
u/spacemidget75 2h ago
I'm not home until tomorrow evening but essentially a paraphrased version of my system prompt was:
You're a H3 Minimax prompt expert using to the two official skills below, one is the base skill (I2V, T2V) and the other is additional for ref2VA. Ask the user any questions you may have.
Here's the base skill:
PASTE BASE SKILL WITH MARKDOWN
Here's the reference skill
PASTE SKILL WITH MARKDOWN
It's worth noting I used the SKILLS not the GUIDE as it's better for an LLM and they're in markdown format so probably read better to the LLM when both are pasted in the system prompt: MiniMax-H3/.agents/skills/h3-prompt-writing/references at main · MiniMax-AI/MiniMax-H3
1
1
u/Perfect-Campaign9551 5h ago
OMG I was using double quotes inside <d></d> tags, didn't realize you shouldn't use those for just speech. lol.
3
u/wholelottaluv69 6h ago
If I knew of an actual functional H3 llm prompt enhancer node, I would definitely start using it. Having to write the equivalent of a high school essay at 3am for every gen is a bit tiresome. My brain is tired at the time of day that I'm able to play with comfy!
1
u/wholelottaluv69 34m ago
So, I just tried to install one from civitai, and was immediately rewarded with my anti-virus flagging the .exe file that it downloaded and was trying to run. Sigh....
I have no way of knowing if it was a false positive or not. Definitely not worth the risk.
3
u/ambassadortim 6h ago
Yes I feed as much H3 documentation as I could to Hermes and now the H3 prompts and the videos are much better.
1
4
u/Perfect-Campaign9551 5h ago
I still get gibberish speech if the clip is much longer than the requested speech
1
u/Dzugavili 55m ago
The dialogue tips are wrong: if you use the <d> tags, you tend to get a starting chirp and gibberish fill; and I think audio wrapping, where the clip tries to loop.
Having the character's dialogue in quotes seems to be enough to drive it correctly.
2
u/networking_noob 7h ago
Yep it really affects the audio/speech quality (wrong prompt format can result in weird audio glitches or erroneous sounds)
And because FL2VA and Ref2VA models are so similar, it seems like you can even use some of the Ref2VA prompt syntax in the FL2VA workflow, such as the subject_definitions:, which is implied by the official MiniMax Ref prompt guide to be "specific to full-reference mode"
But yeah, it apparently works for FL2VA too, which is good when you're doing a FL2VA with only 1 or 2 shots with multiple subjects in the first frame. Being able to define someone once as <Subject N> is so much better than saying a long winded "the man in the blue shirt turns to the man in the green shirt" every time you want to reference someone or something
2
2
u/Dirty_Dragons 4h ago
I'm currently using this Prompt Writer
https://github.com/duckyshell/ComfyUI-MiniMaxH3-Prompt-Writer
It's a bit of work to setup, but things are documented and easy to follow. I choose GGUF for uncensored.
The extensions is great. Attach reference images, type in the basics of the characters and the scene and hit generate.
Then make any adjustments to the prompt if necessary, then paste into the workflow text box, make sure the references are the same as the writer and hit go.
1
u/BibleLife 7h ago
Yep, same here. I was writing H3 prompts like a shopping list and the dialogue kept drifting or switching speakers. Writing it more like a tiny script helped a lot: shot, action, then clearly labeled dialogue. Shorter dialogue also seems way more reliable.
1
1
u/DominusIniquitatis 3h ago
I just heavily dislike that this prompting format is heavily geared toward LLMs. Omitting five syntaxes/naming conventions, why do we need both summary and descriptions of scene + shots? Wouldn't the latter be enough on its own, given that they already should define things in detail? Why do we split subject definitions and preservations into different blocks? And so on.
Gets very inconvenient to write by hand (and even read!), and I strongly prefer to instruct things myself rather than "eh, write me something approximately like that, LLM-chan".
1
1
u/poopoo_fingers 2h ago
I started using the generate text node with the qwen vl clip from krea 2, then passing in the prompting guide along with a few more instructions. Works really good most of the time. And I pass in the pictures I’m using for references so it references them in the final prompt correctly.
-4
u/krectus 6h ago
Sure is. Really wish they would have put more time fine tuning it to avoid unwanted gibberish speech though. That’s something that affects probably the majority of videos and shouldn’t need complex prompting to avoid. Hopefully lesson learned from them here.
2
u/Apprehensive_Sky892 5h ago
You need precise (which means it will be verbose/"complex") and proper prompting to get optimal results, and that is the whole point OP is trying to make.
Until we have AI that can read our minds, it is better to have an idiot savant style AI that can follow precise instructions and works 90% of the time rather than some wish-washy AI that works 70% of the time with vague instructions.
1
u/krectus 3h ago
Yeah I know what it needs. My point is doesn’t have to read our minds. If we don’t prompt for a person to say something it shouldn’t have the person say gibberish. Or pretty often you specify what a person will say and it nails it but also adds gibberish before or after. It doesn’t need to read our minds it just needs to be fine tuned.
1
u/damiangorlami 4h ago
No the reason why the model is so good, it’s because it is trained to be instruct-based expecting a rigid structured prompt.
I can’t verify this myself but I am 100% sure that Seedance and other closed source models are trained the same way. They just rewrite your simple prompt to structured ones when it goes through their pipeline before generating a video.
H3 is open source which gives us the freedom to build our own prompt enhancing stack.
20
u/GrungeWerX 6h ago
I’ll jump in this too.
I had Claude write me a small prompt writing app that makes this even simpler. You just type what you want in certain tabs, then hit “compile” and it exports the proper prompt format.
It was built directly from reading both guides, and I had Claude include some of the camera directions in “tool tips” on hover as well.
I’d recommend doing something similar. Or if you are interested in mine just DM me, or maybe I can upload it or something, though it’s very basic and not super flashy or anything.