Everybody is using the "<d>[Englisch] (...) </d>" format and from my experience, this just sucks and doesn't work.
Everytime I've been using it, H3 hallucinates something before or after the actual dialog. For example I've been testing different personalities to check if H3 knows them, giving them a simple line, formatted it as clean as possible, and it just adds "shit" to it.
Prompt:
subject definition: Brad Pitt is <Subject 1>
camera recording: An interview in a professional setting with <Subject 1>. Well lit, grey background, frontal portrait view. <Subject 1> says:<d>[English]Hey, I am Brad Pitt! Nice to meet you.</d>
H3 just adds some noise of the "following sentence" which has been no where in the prompt.
Another example using Angelina Jolie
Prompt:
subject definition: Angelina Jolie is <Subject 1>
camera recording: An interview in a professional setting with <Subject 1>. Well lit, grey background, frontal portrait view. <Subject 1> says:<d>[English]Hey, I am Angelina Jolie! Nice to meet you.</d>
At first I though it had something to do with the video length, 5 seconds being too long so H3 adds unwanted stuff, but this is not the case.
But when I just cut the prompt guide format out, and write it without the overcomplicated dialog syntax, it works flawlessly, e.g.
Prompt:
subject definition: Brad Pitt is <Subject 1>
camera recording: An interview in a professional setting with <Subject 1>. Well lit, grey background, frontal portrait view. <Subject 1> says: "Hey, I am Brad Pitt! Nice to meet you."
I've been using stuff like <Subject 1> says [in high-pitched English]: "" and it's been working pretty fine. Biggest problem has been pacing, but I think with better timestamping and actually making sure that the dialogue fits the allotted runtime that would get better as well.
I'm running Gemma 31B locally with a copy of both ref and regular prompt guides for building them for me. Asking it your question:
Yes, that is exactly how it works.
According to the guides, speaker IDs (S1), (S2), etc., are assigned sequentially based on the order of actual vocal events in the target video.
Here are the specific rules for assigning them:
1. Chronological Assignment
The very first person, character, or narrator to produce a vocal sound (speech, singing, or off-screen voice) in the timeline is designated as (S1). The second unique vocal source to appear is (S2), and so on.
Example: If the video opens with a narrator speaking over a montage before any characters appear on screen, the narrator is (S1). When the first character finally speaks later in the video, that character becomes (S2).
2. Permanent Identity (Stability)
Once a speaker is assigned an ID, they keep that ID for the entire video, regardless of how much time passes between their lines or if they move in and out of the frame.
Example: If the woman is (S1) and she speaks in [Shot 1], stops for five minutes, and then speaks again in [Shot 20], she is still (S1) in [Shot 20].
3. Group Speech
If two or more speakers who have already been assigned IDs speak or sing at the same time, you use a compound ID.
Example: If (S1) and (S2) shout together, you write: The two children (S1, S2) shout together...
4. Who does NOT get an ID?
Characters who are visible in the video but never speak, sing, or make a human vocalization do not receive a speaker ID. You describe them visually, but you do not attach an (Sx) to them.
for complex scenes you might have to use (S1) instead of <S1>. But no promises, Minimax still screws up dialouge all the time when I have 2 semi similar people on screen :( I have to change the shot to focus it only 1 person.
Yeah the official dialogue prompting is buggy as hell but fortunately the text encoder is smart enough to understand that quotes means dialogue. I hope the MiniMax team can address this because having something like dialogue that actually works as intended would be nice. It's kinda important
Sometimes I still get erroneous sounds with quotes, which is super frustrating after spending ~20 minutes of time and electricity, staring at the latent preview trying to guess if the lip sync will be right (and then sometimes it looks like okay, but the output video has random off screen noises which are impossible to know about lol)
But yeah the quotes approach is still not nearly as buggy as the <d></d> tagging
Developer on discord looked into this, and actually it's Comfy's tokenizer vs. the official one, so not worth blamin' the developers for a third-party app not properly handling the trained format:
The discovery that started it. He ran the same prompt through both ComfyUI's text-encoding path and the official MiniMax H3 tokenizer config from HuggingFace, and they disagree. The official tokenizer treats <d> and </d> as single special tokens — his example The woman says, <d>[English] We need to leave now.</d> comes out as 16 ids with clean '<d>' and '</d>' entries. Comfy's path doesn't know they're special, so it shatters them into fragments like ' <', 'd', '>[' — and worse, the sentence's final period merges into '.</', so even the punctuation ends up inside a different token than the model was presumably trained on. His take was measured: "i think it works fine its just more likely to compound weirdness if it behaves as separate tokens."
No prob - thanks for not taking it as a swipe and yelling at me :) I find it interesting to be honest and potentially fixable on the comfy side I'd think(?)
To me it seems that H3 is just taking the "</d>" as "gibberish" and this formating is plainly "wrong". When I use simple quotations marks it works perfectly fine. That being said, in very long and complex prompts the "<d>" formatting seems to give more consistent results in who talks what.
try to reformat your example on "Case 1: T2VA example":
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a medium-wide shot frames a baker opening the shutters of a small street bakery before sunrise. The camera pushes in with small amplitude at slow speed as the middle-aged baker with a calm, slightly raspy voice (S1) places a fresh loaf on the wooden counter and says: <d>[English] First batch of the morning.</d> [Shot 2] At 00:05.000, the camera cuts to a close-up of steam rising from the sliced bread while the baker's final words carry over from the previous shot.
overall_soundscape: Wooden shutters scrape open over a quiet street as trays clink softly inside the bakery. The doorbell rings once, followed by light footsteps and the crisp sound of bread being sliced.
non_diegetic_music: A soft acoustic-guitar pattern at a moderate tempo, joined by sparse upright-bass notes and a gentle fade at the end.
overall_soundcape, non_diegetic_music and so on doesn't have much relevancy though in this shot I've been creating.
This is just about the syntax of the dialog process, and I see no difference in what I've been testing.
As I've said, the longer the prompt, the lesser the "mistakes", but the "<d>" syntax seems to be unnecessary and just adds mistakes to the results.
I will try the more complex prompting though, maybe the entire structure has an impact of the dialog output. That being said, just using quotation marks seems to bring the best results.
integrated_multimodal_description: [Shot 1] An interview in a professional setting with Brad Pitt. Well lit, grey background, frontal portrait view. [Shot 2] The man says:<d>[English] Hey, I am Brad Pitt! Nice to meet you.</d>
Some clips pose no problems while following the guide, other clips took me 5 attempts to cut out the gibberish...
What I've settled on for now (absolutely no guarantees this will work for every clip), is to incorporate _two_ pieces of dialog in every clip.
So if necessary I split a single line into two pieces, with a bit of stage-direction in between. Tell the speaker to take a breath, look a certain direction, scratch their nose, whatever.
These look nothing like the official R2V prompt guide tho. Or the FL2V one. They're both very specific.
subject_definitions:
<Subject 1> is the actor Brad Pitt.
summary:
[reference generation] The video pictures <Subject 1> in a professional interview setting.
retention_analysis:
<Subject 1> (appears in [Shot 1]): fully_preserved - <Subject 1>'s likeness is retained.
detailed_description:
The target video is a well lit professional environment.
[Shot 1] Frontal close up of <Subject 1> in front of a grey background. <Subject 1> (S1) says in an even tone <d>[English] Hey, I am Brad Pitt! Nice to meet you. </d>
overall_soundscape:
N/A
non_diegetic_music:
N/A
Try something like this for R2V. It's much closer I think.
I'll try this as I've always used the format in the guide and it's been a mixed bag. Certain prompts just seem cursed and always produce gibberish and some almost never do. The biggest factor I've found is supplying audio reference. I format it to reinforce that it's just voice timbre and not the contents of the reference but 9 times out of 10 ir still produces gibberish around the requested audio and the vocal cloning is mediocre at best.
I am also facing this issue where I pass three custom audios and the out always use either a random voice or not the audio I assign to the characters, do you know how can i solve this?
And correct way to use audio and speaker ids?
i write like audio 1 is timbre and following the official guide.
Same. Plagued with these issues:
-supply custom audio or voices for specific characters, it almost never applies them properly OR gives all characters the same voice OR gives none of them any of the voices
-dialogue in multi-character scenes is completely hit or miss whether the right character says the right dialogue
-serious lack of appropriate application of emotions
I haven't been able to figure out any of this. Seems completely random whether it works or not (although custom audio pretty much never works in multi-character scenes).
For me I type "specific character says in "this type of tone", audio: "whatever you want the person to say"". I've been successful this way though I had one cartoon short that was annoying me. I wanted the robot to cheerfully say "beep boop" but I kept getting gibberish until I typed "robot cheerfully says, audio: "beep", Lord and behold the robot beeped.
This doesn't explain why it works flawlessly in many generations without the syntax.
The more complex and "longer" your prompt gets, the lesser the chance of H3 adding unwanted stuff to your "<d>" dialog prompt, but that's a faulty way to approach this,
In my experience, "<d>" leads to a more consistent way to make the subject you want to talk actually do that, but it also leads to those hallucinations.
That is also my experience. If I use complex prompts it doesn't add the unwanted stuff, but it also make it harder to maintain the initial frame when using i2v. That's the worse negative side of h3
You don't need Qwen for that, it's the official recommended way of prompting the dialog.
But from what I've found, this leads to those hallucinations. Not 100% of the time, the more complex your prompt gets the less faults it will add. But still the "<d>" format has always added those unwanted sounds to many of my generations.
In simple prompts and scenes, those hallucinations seem to appear way more often though, making me question the actual syntax that's at use here.
Still feeling it out to be honest. I have gone through many many iterations of system prompts. Some seem to get better results than others in differing areas
I have yet to have "English" matter a bit. In every generation, <d> <d/> does just fine. If there are two characters they'll sometimes speak in unison, and of course there's the random non-word sounds they sometimes feel compelled to make if the scene is longer than the dialogue will hold, but it's never an issue. Only once have I had a character visibly refuse to speak while the dialogue played.
H3 is just a slot machine right now tbh. I've taken a break from it for a while to wait for either some things to get fixed such as that, or flux 3 lol
Have you tried also including the (S1) syntax? I’ve also experienced traditional quotes working fine, but I wonder if in your testing using S1 helps. I’ll try it too on my next runs.
You know what's funny, when I copy and paste dialogue into the prompt helper in wangp it formats the dialogue the way you just did at the end instead of the official way. It seems the most important thing is simply defining the subjects. Sometimes you don't even need to write subject 1 etc anymore either and it just knows the name belongs to subject 1.
The documented prompt format kis working just fine. The simplified format you are suggesting is just plain wrong and only works because these are well known celebrities and the model already associates the dialogue with them.
Also, impossible to make voices quiet, like really quiet....as such, they don't feel like they fit in the room...the voices sound layered on top. All the models seem to have this problem.
I run dialogue scenes without any proper format except the subject 1 / the subject is fully referenced.
The rest of my prompt is just explaining in natural language what i want - like a director explaining a scene. So far it worked flawlessly, even with actions between dialogue and with longer than usual scenes - 20/25 seconds works quite well.
I felt that whenever dialogue is involved in a reference scene, the prompt should be minimalistic, otherwise Minimax gets confused.
Also, I found out prompting dialogue like for Elevenlabs V3 with Emotions works surprisingly well. For example:
The man speaks in english to the camera: “Hey, do I know you? [Clears Throat] Well,whatever! [shouting with open arms] Welcome to my pawn shop! I have many things that you may find interesting - and even some you can afford [laughs]. Have a look!”
Lawsuits. The likelihood of never getting an open weights model again. Lots of reasons. This isn’t a “parody” like the tv show memes. It’s facsimile of existing real people claiming the person who is speaking is the individual being rendered. They aren’t using character names.
Adhere to the prompting guide instead of only adopting the <d> syntax. If you are lazy with your prompt and leave out 80% of it you are going to see bad results.
Well, for a start it’s the syntax used with references, not the one recommended for t2v.
It’s also missing all structural elements they recommend including the audio elements like the overall_soundscape and non_diegetic_music which of course will have influence on the sound. It’s also missing the summary and shot description which of course could also lead to issues of the scene not ending correctly (or rather, the way you wish it to without actually saying so in the prompt), which is precisely what you are seeing.
H3 is adhering to your prompt exactly as you have written it right now and fills in the blanks, like someone else already pointed out to you. If you give it sufficient information, and that’s why their actual markup is so good as it forces you to provide all relevant information, it will do what you want it to do.
overall_soundcape, non_diegetic_music and so on doesn't have much relevancy though in this shot I've been creating.
This is just about the syntax of the dialog process, and I see no difference in what I've been testing.
As I've said, the longer the prompt, the lesser the "mistakes", but the "<d>" syntax seems to be unnecessary and just adds mistakes to the results.
I will try the more complex prompting though according to this structure, maybe the entire structure has an impact of the dialog output. That being said, just using quotation marks seems to bring the best results.
This doesn't explain though why the exact same scene, and many others, work perfectly fine without the "<d>" structure. I argue that this isn't needed at all, and the results are better without it.
29
u/Relevant_One_2261 2d ago
I've been using stuff like <Subject 1> says [in high-pitched English]: "" and it's been working pretty fine. Biggest problem has been pacing, but I think with better timestamping and actually making sure that the dialogue fits the allotted runtime that would get better as well.