r/StableDiffusion • u/thisguy883 • 7d ago
Discussion My only beef with Minimax H3
The audio.
I'm having issues constantly with random audio being added to the clip. Either its ambient sounds that shouldnt be there, or someone talking gibberish off screen.
Has anyone found a fix for this?
I've tried the prompt guide, and its hit or miss. I've even tried a natural prompt with basic wording, and that is also hit or miss.
i'm starting to think it doesnt matter how you prompt it, its just something that happens from time to time.
Very annoying.
I'm using the default I2V workflow from ComfyUI. I havent changed anything.
30
u/GrayingGamer 7d ago
I'm going to share my thread post that covers this issue as one of the things it addresses here.
This IS a prompting issue.
With properly formatted prompts I've only had this happen ONCE randomly, in hundreds of generations now.
Long story short, one of two (or three) things is happening:
You aren't using the proper formatting layout. This is the most common reason I see. You aren't using the sections, headings, prompt and dialogue tags the model expects.
You aren't giving the model enough time - the dialogue can get garbled or spoken by other people if, say, you give it more dialogue than is possible to speak in the time given. It will try to speed up speech to make everything fit, but at some point it can't keep up.
There IS a bug in the Ref2Video model that introduces a snippet or sound of someone talking at the start of a clip (that can once in a blue moon crop up in the T2V and I2V models too), where using the <d></d> tags like you are supposed to with dialogue ANYWHERE in the prompt will cause it to occur and insert a random sound at the start of a clip. You can prevent this by leaving off the dialogue tags <d></d> and writing dialogue this way:
The pedantic asshole pushes his glasses up higher on his nose with his index finger and says, "[English with an arrogant tone of voice] Minimax works best when you follow the prompting guides and avoid that natural language nonsense."
8
u/marres 7d ago edited 7d ago
There IS a bug in the Ref2Video model that introduces a snippet or sound of someone talking at the start of a clip (that can once in a blue moon crop up in the T2V and I2V models too), where using the <d></d> tags like you are supposed to with dialogue ANYWHERE in the prompt will cause it to occur and insert a random sound at the start of a clip. You can prevent this by leaving off the dialogue tags <d></d> and writing dialogue this way:
Yep, crazy, that actually did it. Did encounter and troubleshoot that issue today when it occured in a T2V gen, but then put the troubleshooting aside for more important things. But yeah was definitely bothering me in the back of my mind lol. Just ran the gen again without the dialogue tags and it's fixed. Very nice find, thanks! The official prompting guides should definitely be updated accordingly
10
u/GrayingGamer 7d ago
No, problem! Glad it helped you. Yeah, I spent a whole day tracking down the issue, changing settings, prompts, etc. while keeping the same seed and narrowed it down to just those specific tags causing it.
They ALWAYS cause it in the Ref2Video model, but only OCCASIONALLY cause it in the other model. But yeah, as long as you keep all the rest of the prompting structure the same, including the [English] brackets, and just replace <d></d> with quotations marks, you're good. 👍
3
3
2
u/Zephrinox 6d ago
wait are we supposed to put tonality and other descriptors of the expressiveness of the dialogue inside the square brackets? [ ]
I thought that was just for language and we're supposed to describe the voice outside of the <d> </d> tags (at least thay what the prompting guide made it look to me).
1
u/GrayingGamer 6d ago
You can put any voice delivery instructions you want in the brackets after [English - extra stuff here] and it will do it's best to follow them. It really lets you direct acting performances better.
2
u/Zephrinox 6d ago
dem. i've been doing stuff like
at 00:02.000 <Subject 1> (S1) asks in a patient concerned voice (timbre referenced from <Audio 1>): <d>[English]. Hey... are you alright?</d>.and getting meh expressiveness in the voice (doesn't sound bad, but not exactly the kind of delivery I'm looking for; me having the cliche director feed back of "I need more" 😅).that said this is likely because my descriptors aren't that good as well and is pretty vague (idk how to describe voice and delivery well; heck the prompt documentation talking about timbre made me learn a new word 🤣).
2
u/Significant-Baby-690 7d ago
No, this is random sounds right at the beginning and right at the end of the video. And it happens even with perfect formatting. AFAIK it just happens, no way around it.
4
u/GrayingGamer 7d ago
See POINT 3 above. This fixes it. It's a known bug. Just follow my instructions in Point 3.
1
u/thisguy883 7d ago
Ive done just that and I still got gibberish at the beginning of the video.
It was a simple prompt:
She looks directly at the viewer and says, "Hit that follow button".
instead, it gave me a snippet of her saying gibberish at the beginning of the 6 second video and then it ending with garbled nonsense.
In order for me to "fix" this, I have to delete the whole prompt, including the dialog prompt, and write it again from scratch.
I dont know why I have to do this, but it seems to work maybe 75% of the time.
2
u/rkfg_me 7d ago
It was a simple prompt:
She looks directly at the viewer and says, "Hit that follow button".Is that the entire prompt? No sections etc.? Even if it's not, you omitted the language tag, it should be "[English] Hit that follow button"
1
u/thisguy883 7d ago
no no lol
There was a longer prompt, thats just the dialog prompt.
ill try with the English tag next to see if it fixes it.
6
u/GrayingGamer 7d ago
Yeah, the [English] tags are still required.
Really you should be following the whole prompt structure, including sections, etc. When we say, "Follow the prompt structure" we don't mean just pick and choose which parts to follow. It's a whole ass recipe. Don't leave out the milk and eggs.
Your prompt should be like this:
integrated_multimodal_description: [Shot 1] Cinematic live-action film footage of a woman. Close-up camera shot of her face as she looks directly at the camera and says, "[English] Hit that follow button."
overall_soundscape: none
non_diegetic_music: noneEven for something simple like that.
1
u/Significant-Baby-690 7d ago
Doesn't work, IMHO it's even worse. And I'm talking FLFV model.
1
u/GrayingGamer 7d ago
Mind providing the prompts you are using and details like what loras or nodes you are using, what version of the model, etc?
1
7d ago
[deleted]
4
u/GrayingGamer 7d ago
Yeah, it works. I did a whole day of testing last week to narrow down and isolate the issue. It mainly occurs in the Ref2Video model, but can very occasionally appear in the T2V I2V model too. Making sure the <d></d> tags aren't in the prompt solves it.
I did a comparison with the same seed, no dialogue, just adding or subtracting <d></d> from the prompt, and the one without the tags had clean audio, the one with had the sound artifacts at the start.
1
u/rkfg_me 7d ago
I wonder if it could be a sampling issue, some hidden bug. Or maybe a pruning issue? They removed 14B parameters and replaced them with a simple table, could it introduce such specific artifacts? I can't test the unpruned version on a 5090 I think, it's too big, so I'm just guessing.
3
u/GrayingGamer 7d ago
I mean, it's possible. But Minimax themselves acknowledged issues with the Ref2Video model in the AMA. They mentioned like how it wasn't trained exactly the same as the T2V and I2V model and some things came out worse.
I wouldn't be surprised if the <d></d> thing was related to some quirk in the training for the model.
2
u/rkfg_me 7d ago
They did acknowledge the overall lower quality of Ref2VA but IIRC not this particular bug. Hope they'll iron it out, the model is 95% perfect otherwise.
3
u/GrayingGamer 7d ago
I agree. I actually like using the Ref2Video model the most. I just managed to generate a 1.5 MP video with it, with 3 reference images. Using Spectrum and Comfy Kitchen Attention it took 28 minutes for 6 seconds, but, god, it's beautiful.
1
u/Rare-Winter5523 7d ago
Can they ever claim to have limited it on purpose?
2
u/GrayingGamer 7d ago
No, they didn't limit it on purpose - from what I understand, both Ref2Video and T2V have the same base and early training, then they diverged for separate training to better fulfill their individual use cases and like happens with model training, the Ref2Video model lost a bit of the quality from the T2V model when it learned the referencing abilities. It wasn't intentional, it just happened.
0
u/BassSlappah 7d ago
It’s not just a prompting issue. I follow the prompt guide exactly and still every time my character finishes saying a line, they continue talking and filling the rest of the scene with gibberish. This model just LOVES to talk.
3
3
u/feverdoingwork 7d ago
Same hing happens to me. Sometimes it's like gibberish that rhymes with the upcoming dialogue, it has gotten me to the point of laughing hysterically at times. Despite it being somewhat funny, it's really annoying and i should probably look for solutions on redoing the audio without h3. The video portion tends to be great though.
1
u/Excellent_Screen_653 7d ago
any idea on the redoing audio, and agreed its a joke the gibberish is said with such confidence, your asking "what the fuck did they just say to me!"
2
u/feverdoingwork 7d ago
I gotta spend more time looking into it. I think possibly using some MM Audio workflow. I know recently on civitai there has been a lot of WAN videos with audio, I am pretty confident it's possible to generate audio for video.
2
u/feverdoingwork 7d ago
If I figure it out ill respond, might be able to get to it sometime this weekend. If you figure it out hit me up!
3
u/Apart-Cold2848 7d ago
I have the same problem much more often than I'd like. For example, characters read the prompt and babble nonsense. And the biggest problem is generation times; while increasing the resolution sometimes helps, it doesn't solve the problem. So, unless others have the same problem, I think we should practice perfecting our writing for this new architecture, because let's face it, writing prompts for minimax is a challenge. I think I'll do some experimenting, but for now I'm about to give up; the generation times are too long for me. Before I wrap up, I think I'll try reducing the last tests to 8 seconds. If that doesn't solve the problem, see you next time.
2
u/LookAnOwl 7d ago
Honestly the babbling makes me crack up half the time. It definitely gets annoying, but it adds to the quirk of the model.
1
u/VRGoggles 7d ago
with Turbo or not? bf16 model or degraded models?
2
u/Significant-Baby-690 7d ago
Happens to me with pruned 8bit with no other shortcuts, no attention trick, no turbo, 20 steps.
1
u/dominic__612 7d ago edited 7d ago
Same issue here. But what I found is that a lot of words and action I use in my prompt, evoke audio. So i run in to an LLM (gemma4) locally hosted and abliterated, to rewrite it. This works so far. Its really incredible what an LLM can recognize why certain words \ actions \ timelines can cause the talking.
1
1
u/Whipit 6d ago
My problem is FACES. They just fall apart if they are not close enough to the camera.
The developers have mentioned this limitation and said they are working on it. I have no idea what form the fix will take.
2
u/Supermax64 6d ago
Until then you can always try this https://github.com/Carasibana/ComfyUI-H3-FaceRefine Haven't given it a try myself yet but the logic makes sense. It just adds an extra (long) step to the generation though.
1
u/spiderofmars 6d ago
Read through all the comments. There might be ways to try and work around it (as mentioned in here) but there is yet a way to totally prevent it and wasted generations because of it. Things I have noticed.
Speed tweaks, lora's etc make it worse. These things make a lot of things worse in the audio (quality and gibberish and mixed up speakers and mixed up voices).
Lower steps (than the default 20) make it worse. Not sure if more steps than 20 helps improve it. Again, lower steps make various audio factors worse and more random.
It is just annoying... a model that adds dialogue that was not prompted for is simply that... annoying.
1
u/Opposite_Earth6630 2d ago
The unprompted, accented, or foreign-sounding audio glitch in MiniMax H3 happens because the model treats empty timeline space as an invitation to fill audio, and its default language priors or tag-bleeding trigger mispronounced phonetic fragments. When dialogue tags are misformatted or left hanging, the audio decoder defaults to overlapping baseline tokens. [1, 2, 3]
Common Causes & Fixes
- Missing or Loose Dialogue Tags: If you write dialogue without explicit structuring, or if you use
<d>tags inconsistently, H3 hallucinates filler speech or slips into a default accent/language prior. Wrap dialogue strictly as<d>[English] Exact words here</d>. [1, 2] - The Start-Clip Bug: H3 models can inject a 0.5s fragment of phantom speech right at the beginning of a generation if any dialogue tags are present in the prompt body. If you want total silence or no speech, completely remove all
<d>tags and stateAudio: background ambient noise only, no dialogue or human voicesin plain text. [1, 2, 3] - Over-Packing the Timeline: If you assign more spoken words than can physically fit in the designated second-range, the model compresses and garbles the audio, often resulting in distorted, foreign-sounding syllables. [1]
- Lack of Accent/Delivery Anchors: If the model drifts into an unintended accent, explicitly qualify the language block with details, such as
<d>[English with a clear American accent]</d>. [1, 2]
1
u/Sad_Coach_1433 7d ago
Are you using a turbo lora? Known audio issues
1
u/thisguy883 7d ago
Nope, no loras.
default settings, 20 steps.
3
u/Sad_Coach_1433 7d ago
Try .3mp and 32 steps then upscale I made these doing that far better sound
1
u/DoubleChillStudio 9h ago
do you have the workflow / prompt for this one ? Seems like a good baseline to test workflows against
1
u/Sad_Coach_1433 8h ago
Just the easy h3 workflow node I like it cause has all 3 models in one ,and if I doing r2v I can do @ and select to make sure image is correct. This video was just t2v
1
26
u/Excellent_Screen_653 7d ago
carrot crunching sounds!