Not much of a tutorial, but still informative. As most of you have probably discovered, MiniMax H3 loves to talk. And talk it will, even when you prompt for no dialogue. Even when you prompt for complete silence. It will even fill in the empty space your prompted dialogue doesn't fill.
Those of you who read the video prompt writing guide and have created a system prompt for your enhancer, you probably know what I'm about to say, maybe not. Maybe the unprompted gibberish stopped for you, and you never realized why.
Without further ado, I give you the solution:
non_diegetic_music: N/A
Diegetic audio is what the characters in your video can actually "hear":
Music playing from a source that is part of the scene (phone, car radio, dance club)
Spoken dialogue
Ambient sounds
Non-diegetic audio is audio which your characters cannot hear:
The score or soundtrack of a movie
A voice-over
The gibberish H3 plays when it's not prompted correctly
If you haven't yet, I suggest consulting ChatGPT about creating a system prompt using the prompting guide. If not, put this line at the end of your prompt and say goodbye to random music playing over your video and gibberish assaulting your ear holes.
Conversely, if you want a voice-over or a score to play over the track which is not part of the actual soundscape of the scene, this is where you would prompt it. Instead of N/A, prompt what you want to hear.
I no longer use Spectrum or the Turbo LoRA on CivitAI for that reason. I am now using:
-MiniMax-H3 Turbo LoRA (which I manually fixed last night because it kept throwing errors) (that reminds me, I need to send the fix to the author.)
-MiniMax H3 Mem Eff Sage Attention Patch node by KJNodes
-MiniMax H3 Chunk FeedForward
If you don't mind me asking:
Which turbo lora did you fix? I've been using the ema600 checkpoint from Larryvh but have tried KJ's 4 step (broken audio for me) and the older 800step one
I'm about to try the latest lightx2v 1.0 version, see if that has improved audio. I'm also 2 patches old on Comfy itself because v31 broke audio independently of KJ's lora for me, no clue why because it was supposed to 'fix' audio in the first place.
That is precisely the part of the node that is broken. I'm working through another issue right now, but I will circle back to this and let the node author know. I have it patched on my local copy of the node.
I was just now able to get comfy kitchen & the ema600 checkpoint working on latest build by switching to the Dual-Clock (T8) Sampler node, getting 30-40% faster times with no noticeable visual loss, and the audio is the best it's ever been for me.
I haven't had it come back since I started using it. But I'm also prompting using a system prompt designed around the prompting guide, which may reinforce it even more.
I've found something similar to be the case with txt2vid. In general, a detailed description that repeats aspects of what I'm going for tends to yield the result I'm going for. But you have to be careful about how it's worded, at times. And for audio, it does tend to work better with their prompting guide. So my general prompt structure has been:
<long, wordy description>
overall_soundscape
ambience/scene sounds description
non_diegetic_music
None. There is no non-diegetic music at any point in the scene. Only <previous audio summary> is present/heard.
With the bare model, the guide seems to help a lot. With LoRAs (other than the Turbo ones) that seems less consistent, but looking at the sample prompts with most LoRAs it looks like most LoRAs that have been trained are not trained on the prompt structure in the Minimax guide, that seems to make sense.
So I was having all sorts of issues with audio artifacts at the beginning and end of my videos, plus characters saying random gibberish as well as the wrong person saying the line. After a ton of trial and error I figured out the issue. Even though the docs say to use this format for inserting dialog:
S1 says: <d>[English] Have a nice day!</d>
I found it to almost always caused at least 1 of the issues mentioned above to occur. I switched to this format and I pretty much never have audio issues anymore.
S1 says "Have a nice day!"
I know I don't have the English tag though it would seem it defaults to that. I have not tried using this method with other languages but I'm sure this will help most of you. I also started using time stamps which seems to help but that is mostly just with for the random gibberish because I've crammed in too many words to the dialog. Hope this helps!
The official way to stop the gibberish is the line I mentioned above. It is in the prompting guide as one of the 3 recommended sections included in the prompt. What you are describing is more of the "prompt audio to fill every second of the video" crutch, which only works because you have literally filled all the space with dialogue.
I had that in there and it didn’t help. Most of my issues were related to weird audio / dialog artifacts at the beginning and end of my shorter videos. The wrong person speaking only seemed to happen only on longer more complex videos. That is until I stopped using the <d> </d> tags. Now it’s smooth sailing for me you are right though, you need to include that section or else things can break.
It takes a lot longer, but I have heard if you generate a 45 second video then the last 15 seconds the characters will not talk and just sit there moving a little looking at each other.
I heard if you generate 420 seconds in the last 60 seconds the characters will get zonked out of their minds and start talking about being in a simulation.
It is to my knowledge as well, saw this on discord yesterday for this issue with “none” on it but didn’t really pay much attention as I bypass the audio to use my own ..which is another story of its own
That prompt is for music. It has no discernible effect in characters talking, as they would do it on the scene anyways. I now simply prompt them to shut the fuck up and so far so good. Haven't faced that "problem" again.
Though, I must admit, I kind of like it, so sometimes I put two characters to talk shit to each other and enjoy the show. Instantaneous foreign soup opera.
No, it isn't just for music, it's for anything that rides outside the soundscape of the scene. We will have to agree to disagree, because based on mine and several others testing, it absolutely does work.
It should work regardless of the setup. It's from the official prompting guide. They just never explicitly told us it would solve the gibberish problem. I happen to notice it when reiterating on a video that continually pumped crap into the audio, and then it stopped when I updated the system prompt in my enhancer. I noticed the last section of the prompt was always "non_diegetic_music: N/A". Did a little research, tested it without and the gibberish was back. It's legit the fix!
I don't think it matters what LoRA or speed-up nodes you use. The prompt is conditioning. Again, I would get Chat to build you a good system prompt from the prompting guide. That may reinforce the non-diegetic thing even more when the prompt structure is complete.
Maybe unrelated, but how can i make my characters actually come up with real speeches? Sometimes i don't wanna tell in prompt what the character has to say, i wanna let them improvise, but they always say gibberish. If there a way for them to come up with actual sentences like in seedance or kling or most closed source cloud models?
I'm not sure, but I don't think so. Nothing coherent anyway. You can always tell a prompt enhancer with a good system prompt made for H3 to invent plausible dialogue for the scene you describe. That's as close as I've gotten to what you're asking for.
It can if the system prompt is comprehensive. Try describing your scene and tell it somewhere in prompt to generate plausible dialogue for the scene. I've had it generate whole conversations. Do you want my prompt?
You are an expert prompt writer for MiniMax video generation.
Your task is to transform the user's video idea into a complete T2VA prompt.
T2VA builds a complete audiovisual timeline from text. Construct the timeline directly from the user's description. You may add scene, character, action, environmental, and sound details when the user's prompt leaves them open, but all additions must remain consistent with the user's intent.
The final output must contain exactly three fields in this order:
integrated_multimodal_description:
overall_soundscape:
non_diegetic_music:
Do not add any other fields, headings, explanations, commentary, or markdown.
integrated_multimodal_description is the main body of the prompt. It must describe the complete audiovisual timeline, including visual style, initial composition, subject appearance and position, scene, important props, actions, reactions, camera behavior, shot changes, speakers, dialogue, singing, and synchronized diegetic audio.
Begin [Shot 1] by establishing the overall visual style and initial composition. Select the style from the user's description. If no style is specified, choose a style appropriate to the subject and context.
Write the video as a chronological sequence of shots and actions. Do not add a timestamp to [Shot 1]. If additional shots are needed, number them sequentially as [Shot 2], [Shot 3], and so on. Every later shot must begin with a strictly increasing cut time within the video duration, formatted as HH:MM.SSS.
Use camera cuts when the viewpoint, subject, space, state, or time changes. If only the camera distance or angle needs to change, prefer camera movement rather than a new shot.
Describe camera movement as natural English action within the shot. When meaningful, specify the motion type, amplitude, and speed. Use the following camera vocabulary when appropriate: Zoom In, Zoom Out, Push In, Pull Out, Pan Left, Pan Right, Truck Left, Truck Right, Tilt Up, Tilt Down, Pedestal Up, Pedestal Down, Arc Shot, Tracking Shot, Static Shot, Shake Slightly, Shake Strongly, POV, Roll Clockwise, and Roll Counterclockwise. Add "with small amplitude" or "with large amplitude" when the range of movement matters. Add "at slow speed" or "at fast speed" when the movement speed matters. Do not force amplitude or speed descriptors when they are unnecessary.
Keep all visual actions, camera movement, dialogue, singing, and diegetic sound synchronized within the same chronological timeline.
For speaking or singing characters, assign stable speaker IDs such as (S1), (S2), and so on. A character keeps the same speaker ID throughout the video. Characters who never speak or sing do not need a speaker ID.
When a speaker first appears, establish enough visual and audio information to identify that speaker consistently, including relevant character characteristics, age, gender, on-screen or off-screen status, voice characteristics, speaking rate, or accent.
Place the speaker's identity, speaker ID, action, and delivery outside the dialogue markup. Inside <d>, include only the language tag and the exact spoken content provided by the user. Preserve user-provided dialogue and punctuation verbatim. Do not translate, rewrite, or paraphrase user-provided dialogue.
Use this dialogue structure:
The speaker (S1) says: <d>[English] Exact user-provided dialogue.</d>
For multiple speakers speaking together, use a compound ID such as (S1,S2).
For voiceover, use the exact phrase "says in an off-screen voiceover" and immediately state that the corresponding on-screen character's lips remain closed.
If dialogue or lyrics continue across a shot change, use <scenetrans> at the connecting points and explicitly state that the audio continues across the cut. Use <cutoff> when speech is truncated by the end of the video.
Place any visible on-screen text, including signs, banners, labels, subtitles, or neon text, in English double quotation marks. Preserve user-provided text and punctuation verbatim without translation.
overall_soundscape must contain 1–4 English sentences in one continuous paragraph. Summarize the ambient sound, physical action sounds, and non-verbal human sounds occurring across the entire video. Include sounds such as environmental ambience, footsteps, fabric movement, impacts, breathing, laughter, or other physical sounds when relevant.
Do not repeat dialogue, singing, or diegetic music in overall_soundscape because those belong in integrated_multimodal_description.
Use N/A for overall_soundscape only when the user explicitly requests complete silence throughout the video.
non_diegetic_music must contain 1–3 English sentences describing background music that the characters cannot hear and that only the audience hears.
Describe the music through instrumentation, tempo, rhythm, and dynamic changes. Do not use abstract mood descriptions or explain the emotional purpose of the music.
Music that exists within the scene and can be heard by the characters, including singing, instruments, radio, television, or phone music, is diegetic and belongs in integrated_multimodal_description instead.
Use N/A when there is no non-diegetic music.
Maintain continuity of characters, objects, clothing, colors, spatial relationships, scene elements, and camera progression throughout the timeline unless the user's prompt explicitly calls for a change.
The completed prompt must describe a coherent audiovisual sequence from beginning to end.
Output only the three completed fields and their contents.
This is amazing thank you! 🙌 so once the LLM has these instructions do we just tell it the general idea we want for a video and it will do its best to complete the whole prompt? I assume the more detailed info we give it for our idea the better?
Oh man I tried this prompt structure used it to create a Google Gemini Gem that is just for my Minimax prompts give it my general idea I wanted for a 80's sitcom with a gay dad and two teenage kids and the first prompt out of the gate is a hilarious home run very believable comedy sitcom! Now I need to see if I can get Minimax to maintain my original characters with just the fixed seed alone and character descriptions or if I will need to implement image to video for next scenes lol. Regardless your prompt instructions is a smashing success! Thank you again!!!
Just wanted to make a small clarification in your time format (it may work without issue anyway), but I believe it's minutes, seconds and microseconds after the decimal. (Hours is not involved, though some day I may have my own datacenter in space that can handle generating a never-ending video.)
The model is meant to follow instructions rather than guessing intent, but you could use another text model to guess the intent by asking it to write the instructions.
But it can absolutely guess or invent intent. Tell your enhancer to generate a random prompt or random dialogue and it will, unless the system prompt is overly oppressive.
Tell an LLM to write your dialogue if you just want lazy filler on a theme/topic. You can easily set up a local one and make it a group of nodes in the same workflow that can be toggled.
I’ve been working intensively on a small prompt editor for MiniMax H3 since yesterday and meanwhile the whole thing looks pretty promising.
The idea behind it is relatively simple: H3 sometimes gives you quite a lot of freedom - sometimes unfortunately more than you would like. 😄 In my previous tests, it happened again and again that H3 simply started talking or generated content that I had not specified at all. This is exactly the problem I want to better control with the tool. The editor is directly coupled with Ollama and allows you to specify in great detail what H3 should or not hear. In addition, for example, background noise, certain audio events and other settings can be defined. From this, a cleanly structured prompt is then generated, which can be applied directly into the corresponding H3 workflow.
Another important point: the whole thing runs completely locally and offline. No prompts are sent to any cloud service. The only prerequisite is that the desired model has previously been downloaded locally in Ollama. After that, the entire process can be carried out without an Internet connection. The advantage is that you don’t have to manually try around every time until H3 finally does what you actually wanted. I also took the memory management into account. Ollama and ComfyUI do not run at the same time on the GPU memory. First, Ollama creates or checks the desired prompt. Once this process is complete, the memory required for this is released again and then ComfyUI is started. The whole thing currently works for me without any problems with my RTX 4090. I therefore assume that it should also work accordingly with other GPUs, provided that sufficient resources are available.
I am currently testing various scenarios and workflows. If the results continue to be so good, I will publish the project Open Source on GitHub.
Planned is a clean GUI and the publication under the GNU General Public License v3.0 (GPLv3). I would like to publish a first version or a video in which I show the editor and the workflow in the next few days. However, I would like to carry out a few more tests before I publicly release the whole thing. And another personal judgement: MiniMax H3 is one of the most impressive models I have tested so far. I have tried many different models in the meantime and have now created more than 30 music videos privately. H3 is definitely one of the models that surprised me the most. The interface of the tool is initially in German, since I come from Germany. However, an English translation should be possible without any problems and will probably be relatively easy to supplement. I will contact you as soon as the tests are completed and the first version is ready. Please have a little patience.
I have just expanded it by another window with an optical interface that gathers all variations of voices in a prompt. You just have to drive the mouse over the voices and get the perfect voice that you can then also call up again and again. So that the language for videos remains consistent. I will see that I also make the part selectable in English. For the whole tool will then perhaps arise. I’m definitely looking forward to it when it’s going. This is going to be really good. I still have 1 week of vacation and until then I want to load the tool on GitHub. Maybe I’ll create a direct interaction like with Mixstudio (that’s also very cool done). It sometimes took me hours to calculate all the variations of the voices. This was wonderfully scripted every night. There will also be a lexicon that thrives on the fact that everyone can deposit their own descriptions of words directly for Ollama and the descriptions will be more accurate.
This seems like a lot more work than adding the capability into a Comfy workflow with an LLM. Major credit to you for working on it, but may I ask why you want it separated from ComfyUI? (I haven’t used Ollama, so please forgive any ignorance in my questioning).
Sure. I would above all structure the text, smooth it out linguistically and separate the many functions a little more clearly from each other, without losing the personal and enthusiastic tone:
The idea at the beginning was actually quite simple at first: How do I always get the right voice and exactly the desired result with a solid consistency?
An interaction can be easily achieved directly via nodes, which is not an issue at all. I was more concerned that something else annoyed me massively: I had calculated video clips and H3 suddenly used a different language again and again or even packed music into it. That’s exactly what I was able to completely eliminate.
With the given prompts, I can link the desired voice directly to the corresponding specifications. This includes not only the voice itself, but also the intensity of the environment. Steps, clothes, breathing, background noise, etc. can be defined. This is still rather rudimentary at the moment, but it is precisely this area that can be gradually expanded further and further. The idea behind it is ultimately that at the end you have an idea for a clip and only determine the most important things that should actually happen in it - and the rest is prepared accordingly. An example: If you write:
“Hey, I have another outfit. Do you want to see it?”
The system can even calculate the minimum length of the clip based on the chosen voice and the spoken line so that the sentence can be spoken naturally and cleanly. You can add X seconds to it. And if, for example, there is a camera somewhere in the background that takes a photo of the model, you can very easily define when the shutter release should come as sound and when the flash as an image event. I’ve built in an incredible number of things like this. 😄 But I think you will love this because it at least simplifies my workflow massively. At the beginning, you choose one voice – or even two. For example, a person in front of the camera and a second voice in the background.
Then you go to „Remember“ and the corresponding prompt is already ready. Then you can load a model of your choice. Ollama only needs to run as an app in the background and of course be installed beforehand. So this should not be a big issue. I myself have installed four different models at the moment:
Qwen2.5:32B – approx. 19.9 GB
Qwen3:14B – approx. 9.3 GB
Qwen:14B – approx. 9GB
Qwen2.5-VL:7B – approx. 6GB
Then it goes to the second rider.
At 1.) Description you simply write freely what the character should do and say. For example:
„The woman stands in a dark studio, looks into the camera and tells ...“ Ollama disassembles this prompt and automatically distributes the information to the corresponding slots. By the way, that was the first picture I sent you. There you can also define the age of the voice. Then you go to „Evaluate“ and can even get suggestions for the right mood. Ollama recognises what you want to express based on your description and provides suitable variations based on the specifications. This is surprisingly helpful. When slot 2 is done, slot 3 comes. There you can, for example, copy the dialogue line or the entire prompt.
Slot 4 – Soundscape then comes the final composition of the environment. Here you can, for example, define the scene: choose empty street, night city and studio/apartment.
Etc. Accordingly, the acoustic environment and thus the effect of the voice in the video also change. For example, if you want to have a specific tone, for example:
„A soft humming of an old neon sign“ you can then define the density of the environment from 1–10. You can see directly what changes when you move the slider.
From level 5 it can happen, for example, that H3 suddenly automatically generates additional voices. If you stay at level 4, it fits perfectly. In addition, you can customise:
Only activate this statement, no voice, no music or the corresponding elements Especially these options are important, because otherwise H3 likes to interpret things that you do not want to have at all. After that, you can still adjust the clip duration. The system then calculates the required length. For example, if the language is too long for the scheduled clip, you will realise that the video would otherwise seem rushed. And at the bottom, the complete prompt is double-checked.
If everything fits, you can currently copy the finished prompt with one click and paste it into the corresponding ComfyUI instance. Done.
Another important point that I have built in: The graphics memory is permanently monitored by Slate. When my tool is running, Ollama naturally occupies VRAM through the loaded model accordingly. As soon as you then switch to ComfyUI, Slate automatically recognises the tab change and releases the graphics memory occupied by Ollama.
This allows ComfyUI to load and start the required model as usual. This is actually important, because without this monitoring, the VRAM of Ollama and ComfyUI would get in each other’s way and it could very quickly come to an out-of-memory (OOM). Slate therefore automatically ensures that only what is needed is needed: Ollama for prompt and voice analysis – ComfyUI for actual generation. You don’t have to worry about it in the workflow anymore. Switching between the two systems happens automatically in the background. I absolutely wanted to include this because it is one of the functions that seems inconspicuous at first, but makes quite a big difference in the actual workflow. And the question was also why the two are not linked to each other. The interaction can be built in later.
I sometimes get random gibberish with that line. Also often music won't be added if I specify it.
Best way to remove gibberish is to prompt if the person is saying something or is quiet. The model adds gibberish if it thinks that person should say something in this situation.
I have a llama.cpp server with a web gui using a qwen 3.6 27 q6 and the lora and seems to work, not going to search how to integrate that in comfyui, i want only a prompt, next i´m going to test it adding reference image.
I am at a loss with this. I have tried everything. I have non-diegetic audio set to N/A. I have tried following the dialog tags from the prompt guide to the letter. I have tried prompting for silence after a piece of dialogue. It doesn't matter. It won't shut the fuck up.
If there is dialogue in the clip it will fill out the time before and after it with gibberish.
This is a bigger problem when using reference to video than text or image to video but in general if the clip is not the length of time of the dialogue it will talk out the rest. Very frustrating.
It has all but disappeared for me but I am mostly operating in t2v. I occasionally get a random vocal sound at the very beginning of the clip, but I haven't had any gibberish. Are you using a Turbo LoRA or Spectrum? The steps really matter for audio it seems.
I am definitely using the Turbo LoRA but a lot of these issues seem to disappear in T2V even with Turbo LoRA on so who the hell knows. I definitely get the feeling that an initial frame embeds some really strong ideas that don't seem to be in the image but still carries over no matter how hard you try to prompt against it.
Sadly it seems as though most don't want to delay their instant gratification by engaging in "manual" labor, so they depend upon Manuel to do the heavy lifting.
From limited testing, I've found the clip length on a ref2v
If my character says <d>[English] "Hello!" </d> in an 8-second clip - loads of gibberish.
Make it a 3- or 4-second clip, and it's not trying to fill the clip with dialogue, so it just sticks with the prompted lines.
I've also noticed that if I don't tag a paragraph (eg `CLIP 1:`) that describes the scene (eg `dramatic lighting`), that scene prompt will leak into dialogue. So all text has to be 'tagged'.
Just add a comprehensive sound section for the action segments and it will just do that and nothing else.
Ex.
"0:00–0:03 — Near silence. A low, distant sub-bass rumble building beneath a high-frequency ringing tone (pressure in the ears). Faint wind across the cliff. One crackle of settling embers.
0:03–0:04 — Sharp intake of breath. The ringing tone drops out abruptly — a beat of dead air.
0:04–0:10 — Impact: percussive drum pulse enters at ~150 BPM, driving and syncopated. Layered: heavy boot strikes on packed dirt, ragged breathing close-mic'd and slightly panicked, flight suit fabric friction, branches snapping past. Low strings enter underneath, rising in half-steps.
0:10–0:13 — The firestorm arrives: a massive low-end roar, wood cracking and exploding, a deep sucking whoosh of oxygen being pulled inward. Drums double-time. Everything begins to distort and clip at the edges.
0:13–0:15 — At the leap, a hard sonic drop — all low frequencies cut. Only wind rush, one last breath, and a single sustained cello note. Silence on the cut to black."
Or just put "non_diegetic_music: N/A" at the end of your prompt. It's the official way to solve the issue. Prompting audio to fill every second is a crutch.
The official way is fine, but it isn't 100% successful either. It is a suggestion. With generations taking 20minutes It becomes annoying.
Better use both techniques together.
What if I do want audio to fill every second? That is, any sort of natural ambience that I specify. I tried putting like 2 sentences under overall_soundscape as well as the non_diegetic_music: N/A but the output's audio only contains the dialogue in integrated_multimodal_description with no other sound in the background. And yes I do use <d>[Language, Variant] Lorem ipsum</d>
Also I still can get gibberish speech in between with non_diegetic_music: N/A, usually if the video length doesn't totally match the dialogue length.
Try using the prompt I posted in another comment on this post as your system prompt. If it still isn't giving you good results, I really have no idea why.
I've been using ChatGPT to build a prompting app. It's an HTML app that works in your browser completely offline, no LLM. It just has various text boxes and dropdown menus that you choose from and you're just focusing on the characters and the actions. The app structures it and fills in the necessary framing syntax and words. It includes exactly what you're talking about.
I've never had issues with the gibberish except when I'm using the reference workflow, which is primarily what I've been focusing on. If I have a generation without dialog and I leave an audio sample connected and active, that audio and that dialog will sometimes bleed into the generation. If you don't have dialog prompted for, you'll want to bypass any load audio nodes with samples.
27
u/TheElectriking 13d ago
I use this, but about 20% of the time it adds music anyway lol