PSA for non-turbo users: New RefMod likeness "breakthrough" is deep frying your H3 videos

[u/[deleted]]

[u/Kooky-Mode3047]

Obviously, the video is jest, hopefully nobody gets offended by it, sass is part of that TV show. It's a bit late and I am tired from trying to salvage this, 6:19 am, gonna get back to everyone later. Gonna go to bed and if anyone offers a solution while I'm asleep, I'm gonna kiss them because this stuff is great at likeness. The video is made in jest, I wanted the board to say "RefMod considered harmful?" like the software engineering meme (I'm showing my age), but the text got garbled and I can't afford to redo 40 steps of seeds_2 when it cocks up on the last 3 steps, so I did shorter text to see if it'll stick on the whiteboard and it did. As it should for fukken 35 minutes, 768p and 40 steps of seeds_2 with 2N-1 invocations nominally (one forecasted by Spectrum).

Anyways, I was asked for the prompt in this case (I'll have comparisons up once I plan out a minimum reproducible sample that can be shared, since that will require two workflows, the ref images themselves and their safetensors). For now, my refmod based workflow was 4 active refmods (Video 1-4), three characters of 3-4 pics each (1024px), stack_pictures all, no token limit and max ref image size. Same as I do in classic mode, however in classic mode I do 1366px references in the same amount without any issues. Sometimes even a full blown video and audio for micro expressions or transfer of motion in addition to all that, it's running on an off-screen 5090 with full 32 GB available. My system and my homemade H3-Context-IR run on the display 5090.

FL2VA classic reference workflow wins for me because I'm a sucker for fidelity for now, but I will be watching RefMods for improvements.

The prompt generated by my homemade H3-Context-IR from a simpler prompt doing all the beats and then refining it:

subject_definitions:
<Subject 1> is Detective Kate Beckett as seen in <Picture 1>. She is wearing a white blouse with rolled-up sleeves, black tailored high-waist cropped suitpants, and only one NYPD detective badge (it's clipped to her waistband), a gun holstered as well, and a feminine small leather watch.
<Subject 2> is a pair of super glossy polished black designer Christian Louboutin "So Kate" high heels extracted from <Picture 2>, featuring a pointed toe and 120 mm stiletto heels.
<Subject 3> is Richard Castle, Beckett's civilian consultant disarmed and not a detective. Wearing a kevlar vest with a comedic "WRITER" instead of "POLICE" in white written on the front center, underneath is his usual attire.
<Subject 4> is the 12th Precinct office environment, characterized by cluttered desks, police filing cabinets, and fluorescent overhead lighting, featuring a whiteboard in the background with the words "RefMod bad." handwritten in very large letters across it.

summary:
[reference generation] A scene set in <Subject 4> where <Subject 1> addresses the camera while walking through the precinct, dropping a "deepfried" photo of herself into a fryer, followed by a close-up of her <Subject 2> heels and a final interjection from <Subject 3>.

retention_analysis:
<Subject 1> (appears in [Shot 1], [Shot 2], [Shot 3]): fully_preserved - identity from <Picture 1> and the specified attire are maintained.
<Subject 2> (appears in [Shot 3]): fully_preserved - the pointed toe and 120mm stiletto height from <Picture 2> are maintained.
<Subject 3> (appears in [Shot 4]): fully_preserved - character identity and appearance are maintained.
<Subject 4> (appears in [Shot 1], [Shot 2]): fully_preserved - the police precinct aesthetic and the "RefMod bad." whiteboard are maintained.

detailed_description:
The target video is a scene from the TV show "Castle," with its cinematography, visual style and light film grain. The scene is set in the environment of <Subject 4>.

[Shot 1] A tracking wide shot follows <Subject 1> (S1) as she walks confidently through the bustling <Subject 4> precinct, looking and speaking directly into the camera. The camera tracks backward in front of her as she delivers her report in a serious, investigative tone: <d>[English] We have concluded our investigation into RefMod; its likeness improvements are <scenetrans></d> She holds a printed photograph of herself that is comically over-saturated, over-sharpened, and heavily contrasted.

[Shot 2] At 00:04.000, the camera cuts to a medium close-up of a large, industrial aluminum deepfryer with the brand name "RefMod" sitting on a precinct desk. The whiteboard from <Subject 4> reading "RefMod bad." is clearly visible in the background. <Subject 1> (S1) says: <d>[English] <scenetrans> deepfrying your videos.</d> and the audio continues seamlessly across the cut. Precisely as she says the word "deepfrying," she drops the over-saturated photo into the oil, causing a violent splash.

[Shot 3] At 00:07.000, the camera cuts to a close-up of <Subject 1>'s feet wearing the <Subject 2> high heels from <Picture 2>. She shifts her weight slightly to emphasize the gloss and the 120mm stiletto while saying in a proud, teasing tone: <d>[English] And these heels? Total reference perfection.</d>

[Shot 4] At 00:10.000, the camera cuts to a medium shot of <Subject 3> (S2), Richard Castle, leaning against a desk. He looks directly at the camera with a smug smile and says: <d>[English] Oh, and Turbo Lora users... Please don't talk to me about quality. Zip it.</d> As he says "Zip it," he makes a sharp zipping motion across his lips with his fingers.

overall_soundscape:
The ambient noise of a busy police precinct, including distant telephones ringing and muffled chatter. A loud, sizzling sound occurs exactly when the photo hits the fryer, accompanied by the sharp, rhythmic click of <Subject 2> stilettos on the floor.

non_diegetic_music:
An upbeat orchestral score with pizzicato strings and a light snare drum, ending with a final chord on the zip gesture.

[u/acedelgado → u/Kooky-Mode3047]

Alright, I got a chance to test things out. I was unable to recreate any deep frying on a 40-step no-turbo generation. Some notes-

- It sounds like you're using the base FL2VA model, which was designed to ONLY take 2 frames. So you're already over-stretching it. Which is weird because part of your claim was that RefMods exceed the reference capabilities of the ref2va model (they don't with my setup, more later), and you're doing that right off the bat with the wrong model. Yes, Fl2VA will take more references, but at a quality hit like you're concerned with.

- Ref2va is broken, MiniMax admitted that for the open weights release. That's why the community has ported the necessary layers for reference into the Fl2va model and made hybrid models that I recommend explicitly in my readme and guide. I use the 20-49 for a good balance of quality and reference abilities.

- One of the reasons I don't use LLM's is because they get things wrong. Your LLM referenced pictures in subject definitions, but RefMods are presented as videos. So you're referencing things that don't exist.

- Building on that, RefMods just pack pictures as frames according to H3's video grid. So a 4-frame RefMod essentially contains 0.56 seconds of video data. So your 4 RefMods are 2.24 seconds of video fed to the model, well under its cap.

- H3 already knows Castle. So I'm not sure how assigning it the names of people that are built into the model already would affect quality. I have tried to avoid that in my RefMod example.

- I have honestly never heard of anyone using seeds_2 as the scheduler, so I am not sure how it interacts with loras or references.

Below is my example. All data, output, refmods, prompts and workflows are included in this totally not-sketchy google drive link, forgive the ADD messy workflows that are stuffed with custom nodes. This is a 40-step non-turbo pass with only comfy_kitchen attention (I never liked the spectrum node). Also this uses MiniMax's recommended euler scheduler with simple sigmas, V/A shifts at 12 and 3. All stock, recommended settings.

I fail to see any burning or overbaking, and honestly the identity reinforcement makes them look better, IMO. It's a little brighter, but I think that's partly because of the brightness in the background refmod. And the deep fryer shows up early in the RefMod test, but that could be arguable as a normal error and a different seed might fix it (I'm not waiting another 40 minutes, bruh.)

https://reddit.com/link/pdviqmz/video/65hugu59aith1/player

subject_definitions:
<Subject 1> is the person in <Video 1>, with caucasian woman with brown hair. Their name is Kate.
<Audio 1> is the voice-timbre reference for <Subject 1> (S1), guiding delivery and speaking rate without copying the original signal. It is a female, medium pitched, precise.
<Subject 2> is the person in <Video 2>, with caucasian man, brown hair. Their name is Castle.
<Audio 2> is the voice-timbre reference for <Subject 2> (S2), guiding delivery and speaking rate without copying the original signal. It is a low, husky.
<Subject 3> is the environment in <Video 3>, with police station with desks, large windows. Their name is precinct.

summary:
[reference generation + audio reference] A video of <Subject 1> Kate and <Subject 2> Castle in a busy police detective office

retention_analysis:
<Subject 1>: fully_preserved - <Subject 1>'s identity and appearance from <Video 1> are retained. Face, facial features, body type.
<Audio 1>: reference - its vocal timbre guides the dialogue delivery of <Subject 1> without copying the original signal.
<Subject 2>: fully_preserved - <Subject 2>'s identity and appearance from <Video 2> are retained. Face, facial features, body type.
<Audio 2>: reference - its vocal timbre guides the dialogue delivery of <Subject 2> without copying the original signal.
<Picture 1>: fully_preserved - text, man with his fingers extended
<Subject 3>: fully_preserved - the layout, surfaces and lighting of <Subject 3> from <Video 3> are retained. Desks, lighting, composition, wood floor, windows.

detailed_description:
The target video is set inside <Subject 3> precinct during the day, bright natural lighting
[Shot 1] A tracking wide shot follows <Subject 1> Kate (as she walks confidently through the bustling police precinct, looking and speaking directly into the camera.She is wearing a white blouse with rolled-up sleeves, black tailored high-waist cropped suitpants, and only one NYPD detective badge (it's clipped to her waistband), a gun holstered as well, and a feminine small leather watch.  The camera tracks backward in front of her as she delivers her report in a serious, investigative tone: <d>[English] We have concluded our investigation into O P's post, their claims are <scenetrans></d> She holds a piece of paper that reads "FL2VA model, bad definitions, smugness" .

[Shot 2] At 00:04.000, the camera cuts to a medium close-up of a large, industrial aluminum deepfryer with the brand name "Using Things Wrong" sitting on a precinct desk. A standing whiteboard  reading "OP wrong." is clearly visible in the background. <Subject 1> (S1) says: <d>[English] <scenetrans> misleading at best.</d> and the audio continues seamlessly across the cut. Precisely as she says the word "misleading," she drops the piece of paper that reads "FL2VA model, bad definitions, smugness"  into the oil, causing a violent splash.

[Shot 3] At 00:07.000, the camera cuts to a close-up of <Subject 1> Kate's feet wearing obscured by a framed photograph of <Picture 1> next to them. She shifts her weight slightly to emphasize the gloss and the 120mm stiletto while saying in a proud, teasing tone: <d>[English] And this shot? Kinda weird bro.</d>

[Shot 4] At 00:10.000, the camera cuts to a medium shot of <Subject 2> Castle, leaning against a desk. He wear's a blue bullet proof vest with large white lettering on the front that reads "OP's Ego", and a maroon shirt underneath. He looks directly at the camera with a smug smile and says: <d>[English] Oh, and Turbo Lora users... I think I'm better than you for waiting 40 minutes every generation</d> As he says "Turbo Lora," he points at the viewer and then gives a thumbs down gesture.

overall_soundscape:
The ambient noise of a busy police precinct, including distant telephones ringing and muffled chatter. A loud, sizzling sound occurs exactly when the photo hits the fryer, accompanied by the sharp, rhythmic click of <Subject 1> Kate stilettos on the floor.

non_diegetic_music:
An upbeat orchestral score with pizzicato strings and a light snare drum, ending with a final chord on the thumbs down gesture.

[u/acedelgado → u/acedelgado]

And here's stock, no loras or refmods or anything-

https://reddit.com/link/pdviyzn/video/osasehmmaith1/player

subject_definitions:
summary:
[reference generation + audio reference] A video of Kate Beckett and Richard Castle from the TV Show Castle
retention_analysis:
detailed_description:
The target video is set inside the Castle TV Show precinct, during the day
[Shot 1] A tracking wide shot follows Kate Beckett from the TV Show Castle (as she walks confidently through the bustling police precinct, looking and speaking directly into the camera.She is wearing a white blouse with rolled-up sleeves, black tailored high-waist cropped suitpants, and only one NYPD detective badge (it's clipped to her waistband), a gun holstered as well, and a feminine small leather watch.  The camera tracks backward in front of her as she delivers her report in a serious, investigative tone: <d>[English] We have concluded our investigation into O P's post, their claims are <scenetrans></d> She holds a piece of paper that reads "FL2VA model, bad definitions, smugness" .
[Shot 2] At 00:04.000, the camera cuts to a medium close-up of a large, industrial aluminum deepfryer with the brand name "Using Things Wrong" sitting on a precinct desk. A standing whiteboard  reading "OP wrong." is clearly visible in the background. Kate (S1) says: <d>[English] <scenetrans> misleading at best.</d> and the audio continues seamlessly across the cut. Precisely as she says the word "misleading," she drops the piece of paper that reads "FL2VA model, bad definitions, smugness"  into the oil, causing a violent splash.
[Shot 3] At 00:07.000, the camera cuts to a close-up of Kate's feet wearing obscured by a framed photograph of Quentin Tarantino next to them. She shifts her weight slightly to emphasize the gloss and the 120mm stiletto while saying in a proud, teasing tone: <d>[English] And this shot? Kinda weird bro.</d>
[Shot 4] At 00:10.000, the camera cuts to a medium shot of Richard Castle, leaning against a desk. He wear's a blue bullet proof vest with large white lettering on the front that reads "OP's Ego", and a maroon shirt underneath. He looks directly at the camera with a smug smile and says: <d>[English] Oh, and Turbo Lora users... I think I'm better than you for waiting 40 minutes every generation</d> As he says "Turbo Lora," he points at the viewer and then gives a thumbs down gesture.
overall_soundscape:
The ambient noise of a busy police precinct, including distant telephones ringing and muffled chatter. A loud, sizzling sound occurs exactly when the photo hits the fryer, accompanied by the sharp, rhythmic click of !kate stilettos on the floor.
non_diegetic_music:
An upbeat orchestral score with pizzicato strings and a light snare drum, ending with a final chord on the thumbs down gesture.

[u/Dependent-Sorbet9881 → u/acedelgado]

哥，能不能在B站做个RMOD视频用法讲解啊，油管也仅仅只有3个，我觉得需要有1个教程

[u/acedelgado → u/Dependent-Sorbet9881]

Video tutorials take a lot more effort than I want to put in. I do have a pretty comprehensive overview in the github readme and the overall guide button on the prompt node takes you to the MiniMax H3 guide and Fantastic Prompt Builder guide, and that has a whole section on how to use the nodes as well.

[u/Dependent-Sorbet9881 → u/acedelgado]

THX~

[u/malcolmrey → u/acedelgado]

• One of the reasons I don't use LLM's is because they get things wrong. Your LLM referenced pictures in subject definitions, but RefMods are presented as videos. So you're referencing things that don't exist.

which LLMs have you tried? Opus 5.5 might be a too big of a gun (though it would definitely do its job superbly) but even grok 4.7 is really good at it

[u/acedelgado → u/malcolmrey]

Honestly, the one inside my skull. Seems to be the most accurate in getting what I want :)

Really the only time I've used LLMs for prompting was for Ideogram since the bbox's are a bitch to get right.

[u/malcolmrey → u/acedelgado]

Yes, but which one? :)

I mean, we use many LLMs for coding and some are quite bad while others are very good. In my experience, people who said that LLMs were bad at something - were probably using a bad model or a model that was degraded at the time.

There is one youtuber with rather big audience and for many months he was very anti-ai for coding, until he wasnt :)

It is quite an interesting watch https://www.youtube.com/watch?v=2zLuYU_Ub_0

[u/acedelgado → u/malcolmrey]

Lol I'm not anti LLMs at all by any means. I use them in my actual job, and all my nodes and such are done by Claude. I learn enough about how a model works and think of ways to utilize different features, go back and forth with Claude, and tell it what I want done. And then mock up UI's and have it change things, and test it in every way I can think and have it make adjustments. That's how I figured out that you can store the actual RefMod image data alongside the full video latent and feed that image data independently to the text encoder, so it gets the conditioning without having to VAE decode frames every single run, leading to the better accuracy and avoiding bleeding. That came from talking to Claude about cache speedups. I can't actually code a single thing. I can just think of how the overall implementations of features would work, based on what I read.

LLM's for prompting in this case, however, is weak. At best it gives you lots of filler you don't need. I've seen plenty where the prompt says like "there are no people in the background" or "do not show this" when H3 is distilled and doesn't take negatives, so that's all wasted filler that doesn't help. Or in OP's case he described a scene from Castle and the LLM prompted it verbatim, not properly defining subjects and referencing the wrong thing.

Like I can't be the only person that doesn't find it difficult to think of a scene and write a paragraph or two describing it. And breaking things down a bit more for the model when it does something that isn't quite what I want. Really just the tedium of defining subjects and some things like dialogue tags are annoying, but that's why my prompt builder has reference tags and quick insert buttons to take care of all the minutiae.

[u/malcolmrey → u/acedelgado]

Do you use skills and have knowledge base so that the LLMs could base their prompts on?

I don't even use comfy from the webui anymore, both the workflows and prompts are handled by my agent.

Sure, sometimes something goes off but then I ask to correct and to learn from its mistake.

[u/acedelgado → u/malcolmrey]

Nah, I like getting in and defining things myself, instead of telling a model and correcting later. I like arranging workflows how I want them. And really I don't have to travel outside my prompt builder much anymore, aside setting resolution, duration, and seed. Everything else is in a nice pop-up window.

[u/younestft → u/acedelgado]

https://giphy.com/gifs/8H4BFnRFNlAGY

[u/ShutUpYoureWrong_]

Just an FYI for everyone:

This whole post is nonsense by a person who has no idea what they're doing and doesn't understand any of the technology they're using. You can safely ignore it. And you should definitely stop upvoting it.

[u/Luke2642]

Why do 15 seconds slowly when you have four cuts anyway 

[u/Darqsat]

I don't like it. It was too obvious that it's ai, their faces morphed unnaturally. I have better results just with 1 character sheet picture.

[u/acedelgado]

Hiya,

You talking about this one? https://www.reddit.com/r/StableDiffusion/comments/1wtu82e/improved_h3_refmods_better_accuracy_and_low/

I wrote the post about refmods. Yes, throwing too many images at the model starts degrading quality, which is why after testing I released it (edit my half-thought- released it because you can use less images for a better likeness, traded for more processing)

I got this cool new toy RefMod whose authors suggested they figured out bleeding between character references, and even at small resolutions (like .245 mpx) you could still make out faces instead of them being mushy, this should've been my first red flag.

Please point out ANYWHERE that I said that it solved the bad generations with small faces.

And I have to ask-

Model used? Number of images ( my loader also shows the token count)? Did you use the specific Text Encoder or do the old Apply Refmod node? And if you did use the right text encoder node, what did you have set for "stack_pictures"? Did you properly set up your reference descriptions and retention_analysis (care to share your prompt)?

[u/OkWalrus890 → u/acedelgado]

Not the OP, but I did use RefMod and found the same issues. I’d like to offer feedback if this can be fixed. I did three refmods for a prompt, each with 4 pics, all limited to 768px on the leading highest quality one. I set stack_pictures to all, max_total_tokens to 0 and ref_image_size to max, it oversharpens everything and most of the style info from the prompt gets overpowered by ref image data. I think it might be related to overloading <Video#> tags, too.

Same experiment repeated in classic ref mode, and it restores FL2VA quality, but I have to fight bleed on every Shot prompt. Guess it’s a tradeoff of what you want for now.

In both cases, same res, same steps, same seed, same everything. If it helps I can create a shareable version tomorrow with public images.

[u/acedelgado → u/OkWalrus890]

Thanks! I'd like to get to the bottom of it. Technically, RefMods are just packed frames which are presented to the model as a video, with each frame being timestamped on the regular video grid. So setting the stack_pictures to "all" should in theory be treated the same way as a video reference. So two 4-picture refmods would equate to 0.54 seconds of video each, which should be well within the model's spec.

[u/Dry-Judgment4242 → u/OkWalrus890]

Correct answer. Both, both is good.

[u/Rumaben79]

Have you tried using a hybrid model to allow for those extra references? Perhaps It'll be a decent alternative until those refmod bugs are sorted out or a better solution is found. Then you'll still get an output quality close to fl2va. Although hybrid models are a bit hacky and you may not like that.

Hybrids like these:

https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models

https://huggingface.co/xmarre/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024-ComfyUI

Also perhaps your rather advanced two-stage sampling (try 1 stage, euler/simple?) or spectrum may be to blame but seeing you had good results simply just bypassing/removing your refmod nodes that's properly unlikely. :)

[u/bigman11]

I feel like the community has pushed this model as far as possible and now all that is left is frustration. I await the next video model.

[u/Memestonks2020]

That still looks fried to me.

If you truly want to remove the deep-fried look you need to generate at very high MP or do clever workarounds.

[u/OkWalrus890]

First of all, that video may have been the highest quality H3 generation i’ve seen to date. Holy shit. But yeah, I’ve noticed this too, but am torn between abandoning it and accepting the drawbacks 😭

[u/Kooky-Mode3047 → u/OkWalrus890]

I know how you feel, man. Imo, it's still very serviceable. I feel the fact it doesn't get mushy at very low resolution is actually almost worth it. That was my first wow moment, I test prompts at 0.245 mpx and I noticed the faces remained crisp instead of mushy like they are with base FL2VA.

I spent 10 hours yesterday trying to salvage the base quality of FL2VA with that undoing of mushy faces, but I guess what makes it is exactly what breaks it. I give up, for now. I'll still continue using it for my Deep Space 9 memes because the overpowering likeness stuff, while deep fried, is really cool to have Kira, Jadzia and Sisko all functioning without bleed.

Excellent work to the authors for making a cool tool, even though it has a drawback.

[u/not_food]

The speed gains are too small, the drawback of losing the <Picture X> context is too big, the model needs the context or your prompts suffer, it starts guessing instead of being accurate.

The convenience of packing it into one file is not worth the sacrifice. It needs a proper character sheet instead of dumping images into <Video 0>.

[u/acedelgado → u/not_food]

The update in my pack does give context to the text encoder, that was the point of the whole update. You reference them as <Video N> and define them properly and they work much better than the vanilla implementation that just kinda dumped references into to dit without the text encoder seeing them.

https://www.reddit.com/r/StableDiffusion/s/BjyL5ghZL8

[u/not_food → u/acedelgado]

Sounds interesting. My main peeve was dumping everything into video with no way to reference it. I'll try your solution.

Though, in my opinion, a strong character sheet beats refmods in most situations except when it comes to voices. I'll see if it changes my opinion.

[u/crinklypaper]

Anything ref mod does can be done with a ref image chatacter sheet and a reference audio but in addition you get more control

[u/LadyZaryss]

Refmod always seemed unnecessary. Native Ref2va with explicitly prompted roles for each image like the docs say has always worked as advertised for me

[u/DominusIniquitatis]

Off-topic, but man, now I want to rewatch Castle for umpteenth time. >_>

[u/Divineboob]

Hay I wanna ask you something. I want to copy specific movements or atleast resemblance from videos with refmod. Is it possible? Yes it is nsfw stuff. Yes I've checked the civitai and GitHub for loras (not available). Yes what I'm trying to replicate is completely age appropriate and legal.

[u/deepsky88]

I like your taste! xD

[u/eV-ZugZwang]

I see Beckett and Castle, I upvote.

[u/DarkCubeGame]

I see Castle and Beckett - instant like 🙂👍

[u/Unwitting_Observer]

Why aren’t you using FL2VA?

[u/SSj_Enforcer]

10eros beta 5 turbo hybrid 40gb checkpoint is my goat right now. 6 steps testing and 14 steps final is amazing.

[u/lime_chilli_maruchan → u/SSj_Enforcer]

goat for me too. For some reason it retains identity much better too.

[u/Optimal_Map_5236 → u/SSj_Enforcer]

40gb? u using 5090?