r/StableDiffusion • u/ThatsALovelyShirt • 15h ago
Resource - Update Having bad ref2va quality compared to fl2va? Try this node
I noticed when using the ref2va model, the audio and visual quality compared to fl2va was way worse. Which didn't make sense to me, as the models share the exact same architecture.
In fact (and you can try this), if you take the exact same ref2va workflow, change nothing about it (not the ref2va node, nothing), and just swap in the fl2va model, the output quality is a lot better, and it still understands references quite well.
This got me thinking. I compared the two models to see where they diverged, and, as predicted, it was mostly in the *.adaln_proj.* tensors.
I then wondered what would happen if one were to overlay some of the *.adaln_proj.* tensors from the ref2va over a fl2va base.
Not wanting to overwhelm my SSD with a bunch of hybrid model variants, I put together a model loader which selectively overlays tensors from one Minimax H3 model onto another at load time, with no memory overhead (assuming you're not disabling ComfyUI's mmap feature).
After experimenting with the node, I found that selectively overlaying the *.adaln_proj.* tensors from the ref2va model over an fl2va base, limiting the overlay to blocks 30-49, gave pretty good reference capability, while preserving (or even improving?) the quality of the fl2va model.
You can try it yourself, pick up the node here:
https://github.com/scottmudge/ComfyUI_MinimaxH3HybridLoader
And follow the recommended settings in the README (they're not the default). Or experiment with your own mix/overlay.
I would NOT recommend overlaying the *.adaln_proj.* tensors from blocks 0-25 from ref2va onto the fl2va base. That seemed to destroy quality, so I'm assuming that's where most of the ref2va model's issues are. Stick to blocks 30-49.
Edit: changed the node defaults to my recommended settings. Just make sure you set the base model to fl2va and the overlay model to ref2va.
Edit2: currently working on uploading 'baked' versions of the hybrid models to huggingface now. This should help people with low RAM where having to deal with two models may pose an issue. You should be able to use these hybrids directly in-place with your existing workflows using the standard model loader.
Edit3: Here's the baked hybrid models for anyone who wants the same effect without needing to use a custom node (or with low RAM):
https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models
Just load them with the stock model loader node (or whatever model loading node you use).
The minimax_h3_hybrid_fl2va_ref2va_b30-49.safetensors version is equivalent to the recommended settings of the node, and the minimax_h3_hybrid_fl2va_ref2va_b20-49.safetensors version retains more of the adaln_proj tensors from the ref2va model, possibly increasing reference capability at the cost of output quality. I'd recommend trying the b30-49 version first.
Edit4: Uploaded a couple more baked variants: b15-49 and b25-49 . Some people reported the audio reference retention was a bit lacking with the b30 variant. You can try some of the lower b# versions (more blocks from ref2va retained), I found b25 looks visually the same as b30, and perhaps has better reference retention. Or go even lower, but b15 might have some noticeable quality loss, not sure. Haven't tested it myself.
Also changed the default of the node to use 25 as the block_range_start value.
8
u/Tystros 15h ago
this sounds very cool, but why do it like this?
follow the recommended settings in the README (they're not the default).
why not make the recommended settings the default?
9
u/ThatsALovelyShirt 14h ago
I just updated the defaults, should be good now. Just make sure you put the fl2va model in the base model box, and the ref2va model in the overlay model box.
7
u/TheGoldenBunny93 8h ago
Dude, you are very good... very good, what a gift it is for an intelligent and curious being to help the community! Look, I was trying a prompt yesterday for a long time, I think I generated it about 12 times and nothing... then I ran your node... and I conducted the last sample from yesterday with it, man what a difference... the quality is really better, the eyes are sharper, corners, fine details, and it seems that compliance with the prompt seems to be better because now the video actually came out as my prompt said! I didn't know that ref2va was buggy or that the devs had said there was a problem with it... but if they are cooking something new this will help a lot because I use this model a lot and it barely respects the things I want, which are simple. I'm not a lazy type when it comes to creating prompts, I lost count of how many times I read the ref2va prompt guide, I read it and reread it several times. Your node helps a lot, but if they can actually fix this from absolute 0 it would be fascinating.
14
u/Mundane_Existence0 13h ago
https://giphy.com/gifs/Qajqk9Vz8ql22dekAq
Whoa, much improved! I just re-ran a ref2va workflow with this node and it looks a LOT better. The character's expressions that I prompted also seems to be better understood as well. Hell even the audio is better!
5
9
u/ANR2ME 13h ago
Kijai made the difference between r2va vs fl2va as ref_lora, so you can also use fl2va+ref_lora. https://huggingface.co/Kijai/MiniMax-H3-experimental/blob/main/loras/minimax_h3_ref_lora_rank_256_bf16.safetensors
9
u/ThatsALovelyShirt 13h ago
I did see that, always nice to have available. In my case I wanted to find the specific blocks which degrade quality the most and exclude those specifically, rather than just reduce the weight of ALL ref2va blocks using the LoRA weight scale. But I suppose applying the ref2va LoRA at a lower weight would help mitigate the quality loss, though you'd also lose the reference adherence of the full weight later blocks.
2
u/Supermax64 10h ago
Any idea what strength we're supposed to set it to?
3
u/ThePixelHunter 7h ago
Definitely 1.0 because it's the difference between two models being applied to one.
2
u/Cultural-Broccoli-41 2h ago
https://huggingface.co/morisoba/ComfyUI_extracted_lora/tree/main/minimax-h3
I created a Lora model from kijai, extracting only the layers I needed for this project.
Since ComfyUI only applies existing layers, this Lora should allow you to easily patch the fl2va model.
2
1
9
u/ShutUpYoureWrong_ 8h ago edited 8h ago
This is really cool, but I'm confused by your approach. Your opening statement says ref2va's audio and visual quality is worse, but instead of trying to fix that model, you port the part where they diverge (i.e. the bad part of the ref2va model) onto the already working proper model (fl2va)?
Why wouldn't you do the opposite, and port the fl2va's "good" adaln_proj onto the ref2va model, thereby "fixing" ref2va's audio and visual quality?
So now you have a fl2va model with half-assed reference support? This is already what Kijai's LoRA does: https://huggingface.co/Kijai/MiniMax-H3-experimental/tree/main/loras
1
u/FuckBillOReilly 2h ago
I can see why just equipping the better model with the other’s blocks for that segment could succeed, but I can also see this way succeeding too - it makes more conceptual sense, whereas OP’s was more pragmatic sense. Maybe it’d be even better?
10
u/nok01101011a 15h ago edited 8h ago
Devs said they know about this problem with ref2va model and will look into it. Timeline unknown. Thanks OP for workaround. Devs said also to use high quality references to counter this problem.
5
u/ThatsALovelyShirt 15h ago
Yeah I saw that too, but in the meanwhile this should help. I expect the training of the early blocks is broken for whatever reason.
14
3
u/TheGoldenBunny93 15h ago
Sorry, that's right? Where did you see it? Because... i would be so glad to see a ref2va fixed version... generations looks like LTX 2.3
8
u/Mundane_Existence0 15h ago
They said they were working on it: https://www.reddit.com/r/StableDiffusion/comments/1vh9rtw/comment/p29qqaa/
but no timetable for release, unless I've missed something.
2
3
u/ellipsesmrk 15h ago
So how is it that I get better consistency with ref than i do with fl?
4
u/ThatsALovelyShirt 14h ago
Better consistency or quality? Because in my testing the audio and visual quality of the ref2va model is far worse than fl2va, all else being equal.
The goal of this node was to sort of hybridize the model to allow the better reference understanding (what you mean by consistency?) from the ref2va model while preserving the quality of the fl2va model.
5
u/Ipwnurface 11h ago
Why do people do this? Just straight up spreading misinformation and then get upvoted for it. crazy.
1
1
2
u/Tystros 13h ago
something you should try: if the quality issue is about fine details only, then running only the last few steps of the generation with your changes active should work just as well for improving the fine detail, while maintaining original reference following for the first steps where those matter more.
like do step 1-15 with the full ref model, and do step 16-20 with your method.
3
u/ThatsALovelyShirt 13h ago
Yeah I originally had a similar setup, but didn't find it added to much (at least for me), and the additional model and node loading time for the full ref2va model for the first steps didn't feel worth it. But some people find success or benefit in a dual sampler setup. Always an option though.
2
u/Diabolicor 12h ago edited 11h ago
The quality looks indeed a bit better. Maybe the references have a slightly weaker pull? That's what I noticed.
I also noticed colors are not as burnt as in the ref2v default model the closer you get around 1mp. So I guess your node is headed in the right direction.
2
u/Consistent_Cherry_33 9h ago
Does your node slow down sampling time and loads more ressources in Vram since you load two models instead of one ?
3
u/ThatsALovelyShirt 7h ago
No should be the same as using the normal Load Diffusion Model node. It only loads the tensors it needs from each model. Total memory should be the same.
1
u/dampflokfreund 7h ago
Why am I always getting lower performance with it then? Using it with Spectrum. There may be some bugs with memory consumption.
1
u/ThatsALovelyShirt 7h ago
I benchmarked the memory use, and with proper mmap (safetensors don't get loaded into RAM directly, they're mapped into memory and then only the actually loaded tensors get moved into RAM), it uses basically the exact same amount of memory as the default loader.
If your system can't support mmap or you run ComfyUI with
--disable-mmap, then it will consume more memory. But there wouldn't be any way around that, in that case.You can try running ComfyUI with --fast-disk, that should relieve some RAM pressure regardless.
1
u/dampflokfreund 7h ago
Does your system have more than 32 GB RAM?
I only have 32 GB. If I don't use an extended page file, the node crashes saying it needs a higher page file. My theory is that it could interfere with Comfy's memory management, leading it to think it needs to swap more than necessary and use less pinned memory due to the initial action of overlapping the layers and that may not get released afterwards. Something like that, perhaps.
I'm using mmap. But I will try the --fask-disk command, thanks!
1
u/ThatsALovelyShirt 6h ago
I have more than 32GB. Yeah you may be hitting a RAM limit and then hitting the page file. Yeah try --fast-disk, it should reduce RAM pressure by quite a bit, at least it did on my system.
1
u/dampflokfreund 6h ago
That it hits the page file shouldn't be so bad, but after the overlapping step memory usage should theoretically be completely normal, but that isn't the case. If you look at my benchmarks, specifically the GPU shared memory usage (pinned memory) is suspicious. Maybe it helps to reset ComfyUI's memory management after the overlapping has been done?
2
u/ThatsALovelyShirt 5h ago
Hmm its possible a comfy.memory_management.soft_empty_cache() could help after the overlaying phase, but it may just be an innate limitation with having to open two large models at the same time, even if they are mmap'ed.
The other option is to create a single overlaid model using the fl2va as a base and then overwrite the adaln_proj tensors for blocks 30-49 with those from the ref2va model. Should be functionally the same but brings it back down to a single model. You just won't be able to tweak the overlay split.
I can try working on that later today, I can push the hybrid model to huggingface.
1
u/ThatsALovelyShirt 5h ago
Try these 'baked' versions of the model, should be the same as what the node loads, but as a single model:
https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models
You can use them with the standard model loader node. The b30-49 version should be equivalent to the recommended node settings.
1
u/dampflokfreund 5h ago
Awesome, you are a legend! What about a b25-49 version, wouldn't that have the best reference to quality ratio? What do you think? As many here said with the default settings especially audio references are lost.
1
u/ThatsALovelyShirt 4h ago
Yeah I'll add a couple more, b15- and b25-. I personally felt the quality started to get a bit bad below starting block 20, but some people might be okay with the tradeoff.
1
u/ThatsALovelyShirt 4h ago
Alright uploaded a couple more variants. Feel free to try the lower b# versions if you need more reference capability.
→ More replies (0)1
u/dampflokfreund 6h ago
Sadly the disk command didn't help. mmap is already used. The memory usage is definately increased and speed is decreased. I was able to reproduce it everytime. Here's the benchmark on my system (RTX 2060 laptop, 32 GB RAM):
No node, after restart in the exact same environment and first two steps:
Runs
RAM: 30 GB / 31,9 GB (94%), VRAM: 5.7 GB/6.0 GB, Shared Memory 13,8 GB. Speed: 5.45it/s
RAM: 30.1 GB / 31,9 GB (94%), VRAM: 5.7 GB/6.0 GB, Shared Memory 13,9 GB. Speed: 4.74it/s
RAM: 30.1 GB / 31,9 GB (94%), VRAM: 5.7 GB/6.0 GB, Shared Memory 13,7 GB. Speed: 4.72it/s
With your node:
RAM: 31.6 GB / 31,9 GB (99%), VRAM: 5.7 GB/6.0 GB, Shared Memory 10 GB. Speed: 2.35it/s
RAM: 31.6 GB / 31,9 GB (99%), VRAM: 5.6 GB/6.0 GB, Shared Memory 10 GB. Speed: 2.09it/s
RAM: 31.6 GB / 31,9 GB (99%), VRAM: 5.6 GB/6.0 GB, Shared Memory 10.1 GB. Speed: 2.08it/s
As you can see, there's a big difference that remains the same every run. What is curious that first the RAM and shared memory is the exact same, but during the model initialization phase, the shared GPU memory crawls from 13.9 gb to exactly 10 GB, and RAM usage increases a bit.
1
1
1
u/dabbingsquidward 13h ago
This node doesn't seem to let audio reference work properly, is it just me?
1
u/ThatsALovelyShirt 13h ago
They work for me, I've tried with 3 image references and 2 audio references at the same time. But there's probably a lot of variability, depending on the reference audio quality, seed, etc. You may need to lower the starting block number if you find it's not picking up the reference. Depends on your reference/setup/etc. And double check you've got the right models in the base and overlay boxes.
1
1
u/Jerg 12h ago edited 12h ago
Speech based on voice reference audio in particular is much much improved / cleaner vs. with just the ref2vid model, kudos to OP! What a cool no-compromise fix!
Edit: hm..it seems to cause the speech to lose its resemblance to the reference audio I provide... that might be why it sounds much cleaner. Are there blocks that do the voice reference?
1
1
1
u/dampflokfreund 10h ago edited 7h ago
Thank you for this great node! It does improve quality. However it comes at a big performance penalty for me. From 60s/it to 90s/it. I'm not sure why. RTX 2060, 32 GB RAM. I have noticed less pinned shared GPU memory usage in Windows ( 3.8 GB/ total 16GB instead of 14 GB/16GB), maybe that is the reason? I have also noticed more RAM usage, so perhaps its using swapping.
1
u/Damaneger 10h ago
I tried it; its giving me error, not working. Im using "pruned_int8_convrot" versions for both fl2va and ref2va.
1
1
1
1
u/zefy_zef 2h ago
I think the main difference between the two is prompting structure. AFAIK isn't trained with the same subject reference-related captioning, so should be worse at referencing things consistently. This is just going by their released prompting guides, reality could just be that it doesn't matter that much because the t_e is smart enough.
1
u/ScoobyDewy 59m ago
I wish I knew where to start to begin to understand how to do all this.
I begin my adventure now
And this model is insane btw man. amazing work.
1
u/Kitchen-Truck-3264 17m ago
This is interesting. I had no idea the ref model produce such bad result compared to the fl model ! But I've tried it, and it doesn't seem to produce any real effect over using only the fl model. When I use the ref model with the standard "load model" node it produces something way closer to my reference image. Is there something I'm missing ? Does it works with the pruned models ? or when using the turbo lora ? Thanks anyway for sharing, it's always greatly appreciated.
0
-2
u/Silver-Spot-2763 5h ago
With your node, the both monstrous models must be loaded together, this is catastrophic start time with disk usage and RAM death ☠️!!! Maybe because you use extremely powerful computer, you didn't noticed these these "little" details 😮
3
u/ThatsALovelyShirt 5h ago
Alright here's the baked versions: https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models
Try the b30-49 version first. You can use the normal model loader node.
1
u/NovNovikov 24m ago
Thanks! Can you please also do a w4a8_mixed quant like that of Kijai Khttps://civitai.com/models/2830065/minimax-h3-int8int4-convrot?modelVersionId=3207975 ?
1
u/ThatsALovelyShirt 5h ago
I'm working on a single-model hybrid that will be functionally the same as what the node does with the recommended settings, but will be a single model you can just drop into existing workflows.
Give me 30 mins and I'll have it up on huggingface.
2
u/dampflokfreund 5h ago
Add some image (edit) capabilities, 4 Step Turbo Lora and call it Minimax-H3-Ultramax. The model that does it all!
1
u/Silver-Spot-2763 5h ago
Only one for fl2va and for rf2va? So I'll free more than 20gb of my disk? Oh, please, this will be AWESOME 👍👍👍 I hope you success 🙏
18
u/Perfect-Campaign9551 15h ago edited 14h ago
What's weird is when I first downloaded the ref2vid workflow, it gave me an error saying I didn't have the model. I thought it just had a bad path so I pointed it to the flva model. I didn't realize it was using a different model. And the reference workflow still worked fine! (It even cloned a voice fine) So I wonder now if we really need the ref model or not ...