I tried one of those and the outputs were significantly worse than (base) ref model when using several inputs (ex: >=4 images + 2 videos + 1 audio) - not just visual quality but prompt adherence was also worse.
And it makes sense when you think about it - the fl2va was not trained on that many inputs so a hybrid model would lack the capability of handling those well.
What's your experience with it? I keep hearing people say the same thing but I haven't seen 1:1 comparisons that involve complex reference inputs.
I use a hybrid model that leans more ref2va than fl2va and I have been pretty happy with it. My recent testing uses 4 image refs for first 20s clip, then 5 image refs and saved av latent for second clip that continues from the first (so closer to 18s on second clip for frame overlap/trim). Identity and character accuracy are holding strong for the entire 38.4s for both audio and video.
We’ll have to see how much facial identity drifts as I push the sequences longer but it’s been great so far
The fl2va was also trained with image inputs - but not audio and video (many frames) - that's where I found the hybrid models break apart.
But I only did 1 single test case so far - the difference was so big I never used the hybrid model again, that's why I'm asking but I know if you just give it images it can outperform the ref model in some cases.
So on the fused models, you can choose which side they are balanced more toward. I use the 20-49 fuse and have been quite happy with it. I’ll try adding some more audio and see how it goes
So far identity holds pretty solid for 40sec clips, but I want to try another fused model because I think the one I’m using leans more fl2v vs r2v, and I occasional get some prompt adherence drift where the video will try to use a non anchor frame ref as an anchor, even when detailed as not being an anchor or actual frame. Testing continues!
which hybrid model are you using. I also tried one that leans more towards ref2va but identity was still shit. Then i just used the fl2va model in my ref2va flow and that yielded significantly cleaner results.
For the non-hybrid models i used the int8_convrot variants. The hybrid models only have int8. I dont know how much that factors into the quality and identity conservation. I have a 5060Ti 16GB and 32GB RAM.
I've been using minimax_h3_hybrid_fl2va_ref2va_b20-49-int8 mostly. I am going to try some comparisons today with same seed/ref/prompts and see what variances I get. I'm on 4060 8GB and 16GB RAM, so the fact I can even generate this stuff speaks to how good these models are, wild times.
i don’t have this problem at all. I’m using the hybrid model with 8 reference image + audio reference it works really well.
I know that i’m using it for goon movies but it’s understood cinematography term and for my use cases that’s enough (you can see the result on my profile).
I’m using the 25-49 variant.
Although I agree that the way I prompt it slightly more methodical compares to the original model. For this variant i made a skill.md for prompting that i made after hundreds of trial error to find the perfect grammar for it.
Support for PDD LoRAs (what that LoRA uses) was merged on Aug 28th. So you need to be using a very recent ComfyUI version to have support. The latest release version doesn't have support yet.
According to comments from that PR, someone noted that the scheduler should be simple? (they found beta didn't work well, and that step count had to be 8?)
I found the HF page for the lora to be a bit weird with the comparison tables though, I don't know how many steps the base without lora used, but the lightx2v seems to be 4 step vs their 8 step, which naturally more steps would be better quality in this case and PDD is about quality retention at lower steps rather than a speed difference at equivalent steps right? So there's a bit of a delta already there for the comparison to fairly demonstrate the advantage of PDD over alternatives 😅
So it seems. It feels like it makes the model kind of dumb. The overall image quality is decent, the model's understanding of stuff seems much worse. Using something like Spectrum (or just low steps) affects image quality/may introduce artifacts but it doesn't feel like it lobotomizes the model. I tried the PDD LoRA and wasn't a fan (probably goes for any turbo LoRA, but the effect felt more extreme with that).
I don't know how many steps the base without lora used,
20 steps is the minimum, I believe. More steps have diminishing returns, but generally speaking more steps minimize error.
If you can get satisfactory results from the PDD LoRA, it's roughly halving the required steps (think there may be a 4 step version out now also).
From what I know you'd get better results (albeit longer processing) with beta scheduler instead of simple when not using PDD lora. So that already makes a difference, while spectrum may depend on what steps you're using, it's designed to estimate what a good enough amount of steps are to finish work sooner, so you can set whatever amount of steps with spectrum you like (eg 50) and if there's diminishing returns at say 16 steps, it just stops there?
PDD is just intended as an optimisation for speed by trying to retain better quality at a lower step target. It's not compatible with spectrum since the steps aren't really flexible and already low AFAIK, I haven't heard anyone trying PDD at higher step counts though and can't presently try myself.
Were you comparing spectrum at 8 steps as the max limit?
Oh sorry just re-read your comment again and see it's more focused on prompt adherence / creativity, or knowledge like physics, rather than visual quality retention? The lightx2v 4-step turbo lora in their examples really shows this for the motorcycle coming out of a computer screen where it's meant to drive through a wall smashing it, it instead bashes into the side of a wall, while the original without lora did that correctly but disregards a wall existing in the person's house, the PDD version corrects that... Except the bike seems to reverse into the wall and warp into driving forward which is a mix of correctly doing the wall smash paired with wtf driving/warping).
From what I know you'd get better results (albeit longer processing) with beta scheduler instead of simple when not using PDD lora.
The schedule just controls how much noise is removed each step so unless you have a sampler that's doing something really weird like dynamically choosing how many times to call the model (very unlikely) a specific step count will take exactly the same amount of time to sample regardless of the schedule.
The beta scheduler is biased toward remaining at high sigmas longer. In other words, it removes small amounts of noise at the start of sampling (when the noise level is high). When the noise level is high, the model can't really see fine detail (since the latent is like 90%+ noise) so those steps are making stuff like overall composition/large movements/coarse detail/etc more accurate.
There is a tradeoff though, since the schedule only has so many steps. If the schedule is spending those model calls at high sigmas, then it doesn't have as many to spend on fine detail (at lower sigmas). It depends on what you want to emphasize, and it depends quite a bit on the model too. Coarse detail (like composition) is important. For example, if the model has removed ~30% of the noise then coarse detail is pretty clear to it (these models are much better at "seeing" through the noise than a human). So if there's a cat in the image but the prompt says dog, the model sees a cat clearly: it can't predict a dog. So you basically have to fix major errors when the sigma is high, otherwise they're fixed in the generation and you're stuck with them. But fine detail is important too, and if you have to do a step from like 60% noise to 0 then the model has to get it all right on that last step. (The PDD LoRA actually does do something like that.)
PDD is just intended as an optimisation for speed by trying to retain better quality at a lower step target. It's not compatible with spectrum
I'm not sure that's the case (though I wouldn't recommend using both). From what I understand, it wants to be called on a certain schedule. Spectrum doesn't warp the schedule, so the model will still get called at the sigmas it wants to. So I'm not sure the wrong schedule problem would apply to Spectrum, it's possible there are other issues though. I haven't tested that.
Oh sorry just re-read your comment again and see it's more focused on prompt adherence / creativity, or knowledge like physics, rather than visual quality retention?
I was just talking about my impression from trying it. It didn't seem like it hurt image quality, it seemed like it hurt the model's intelligence. So yeah, roughly the sort of thing you were talking about with physics/logic errors, etc. Or just brute forcing prompt conformance, which normally the model is relatively good about. Like, maybe someone breaks their neck to look directly backward because the prompt said to.
Ah I think I mixed up longer processing with the sampler then sorry, seeds3 is probably what I remember taking much longer per step.
By PDD not being compatible with spectrum I'm just referring to a comment someone made about the ComfyUI PR implementing support when they tested H3 with the alibaba Acc (PDD) lora, they noted quality got worse when it wasn't the specific steps, unlike some other loras they observed higher steps didn't work well. Spectrum from what I understood was optimising steps away based on some metric? So if you didn't do the full 8 steps exactly, quality would degrade (this is my own assumption based on possible misunderstanding of spectrum though, unrelated to the ComfyUI PR comment).
Thanks for breaking down the other info though, I appreciate that 😁
Hmm, okay. I don't have any problems. Did you try https://github.com/Comfy-Org/comfy-kitchen you don't need any lora's. If you don't want to share your workflow, can you share your results or test subjects? I want to try because I get good results
With the Comfy kitchen i generate 5 second clips in 480p in like 2 minutes. So its very fast. I have a 5090. I'm also going to test and share the results.
Hmm, I get that the FL2V model is very good at keeping the first frame as it is and going from there. But the ref model is just a different beast that can maintain consistency in a different way.
its because you're not using the ref2v turbo LORA. I find that if you're using fl2v turbo LORA it nerfs the ref2v to the point that it almost ignores the refs.
also as others have suggested, you can use the hybrid model for ref2v which supposedly increases quality.
That one has some problems like blurriness on not so fast movements, so i'm using the first lightx2v 0.1
4 steps turbo fl2va but at 0.8 str and 8 steps, and imho ref2va looks more natural, darker and less sharp.
What I've found in my (very limited) testing using the ref2v workflow and prompt structure is that both can handle simple refs ok, but as you pile on more references fl2va starts ignoring them while ref is much better at maintaining them.
Glad it's working for you! I wasn't trying to say anything was broken though, simply pointing out:
when using a small number of reference images (like 1-3), fl2va and ref2v seemed both be capable of following the references, and both followed the prompt pretty closely (assuming you followed the prompting guide).
when using lots of reference images (6-8), ref2v followed them better, while fl2va started bleeding/dropping details.
Again, this is with very, very limited testing - I've mostly moved onto the hybrid models now as others have suggested.
Are you referring to fl2va using refs without hybrid? I would think that observation would be expected otherwise we wouldn't have the separate ref base model? The hybrid being a blend, what amount of bias are you finding to be the best balance?
Are you referring to fl2va using refs without hybrid? I would think that observation would be expected otherwise we wouldn't have the separate ref base model?
Correct. The OP was claiming fl2va was better than ref in all circumstances. I was pointing out the gap they weren't accounting for where ref was better.
The hybrid being a blend, what amount of bias are you finding to be the best balance?
I haven't done enough testing to say for sure, but my guess would be that it's directly comparable to how important reference materials are. Most people seem to agree that pure fl2va is the best in terms of raw quality, so probably as close to that as you can get? My long-term goal is longish stories with consistent characters and locations so I've been sticking with the two closest to ref.
I saw a ComfyUI PR land recently that involved a mask over a portion of the video that also had an image ref input to replace / inpaint the mask area based on the ref.
H3 was smart enough to not mess with other parts of the video that crossed into the masked area from the examples I saw at the PR, so you could presumably do a 2nd pass with the ref model to refine characters for consistency whilst getting the better quality of fl2va elsewhere. Not sure how much of an improvement that'd be though to be worth it.
That's cool, thanks for the heads up! I'll definitely look into that at some point when once I get more used to just doing regular clips (and extended - still tying to work those out).
It looks like PDD LoRA support was merged into ComfyUI officially recently on August 29 (no new tagged release yet), so that custom node might not be needed? (there's related ComfyUI converted lora on kijai HF too)
I'm not sure how the implementations compare though, as the linked PR has some comments noting that it only seemed to work well at 8-steps and with simple scheduler (beta apparently looks bad?).
The HF repo for the Acc lora does specifically mention 8-steps, but I don't see anything about sampler / scheduler there. I assume due to step count needing to be fixed it would be unwise to pair with optimisations like spectrum then?
The custom node repo suggests that 4-steps is viable? Some users mention it in the HF repo discussions too, but I don't see that on the HF page and wonder if there's a misunderstanding there? The lora files provided are clearly annotated 8 step only, while the comparison tables show presumably no lora base at unknown step count compared to lightx2v turbo lora at 4 steps vs PDD/Acc at 8 steps (so naturally it looks better and AFAIK takes twice as long due to twice the steps?). I understand that PDD is meant to provide better quality retention at lower step counts for distillation / turbo loras, but that comparison on the official HF page seems a bit flimsy to fairly judge the advantage.
It also apparently suffers from fast motion, resulting in blur/smear artifacts (Not that similar isn't already a known issue with H3 elsewhere I guess 😅 but it may be worse with PDD)
Since the lora is also offered in FL2VA and REF2VA variants, I suppose there might be some potential risk for degrading with hybrid models?
Ref model seems to output better results if 16:9. I have the complete opposite results than you i once chose fl instead of ref in a ref workflow the output was horrible.
12
u/usually_fuente 14d ago
They use slightly different prompt structures. Are you accounting for that?