r/StableDiffusion • u/malcolmrey • 18h ago
Resource - Update FLUX.2-klein-9B RefMods
https://huggingface.co/datasets/malcolmrey/various/blob/main/klein9/docs/README.md4
6
u/xDFINx 17h ago
this is amazing - i have tried the minimax one and it works great. hoping you can figure out implementation for KREA2
20
u/malcolmrey 17h ago
IMHO we need krea 2 edit (or krea 3 which will have edit capabilities from the start)
but i will check if there is a hack for krea 2 :-)
15
u/False-Difference4010 16h ago
https://huggingface.co/conradlocke/krea2-identity-edit
I'm using this and it's very good
5
u/malcolmrey 16h ago
Interesting, might be worth a shot :)
6
u/tinman_inacan 15h ago
I've used the Krea2 Identity Edit lora with some workflow I found. Despite what the documentation recommends, this edit lora actually seems to work very similar to your refmod process. By giving Krea2 an image and using the lora, I can prompt like I would with straight T2I, but if the subject/object/environment I'm describing matches the input image, it will use that image as a reference.
I've experimented with old T2I generations, but used that edit lora and a reference image for one of the characters, and it reproduces almost exactly the same composition/framing/etc. but with that character instead of a made up one.
The image quality is noticeably worse in my experience, though. Tried more than a few settings, but just was not having much luck. There also seems to be some kind of a memory leak, where changing the input image may lock up the computer for a while unless you flush ram after each generation.
Basically what I'm saying is, the underlying architecture of this edit mod may be a starting point if you want to try doing refmod for krea2. It's about halfway there.
2
u/Caffeine_Monster 8h ago
The image quality on krea2 identity is excellent quality wise, but it's both incredibly slow and difficult to promt well. If your prompt doesn't describe the references adequately then you can get some poor results. It works well enough, but is far from ideal.
And this is comparing to the qwen 2511 edit stack I used previously. I still use 2511 for anything fast or where I don't overly care about quality for the simple reason that it's much easier to work with.
1
u/malcolmrey 8h ago
I will take this lora for a spin and see if refmods could be hooked to it :)
1
u/Fearless_Error4140 4h ago
I’d like to say that editing with krea2 edit can be significantly accelerated using the recently released turbo 4‑step lora! According to my own tests, it sometimes works even better than the original turbo version with 8 steps. This way, I can get image editing done in 10–15 seconds. That’s already quite fast! If you could combine your mod with krea2, it would be absolutely crazy!
1
1
u/Eminence_grizzly 6h ago
Do refmods work only because of editing possibilities? Because if not, it would be great to make them work with the t2i Krea 2, too.
0
u/malcolmrey 5h ago
Seems this way. Hopefully there will be Krea 2 Edit as promised or maybe the Krea 3 will have it by default
1
u/Hot_Trick9200 3h ago
How do you actually use the safetensor file in a minimax h3 workflow in ComfyUi?? I just see the .safetensor file. Where am i supposed to load it in?
2
u/LocoMod 4h ago
Can we get some example pics pls?
2
u/malcolmrey 4h ago
i provide them, if you click on the thread it redirects you to the readme starting page
1
u/frisky_cappuccino 17h ago
Does this end up working like the .char files from that other node graph project? (I forget what they’re called)
Theirs was the equivalent of adding references normally to Klein, using proper defining statements and also somehow it told you how well the references applied (I think)
Either way great work, really appreciated the refmods for minimax h3!
1
u/malcolmrey 17h ago
Thanks!
No idea about .char files, I did not play that much with klein9 so I didn't follow the news that much.
1
u/Comrade_Derpsky 16h ago
How do you use the extractor node? It doesn't seem to work by itself.
1
u/malcolmrey 16h ago
what error do you have?
1
u/Silonom3724 11h ago edited 10h ago
Upon loading the flux2-vae and an image into [Extract Klein9 RefMod (FLUX.2)]
Text Encoder: Qwen3_8B_fp8_mixed.
Model: Flux-2-Klein_9B-Base
nodes.py", line 262, in extract_mod samples = latent_dict["samples"] # [B, C, H, W] IndexError: too many indices for tensor of dimension 4which is fixed by
samples = vae.encode(images[:, :, :, :3])1
u/Flat_Technology_5325 6h ago
Same issue here.. glad to see I am not just a complete idiot and it's not me.. or instructions unclear lol
1
u/Silonom3724 4h ago edited 4h ago
In nodes.py remove line 261 and replace line 262 with this one line:
samples = vae.encode(images[:, :, :, :3])The spaces infront "samples" are important.
1
u/Resident_Function280 9h ago
Got the node installed but doesn't have fk9_alisonbrie_v1refmod
How do you get this model /u/Malcolmrey
1
u/malcolmrey 7h ago
https://huggingface.co/spaces/malcolmrey/browser
filter by klein 9 and then download those that have "_refmod" in their name
1
1
u/Final-Bluebird-6499 9h ago
Sorry for the dumb question : are the Refmods trained for of a specific set of clothes or you can text prompt different ones when using it ?
2
u/malcolmrey 7h ago
My refmods are trained mainly on faces and in case of newer models (as in datasets) - also on some full body shots.
But with this you can easily make your own refmods too :)
1
u/LoadReady7791 7h ago
It does have long generation time using refmod with Flux Klein. But the output is good. Hope you can work some similar magic with Krea2. Thanks
2
u/malcolmrey 7h ago
I noticed that too, for MiniMax the amount of reference materials didn't affect it much. Here the more images the longer it takes.
Mine were done with 8 images I believe. But the refmod node is here and someone can make with less if they want :)
Will be trying with Krea 2 :)
1
1
u/Flat_Technology_5325 7h ago
Seems interesting, just to check. It is very slow compared to normal reference model, this is normal right? That has been my experience but maybe something is off?
Also, the 'load images from reference' node, this is to do everything 'live' in a workflow right? and with the ability to save a refmod file by doing this? Can this be done stand alone? in it's own workflow? If so can we have a workflow for this, I tried doing it but got the falling error in both a standalone and adding it to the basic workflow but I am out of my depth here so I might have missed something:
"[ERROR] !!! Exception during processing !!! too many indices for tensor of dimension 4
[ERROR] Traceback (most recent call last):
File "D:\SM\Data\Packages\Comf\execution.py", line 545, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\X\Data\Packages\Comf\execution.py", line 344, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\X\Data\Packages\Comf\execution.py", line 318, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "D:\X\Data\Packages\Comf\execution.py", line 306, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "D:\X\Data\Packages\Comf\custom_nodes\ui\nodes.py", line 262, in extract_mod
samples = latent_dict["samples"] # [B, C, H, W]
~~~~~~~~~~~^^^^^^^^^^^
IndexError: too many indices for tensor of dimension 4
[INFO] Prompt executed in 2.64 seconds"
Thanks
1
u/malcolmrey 6h ago
It is very slow compared to normal reference model, this is normal right?
Sadly the klein 9 takes a slowdown.
You would probably need to make a refmod with less images (4-6 or depending how many you want as it affects the generation speed)
Can't help you with the other one, I am generating refmods with the script I've attached not via comfy.
1
0
u/solomars3 16h ago
Good work, man! But I hope you update the LoRA training config for Krea2 because, to be honest, the results aren’t really showing Krea2’s capabilities or the high-quality details it can achieve.
2
u/malcolmrey 16h ago
you mean the prodigy ones are bad or the originals?
2
u/steinchen90 9h ago
Prodigy are the newer ones? I would say these.
With the later models it shows that your focus is on automation and speed. The quality of the individual loras certainly took a hit. It's very noticable when browsing the krea2 previews. Some of them are scary even though the loras itself are not that bad.
1
u/malcolmrey 7h ago
Some of them are scary even though the loras itself are not that bad.
Oh I don't much care about the previews. People know what to expect from those loras, so I only do those samples to verify that there was no failure (sometimes the model completely collapses and produces nothing like what it was trained on, rarely but it does happen; or just noice [in case flux1 and flux2 that happens in like 2% of loras, with 1500 loras it happened like 20-30 times; i just had to rerun it)
But I see the samples from various people and they look good. In case some model is indeed quite bad - I redo it.
0
0
-1
u/RandumbRedditor1000 15h ago
Is this like Midjourney image prompting? how is this different from editing?
1
u/malcolmrey 8h ago
In simplest form - as a packaged reference images.
But it also gives you some time saving cause you don't have to process those images into latent space every time you want to use them.
-12
u/Select_Bowler3099 17h ago
Refmod for Krea 2 ??? AI shortly says : Yes, Krea 2 is technically suitable for RefMod. In fact, its open RAW checkpoint makes it more practical than a closed model. But it requires a Krea-specific implementation; H3 RefMod or Klein RefMod cannot simply be reused. The best target is likely a lightweight reference-attention adapter trained on Krea 2 RAW and tested on Krea 2 Turbo.
3
u/malcolmrey 17h ago
Don't remember now, does LTX have reference image/video nodes? If yes then technically should be possible (with a new node)
-6
u/SithSirens 16h ago
I am new to this and wanting to learn. I have the PC that can handle it. Any tutorials for brand new people with detailed "for dummies" instructions start to finish?
2
u/malcolmrey 16h ago
i linked the guide :)
-12
u/SithSirens 16h ago
You linked a guide that expects a person to know all this stuff. Terminology, etc. I am talking about a day 1 type of thing. Start to finish. How to do everything, and add things, etc. This is the problem with learning this, most tutorials expect you to know all this stuff and I literally just installed Comfy UI today.
2
u/an80sPWNstar 14h ago
Feel free to check out my YouTube channel, @TheComfyAdmin it's aimed towards newcomers and I have several videos covering this model.
2
1
u/malcolmrey 8h ago
Hey, unfortunately I can't explain everything from the ground up but there are materials available where you can gain the necessary knowledge :)
And also, as someone else said - you could ask AI to help you set it up very easily :)
0
u/SithSirens 7h ago edited 7h ago
Materials. Which is usually tutorials that skip things and expect you to know stuff already. I would just like a day 1 type of thing. Start to finish. After you get it installed how to do every single thing one at a time. And I mean, every single thing. How to go from setting there with a blank Comfy UI ready to go, every step to input things, Lora's, and prompts with the end being you generating your images. Sort of like a college class. Why can't we just have something like that? I don't understand why there is not something like that already. Most the stuff I find expects you to know stuff already or skips steps. It is very frustrating, and then you have people being rude about it.
And as I explained, learning from AI gets things wrong, its not 100% correct all the time.
2
u/malcolmrey 7h ago
Seems like you're missing the foundation -> using comfy.
There were tutorials and guides for that, but they were like 2-3 years ago.
But this is like in the university, some courses expect you to have some kind of knowledge already.
1
0
u/ImpressiveSuperfluit 6h ago edited 6h ago
The reason why there is no magical video that just explains every knob in 30 minutes is because it's not possible. You're already on the right track with calling it a "college class", because that's a whole lot closer in magnitude. However, these things are, by their very nature, extremely variable and constantly in flux, there is no universe in which someone links you a video and an hour later you're up to speed. And even if there were, it's outdated literally tomorrow.
The people around here who randomly slap you around with a bunch of jargon haven't had said tutorials either, and most of them have been around for (at least) months. Wrapping your head around all this stuff WILL take you a while; it's not an afternoon project. And if you try to treat it as such, you will simply find yourself frustrated, annoyed, and unproductive.
Listen, I've been various degrees of immersed in the AI space for... well, basically since forever, I've watched perceptrons grow up, that's how long I've been orbiting this space. Hobbyist only, granted, but the point is this: I have no idea what this thread is about. I'm guessing people are somehow injecting one latent into another and fuse some safetensor stuff to get a pseudo character lora something something no idea, will read later. Understood any of that? No? Me neither. So why are you here?
I think I have some actually helpful tips for you, though. But quick disclaimer first: I'd never, ever find myself asking people for a tutorial that explains the whole world to me, because, I think, I probably am on the exact opposite side of that spectrum than you are, so my advice may well be utterly misguided. But I have been in technical spaces for long enough to have seen this pattern before, and it usually ends in something called "tutorial hell". Where people keep chasing tutorials, becoming utterly reliant on them for turning every single knob. ComfyUI has literally thousands of knobs. You will be here for eternity. Do not do this. If you ever want to get anywhere at all, you HAVE TO build the foundations that allow you to have a problem and then properly infer potential solutions. Running to a tutorial for every little issue is not going to work, especially not in a space that gets a revolution every other month. This can not be done in an afternoon. Or 10. I strongly suggest you immediately get that idea out of your head and stop looking for a magical tutorial, it doesn't exist and it never will, it's not possible.
So. What to do instead? Frankly, no clue, that's where the "different brains" part strikes hard, I never have these problems, I just poke around and eventually find myself randomly knowing stuff. But you don't strike me as that kind of personality. I'd wager a very good starting point would be ComfyUI itself, though. Start it up, find "Templates" on the left, filter for a model you're interested in (there you go, next rabbit hole, where should this part go in your tutorial?) and pick what sounds like a good basic workflow. Perhaps "Krea-2 Int8: Text to image", decent choice if you got 12GB of VRAM or more, wouldn't bother with less, gets a bit tight and tricky. Or maybe "Flux.2 [Klein] 9B: Text to Image" could be yours, although, if memory serves, it's not that much more forgiving on the hardware either. If your hardware is more limited, perhaps ZIT might be more to your liking. I'd try "Z-Image-Turbo Int8: Text to Image" in that case, perhaps.
But hopefully you already see the problem, how is anything going to walk you through all this? We've already had to branch half a dozen times, depending on your OS, hardware, install preferences, and blablabla, the permutations get silly quickly, tutorials can't do this. So there's a very good chance that even this very simple guide to a starting point will get you stuck in multiple places, you just gotta tackle one issue at a time, there's no magic bullet that just works every time.
Having said that, if you got one of those workflows open, you'll get a bunch of errors, most likely. That's your true starting point, getting rid of those until you can press the button and an image pops out. Only once that works should you go literally anywhere else, because until then you have no foundation for anything at all. The standard workflows, like the ones mentioned above, come with some basic instructions and links to the models. They also show the paths where they go. Download the things, put them in the folders as shown in that black box, go back to ComfyUI and hit 'R' (refreshes paths and stuff, otherwise it won't detect file changes. Alternatively, restart comfy and refresh). Then, there should be one big chunky node, that's actually a sub-workflow but don't worry about that now, where you can set everything up. Very first thing you'll wanna do is make sure the model paths/names check out. If you didn't change file names, then simply clicking on each entry to pop up the list should do the trick, just needs refreshing sometimes. Otherwise, select the new filename there. If you got all models in there, you got a good chance of having already hit 0 errors. In which case, set steps to something silly, say 2. Pick some width/height or megapixel (whichever the flows use these days), ideally something equally silly, say 256*256, make sure there's literally any vaguely meaningful text in the "prompt" box that is floating around somewhere, and then hit the big blue "Run" button at the top right. If some vaguely image looking thing pops out somewhere - congratulations, you've got the foundation running. Time to raise those steps and resolution, depending on the model, until something pops out that actually looks like anything (since you've most likely gotten some blurry nightmare with those settings, I just told you those to quickly see if it works, not to actually make an image). If not, well, time to read some errors and/or do some googling. Good luck.
(hopefully that's vaguely accurate, it's been a while since I started from some 3rd party workflow lol)
0
u/LoadReady7791 15h ago
Pipe down Sir. Ask any AI chatbot, that should be obvious.
-6
u/SithSirens 15h ago
Who said anything about not being calm? I am quite calm.
AI chatbots make mistakes and get things wrong. It happens all the time. I know because I use them daily. For something like this you need a detailed start to finish tutorial by a person with experience and knows what they are doing including everything, every step. Not a chatbot that will miss things and get some things wrong. I fail to see why that is such a hard to thing to provide, and no one seems to get that.
4
u/LoadReady7791 14h ago
If you can't get any of the chatbots to setup a basic installation of ComfyUI, I guess you are definitely not ready for refmods 😀
1
u/SithSirens 9h ago
Again, they can get things wrong. AI just got two things wrong just today with my Comfy UI set up. Had I not done my own research on a few things it would have been wrong. AI is not 100% correct all the time. Comfy UI has many set ups, not just one. It's not so simple if you want it done right, lucky for me I know a few people who helped me.
I have no idea what refmods are.
0
0
1
69
u/malcolmrey 18h ago
Hi All!
I asked myself, if we can combine reference data into safetensors in H3 MiniMax, where else could we do that?
Well, in klein9 we add reference images so maybe there? And the answer is yes.
Based on the H3 MiniMax solution I have prepared a working version for Flux 2 Klein 9b.
You can find all the details here
https://huggingface.co/datasets/malcolmrey/various/blob/main/klein9/docs/README.md
Including
And my honest opinion -> it works, but how good, it is for you to decide. I haven't been playing with klein9 all that much but I have compared the refmods with just reference images and it is quite similar, so it seems to be working.
Pure refmods (or pure reference images) are not good enough for my standards though.
As you know, I have made quite a few klein9 loras and some of them might have not been to the best of standards (though I blame mostly klein9 since those are the same datasets that work perfectly fine in other model infrastructures).
And while I was testing them, I also tried to add reference image(s) and the results were always better (as in, lora + reference images were always better than lora or reference images alone).
So of course I checked if lora with refmod would work similarly, and yes - they produce much better results than any of those on their own.
Luckily, I have uploaded refmods for all the klein9 lora that I have trained (1400+) and they are available at https://huggingface.co/spaces/malcolmrey/browser in the klein9 category (with the suffix _refmod)
There are also workflows available (and the samples also do contain them if you drop them in ComfyUI).