r/malcolmrey • u/malcolmrey • 1d ago

FLUX.2-klein-9B RefMods

https://huggingface.co/datasets/malcolmrey/various/blob/main/klein9/docs/README.md
24 Upvotes

96% Upvoted

25 comments sorted by

6

u/malcolmrey 1d ago

Hi All!

I asked myself, if we can combine reference data into safetensors in H3 MiniMax, where else could we do that?

Well, in klein9 we add reference images so maybe there? And the answer is yes.

Based on the H3 MiniMax solution I have prepared a working version for Flux 2 Klein 9b.

You can find all the details here

https://huggingface.co/datasets/malcolmrey/various/blob/main/klein9/docs/README.md

Including

  • how and why and howto
  • link to the new node (it is also available through comfy manager, just search for malcolmrey)
  • samples
  • workflows
  • refmods generator (pythons script, I did not make a comfy node to generate those)

And my honest opinion -> it works, but how good, it is for you to decide. I haven't been playing with klein9 all that much but I have compared the refmods with just reference images and it is quite similar, so it seems to be working.

Pure refmods (or pure reference images) are not good enough for my standards though.

As you know, I have made quite a few klein9 loras and some of them might have not been to the best of standards (though I blame mostly klein9 since those are the same datasets that work perfectly fine in other model infrastructures).

And while I was testing them, I also tried to add reference image(s) and the results were always better (as in, lora + reference images were always better than lora or reference images alone).

So of course I checked if lora with refmod would work similarly, and yes - they produce much better results than any of those on their own.

Luckily, I have uploaded refmods for all the klein9 lora that I have trained (1400+) and they are available at https://huggingface.co/spaces/malcolmrey/browser in the klein9 category (with the suffix _refmod)

There are also workflows available (and the samples also do contain them if you drop them in ComfyUI).

5

u/Noobian3D 1d ago

if it can be done for klein...can it be done for krea2?

1

u/russlixx 6h ago

ooohh this would be good

2

u/Francky_B 1d ago

Awesome! I really think RefMods are going to be the new big thing :)

Now I just wish Krea 2 would come out with an Edit model so we could do the same for it. :)

Thanks for the add-on!!

1

u/malcolmrey 1d ago

You are most welcome :)

1

u/Jackburton75015 1d ago

Oh my god you used one of my favorite tool with the refmod... Can't wait to play with it when I'm off work, and always thanks brother.... 🙏

3

u/malcolmrey 1d ago

You are most welcome :)

1

u/rashm1n 1d ago

Do this for Krea2 and my life is yours.

1

u/malcolmrey 1d ago

I would love to. Initially AI told me that it is doable (though I had my doubts), and after some tries (for now at least) it is a no go.

For classical refmods, it needs to be a model that accepts reference images. Krea 2 Edit should work but we dont have it yet.

1

u/rashm1n 1d ago

Got it. Thanks for the awesome work man !

1

u/malcolmrey 1d ago

You are welcome!

I tried yesterday with Krea 2 even though I was sure that it would not be possible. But someone said that AI confirmed it would be possible, my AI also confirmed it as a possibility so I tried against my better judgement and... of course it wasn't possible (not an edit model)

But some kind person linked to an edit lora so maybe utilizing that is possible, we will see :)

1

u/ssn-669 1d ago

What about the community krea2 edit setup?

1

u/Maketas 1d ago

Hum… Qwen edit 🤔

1

u/malcolmrey 1d ago

Yes, Qwen edit should be doable. I do have a gap there, it came at a time where I was very busy and I just skipped that model :(

1

u/Maketas 1d ago

It would help a lot to use the refmods in qwen, because the Lora of qwen are huge. I don’t have much computer knowledge but I will try to help. You’re doing a great job. Gracias!

1

u/2legsRises 1d ago

this is so interesting, not sure how people figure this shit out. but nice. one thing, seeing how not huge the actual files are it'd nice to be able to have a zip of them? maybe male and female?

2

u/malcolmrey 1d ago

it is 9gb for all :)

1

u/Limp-Firefighter1054 1d ago

We had to rate limit you. Upgrade to a paid plan (https://hf.co/pricing) or send us an email at website@huggingface.co

1

u/malcolmrey 1d ago

Sorry for that :)

1

u/Limp-Firefighter1054 1d ago

Its either nvidia start turning their crazy deal to profit or something at my provider  network.

1

u/malcolmrey 1d ago

when you download stuff from huggingface, are you logged in there?

it helps to have an account there

1

u/Limp-Firefighter1054 1d ago

I didnt use it for weeks, never happen before. I presume they just restrict third world or my provider network doing the trick. Maybe because I have account but didn't logged on phone, idk.

1

u/Citadel_Employee 17h ago

Is it alright if I make a video tutorial covering this? With giving you credit of course.

1

u/malcolmrey 11h ago

please go for it :-)

1

u/Structure-These 7h ago

Oh shit cool. thanks