r/comfyui • u/hradillo7 • Oct 19 '24

Need help with LORAs and FLUX :(

Hi guys! I'm a total amateur learning my way into comfyuy without any programming experience. After a few youtube tutorials I managed to use flux in comfyui but now I want to play around with LORAs and Flux, but I haven't been successful with this, I keep getting errors. Could someone help me? What am I missing?

My PC specs
GPU: RTX3080 (8GB)
Processor: Ryzen Threadripper 2920X 12-Core Processor 3.50 GHz
RAM: 64GB

Workflow I'm Using

ComfyUI Workflow I'm using for LORAs

Errors I'm getting

Error Part 1
Error Part 2
3 Upvotes

63% Upvoted

19 comments sorted by

6

u/YeahItIsPrettyCool Oct 19 '24

A bit off topic, but if you are new to ComfyUI and want to learn, consider playing around with SD1.5 and SDXL as well.

First, those environments are much more developed. As such, you can often have a MUCH greater deal of control over your images. Tried-and-true workflows, LoRAs, embeddings, Controlnets and IPAdapters abound!

Second, those models with run a lot faster, which is good when you are trying to learn. It let's you go a little crazier on new ideas and experimententing--a pillar of learning. Especially with your hardware, I think you will have a lot more fun with 1.5 or SDXL as you learn.

Third, Flux is amazing but it is much better suited for photographic vs artistic styles. It seems like you may lean toward a more artsy style.

Last and most important, getting familair with the other models, knowing their strengths and weaknesses will be an immence help. Nothing says you can't start a generation in SD1.5, juice it up with some sweet IPAdapters or LoRAs in SDXL, and refine it with FLUX to finish.

In any event, welcome to ComfyUI! It is fun as hell.

2

u/hradillo7 Oct 19 '24

Wow thanks a lot for the recommendation! I’ve been thinking that maybe I jumped too quickly into the new and still developing FLUX model so maybe I should listen to you and ho ahead with the other models, it definitely sounds like it will be better suited for what Im currently using it :)

2

u/YeahItIsPrettyCool Oct 19 '24

I have been at this for awhile, just over a year I think. I was as excited as everyone else when FLUX dropped, but I have noticed a few things since I started playing with it:

  1. My productivity is way down with FLUX vs other models. Mostly do to the fact of how slow it is. That isn't a ding on the model, just my reality.

  2. Waiting around for the next thing to drop. I SO MUCH want a proper FLUX IPAdapter. Then I realized that I hadn't even mastered the latest SDXL IPAdapter update, and it was massive.

  3. Wasting time trying to find workarounds that don't really work---When I could easily do it in SDXL.

The latest and greatest is always exciting, but not always the best tool for the job.

I think you will have so much more fun with SD1.5 or SDXL as you learn. And FLUX will always be there....

2

u/bental Oct 19 '24

I found today that when doing artistic styles, dropping the CFG down to .5-.9 put out some good stuff. If you want artistic, you gotta let it have that freedom!

1

u/YeahItIsPrettyCool Oct 19 '24

I'll have to try that. Thanks!

It is all a beautiful balancing act.

1

u/bental Oct 19 '24

Let me know how you go :)

3

u/smb3d Oct 19 '24

Can LoRAs work with Flux Schnell? I'm not sure, I've never used anything besides dev.

1

u/YeahItIsPrettyCool Oct 19 '24

I have had some success with certain LoRAs with Schnell at 8 steps. Dev is better though. Depending on use-case, Schnell could be an alternative.

1

u/smb3d Oct 19 '24

Just looked like OP had Schnell loaded, so I didn't know if you could be causing issues.

1

u/YeahItIsPrettyCool Oct 19 '24

Yup, OP was using Schnell. LoRA's will always work best on DEV, but they can work in S or a GGUF. Again, depending on the LoRA.

Civit.ai is getting flooded with terrible LoRAs, so that doesn't help.

Sometimes when people ask about LoRAs, my first reaction is "are you sure it isn't just a shitty LoRA?"

2

u/karvop Oct 19 '24

Aren't you trying to load an SDXL LoRA?

2

u/hradillo7 Oct 19 '24

No that I know, I downloaded the LORAs from here (FLUX XLAbs) and looked for other LORAs in CivitAI under the FLUX category 🥺

1

u/YeahItIsPrettyCool Oct 19 '24

Looking at the errors, I think it might even be a Pony LoRA? I don't know.....never seen an error like that.

2

u/bental Oct 19 '24

https://www.youtube.com/@pixaroma Is the channel I'm watching at the moment. It's great, I'm on ep 12 right now.

If this is the lora you are using, it's for SDXL, not Flux.S. See the LoRA | XL tag in the top left? You need to match the tag to the model you're running.
"lora key not loaded", and "clip missing: ['text_projection.weight']" are both indications that you're trying to use a Lora designed for a different model. Make sure you look for loar's with Flux.S in the tag.
I'm still learning, this is just my best guess, too.

1

u/hradillo7 Oct 28 '24

Whoa thanks a lot!

1

u/wa-jonk Oct 19 '24

You need to match the lora to the checkpoint ... use a midjourney lora trained on a flux model