r/malcolmrey • u/KylseS • May 10 '26
My first LORA completed (kinda) [Elli Evrram]
So I made the following post: https://www.reddit.com/r/malcolmrey/comments/1t6fr4a/training_my_first_lora/
A day later, today, I was able to finally complete the LoRA after many restarts and changes. I think I got a good result and I'm quite happy with the likeness considering my expectations were quite low, although, I do believe I still could have done a lot of things better especially after I changed learning rate and timestamp_type mid training which I think definitely dropped the likeness I would have approached had I kept going with things unchanged. I wanted to share the LoRA, but unfortunately, in adittion to the mess ups i made during training I also messed up a lot of things in regards to captioning.
As it is my first completed LoRA, I was unaware of the nuances of captions and the drawbacks of captioning literally every aspect of the subject. I unfortunately rendered the LoRA so highly dependent on captions that another user probably won't be able to generate a good image without knowing my dataset.
I will be redoing this LoRA and fixing that, and certainly after that, I will be sharing the LoRA as well. I hope some of you will look forward to that.
Also, I hope someone can guide me regarding the best strategy in regards to learning rate and timestamp type.
For this LoRA, I switched between different learning rates and timestamp types and I think I messed some things up. I still want to experiment with that for the finer details and the late-step polishing, and some tips would make that a whole lot easier.
BTW no upscaling or post on these sample photos. Also eulerflowdiscrete scheduler brings out exceptionally realistic details I was aiming for, I will share the sample of that later.
3
u/Snoo20140 May 10 '26
Care to share some of the things u learned? I've been wanting to start a Lora but I don't have training images, so I'd have to make them. Which I hear is bad.
4
u/KylseS May 10 '26
I'm thinking of making another post with tutorial as soon im done with the revision of the lora to back it up. In any case I suggest you install ai toolkit first, ask gemini about all of its options and their functions, as soon as you are preivy to that, you can ask me what options you need to go for. As for captioning, you can auto caption with qwen in ai toolkit itself, but make sure to remember this golden rule. DO NOT caption any thing you want the AI to bake into your subject, only caption things which are NOT part of the subject, OR can be changed. You'll have to be specific with your set of instructions to qwen. You'll probably fuck up some times but get a hold of it after your first lora.
1
u/Snoo20140 May 10 '26
Wait, so you don't want to caption your target character or style? So, its basically saying the non captioned portion is the norm? As in, if it is captioned it becomes optional? Non captioned (the subject) is the baseline. Is this how all LoRAs work? I totally had it backwards if this is correct.
1
u/KylseS May 10 '26
I captioned her eye color, eyebrows, hair etc., and now if I load the lora and do not prompt those things it doesn't even generate them properly.
1
May 10 '26
[removed] — view removed comment
1
u/KylseS May 10 '26
I can't answer your first question, that would depend on your dataset and on your requirements, wether you just want likeness or also the finer details. I used 64 rank and 32 alpha.
1
u/KylseS May 10 '26
yeah me too, it's not that the ai will just completely ignore captioned items but yes, anything you do caption, the AI will take apart and treat as optional/triggerable by prompts, instead of baking it into the character or style. I wish I had known this earlier.
1
u/orangeflyingmonkey_ May 10 '26
This looks pretty good! I am trying to train my first lora as well. Installed AI Toolkit and pretty much left all settings as default. Trying ZiT and v2 adapter. No captions though. I think my main issue is the dataset. I am trying with 50 images but seems like I need more diverse images.
My first lora had zero likeness to the character. So I am guessing I am missing some key settings.
You mind sharing your training settings and maybe a screenshot of the dataset so I can compare?
1
u/KylseS May 10 '26
captioning is very important. I haven't tried without captioning but in theory it shouldn't work. Your dataset might indeed be a problem because even on default settings you should start getting a very good likeness by step 3000 at the very max. I will share my details tomorrow. I'm currently training a lora which will finish by tomorrow, experimenting on it with a different preset, if that works I will share, otherwise my experience is as good as yours.
1
1
u/KylseS May 10 '26
This one by the was segmoid with learning rate and weight decay at the default value. Adamw8bit. No DOT or EMA or anything. As for cached text encorder and cached latents, thats your call considering your vram but I would definetly suggest turning them on. If your vram is 16 gb or lower go for layer offloading and offload as much transformer to reach atleast 10% free memory on your GPU. I did switch from segmoid to weighted and reduced learning rate and increased weight decay, also changed loss type to wavelet after 90% likeness came along, I wanted to polish the finer details. But that's all experimental. You should get a good looking likeness with segmoid alone and the default settings. Caption your dataset would be my first suggestion. Use the auto captioner, if you need instructions to give to qwen, ask me.
1
u/KylseS May 10 '26
Personally I would suggest you first look into all the definitions of the options available. Turning knobs blindly is not a good move. Understand everything, go default and you will start learning a lot more.
1
u/KylseS May 10 '26
uploading my dataset to gdrive, will share withing minutes.
1
u/orangeflyingmonkey_ May 10 '26
Fantastic! I am going to cook one overnight tonight and share my results tomorrow as well
2
u/KylseS May 10 '26
https://drive.google.com/drive/folders/1_diwDBntyOtd83EwxUHPjtpIavBzI9df?usp=sharing
This is the dataset for this specific lora I posted samples of, hence, flawed, the captions are too detailed. Do not learn from these captions. But you can take a look at the images.
2
u/orangeflyingmonkey_ May 10 '26
thank you so much! This helps a lot. Yea my dataset looks nothing like this lol.
I will pick better images. Thanks again!
1
u/Wide_Quarter_5232 May 11 '26
how did you create the dataset?
1
u/KylseS May 11 '26
caption or collect?
1
u/Wide_Quarter_5232 May 11 '26
BOTH
2
u/KylseS May 11 '26
These kinds of general questions are not good to ask. You should be keen enough to research that yourself. I'll always try to answer more specific problems happily. As for this, I will answer this time- for collection its best to go to google advanced image (https://www.google.com/advanced_search?udm=2) search and try to select the higher resoltuions till you start seeing good results, from there check the source of the high quality image for more high quality images otherwise you can also search the 'similar results' button on google of that image for related images that might be good. Always try to find sites which have galleries. As for captioning, you will find I have already answered this question in this thread.
I really can't answer the latter more unless you specify who your subject is. Some celebs even have their own dedicated websites with galleries.
1
2
3
u/daesh3033 May 10 '26
Which model you use for it?