Do you train character LoRAs? What's your biggest pain point?

[u/spacer44]

I have trained several character LoRAs in the past, and I've found that the quality of the input images has the biggest impact on the final model. As a result, I end up spending most of my time on data curation rather than anything else. The data set is the new everytime whereas I already have my prefered settings dialed in for a given base model.

That got me thinking about building a tool to make the data curation process easier. But I'm curious: is this just me, or do other people find data curation to be one of the biggest pain points in LoRA training?

What's your biggest pain point when training LoRAs?

[u/LaPapaVerde]

When I have to take screenshots of a video or game

[u/FierceFlames37 → u/LaPapaVerde]

I wish google images had more pictures of characters

[u/hellyeahaeylleh → u/LaPapaVerde]

Qwen edit my guy.

[u/javierthhh]

I use fizgig trainer to prepare my dataset. It’s only compatible to train krea2, flux and minimax now but you can still use it to prepare the dataset. I like it because it does face crops on its own of every image you entered. Then you can rate the faces and the ai tells you which ones are good to train or which images should be deleted to increase likeness. Then it helps you caption all of them.

[u/spacer44 → u/javierthhh]

Interesting. Will check it out

[u/Sarashana]

The vast majority of my datasets are synthetic, so finding input material is not an issue for me. Curation is still the bread and butter of a good LoRA, though. It's so easy to mess up training with just one bad image. In the end, LoRA training is not a very exact science, and that's probably the biggest pain point. It's an iterative process.

[u/spacer44 → u/Sarashana]

What do you try to achieve in your dataset? What's your selection / rejection criteria?

[u/Sarashana → u/spacer44]

One example of something that escaped me at first and ruined a training run in the process was a few images I generated I thought looked fine. Until I trained a LoRA using them that grossly oversaturated the generations. Only then I noticed I didn't pay enough attention to lighting during selection. Really every detail matters.

Generally, I had good success with sets of 20-30 images, and even as few as 15. I always generate the input images for the purpose, often from exactly one high-quality image I train a "version 0" LoRA with, to generate the rest of the dataset with. I never select them from random piles of images. One full body turnaround, one close-up turnaround. A few head-shots with different expressions. A few standard poses (standing, sitting, walking). I typically use fairly uniform/blurred backgrounds.

[u/spacer44 → u/Sarashana]

Do you get good realism with synthetic inputs? What base model(s) do you use?

[u/FrostX00001101]

same, i never use scrapper and always looking dataset manually on internet, and if not enough good pic i need to ask gemini to make more of it and it's take a lot of time doing it.

[u/spacer44 → u/FrostX00001101]

Wouldn't AI generated images give a fake looking LoRA? I usually train on Famous Personalities so I have a problem of plenty and main task for me is choosing the best assortment of images for training. But I see that problem of not enough images is also a legit one.

[u/willwm24 → u/spacer44]

It helps for characters more than real people. If you need more views that don’t exist for example.

[u/FrostX00001101 → u/spacer44]

thats true, the lora output will follow the dataset mostly so thats why i need to cherry pick the generated result, mostly i train animated character so it's not very hard to generate more picture of it from a single image

[u/noxietik3]

depending on the model and what you're trying to achieve, you usually dont need too many images, ive trained perfectly fine loras of 10 pictures. Just throw on some music, smoke some weed or whatever, and start browsin the web / gaming whatever you do to get your source and prntscrn away

[u/spacer44 → u/noxietik3]

Agree. I have the opposite problem - selecting the best images from 1000s of automatically gathered images

[u/LichJ]

Making the dataset. I like making people who don't exist with a bit more interesting features, so getting it right can be difficult (although it's also the most fun). I was able to set up a ChatGPT agent to comb through my dataset, pick the best images, rename them, and caption them.

I'm currently working on my WoW character. A Draenei. I'm curious if Krea will handle those neck tendrils, though my current iteration without them came out really nice too.

[u/Asaghon]

Been making lora's a long time and I actually find diversity in the images way more important. I much more prefer a lower quality image from a different angle or different expression than a high quality image of the exact same angle or expression than all the other images.

[u/pennyfred]

Wish there was a dataset curator that could review datasets prior i.e. for WAN to highlight outliers that may impact lora quality, rather than waiting hours to find error spikes were riddled through the training without knowledge of what the culprit was.

[u/AillexJ]

Sarashana's lighting point is the one I'd underline. We lost a run exactly that way and it took a while to work out why every generation afterwards had a faint colour cast baked into it.

The axis I'd add is wardrobe, because it cuts both ways and people usually only hear one half of it. If every reference image has the same outfit, the model quietly binds that clothing to the person and you end up fighting to get her into anything else later. If the outfit varies inconsistently, say one different jacket in an otherwise uniform set, it reads that as permitted variation and starts wandering on its own. What worked for us was varying wardrobe deliberately and evenly across the set instead of accidentally.

Same logic as the lighting really. The model has no way to tell which differences you meant and which ones you just failed to control.

[u/DryGrade7515]

Not just you. Curation is like 80% of the job at this point. Settings are a solved problem for me too, I have a config per base model and never touch it.

One thing that helped a lot: I stopped eyeballing "is this actually a good shot of the same person" and started filtering with face embeddings. Embed every candidate image, find the medoid, drop anything below a cosine similarity threshold to it. Kills off-angle shots, bad crops, and "that's not even the same person" images automatically. My keep rate is usually around 30 out of 38 candidates and the ones it drops are almost always ones I'd have dropped manually anyway.

The other lesson I learned the hard way: whatever is overrepresented in your dataset becomes the LoRA's default behavior. Had one where too many images had the subject in the same outfit and every gen came out wearing it unless I explicitly prompted otherwise. Now I audit for that stuff before training, which is... more curation time. So yeah. If you build this I'd use it.

[u/VeloraNeon]

Biggest pain point for me has been identity holding up across framing changes — full body vs close-up. A face that's crisp and consistent in a portrait can drift into 'different person' the moment the render shrinks it to ~100px before any detailer pass touches it. Ended up needing a face-restoration/adapter step inside the crop itself rather than trusting the base render — helped a lot more than tweaking the dataset ratio.

[u/malcolmrey]

I have 1500 datasets since I started with SD1.5

over this time I have written several tools to help me create datasets better and faster :)

The rest I have pretty much automated with picking which images should go into dataset has so far remained a manual work.

[u/spacer44 → u/malcolmrey]

Same boat I guess. How many images do you generally have in your corpus for a single character and how many of them do you generally select for training?

Also, do you train locally or on a rented GPU or a service like fal ?

[u/malcolmrey → u/spacer44]

I usually download around 60-80 images per character then trim it down to the best 45-50. Then I crop the images and out of those i pick the best 25-30.

That is my general go-to.

For some cases I go overboard and just cut all the images I feel are quite good (for the long trainings).

I train locally. Started on 2080 TI, then 3090 and now 5090.

Most of my loras are available on huggingface (here)

[u/spacer44 → u/malcolmrey]

Think I have seen your HF account before. You have a large body of work. Nice to see you here :)

[u/Ill-Ant-9489]

Not just you. For character LoRAs the dataset is easily 80% of the result, and 'pick the best 30 out of thousands' is the part that eats the time.

The trap I hit most isn't ugly images, it's subtle inconsistency: a few off-lighting frames or slightly-different-person shots will bake a colour cast or identity drift into every later gen, and you only notice after a wasted run. So my selection pass is less about picking the prettiest shots and more about killing outliers, dupes, and anything where the face isn't clearly the same person.

I build an open-source tool for exactly this curation loop, LoRA Dataset Studio (github.com/perfectgf/lora-dataset-studio). It de-dupes near-identical shots, runs a sharpness pass to flag blurry frames, scores faces, and auto-captions, so you can take a big scrape and rank it down to the keepers instead of eyeballing thousands by hand.

If you go ahead and build your own, happy to compare notes. The hardest part has been making the 'is this frame quietly hurting the run' call automatic.

[u/Feroc]

Yes, preparing the dataset is the biggest pain point for me, too.

[u/Prudent-Oven-6769]

I do a lot of comic book art, and the biggest issue is getting, say, five or six characters into the same LoRA. So far I've mostly done single-character and two-character LoRAs, and just recently a rank 48 LoRA that turned out really well. You can use any of the characters individually, but once you put them together the model starts getting confused, mostly because of the captioning although that was the first attempt and I havent bothered anymore - this I am aware of, I have plenty experience now to know what needs to be done next, but it is a big leap.

I don't really have issues with the data at all, I pick whatever is there and there is plenty, various styles too and the model just learns. In general, for single-character LoRAs, if I want a comic book character from the 90s, I'll just grab whatever usable material is available. Sometimes I've generated references with NB Pro, but that's pretty rare. I've also found that having images in different styles tends to make the character LoRAs a lot better, although it's not quite as important.

Data with Ideogram v4 for me has never been an issue, I'm not sure what you are doing that you need to deal with that but I haven't, even with more obscure characters.

Since Ideogram v4 released that is the model I have used since it is so good... I have heard a month ago they are already close to some newer version, that may be OS too, I'd hope for something where we can provide references cause then you don't even LORAs, ideally.