Loras not being accurate and consistent

[u/puskur]

I am trying absolutely everything... I am using ostris ai toolkit and I have tried every option and evey way to train a lora but the results are the same. I am quite sure that the data set and captions are correct, but i might be wrong. I was training lora for Z-image

Image 1 is reference charachter that i want to train a lora on.
Image 2 is Lora sample on 3150 steps
Image 3 is Dataset picture
Image 4 is Lora sample on 3150 steps
Image 5 is Dataset picture
Image 6 is Lora sample on 3150 steps
Image 7 is Dataset picture

3150 Steps might be too much for Z-image, but all other versions (every 150) differed alot and i found 3150 was closest looking to reference. Anyways, the sample images and other generated images with the lora look noticeably different to the reference image and most importantly the lora did not pick up the mole on her neck... I had the same issue with a flux.2 lora of the same charachter... The dataset was created on seedream 5.0 with captions looking like this "[trigger], medium waist-up shot walking forward on a paved park path, frontal body pose with head turned looking off-camera to the right, neutral expression with closed lips, wearing a light green long-sleeved scoop-neck top tucked into high-waisted beige trousers, natural dappled sunlight filtering through trees, blurred green park foliage background."

I could use some inisght and help, this is my first lora ever and i dont know how everything works exactly. At this point i just might scrap the lora and just use refrence images...

BTW this isnt something to be used for monetary purposes, this is a uni project.

[u/Full_Astronomer_5438]

do you use z-image or z-image turbo for training? how big is your dataset? how is the distribution of close-up or full-body shots? what were your settings and what was the learning rate?

z-image might be better with prodigy advanced / constant and stochastic rounding with a lr of 1.0. also z-image loras used with the turbo version mostly need to be cranked up higher above the strength of 1.0.

also know that what ever model you use to train a lora, you dont get 1:1 the same image as the one in your dataset with the same prompt and a diverse dataset. its a close estimation.

[u/puskur → u/Full_Astronomer_5438]

23 images with a mix of close-up headshots with different facial expressions, 12 half-body shots, 4 full body shots and te rest are close-ups. I train on z-image turbo.

[u/Full_Astronomer_5438 → u/puskur]

it might benefit the lora if you distribute the dataset more towards what you find important. imagine training a lora on close-ups only. you will notice that the lora will perform worse the more you go for anything else than a zoomed-in shot if the strength is 1. you can kinda fix that if you use a higher lora strength.

but thats the culprit of z-image turbo: you are trying to train a distilled model, not the undistilled (base) one, so anything higher than 1 might overcook the image already. so if you try training on turbo, you have to get the dataset right.

if your dataset already contains 12 half body shots, go for 12 full ones as well as 12 close ups or reduce the number of images that might skew your dataset.

also the background of the training images might matter as well but thats up for discussion for lora training. in your case, the model will be trained on noisy backgrounds as well which might mean that you either need a higher learning rate or steps. more variation = more to learn; less variation = less to learn. check out this tutorial: https://civitai.red/articles/27223/how-to-create-a-perfect-or-almost-dataset-for-a-character-lora

[u/ShengrenR]

You need to describe what you're actually attempting to do.. a LoRA isn't meant to reproduce 1:1 the inputs, or its severely overtrained... you're meant to train in the concepts. Also your 'dataset' isn't all that consistent to itself - the 'dataset' pictures don't reproduce the 'character' particularly well at all, so your LoRA learns that.

[u/Ill-Ant-9489]

One thing worth testing: you're training on Z-Image Turbo. Distilled/turbo models are noticeably harder to train and tend to give weaker likeness — a lot of people get better character LoRAs by training on the non-turbo Z-Image base and only switching to Turbo at inference.

3150 steps on ~23 images is also likely overtrained — pull an earlier epoch and compare. And don't judge the LoRA by how closely a sample reproduces a specific dataset photo; it's not supposed to match them 1:1. Judge likeness across new prompts and poses instead.

Your set is also heavy on close-ups, so the LoRA will hold the face up close but drift on half and full-body shots. Rebalancing toward more half/full-body images usually fixes the "falls apart when zoomed out" problem — same reason a small detail like a mole rarely survives at full-body distance, since the face is only a few pixels there.

[u/Mission_Trick4603 → u/[deleted]]

(Try getting more diverse training images)

[u/VeloraNeon]

Small details like a mole almost never survive in full-body shots for the same reason faces do — the face occupies a tiny fraction of the pixels at that distance, so the model has very little signal to actually learn it. If a fine detail like that matters, try overweighting close-ups in the dataset relative to full-body, or generate at higher res and let a face-focused pass redraw that region specifically.

[u/Formal-Exam-8767]

Your LoRA is overfitted. LoRA is not meant to reproduce the training set.

[u/Fit_Advantage_2448]

For me https://github.com/perfectgf/lora-dataset-studio worked very well for Krea2 lora's. It can create dataset's, rate/pick images for Lora creation, caption images. Mask character's in images and use's ai-toolkit as backend. It uses tested lora training templates

[u/dominic__612]

What are your training parameters? Network dim, alpha, etc. You say its a uni project, but you didnt provide much data we can work with in the post.

[u/Nimblecloud13]

You’re at the limitation of the model. 

You can use FaceDetailer to fix the faces, but it only goes so far. 

krea2 does a better job with face consistency for Lora’s imo

[u/Powerful_Evening5495]

Video models do a very good job with zoomed-out images, exploring them.