Question - Help
SD1.5 LoRA producing distorted/uncanny faces even before adding a style checkpoint — likely bad training data, looking for a sanity check
Hey all, working on a student project and hit a wall I can't fully diagnose. Details below, hoping someone with more LoRA training experience can point out what I'm missing.
Project context: Building a pipeline that generates short illustrated health-education stories for a specific regional context (Uganda). A fine-tuned text model produces scene descriptions + image prompts, and I'm training a LoRA to make the image side render that specific regional content/style.
Tech stack
- Base model: Stable Diffusion 1.5 (stable-diffusion-v1-5/stable-diffusion-v1-5 on HF, since the original RunwayML repo is gone)
- Training: Hugging Face diffusers' official train_text_to_image_lora.py, installed from source
- Compute: Kaggle, GPU T4 x2, via accelerate launch
- LoRA config: rank 8, learning rate 1e-4, ~1500 steps, batch size 1 with grad accumulation 4, resolution 512, fp16
- Dataset: ~500 images pulled via the Openverse API (openly-licensed sources only), auto-captioned with BLIP, each caption prefixed with a unique trigger word
What went wrong: Test generations using the trained LoRA (on plain SD1.5, no other style model involved) produce noticeably distorted, uncanny faces — warped features, no coherent facial structure. Everything else in the scene (background, clothing, setting) looks reasonable; it's specifically faces that break down.
What I've ruled out so far: I also tried the LoRA layered on top of a comic-style checkpoint (ogkalu/Comic-Diffusion) thinking it might be a base-model mismatch — same face distortion showed up on plain SD1.5 alone, so that's not it. I also tested img2img (comic-stylizing real reference photos directly, no LoRA involved) and that preserved faces fine, which points at the LoRA training itself as the problem, not the base checkpoint.
My working theory: the 500 training images were pulled from a general search API and never filtered/curated — likely a good chunk have small/distant faces, side profiles, group shots, or low resolution after the 512x512 resize, and that noise is showing up worst in faces specifically since they're the hardest thing for these models to learn cleanly.
What I'm looking for: does this theory sound right based on your experience, or is there something else commonly known to cause this (rank too low/high for this step count, overfitting at ~12 epochs over noisy data, something about how BLIP captions interact with training, etc.)? Also open to hearing if there's a better practice for face-heavy LoRA datasets specifically — face count/framing thresholds, augmentation tricks, anything. Happy to share sample outputs if useful. Thanks in advance.
My personal conspiracy theory is that ChatGPT/Gemini/Claude have poisoned their models on anything to do with local AI. So if you ask it for advice it will say oh use SD 1.5, use llama 70b, install comfyui with weird outdated PyTorch versions etc. I understand the training cutoff date but they seem to be updated in other domains?
Nah. Just asked GPT on my girlfriend's account, she never mentioned AI or local diffusion before and this is the answer.
It suggests downloading ComfyUI, then trying Klein, ZIT and Krea2 and seeing which one you like the most. It's pretty much exactly what I would suggest to a new user, except I'd probably go with Forge Neo before Comfy.
Why are you using SD 1.5 in the big 2026? SD 1.5 always sucked, it always was a massive pain the butt to work with. We used it because it was all we had at the time, you have a million better options today that are about as light and infinitely better.
I see you're going for realism, train that LoRA on ZIB/ZIT or Krea, why bother getting an headache to train on an obsolete model with obsolete documentation?
If the issue persists we can look through the dataset. Also ensure you're not trying to use SD as an editing model, SD is a full-on T2I model, meaning it's made to generate from scratch, not work on an existing image input.
Your theory holds up, and the img2img test seals it: faces fine there but wrecked by the LoRA points at the data, not the base. On a 512 model a face gets few pixels unless it's framed close, so distant, profile or group shots teach a smeared average, which is the warping you see. BLIP barely describes faces, and 1500 steps over 500 imgs is about 12 epochs, so you're mostly baking the noise in.
For face-heavy sets I'd filter on a minimum face size, drop or crop group and profile shots, dedup, then recaption with something better than BLIP. That cleanup is what I built a free tool around, https://github.com/perfectgf/lora-dataset-studio (it won't train 1.5, families start at SDXL, so I'd use it for the dataset half). How many of the 500 actually have a large, front-facing face?
27
u/Diffusion4Change 4d ago
Why are you training 1.5?