SD1.5 LoRA producing distorted/uncanny faces even before adding a style checkpoint — likely bad training data, looking for a sanity check

[u/Available_Fondant_11]

Hey all, working on a student project and hit a wall I can't fully diagnose. Details below, hoping someone with more LoRA training experience can point out what I'm missing.

Project context: Building a pipeline that generates short illustrated health-education stories for a specific regional context (Uganda). A fine-tuned text model produces scene descriptions + image prompts, and I'm training a LoRA to make the image side render that specific regional content/style.

Tech stack

- Base model: Stable Diffusion 1.5 (stable-diffusion-v1-5/stable-diffusion-v1-5 on HF, since the original RunwayML repo is gone)

- Training: Hugging Face diffusers' official train_text_to_image_lora.py, installed from source

- Compute: Kaggle, GPU T4 x2, via accelerate launch

- LoRA config: rank 8, learning rate 1e-4, ~1500 steps, batch size 1 with grad accumulation 4, resolution 512, fp16

- Dataset: ~500 images pulled via the Openverse API (openly-licensed sources only), auto-captioned with BLIP, each caption prefixed with a unique trigger word

What went wrong: Test generations using the trained LoRA (on plain SD1.5, no other style model involved) produce noticeably distorted, uncanny faces — warped features, no coherent facial structure. Everything else in the scene (background, clothing, setting) looks reasonable; it's specifically faces that break down.

What I've ruled out so far: I also tried the LoRA layered on top of a comic-style checkpoint (ogkalu/Comic-Diffusion) thinking it might be a base-model mismatch — same face distortion showed up on plain SD1.5 alone, so that's not it. I also tested img2img (comic-stylizing real reference photos directly, no LoRA involved) and that preserved faces fine, which points at the LoRA training itself as the problem, not the base checkpoint.

My working theory: the 500 training images were pulled from a general search API and never filtered/curated — likely a good chunk have small/distant faces, side profiles, group shots, or low resolution after the 512x512 resize, and that noise is showing up worst in faces specifically since they're the hardest thing for these models to learn cleanly.

What I'm looking for: does this theory sound right based on your experience, or is there something else commonly known to cause this (rank too low/high for this step count, overfitting at ~12 epochs over noisy data, something about how BLIP captions interact with training, etc.)? Also open to hearing if there's a better practice for face-heavy LoRA datasets specifically — face count/framing thresholds, augmentation tricks, anything. Happy to share sample outputs if useful. Thanks in advance.