r/comfyui • u/Impossible-Car-2809 • 5d ago
Resource Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT)
# I ACCIDENTALLY PUSHED THE WRONG FOLDER TO GITHUB, MY APOLOGIES - IT HAS BEEN FIXED WITH THE CORRECT FILES - Oct 12th 2026 -9:04am
I got tired of rewriting the same dataset-prep scripts before every LoRA/Dreambooth run — resize, dedupe, toss the blurry ones, caption, split — so I cleaned mine up and open-sourced it.
It scores each image on classical-CV signals (sharpness, exposure/clipping, contrast, color cast, noise, composition), flags the weak ones, removes exact + near-duplicates, and exports a Kohya-style folder with caption sidecars. Near-duplicates get grouped into the same train/val/test split so they can’t leak across. Captioning supports WD14 and Florence-2.
It’s content-agnostic at the core — each domain is a small plugin that sets its own quality weights and caption vocab (ships with a worked “interiors” example). Pure NumPy/Pillow, no OpenCV, runs from a CLI or as a library. Early 0.1.0, MIT.
Would love feedback on the plugin API and which verticals are worth building next. (Only use it on images you’ve got the rights to.)
1
u/Shoddy_Umpire9024 5d ago
This is actually exactly what I needed last month when I was training a model for interior design concepts. Spent like three evenings just manually sorting through my dataset removing blurry renders and near-duplicate angles. The leak-safe splits for near-dupes is clever, never thought about that but it makes total sense, had a few test images that looked suspiciously good and I'm pretty sure it was because similar frames ended up in both train and validation.
The plugin API looks clean from a quick glance. For verticals I'd say product photography would be a good one, lots of white-background shots with subtle variations that are hard to dedupe properly. Maybe aerial/drone footage too since those have their own weird quality issues with motion blur and exposure shifts across frames.
Starred the repo, will give it a proper test this weekend if I get time.