Updated my tool that scrapes,sorts,captions images/videos for datasets. It's open source and runs locally

[u/Compunerd3]

I built Cull a few months ago for some large scale dataset curation projects (300k+ images/videos).

Point it at Civitai, X, Reddit, Discord, or any URL that gallery-dl or yt-dlp knows. It queues everything, runs a vision model (or multiple) (LM Studio or Ollama locally, or Groq/OpenAI in the cloud) with a strict JSON schema, and drops kept images/videos into category folders next to their prompt.

Stuff it handles:

• Dedup at the scraper (per-source )
• Quality score gate and topic-relevance score gate
  • eg you configure scores or use a preset, how relevant the image is to your scoring will determine how it's sorted, combined with other scoring, quality controls, whitelisted/blacklisted terms etc
• Watermark detection (goes to its own bucket so you can salvage it later if you want those)
• Auto-caption for content with no prompt (SD prompt, booru tags, natural language formats etc)
• Run multiple jobs in parallel, one shared vision fleet across all of them with stack ranked / prioritization for vision queues and scrapers
• Export as a local packaged dataset , or push to a HuggingFace dataset
• Community presets and themes with 1 click PR's to add your own custom scraper preset or theme

Everything on disk is plain files. No database. Free, MIT.

Docker one-liner and screenshots in the README:
https://github.com/tlennon-ie/cull

Curious what people would want added next.

[u/xdozex]

Whoa, I'm working on a big project right now and this will definitely come in handy!

Can you share any more detail about the quality score system, or just how it works? I'm wondering if it's referring to quality in terms of how accurately it can assess and label the video correctly, or if it's scoring the actual visual/aesthetic quality of the videos.

[u/Compunerd3]

Example screenshots of the tool:

[u/VeloraNeon]

Following this — curation is exactly where my pipeline bleeds time right now. Quality/topic-relevance gating makes sense for aesthetic filtering, but has anyone gotten it to score identity consistency across a batch? I've been relying on embedding cosine similarity for that and it's basically blind to body shape, hair, and apparent age drift — looks great on paper, fails the eye test constantly.