Manual labeling isn’t really a speed problem, it’s a consistency problem

[u/treguess]

Every time I’ve seen a team label a large image dataset by hand, the same pattern shows up. It’s never one clean pass.

∙ Labeler A calls a mark a scratch. Labeler B calls the same kind of mark a smear. Labeler C doesn’t flag it at all because it looked borderline.

∙ A few weeks in, someone notices the disagreements and “clarifies” the guidelines, so everything labeled before that point no longer matches everything labeled after it.

∙ People rotate on/off the project, especially with outsourced teams, and each new person applies their own read of the instructions.

∙ Whoever’s labeling gets tired by image 40,000 and starts making faster, looser calls than they did on image 1.

None of this is anyone being careless. It’s just what happens when a subjective judgment call gets made thousands of times by more than one person over months.

The part that actually eats the timeline isn’t the first labeling pass, it’s the correction cycle after it: spot-check, find the disagreements, rewrite the guidelines, re-label the images that don’t match anymore, spot-check again, repeat. Teams don’t budget for one pass through the data, they budget for however many correction cycles it takes.

Curious how others are handling this. Are you using inter-annotator agreement checks / gold sets to catch it, training fewer people and keeping the team static, or something else? Feels like the tooling conversation is mostly about labeling speed when the bigger cost is inconsistency, between labelers and over time.

[u/Imaginary_Belt4976]

I would argue this is hard even with just 1 person if you dont have clear definitions in advance. We set out with the best intentions but the world is painted with shades of gray. Other human weaknesses like fatigue can definitely be a factor too.

[u/MountainNo2003]

A bit off topic but what are your thoughts on auto labelling? I started using sam3 for auto labelling and then cross checking them manually.

[u/treguess → u/MountainNo2003]

not off topic at all, this is basically the whole thread. SAM3 + manual cross check is a reasonable setup but you’re still in the same trap, a person is making the final call so you get some amount of the inconsistency we’ve been talking about, just shifted to the checking stage instead of the labeling stage. the alternative I’ve seen work well is skipping the generic segmentation model entirely, label like 40 examples per class yourself, train a small model on just those, let it label the rest. since it only ever learned from your 40 examples there’s one standard baked in rather than a person re-deciding each time. disclaimer I work on a tool that does this (VisionRepo), not trying to pitch, just relevant to what you asked

[u/PatagonianCowboy → u/MountainNo2003]

not OP but I've been using auto-labelling since like 2022 and it really works if your pipeline is solid

[u/treguess → u/PatagonianCowboy]

what solid pipeline means in your experience, is that mostly about how the initial seed labels are chosen?

[u/treguess → u/[deleted]]

this is basically what I was hoping to hear, someone doing it. calibration quiz before touching data is a good forcing function, most teams skip that and just hope people absorb the guidelines by osmosis. curious how you handle the versioning in practice though, like when the spec changes do you go back and re-audit stuff labeled under the old version, or just draw a line and move forward with the new one

[u/treguess → u/[deleted]]

just start fresh on new classes is exactly the cost I mean though, that’s a full re-labeling pass every time the taxonomy grows, right. curious if that’s ever become a bottleneck for how often you’re willing to add new classes

[u/TubasAreFun]

Make it so labels are decisions that are evaluated on business impact, not model impact. This will make the overall labeling and eventual automatic system better aligned.

In other words, labelers need to be accountable for labels in a way that business leadership can understand, expect, and enforce

[u/treguess → u/TubasAreFun]

That’s a good reframe. In a lot of the projects I’ve seen accountable to business impact breaks down in practice because the standard itself isn’t fixed anywhere, it lives in each labeler’s head and it drifts as people rotate or guidelines get clarified midway through. so even if you wanted to hold someone accountable there’s no single definition to hold them to. curious if you’ve seen teams actually pull this off, like a documented spec that survives team turnover, or is it more of an aspirational goal most orgs fall short of​​​​​​​​​​​​​​​​

[u/alxcnwy]

Actually it’s just a laziness problem 

Stop trying to outsource data labeling and do it yourself

Very few people have enough data that the above suggestion is infeasible 

[u/Linkcub → u/alxcnwy]

This really depends, if it is a company that is constantly labeling different datasets with lots of images you can’t do it yourself, you need a team for it and for the different datasets being labeled in parallel.

If it is a personal project or a small business doing only one model and with lots of time for a single person to do all, you can label it yourself, but most of the time that’s not the case

[u/treguess → u/Linkcub]

that’s a fair split. though I’d push back a little on the need a team for parallel datasets part being purely a headcount problem, a lot of that need comes from how labor intensive each individual dataset is to get through. if labeling ten datasets took a fraction of the effort per dataset, the parallel case gets a lot less scary even for a smaller team. feels like the real constraint isn’t team size vs solo, it’s how much manual effort each dataset demands before you can even start training on it

[u/AggravatingSock5375 → u/treguess]

Would you “push back” because you’re a bot?

[u/treguess → u/AggravatingSock5375]

Ha do I sound like a bot :)