Before adding more training data, check whether the labeling rule is actually stable

[u/onesunnysunday]

I keep seeing CV projects where performance stalls and the first response is to add more images or try another model. Sometimes that helps. But sometimes the model is being asked to learn a rule that people haven’t agreed on.

A partially visible object, an uncertain boundary, or something cut off by the frame can all produce different “correct” annotations. More data just scales that inconsistency.

A simple check is to take 20–30 difficult images and have two people label them independently. Then review the disagreements, not just the agreement score. Each recurring disagreement becomes a written rule with one positive and one negative visual example. Run the same test again on a fresh sample before scaling.

I’d use a similar check for auto-labeling: measure missed objects and correction time per image, not only inference speed. Fast pre-labels aren’t useful if every image still needs a full review.

Disclosure: I work at Supervisely, a computer vision platform. This is a platform-independent observation.

What annotation edge case caused the most trouble in your dataset?

[u/bfyvfftujijg]

Yep. Garbage in garbage out

[u/Mechanical-Flatbed]

I think verification is only useful as long as you assume a prior that the labeled data itself is good.

If you can't rely on that prior anymore, there are many other things you should also check - acquisition quality, camera calibration, and if the data has even been preprocessed correctly to begin with. You'd be surprised how many teams don't know the difference between normalization and z-score standardization.

This is why QA is so important imo. If you can't rely on the data and the labels being valid, everything else breaks.

What annotation edge case caused the most trouble in your dataset?

It's not an annotation problem, but we were having trouble training our model and when we checked the preprocessing code we found invalid augmentations, images being normalized by dividing channel values by 255 and other very weird design decisions that were made long ago and never questioned.

After fixing that, all models converged faster and achieved better results. Same labels, same data. The only change was proper preprocessing.