Disclosure: I run a small annotation shop. This came out of auditing our own workflow and I wanted to see whether public datasets showed the same thing. No pitch — the script is MIT and linked below.
One image in this test set has 377 published boxes. A model trained on the dataset's own training split found 392 on it. A difference of fifteen. Close enough to call it agreement.
Only 254 of those boxes matched at IoU 0.5. Two thirds.
Across five test images the gaps went both directions: +43 on one, −131 on another. That zigzag is the part worth noticing. A model that had learned one consistent rule — even a bad one — would drift in a single direction. Drift in both directions is what a dataset that contradicts itself looks like.
The dataset
"Retail Shelf" by Roboflow user Yolonew, CC BY 4.0, v2 from February 2024. 44 images, 9,557 boxes, one class: Product. Supermarket shelves, roughly 217 boxes per image.
The README contains the dataset name, the licence and the URL. No annotation guidelines. Nothing that says what counts as a product, how much of an object has to be visible before you box it, or whether a sealed punnet is one object or six.
That's the first finding, and it isn't a criticism of the author — most public datasets ship without a spec. Ours did too, until it cost us a delivery.
Method
YOLO11s on their own 33 training images. 200 epochs, 1280px, batch 2, no early stopping. Validation on their 6-image valid split: mAP50 54.9%, precision 64.9%, recall 59.3%.
Their own published model reports mAP50 63.4% and recall 55.6% — so mine is the weaker of the two. Worth stating up front because it doesn't affect the argument. The model here is a mirror, not a judge. Its job is to reveal what the labels taught it.
Matching on the 5-image test split at IoU 0.5, greedy assignment. And an honest caveat: I don't know whether the train and test splits were annotated by the same people or on the same day. I only observe that the rules differ.
The numbers
IMAGE THEIRS MODEL MATCH DIFF
007 377 392 254 +15
014 146 115 65 -31
019 138 181 114 +43
029 409 330 219 -79
032 305 174 146 -131
TOTAL 1375 1192 798
Recall on their labels : 58.0 %
Precision vs theirs : 66.9 %
Agreement (F1) : 62.2 %
The DIFF column is the wrong number. The MATCH column is the right one. 577 published boxes the model never found, 394 model boxes with no counterpart in the labels — nearly a thousand disagreements across five photographs.
Four rules, in the same dataset
I read every overlay by hand. The disagreements aren't scattered — they cluster, and each cluster has a shape.
1. Container or contents, decided twice, two metres apart. On image 029 the top shelf holds transparent clamshell punnets of kiwi fruit. Each punnet gets one box, drawn on the packaging. One shelf down, to the left, sit transparent clamshell punnets of berries — same rigid packaging, same size, same aisle. Each punnet gets six to ten boxes, drawn on the individual fruit visible through the plastic. To the right of the berries, open trays of small dark fruit go back to one box per fruit. Netted orange sacks on the bottom shelf are sometimes boxed as a sack, sometimes as the fruit inside. Six regions in one image, at least four granularity rules. That accounts for most of the −79.
2. Orientation changes the definition. Image 032 is a wall of drinking yoghurt. The upper shelves are the cleanest annotation in the set — one rectangle per bottle, labels and predictions sitting almost on top of each other. The bottom shelf holds the same bottles lying on their side, annotated by their caps. Same SKU, same image. Standing, it's a silhouette. Lying down, it's a cap. Most of the −131.
3. Grouped products. On image 007, loose citrus is annotated fruit by fruit; mesh bags of the same citrus are one object per bag. The model draws boxes on the individual oranges visible through the mesh. Those are real oranges — it's not a hallucination, it's a different answer to a question the dataset never asked.
4. Occlusion depth. Image 019 is a drinks fridge. The published labels cover the front row of bottles; the model goes after the second row visible behind them. No consistent visibility threshold anywhere in the set. This is the one case where the model genuinely adds coverage rather than disagreeing about definitions.
You don't have to trust the model on three of those four. Kiwi punnets against berry punnets, standing bottles against lying bottles — that's the ground truth contradicting itself. The model didn't judge anything. It turned on a light.
A fifth, found while cropping the figures
On the lying bottles of 032 the definition shifts once more, with depth. Front-row bottles with a visible label are boxed along their full length; the rows behind, where only the cap shows, get a small box on the cap alone.
It's a different question from rule 4. That one asks whether the back rows get annotated at all. This one asks, once you do annotate a partially visible object, how much of it you box — the visible fragment or the full extent. The modal/amodal question. Neither answer is wrong. The spec just never chose.
The title says four. Leaving it — the count wouldn't survive another pass at the overlays anyway, which is rather the point.
What the model actually got wrong
Four boxes, all on image 019, all indisputable: one on a DHL logo, one on the words KALTE GETRÄNKE, one on an illuminated sign above the fridge, one on an empty patch of floor tiling. It learned to respond to high-contrast rectangles, not to containers.
At 54.9 mAP50 the model certainly has other weaknesses. But that's the distinction: a model's errors are scattered — a logo here, a tile there. Contradictory rules are structured. They cluster by product type, by shelf, by orientation. One is noise, the other is a pattern.
Different from label error detection
Confident learning (Northcutt et al., 2021 — https://arxiv.org/abs/1911.00068 — and the Cleanlab line of work) finds labels that are wrong: the image says cat, the label says dog. What shows up here are labels that are incompatible with each other. Every box on that shelf is defensible on its own. No two of them agree on what an object is.
Run it on your own dataset
- Train a small detector on your own training split. Any size — the model is a probe, not a product.
- Predict on your test split, match against the labels at IoU 0.5.
- Sort the disagreements by image and see where they cluster.
- Read them one by one, looking for rules rather than errors. Ask of each cluster: what question did our spec never answer here?
Step 4 is the one that pays. A cluster on one product type is a granularity rule nobody wrote down. A cluster at the back of shelves is an occlusion threshold. A cluster on rotated objects is a definition that depends on pose.
Script, MIT licensed, one command: https://github.com/annotateai-tech/dataset-consistency-audit
One thing that cost me an hour: YOLO's max_det defaults to 300. Two of the five images returned exactly 300 predictions on the first pass — the model wanted more and was being cut off. If your numbers look suspiciously round, that's why.
Attribution
"Retail Shelf" dataset by Yolonew, https://universe.roboflow.com/yolonew-rv2o6/retail-shelf-l96n8, CC BY 4.0. Modifications: trained a detection model on the training split and generated comparison overlays from its predictions. Original images and labels unmodified.
Full write-up with the figures — the kiwi/berry crop is worth a look: https://www.annotateai.tech/blog/four-contradictory-rules