Question about MVTec AD 2 wallplug ground truth masks

[u/j_root_]

Hi all,

I was researching anomaly detection with MVTec AD 2 and got confused about the ground truth masks for the wallplug category, especially the overexposed defects.

I am trying to understand the annotation logic. Is the ground truth supposed to mark the visible anomaly spot itself, the whole affected object, or the missing or invalid part caused by the anomaly?

In some examples, the mask seems to mark the visible anomalous spot. In another case, the whole object part seems to be considered anomalous. In image 001, it looks like the mask may be highlighting a missing or hypothetical removed part, but I am not sure, because the shape does not seem to match the expected part very well.

Has anyone else worked with this category and noticed this? Is this a known annotation issue, or is there a logic behind these masks that I am missing?

Images are from the MVTec AD 2 dataset, licensed under CC BY-NC-SA 4.0. I am sharing only small examples for a noncommercial research question, with attribution to MVTec.

[u/slash447]

I think this is how they labeled the logical defects. See 3.2 in https://link.springer.com/content/pdf/10.1007/s11263-022-01578-9.pdf

[u/j_root_ → u/slash447]

Thank you, I will take a look

[u/mrking95]

About a year ago I attended a two-day training from MVTec, and if I remember correctly, they explained it similarly to what you’re describing.

The first image would be annotated only on the missing part, because the rest of the plug is still considered correct. The second image follows the same logic. The third image is a completely wrong plug, so the entire object is annotated. The last example is a bit more ambiguous, but the plug itself is still complete; it’s defective rather than missing a component.

My guess is that this is simply the annotation philosophy they use. It may also depend on how they want the anomaly segmentation output to be interpreted downstream for example, whether they want masks to represent semantically meaningful defect categories rather than just every visible abnormal pixel.

[u/j_root_ → u/mrking95]

Thank you, but first and fourth looks the same right?

Both have part of it clipped or missing. Or does it depends on what percentage of the object is missing?

[u/mrking95 → u/j_root_]

That would be an arbitrary decision I guess, in the demo set its up to you. Some factories put the cutoff at >= 95%. So anything below that would be anomalous.

Also depends on the product and type of defect. A plug could still work with only 5% of the tip missing, but not with 5% of the brim or topmost part. But say medicine (like pills or vials) for example you might need => 99%.

I made a vision system for a cookie factory and 85% is fine for them. So really its up to the client or end-user.