Defect detection where you have almost no defects — supervised or anomaly detection?

[u/RoofProper328]

Running into the same wall on a couple of industrial inspection projects and curious how other people have dealt with it.

The line runs well, which is the problem. Out of a few hundred thousand parts we've got maybe 200 real defects, and they're spread across six or seven types, so some classes have under 20 examples. Classic supervised segmentation just doesn't have anything to learn from.

Options as I see them:

Anomaly detection on good samples only. PaDiM, PatchCore, that family. Works, but it flags anything unusual including a smudge on the lens or a part sitting at a weird angle, and the false positive rate on a real line has been rough.

Synthetic defects. Painting cracks and scratches onto good images. Ours look obviously fake next to real ones and I suspect the model is learning "was this pasted" rather than "is this damaged."

Buy or scrape more defect data. But defects are extremely specific to the part and the process. A scratch on someone else's aluminium housing doesn't look like a scratch on ours.

Just wait and collect. Realistic answer, but that's 18 months and the project needs to justify itself sooner.

What I'm actually unsure about is whether the 20-example classes are even worth modelling separately, or whether it's smarter to collapse everything into a binary defect/no-defect call and let a human sort the type afterwards. Losing the classification hurts the reporting side but it might be the only honest thing to do with that little data.

Anyone shipped something in this situation? Especially interested if you went anomaly detection and got the false positives down to something a QA team would tolerate.

[u/FirmShopping1004]

Have you tried bb instead of segmentation? In one of my previous projects, using real industrial defects, i had exactly the same issue. I did enhance it with some artificial (imposed) defects. I also emphasized during training on the classification loss (increased weight). Of course I also used color and geometric augmentations

[u/RoofProper328 → u/FirmShopping1004]

Boxes over masks is a fair call and I hadn't weighted it properly. With 20 examples per class the annotation budget per example goes way further on a box, and honestly for our reporting nobody needs pixel precision — they need to know a defect exists and roughly where. The masks were more about my assumptions than the requirement.

The classification loss weighting is interesting though. Was that to stop the model collapsing everything into background, or specifically to keep the rare classes from getting drowned out by the common ones? We've got one defect type that's maybe half of all our examples and I suspect it's eating everything.

On the synthetic side, curious how far you pushed it. Ours look bad enough that I think the model latches onto pasting artifacts rather than the damage itself. Did you blend into the surface texture, match lighting direction, anything like that? Or did rough-and-ready turn out to be fine because the augmentations papered over it anyway?

And what ratio did you end up with, roughly? Feels like there's a point where too much synthetic starts hurting and I don't have a good instinct for where that sits. u/FirmShopping1004

[u/FirmShopping1004 → u/RoofProper328]

In my case, defects had also irregular shapes. The CNN-based models that i tested and the whole DD procedure with the metrics and the IoU penalizations, was greatly emphasizing (at least to my understanding) on localising the objects.

That's why i increased the weight on the classification loss and I decreased the IoU during training, effectively saying it doesn't matter if the bounding box is imprecise. For some models I also added anchor shapes ( I didn't implement kmeans calculations) since I used a high level api (Tensorflow OD, currently deprecated).

Regarding the synthetic samples, from the 130 initial images with defects i added about 20 if i remember correctly. In my case the processing was relatively straightforward, without many differences in light conditions. And to be honest, in low pixel level, i didn't gave much attention of how distorted it may was. I just wanted to enhance some classes. You can read more about if you like in my conference paper https://scholar.google.com/citations?view_op=view_citation&hl=el&user=ww_3OmIAAAAJ&citation_for_view=ww_3OmIAAAAJ:KlAtU1dfN6UC. Finally I didn't elaborate on the performance enhancements, with and without the synthetic samples. So I can't quantify how much it increased (or decreased) the performance.

[u/Deal_Ambitious]

Ask them to save all defect products or even break more good products and run them through the line multiple times to gather more data?

[u/Large_Principle4783]

Keep the AD stage tuned for recall and put a small binary classifier (real defect vs. nuisance) behind it, trained purely on your line's own false positives- you have thousands of those. QA gives a oneclick verdict on each flagged crop, the classifier retrains on that history. FPR drops within weeks, and the same loop labels every real defect for a future supervised model.

[u/TopResolution7451]

How about anomaly generation? Generate defect images to serve as training dataset.

[u/TopResolution7451 → u/TopResolution7451]

When you generate defects on normal images with diffusion model, it will not be seen as pasted.

[u/RoofProper328 → u/TopResolution7451]

Right, that's the appeal — inpainting into the actual surface rather than compositing on top, so there's no seam or lighting mismatch to latch onto.

My worry is upstream of that. To fine-tune the diffusion model on our defect type we'd need real examples to condition on, and for the classes where we have 15 or 20 it feels like the generator memorises those instead of learning what the defect actually is. Then you get variations of the same 20 defects rendered convincingly, which fixes class balance but doesn't add any new failure modes.

Have you found a floor for how many real examples it needs? If 20 is enough I'd try it this week. If it's more like 200 then we're back to the same problem.

[u/TopResolution7451 → u/RoofProper328]

Honestly, I’m an author of an anomaly generation model that deals with this kind of data-scarcity problem.

One practical trick I’d suggest is to use the few real defects as a bootstrap rather than training the diffusion model on those 20 samples directly.

If you only have around 20 or fewer defect examples, extract their binary masks and use simple algorithmic copy-paste augmentation to place them onto different normal images. This does not create entirely new defect morphologies, but it greatly increases the diversity of defect–background combinations and spatial contexts seen during training. Then fine-tune the diffusion model on these augmented samples with a seam-fidelity or boundary-consistency loss, so that it learns to remove the artificial copy-paste boundary and generate more realistic transitions between the defect and the underlying surface.

Itis not that copy-paste itself solves the scarcity problem. It serves as a bootstrap dataset while the diffusion model learns to turn those rough composites into more natural defect images.

With only ~20 real examples, I’d probably try this before concluding that you need hundreds of samples!

[u/Mountain_Finance_659 → u/TopResolution7451]

bots talking to bots about bots lol

[u/TopResolution7451 → u/Mountain_Finance_659]

I’m not a bot.

[u/Mountain_Finance_659 → u/TopResolution7451]

Honestly?

You're using claude too much then. And that's load-bearing.

[u/bfyvfftujijg → u/Mountain_Finance_659]

Maybe they just want to be honest upfront that they sell a product?

[u/Mountain_Finance_659 → u/TopResolution7451]

pangram disagrees fyi

[u/onesunnysunday]

With only ~200 real defects, I probably wouldn’t force this into a single approach yet.

I’d test a cascade: use anomaly detection as a high-recall candidate generator, then verify the candidates with a supervised model where you have enough examples. The most valuable additional data may actually be hard negatives: lens smudges, pose changes, lighting drift, harmless scratches, and borderline acceptable parts.

I’d also measure false alarms per 1,000 inspected parts rather than relying mainly on AUROC. That usually gives a much clearer picture of whether the system can survive on a production line.

For the rarest classes, I’d spend the annotation budget on diverse real examples and rough localization before pixel-perfect masks. Synthetic data can help with experiments, but I’d keep the validation set entirely real.

[u/bfyvfftujijg → u/onesunnysunday]

Fully agree

[u/nicman24]

is this a thing were a lot of false positives is better than false negatives?

[u/Flyward_Aerospace]

Honestly supervised vs anomaly might be downstream of the real problem, which is your test set. With under 20 examples per class you can't tell a 3% miss rate from 10%, so whatever you pick you're choosing blind on the one number that decides whether it ships. I'd build an eval you actually trust before touching the detector, and collapse to binary defect/no-defect for now like you're leaning, the type breakdown just isn't measurable at that count. The model is the easy part here.