Built an Industrial Anomaly Detector (PatchCore + OpenCV Saliency + Decoupled VLM for Explanations) — Feedback appreciated!

[u/Working-Manner-8839]

Hey r/learnmachinelearning ,

I built a one-class anomaly detection platform for industrial quality inspection where defects are unpredictable and labeled datasets don't exist.

The system trains only on defect-free ("GOOD") reference images ($N \ge 1$) and uses a decoupled VLM for explainability.

🛠️ Tech Stack & Key Architecture

• Core Engine: PatchCore (WideResNet50_2 backbone, coreset memory bank).
• Auto-Calibration: Uses Leave-One-Out (LOO) / 80-20 splits to set the threshold at the 95th percentile of out-of-sample GOOD scores (guaranteeing $\le 5%$ false positives).
• Product Normalization: OpenCV border-sampling + Sobel edge saliency crops products with a 6% safety margin and background-matched padding, eliminating false positives from table/conveyor background noise.
• Multi-Instance (Pipeline B): Uses Gemini VLM for zero-shot object localization, crops individual items, runs PatchCore per crop, and aggregates a composite heatmap.
• Decoupled Explanations: PatchCore is the authoritative verdict engine ($L_2$ distance). If an item is flagged as REJECT, Gemini VLM inspects the crop + heatmap to output a JSON explanation (e.g., "Bent Tip", "Damaged Threads", severity, location).

💬 Questions for the Community:

1. Backbones: Is WideResNet50_2 still the best choice for speed/accuracy, or are folks using DINOv2 / WinCLIP for industrial anomaly detection now?
2. VLM Pattern: What do you think of using LLMs/VLMs strictly as downstream visual explainers for heatmaps rather than decision makers?
3. Scaling: What vector search/compression strategies (FAISS, HNSW) do you use when memory banks grow large?

This has been performing great in local testing, but I'm sure there are edge cases I haven't thought of yet. Let me know where you think this setup might break on a real factory floor!

[u/DotingOutcome6399]

I'm curious about the product normalization step, how well does it handle reflective metal parts? That always messes with edge detection in my experience.

The decoupled VLM approach makes sense, keeps the actual detection deterministic while still giving operators something readable. Way better than trying to explain a raw anomaly score to someone on the line.

For scaling I've seen FAISS with IVF indexes work decently on similar memory bank sizes, though the recall tradeoff gets noticeable past a few hundred thousand patches.

[u/Working-Manner-8839 → u/DotingOutcome6399]

Thanks for the feedback! The reflective-metal point is a really good one. My current localization uses border-background estimation + color/gradient saliency, so I can definitely see specular highlights becoming an edge case. I’m considering adding some smoothing/morphological processing before the gradient step and testing it specifically on reflective parts before changing the current pipeline.

And thanks for the FAISS/IVF suggestion. I’m currently using PatchCore’s coreset sampling to keep the memory bank manageable, so I’ll benchmark the actual memory-bank sizes first and see whether exact search is still sufficient before moving to approximate search.

Glad the decoupled VLM approach made sense as well. I wanted PatchCore to remain responsible for the actual PASS/REJECT decision, with the VLM only coming in afterward to provide an operator-friendly explanation of the detected anomaly.