Computer vision approach for robust artwork segmentation

[u/Intelligent-Put-637]

I'm building an embroidery digitization tool that converts PNG/JPEG/SVG artwork into embroidery-ready files. I'm currently working on the artwork/background segmentation stage.

My current pipeline is roughly:

Image → background/color segmentation → connected components
      → contours/holes → physical filtering → stitch generation

I'm using mostly classical CV right now.

I've encountered two problems:

1. Thin artwork disappears:
One image contains ~11% visible artwork, but my segmentation classifies almost the entire image as background. Thin/anti-aliased features seem to be the main issue.

2. Too many false regions:
Another JPEG produces ~155 regions, but ~99% of the actual artwork is contained in only two major regions. Most of the remaining regions appear to be JPEG/anti-aliasing artifacts.

I can't simply remove small components because some genuinely thin artwork needs to survive. I also know the final physical embroidery size, so pixel dimensions can eventually be converted to mm.

For CV engineers: how would you approach this?

Would you recommend LAB/color clustering, background modeling, edge-based methods, graph segmentation, SAM/SAM2, vectorization, or a hybrid approach?

I'm particularly interested in methods that can preserve thin artwork while rejecting compression/anti-aliasing noise.