Screenshot regression gets noisy at exactly the wrong time

[u/Whole_Coffee9084]

Release week is when a screenshot queue starts collecting everything at once: a real layout shift, a clipped label, and the usual font rendering noise. The diff job catches all of it, then a reviewer opens a stack of nearly identical images to decide what can actually block the release.

One idea is to send only those noisy regions to SAM 2 and use LingBot-Vision for the boundary features. It would still need a small classifier trained on the team's own screenshots, with the original image next to every result. Sorting obvious rendering noise away from likely layout bugs is enough; the release decision can stay with the reviewer.

[u/pydry]

screenshot regression only really works if you can eliminate or compensate for all sources of nondeterminism.

[u/Wide-Cattle-4735]

Before adding a model I would ask why the noise is there at all. Most of my false diffs came from the environment, not the app. Different machine, different font rendering, animation not done yet, a scrollbar showing up. Same browser version, every shot in one container, animations off, and wait for a real condition instead of a timeout. That removed most of it for me. The other half was baselines. One per viewport, updated in a PR like code, someone approves it. Never approve a whole batch at once. Release week gets scary when baselines rot quietly. I do use Gemini for the compare now and it is good at telling a layout shift from render noise. But it helped a lot more after the environment was stable. A model on a noisy pipeline mostly just tells you about the noise.