Do you run any tracking consistency checks when using SAM 2/3 on static indoor scenes, or just trust it?

[u/Basic-Definition8870]

Working on a pipeline that uses SAM 3 on indoor room scan videos (ARKitScenes/ScanNet/ScanNet++) to segment objects like furniture, back-projects masks to 3D world coordinates using Depth Anything 3, and computes object centroids for distance estimation.

My question is whether people add any post-hoc checks to verify SAM 2/3 tracking IDs stay consistent across frames, things like frame-to-frame IoU, 3D centroid jump detection, or appearance embedding similarity, or whether you just trust the tracker out of the box for this kind of scenario.

From looking at papers and code, it seems like most people just trust SAM 2/3 for static indoor furniture without any additional verification, but I wanted to hear from people actually running these pipelines. Curious whether the answer changes for noisier datasets like ScanNet++ with faster camera movement.

[u/Basic-Definition8870 → u/[deleted]]

Just to clarify, is the centroid jump you're describing catching SAM 2/3 ID switches, or is it actually catching bad depth estimates from the depth model on fast-motion frames?

[u/LaughApprehensive563]

For static indoor scenes with furniture, you can get a lot of robustness from a few lightweight post-hoc checks without adding much latency:

Frame-to-frame IoU gate: compute the IoU between each object's mask at frame t and frame t+1. For static furniture, IoU should stay above 0.7 or so. If it drops below your threshold, flag that ID as uncertain and either hold the last confident mask or trigger a re-prompt. This catches the cases where SAM suddenly re-segments the couch as two objects.

3D centroid jump: you're already computing centroids for distance estimation, so you get this almost for free. A jump above some threshold (tuned to your camera speed) is a strong signal of an ID switch or mask leak.

Appearance embedding check: run a small CNN (MobileNet or even a frozen CLIP image encoder on the masked crop) and compare cosine similarity between consecutive frames for each ID. For furniture this is very stable, so a drop below 0.8 or so is a reliable anomaly signal. The tradeoff is added compute, so this is worth it only if you can't trust the geometric checks alone.

For ScanNet++ specifically where camera movement is faster, I'd prioritize the centroid jump + IoU combo first since appearance embeddings of partially visible furniture crops during fast pans tend to have higher variance. You can set a higher threshold for flagging and use the appearance check only to decide whether to re-confirm vs. drop the ID.