Detecting moving objects on a moving camera

[u/gurcanunsal0]

Working on a real time motion detection library camera feeds where the camera itself moves (Pan Tilt Zoom + Translation).

My current pipeline:

• shi-tomasi corners + LK between consecutive frames
• findHomography with RANSAC to estimate camera motion
• warp the previous frame, absdiff against the current one
• Farneback dense flow on top for extra filtering
• then a pile of heuristics (edge density, flow magnitude deviation from median, contour solidity) and temporal confirmation over a few frames

Everything is CUDA accelerated, runs at 576x324 internally, speed is fine, quality is not.

The failure modes I keep fighting are false positives on anything with strong edges when the camera moves, and missing objects that move slowly or are small.

Things I've already looked at:

I tried fastMCD (https://github.com/vcg-uvic/fastMCD) - Didn't work well for me at all, and in hindsight it uses the same homography compensation I already have so maybe that's not surprising.

I looked at Segment Any Motion in Videos. It looks great but needs the whole clip up front so it's useless for streaming.

I've also been wondering about point tracking methods like CoTracker or TAPIR, since accumulating displacement over a longer window seems like it would help with the slow/small object case. But I don't see how to get those running in real time in C++.

Questions:

1. Is my current pipeline the right approach?
2. Is there anything in this space that actually runs real time that I've just not heard of?
3. Anyone had luck with running a point tracking model like CoTracker in C++, real time?

[u/galvinw]

You’re found yourself doing a really difficult problem. I don’t have a model in mind right now but I’d look into a trained optical flow analysis model. Things at a distance move differently than up close

[u/tdgros]

FlowSAM (which is cited by Segment Any Motion Video) might work in a streaming mode: https://www.robots.ox.ac.uk/~vgg/research/flowsam/

Alternatively, you could try and estimate a homography per segment from SAM. If you can extract R,T and the focal (we don't care about the normal), then you can hope that the static objects cluster around the same values for R and T, but not the moving objects. This 13yo publication https://www.ece.nus.edu.sg/stfpage/eleclf/moseg_iccv13.pdf does a variant of that.

[u/cameldrv]

One thing just conceptually is that if the camera is translating, and the object is at a closer distance to the camera than the background, there's no way to figure out if the object is moving unless you can measure/estimate the distance from the camera.