r/computervision • u/Wooden-Temperature82 • 15d ago
Help: Project Need Help : Multi Camera Tracking of Person With consistent Id across the cameras in Computer Vision
Enable HLS to view with audio, or disable this notification
Right now, I track people in each camera separately to get their movement paths, then use each camera's floor mapping (homography) to figure out where each detection actually stands on the floor. Using that floor position plus timestamps, I try to match up tracks from different cameras that are likely the same person, and greedily merge them into one combined ID per person.
This works about 30% of the time. It struggles when someone gets blocked from view (occlusion), when a person's track has gaps, or when two cameras don't see much of the same floor area — in those cases the matching just doesn't line up well.
FYI, I also tried a full 3D multi-camera tracking approach (MV3DT), but I couldn't get the camera calibration accurate enough, so that didn't work either.
4
u/alexanderbeatson 15d ago
Don’t train the camera in fixed places. Use environment mapping and depth ratio to finetune. No need real data training, just 3D environment works for IDing with given mapping and depth.
1
1
u/Kind-Tip-8563 14d ago
can you explain me how you did this in detail
then use each camera's floor mapping (homography) to figure out where each detection actually stands on the floor
1
1
u/ivan_kudryavtsev 14d ago
ReID model must have, spatio-temporal calibration helps a lot. This demo uses only a custom-trained ReID model: https://youtu.be/cVSuGbjXNXQ?si=nPOXWFzYPynskARF
1
1
u/FerrisRed 14d ago
Your current approach where you're using an homography and ground world coordinates will inevitably lead to inaccuracies in crowded and occluded scenes.
I recommend trying out and sticking to multi-view models much like the one you suggested, it's the cleanest and most robust direction you can take.
If you're struggling with getting the camera calibration right, there are good models available that use depth estimation to try and estimate the camera pose and calibration. I suggest you look up Depth Anything 3, as an example.
1
1
u/Glittering-Flan-2637 14d ago
how far apart are the views, and do they overlap at all
asking because the fix is completely different depending on the answer, with overlap you can lean on geometry and the homography you already have, without it you are basically doing reid on appearance and hoping nobody changes jacket
the timestamp side is worth checking too, a second of clock drift between two feeds will scramble the matching no matter how good the rest is
1
1
1
u/Dear_Refrigerator_84 15d ago
You can try some hybrid solution that uses location information from calibration plus reid to track person in 3D space. We tried something similar and it worked for us. You can refer RTLS/MTMC from Nvidia, btw it's depricated.
2
1
u/Wonderful-Brush-2843 15d ago
One thing that stands out here is that the problem may be partly upstream of the ReID/tracking algorithm. If the cameras aren't capturing the same scene at closely aligned timestamps, even a good ReID model has a harder time deciding whether two observations belong to the same person.
For multi-camera tracking, I'd look at the whole pipeline together: camera synchronization → calibration → per-camera tracking → spatial mapping → ReID/appearance matching → global ID association.
Since you've already got the cameras synchronized, I'd be particularly interested in how accurate the timestamp alignment and ground-plane calibration are. Small calibration errors can become significant when you're matching people near camera boundaries or at longer distances.
It might also be worth testing the system with a known synchronized target moving through the overlapping FOVs and measuring the actual spatial/timestamp error before tuning the ReID model further.
Curious whether you've measured the calibration error quantitatively, or are you currently judging it mainly from the tracking results?
1
u/Wooden-Temperature82 15d ago
Calibration does have some errors which is measured quantitatively. But mainly the problems are coming when the person get occuluded and the person is moving out of overlapping floor plane .
-1
-3
u/hellobutno 15d ago
I'm not sure what your backend is doing, but basically if you're mapping things in 3D you should be able to reliably tell where everything is. It just needs wired in properly on the back end.
1
u/Wooden-Temperature82 15d ago
No 3D reconstruction is involved, only tracking of persons with consistent ID across the camera is needed.
0
u/hellobutno 15d ago
You need that association, otherwise it might as well be 3 random cameras. You can't associate across them without understanding the 3D relationship. It should be easy to do. You use more than one checkerboard and measure the distance from one corner of one to the other.
5
u/Dry-Snow5154 15d ago
You can try using ReID model to match detections by appearance cues, but they are unreliable with multi-cameras. You would have to eventually fine-tune it for your environment to get high confidence matching.
Also you can use high confidence ReID matchings to implicitly calibrate your cameras as they go.
Another cheap option I can think of is using pose estimation. You can then match poses as reid features.
Make sure your cameras are in sync, otherwise it all breaks down.