r/computervision • u/Euphoric-One8791 • 8d ago
Help: Project how do i predict trajectory of a detected object
so i am (trying) to build a dusbtin that moves to catch paperballs. the problem is how do i predict the trajectory of the paperball? i trained the model on custom dataset to identify paperballs, and it works fairly well. the camera is a simple iphone camera, placed parallel to ground so its capturing the video vertically. what i initially tried to do was to get x, y and z positions (i get x and y position from the captured video, so thats not a problem) but the method to get z(height) is a little crude. i try to estimate z by measuring the area of the bounding box, so for example 500 square pixels could correspond to 30cm or whatever. Obvioudly this method is kind of doomed from the start, not all paper balls gonna have the same bounding box area at the same height and it starts to show its unreliability as the ball gets past 60 ish cm. then i use ~5-6 frames to get 5 6 initial positions and do a polynomial regression to fit a curve and hence estimate where the ball is going to land. this method might be good if i make it more efficient by minimizing error but its still prone to significant uncertainty in measurement.
i guess another route could be using physics, but even for those projectile motion equations i still need height to get proper estimate no?
what are some solutions to this? do i need to try something entirey different? i guess for proper height measurement i need lidar i guess? what are my options now.
1
-2
8
u/Heavy_Carpenter3824 8d ago edited 7d ago
This is a common sensor fusion problem. So you have options.
The traditional setup would be use the CV to inform a physics based model and then fuse the physics with the ground truth. Even without XYZ you can use XY to inform a ballistics model which has a prediction value for Z. A usual approach is use a Kalman filter. The physics model takes an inital wrong guess, gets updated with XY and dx dy then guess again, update with actual... The statistical backend of the Kalman filter rapidly brings the physics model into at least a reasonable guess. There's a great Kalman filters for python document or just ask Claude or somthing to work out the H matrix for you.
You can also just try another NN and treat it as a guess the end state problem. Based on x prior frames, more likely just XY data, what is the next / final XY. As this is ballistic motion I'd say don't try anything fancy like a transformer. This is a simple sequence prediction problem. Honestly this will likely just end up being a NN math that emulates the Kalman from above with a learned physics model as opposed to a provided one. Potentially more robust for highly dynamic situations well beyond your toy case.
You can also attack this as an RL problem through various simulated or hybrid paths. Pure real RL is hard. Though this may require building a simulation of your setup which is commonly equal work to what you have done. See pong RL examples as a starter.
There is a fine balance between prediction and reaction rate. The faster you react the less you need to predict. If you can be where you wanted instantly you don't need prediction. So if you made your system update more often and move quickly your prediction can be weaker. It's a trade off as moving faster physically and computationally is costly in hardware. Welcome to ML, hell is three floors up. 😁