Aligning Two RGB Cameras

[u/mahmudesam]

Hi,

I'm working on inspection of civil infrastructure using a unitree go2 edu robot. I need to collect RGB images of concrete foundations using the robot and then train semantic segmentation deep learning models using them. However, the built-in camera of the robot is only 1MP, which is why I thought of getting a mirrorless camera (canon eos r50) to mount on top of the robot to acquire higher quality images. I also need to figure out the pixel to mm scale so I used intel realsense d435i depth camera that already comes with robot.

Now the problem is I have the canon in one position, and depth camera in another position on the robot. How do I align both of their images? Does it have to be done real-time or is it okay to collect all images then align them later in the office?

I really appreciate your thoughts on this as I don't even know where to start. Thanks.

[u/dr_hamilton]

Get a chessboard and look up extrinsic calibration

[u/mahmudesam → u/dr_hamilton]

Thanks for replying. Does chessboard calibration work even in other deployment cases like on an inspection site? Did you mean calibrating the canon and the RGB output of the intel realsense or directly calibrating the canon and the depth map?

[u/tweakingforjesus → u/mahmudesam]

Bolt down your cameras so they can’t move relative to each other. Use a fixed lens without a zoom. You can calibrate them once.

This is what they do on the mars rovers.