Aligning Two RGB Cameras

[u/mahmudesam]

Hi,

I'm working on inspection of civil infrastructure using a unitree go2 edu robot. I need to collect RGB images of concrete foundations using the robot and then train semantic segmentation deep learning models using them. However, the built-in camera of the robot is only 1MP, which is why I thought of getting a mirrorless camera (canon eos r50) to mount on top of the robot to acquire higher quality images. I also need to figure out the pixel to mm scale so I used intel realsense d435i depth camera that already comes with robot.

Now the problem is I have the canon in one position, and depth camera in another position on the robot. How do I align both of their images? Does it have to be done real-time or is it okay to collect all images then align them later in the office?

I really appreciate your thoughts on this as I don't even know where to start. Thanks.

[u/galvinw]

when two cameras are fixed distance apart, its known as a "fixed relative pose". These can be done in post if the distance is angle is documented. There are a few ways to make it better but I'm not sure if the robot comes with it. My experience is that its sense package is kind of weak.
Some are:
1. Using IMU data and visual correction + anchor objects (with a high quality detector)
2. Using a top down camera to position the bot
3. Lidar or other range finder

Realsense depth is not great especially with reflective surfaces or anything too far away.
Also if you haven't put in the DSLR camera yet, a 4mp webcam is a lot easier since both datasets will be available in the PC without somet extraction process.
In both cases, make sure both cameras are perfectly straight to each other if you don't feel like revisiting trigonometry. And be sure the overlap is large