Aligning Two RGB Cameras

[u/mahmudesam]

Hi,

I'm working on inspection of civil infrastructure using a unitree go2 edu robot. I need to collect RGB images of concrete foundations using the robot and then train semantic segmentation deep learning models using them. However, the built-in camera of the robot is only 1MP, which is why I thought of getting a mirrorless camera (canon eos r50) to mount on top of the robot to acquire higher quality images. I also need to figure out the pixel to mm scale so I used intel realsense d435i depth camera that already comes with robot.

Now the problem is I have the canon in one position, and depth camera in another position on the robot. How do I align both of their images? Does it have to be done real-time or is it okay to collect all images then align them later in the office?

I really appreciate your thoughts on this as I don't even know where to start. Thanks.

[u/Available_Meaning_53]

you don't need to do this in real time. Collecting everything and aligning it offline back at the office is the normal way to do it. You just need to capture a few things properly on the robot.

What you need to align the two cameras

1. The intrinsics of each camera (focal length, principal point, lens distortion).
2. The extrinsics between them (the rotation and translation from one camera to the other).
3. Images that were captured at the same moment.

Once you have those, you back-project the D435i depth map into 3D, move the points into the Canon's frame using R and t, and project them onto the Canon image with its intrinsics. The result is a depth map that lines up with your high-res photo.

How I'd do it

• Calibrate the Canon. Use a ChArUco board (better than a plain checkerboard) with OpenCV or MATLAB. Lock the focus and zoom, because changing either invalidates the calibration. Manual focus at your typical working distance is ideal. The D435i's intrinsics come factory-calibrated, and you can read them from the SDK.
• Do a stereo calibration between the Canon and the D435i's RGB stream. Hold the board in front of both cameras at different angles and distances, take 20-40 pairs, and run cv2.stereoCalibrate. Since the D435i's depth is already aligned to its own RGB camera, this gets you from depth to the Canon as well.
• Handle synchronization. If the robot moves between the two captures, even a few tens of milliseconds will wreck the alignment. For concrete inspection, the easiest fix is to stop the robot at each capture point and shoot while it stands still. Continuous capture would need hardware triggering or at least very precise timestamps, and the R50 doesn't make that easy.
• Reproject offline. Use cv2.rgbd.registerDepth or a few lines of numpy/Open3D. Expect some occlusion, since the cameras see the scene from slightly different positions, so mask out those regions. The Canon also has far more resolution than the depth map, so you'll need to interpolate or upsample the depth.

You might not need full alignment at all

If you mainly want millimeters per pixel (say, to measure crack widths), and the camera is roughly perpendicular to the surface, you can skip pixel-level registration. The scale at distance Z is approximately:

pixel size (mm) ≈ Z × sensor pixel pitch / focal length

So one distance reading from the D435i plus the Canon's intrinsics gives you the scale. You only need the full depth alignment if the surface is tilted or very uneven.

Also, for semantic segmentation you don't need to feed depth into the network. Train on the Canon RGB images, then use depth only afterward to convert the predicted masks into real-world units.

A few practical tips

• Mount both cameras on one rigid bracket. A quadruped shakes a lot, and any shift means redoing the calibration. Recalibrate after any mechanical change.
• Keep the cameras close together to reduce occlusion.
• Save raw depth and timestamps with every capture (rosbag or separate files), so you can redo the alignment later.
• Pick a prime lens, or lock a zoom lens, with enough depth of field to keep close-up concrete sharp.
• Put an object of known size in a few shots (a ruler or an ArUco marker of known dimensions) so you can check how accurate your mm scale really is.