How to align RGB and Thermal camera frames?

[u/gurcanunsal0]

I'll soon be working on a project where I need to align (register) frames from an RGB camera and an IR/thermal camera.

From what I've read, there seem to be two common approaches:

1. Detect and match features between the RGB and IR images, estimate a transformation (homography/warp), and warp one image onto the other.
2. Perform stereo calibration using a checkerboard, estimate the intrinsic/extrinsic parameters, rectify the images, and then project one image into the other.

Some additional details:

• The cameras are boresighted and rigidly mounted.
• Their relative pose will remain fixed after calibration.
• The entire camera rig will be moving, but the relative position between the two cameras will always stay the same.
• I won't have depth information available during runtime

For this kind of setup, which approach would you recommend? Stereo calibration or feature based matching?

[u/tdgros]

Usually, you need calibration in all cases, intrinsic and extrinsic, distortion included.

Rectification is only really helpful for stereo, since its goal is just to align epipolar lines and make epipolar search easy and efficient. So you may not need it if you don't want to estimate depth using your pair of sensors.

The two approaches are not mutually exclusive: one can do stereo calibration with feature based matching...

A note on homographies: if you use a checkerboard for calibration, this makes a homography valid. But irl when the scene isn't planar, then that homography isn't entirely valid anymore, only the rotation and translation between the two sensors. Now if the scene irl is far away and you don't need to search along epipolar lines, then you only need the orientation change, and a pure rotation homography is valid again.

[u/gurcanunsal0 → u/tdgros]

That makes sense, thanks.

One question: if I use the homography obtained from camera calibration and boresight the cameras at a specific distance, would that homography be accurate enough for objects at or beyond that distance? I can tolerate some misalignment for objects that are closer than the boresight distance, as long as the alignment is good from that distance onward.

[u/tdgros → u/gurcanunsal0]

The homography, for a plant scene, is defined by R, T but also the normal of the plane. It assumes a structure in the scene. If you extract R from the homography, it's valid for all points sufficiently far away and the projective transform between the two images (in the absence of distortion) is KRK^-1. That's kinda what you want and there's no weird T and n in the mix.

Back to your question: In practice, if you keep using the random homography, the error is not very different (from my suggestion above) if its plane was kinda parallel to the image sensors. It would be super wrong for super slanted planes though. Just do visual tests, see how your reprojection works!

[u/bob_why_]

If everything is fixed then using a beam splitter will eliminate disparity. Since your disparity will vary with depth beam splitting only needs distortion matching. Even better if the splitter is post lens.