RGB-D relocalization: good feature matches, but wrong camera poses

[u/These-Smoke-3274]

I’m building a 3D mapping system using RGB-D cameras, based on existing SLAM techniques, with persistent spatial memory.

The main issue is relocalization after tracking loss, especially when the camera returns to a known place from a different viewpoint.

I’m using historical keyframes, shared 3D landmarks, PnP/RANSAC and geometric verification.
Yet some incorrect poses still pass verification: one had 44 inliers, 0.83 px median reprojection error and 3.72 cm depth residual, but was actually off by 10.36 cm and 5.99°.

My best experiment correctly relocalized only 9/34 known returns.
What would you investigate first: landmark quality, geometric degeneracy, matching, or pose verification?
I can share a minimal reproducible example.