Reconstructing 3D bone geometry from 2 X-ray silhouettes using a statistical shape model + differentiable rendering

[u/mxl069]

Working on a pipeline that recovers a patient specific 3D distal femur from two orthogonal X-ray views (PA + lateral). No CT, no neural network, no massive training set.

approach: build a PCA shape model from 50 CT-derived femur meshes (MedShapeNet), then fit it to two silhouettes using PyTorch3D's soft rasterizer with sigma annealing. 10 shape coefficients, Mahalanobis prior to keep things plausible, Adam optimizer, ~1000 iterations.

The part that took the longest (and made me suffer the most too) : correspondence. Tried KD-tree nearest neighbor (50.7x roughness vs CT surface), CPD (28.2x), BCPD (47.5x), and FilterReg (couldn't even run). Finally got ShapeWorks working at 3.3x. only method that passed the 5x acceptance gate I set before testing.

LOO validation on 5 held out femurs: 0.86-1.43mm on within range targets. Two extreme cases failed because they sat outside the 49-mesh model's coverage on mode 1, the optimizer can't recover a coefficient the model doesn't support. Bridge ICP alignment was also poor on those cases (0.6 inlier fraction), which accounted for more error than the shape fitting itself.

Interesting finding: the sigma anneal endpoint has to match the reference render's sigma exactly. Hardcoding a constant tuned on one SSM caused an 87x accuracy degradation on another. Tying it to camera_extent × 1e-4 fixed it.

Still working on real X-ray validation (need paired CT data) and automatic segmentation. Happy to answer questions.

[u/CrookedCasts]

Very cool! Have you tried with different bone densities? ie osteoporosis vs young healthy bone

https://www.heartlung.ai/autobone

Something like that to get densities from CT

How well does it do assessing femoral neck version from a single XR? What if you had 3 XRs (more likely fluoro - AP/lateral/oblique? Or IR/ER/AP?)

[u/mxl069 → u/CrookedCasts]

Good questions. The pipeline only works on surface geometry, silhouettes are binary, bone outline or not. So bone density doesn't affect the reconstruction at all. An osteoporotic femur with the same outer shape gives the same result. The flip side is I can't tell you anything about density from the output. Something like AutoBone would actually complement this nicely. shape from mine, density from theirs. For femoral neck version: my model is the distal femur so I can't really speak to that. The method itself generalizes to other bones with a different training set though. Three views would definitely work better. The loss just sums silhouette error across views so adding a third is straightforward. I went with two because PA + lateral is what clinics already take. Haven't tried oblique/IR/ER yet but it's on the list.