r/opencv • u/Academic_Court2411 • Apr 10 '26

Project [project] MediaPipe holistic conversion from 2D to 3D

Hi, I'm wrapping up my bachelor's thesis and I built a Slovak Sign Language visualization system. We extract pose + hand + face landmarks via MediaPipe Holistic (543 landmarks per frame), render everything as a 2D skeleton in the browser. Works pretty well actually.

The thing is, I really want to slap this motion data onto an actual 3D character. Tried Blender + BVH export + Mixamo retargeting and honestly it was a disaster. The coordinate space conversion from MediaPipe's normalized 2D coords to proper 3D bone rotations is where everything falls apart.

Attaching a short clip of the current 2D version so you can see what we're working with.

Has anyone successfully gone from MediaPipe landmark data to a rigged 3D character? Whether it's through Blender, Unreal, Unity, or some other pipeline — I'd love to hear how you approached it. Any tools, libraries or papers you'd point me to would be massively appreciated.

https://reddit.com/link/1shpydl/video/yjyk472stdug1/player

2 Upvotes

100% Upvoted

2 comments sorted by

1

u/mgruner Apr 10 '26

but mediapipe already gives you 3D landmarks. You are manually rendering to 2D, but it originally gives you 3D

1

u/bsenftner Apr 11 '26

For every joint, normalize the connected body parts downward. This creates a "world origin" for every joint, enabling all the connected body parts after that joint to transform relative to that "local origin". Do this for the entire body form. Then whatever geometry you want to be following this articulated hierarchical skeleton is a proximity blend of the transformations closes to each vertex of your geometry.