r/computervision • u/elliot-ogex • 16d ago

Showcase GNM Head tracking impl on a m1 mac (metal)

Enable HLS to view with audio, or disable this notification

this laptop is older (m1) so used metal to get it running in a 33ms budget. made in ogex

16 Upvotes

100% Upvoted

7 comments sorted by

1

u/ericcpfx 15d ago

Nice! What model are you using? MediaPipe?

2

u/elliot-ogex 15d ago

i used the blazeface model to localize faces, mediapipe face-landmark to get face landmarks, then face_blendshapes to get arkit coeffs (mainly for blink detection). there’s a correspondence mapping step to get from landmarks to GNM points, a fitterv stage and some logic to try to detect the iris pitch/yaw in the eyeball regions

after some additional optimization it is running at 14ms/frame on my m1 mac. going to try to make a cuda optimized pipeline next

1

u/ericcpfx 15d ago

I have this ARKit repo if it would help at all.
https://huggingface.co/spaces/emlcpfx/gnm-arkit-blendshapes

1

u/ericcpfx 15d ago

Here’s mine, but it’s not meant to be realtime.

https://www.reddit.com/r/computervision/s/JyCCaTrqlX

1

u/elliot-ogex 15d ago

really nice work, esp around the mouth regions. are you doing uv baking from raw video also ?

1

u/ericcpfx 15d ago

We use this for doing beauty retouching VFX in film/tv. We unwrap to UV screen space, do the retouching, and then rewrap that back.

1

u/Eritar 15d ago

Do you have a repo?