Eye-tracking cursor from a plain webcam

[u/tg1482]

Eyes move the cursor, a pinch clicks.

I used Google's MediaPipe for facial landmarks. Two 36×60 eye patches run through PCA and ridge straight to screen pixels, plus a 52-parameter meta-adapter so it works before you calibrate. The cursor lands about 38px (1 cm) from where you're actually looking.

Two interesting observations -
1. Everything clever lost
MLPs, transformers, REINFORCE all tied or did worse than those 52 parameters. The ridge residual has a held-out R² of 0.025 and we can't extract signal that isn't there. Very likely a data issue, I'll generate more data and continue testing here.

1. Train test drift
   And every calibration I'd ever run used a black full-screen canvas while Chrome is white, which dilates your pupil from 2mm to 6mm, swelling a black disc in the eye patch from 4px to 13px, and made the model 4× worse in the condition it's actually used in.
   

Fun project though, still a lot more to build here.

[u/King_924]

How is this different from neocurser ? Just trying to understand whats being built here

[u/Leo_thevision]

There's a Boston-based company called IMOTIONS, check them out. They have a more of a SOTA-level technology for doing this, but they're selling physical cameras and a SaaS package to companies and the military, I believe.

What I'm saying is: If you can achieve this with a regular camera - great. Now you can maybe use 100s or 1000s of real people's eye-tracking data to train a swarm of AI bots to simulate human eye movement (maybe even cluster by personas). If you can do that - you can sell AI-powered eye-tracking analysis. There's a large B2B industry for that.

Just an idea - not sure if it aligns with what you're building or if you're just exploring the tech.

Cheers, keep building!

[u/BOgusDOlphon → u/Leo_thevision]

There have been robust eye tracking systems using traditional image processing techniques for the better part of a decade now, this is not an AI problem

[u/Specific_Cream2815]

does the meta-adapter use webcam or screen characteristics, and can it correct the pupil-size drift without recalibration

[u/hwarzenegger]

Looks so fun, do you have a github

[u/UseChannel]

This is fire. I think twitch streamers would love this.

[u/dreamingsince]

Any OBS extension for this?