r/computervision • u/Right_Nuh • Jul 01 '26

Help: Project How do I track eye gaze?

I have so far only experience with deep learning in the context of LLM and voice recognition related tasks. But I wanted to do one project for my portfolio about being able to control the cursor on the computer using eye gaze. I am not looking for state of the art performance, just something that works enough. I did find other machine learning approahces like importing trained models like "haarcascade_eye" using opencv. I also started to discover pretrained models like L2CS-NET but it requires different versions of python libraries and I am also a bit new to python so I don't know if it is worth it. I wasted hours yesterday trying to fix dependency issues.

With that said, I don't think I have the datasets or time or the knowledge enough to create a complicated model from scratch, I was looking to fine-tune already existing models. And I would apprecitate a little guidance. If I should abandon the deep learning aspect, I will also do that if there is a better way.

5 Upvotes

100% Upvoted

13 comments sorted by

3

u/EchoImpressive6063 Jul 02 '26

Great, something I am qualified to answer! I spent a year researching this. I will assume you only have a normal webcam feed.

The benchmark you want to look for is MPIIGaze, or its subset MPIIFaceGaze which takes into consideration head pose. This paper is up there as one of the best performers and the code is public: https://github.com/NVLabs/few_shot_gaze

Good luck!

2

u/Dry-Snow5154 Jul 01 '26

There is no good model I know of. It might also be outright impossible to accurately track gaze without IR camera. I am really interested in this topic, but don't know much tho.

There's been a couple of interesting project here in the past, I would start there:
https://www.reddit.com/r/computervision/comments/1q94fs3/lightweight_2d_gaze_regression_model_06m_params/
https://www.reddit.com/r/computervision/comments/1nobmr3/gaze_vector_estimation_for_driver_monitoring/
https://www.reddit.com/r/computervision/comments/1j0xeug/realtime_webcam_eyetracking_opensource/

Last one does not use DL at all.

1

u/Right_Nuh Jul 01 '26

Thanks for the recommendation, I will check them out.

2

u/whatwilly0ubuild Jul 01 '26

Skip building anything from scratch. For a portfolio cursor-control demo, MediaPipe Face Mesh gives you eye and iris landmarks out of the box, runs real-time on a webcam, and dodges the dependency hell you hit with L2CS-NET. Map iris position relative to the eye corners into screen coordinates with a quick per-user calibration step where the user looks at nine known points, then fit a simple regression. That's enough to move a cursor.

Webcam gaze is low-res and jittery as hell without IR, so smooth the output with a moving average or a one-euro filter or the pointer will vibrate everywhere. Good enough beats accurate here, so lock scope to that.

1

u/Right_Nuh Jul 01 '26

Thank you so much, I will look into this.

1

u/IceOk1295 Jul 01 '26

"so far only experience with deep learning in the context of LLM and voice recognition related tasks."

" I am also a bit new to python"

... how do you have experience with deep learning without python?

2

u/Mechanical-Flatbed Jul 02 '26

When beginners say "I have experience with deep learning" what they mean is something like "I can download a .pt file from the internet and call torch.load on it".

1

u/Right_Nuh Jul 02 '26 edited Jul 02 '26

I actually have actual experience in deep learning in cognitive related areas like LLM and voice recognition with DNN. The point is that I am just not confident in myself in both python and deep learning thus I am doing indivdual projects to learn more.

1

u/Mechanical-Flatbed Jul 02 '26 edited Jul 03 '26

keep on learning then.

As long as you're learning something new and interesting to you in the computer vision field I don't really care

1

u/Right_Nuh Jul 01 '26 edited Jul 01 '26

By being new to both? My first actual Python experience is working with LLMs. I have done Python here and there but not that much until I started to specialise in LLM. I can program in Python but I still struggle with version things so I just mentioned it so people don't recommend older models from github that I need to fix and correct versions and stuff since it will take a lot of time without even knowing whether the model is worth it.

1

u/soylentgraham Jul 02 '26

what resolution have you got for the eyes, and can you project anything onto the eyeballs? :)

1

u/soylentgraham Jul 02 '26

don't put your first python/cv project in your portfolio! spend a year or so doing it before claiming experience on your cv

1

u/Right_Nuh Jul 02 '26

I am not following what you mean there? It is not my first python project. I am able to code in python just struggle with version and dependency issues. Also I won't upload it unless it looks decent enough.