Can a webcam follow your gaze?
First-author research on low-cost gaze estimation using multimodal learning and facial landmark data.
IEEE REEPE 2024 / MULTIMODAL GAZE ESTIMATIONEye tracking is expensive and usually needs specialized equipment.
First-author multimodal model combining webcam images with eye-landmark metrics.
What can a normal webcam see?
Eye trackers can be expensive and awkward to set up. We wanted to know how far we could get using a webcam someone already had on their desk, without relying on a specialized infrared tracker.
We used MediaPipe to find facial and iris landmarks, cropped a strip containing both eyes, and fed the image and numerical eye measurements through separate neural-network branches. The model combined them to predict where on the screen the person was looking.
Why use two kinds of input?
The images give the model visual detail; landmark measurements capture eye geometry and approximate orientation. Combining them let us work with both signals rather than choosing one.
We also wrote a data collection tool that recorded the webcam frames, landmark measurements and the mouse pointer position together. We gathered more than 70,000 training samples from seven people and used 10,000 samples for validation.
What worked—and what didn’t.
In our desktop experiments we reported a mean gaze-distance error of 8% of the screen dimensions, with about 6 percentage points of standard deviation. It wasn't a drop-in replacement for a specialized tracker, but it was a convincing result for an ordinary webcam.
Related research
At the American University in Cairo, I also worked on eye tracking and interactive tasks as instruments for studying learning and cognitive psychology—an adjacent, but distinct, research thread.
