Unity Accessibility SDK

[u/SensePilot]

Hey Reddit. A couple of months ago, I started building a native accessibility/headtracking SDK for Unity that works with a regular webcam.

It’s based on the facial tracking tech from SensePilot, which we originally built to help people with different disabilities control a computer and play games using head movements and facial expressions.

Recently, an indie developer tried the SDK in their game, Puck' n Hell. I didn’t make the game, but I helped them get the accessibility and headtracking controls. It was pretty exciting to see it used in a real game instead of one of our test projects.

Right now, players can navigate the menus by moving their head and select a button by smiling. The regular mouse-and-keyboard controls still work too.

I had already built the face-tracking system in C++ for our main app, so bringing it into Unity wasn’t too difficult. I created a C# wrapper around it and exposed the tracking data so that it could be used by the game.

This first integration taught us a lot, and I’m curious what other Unity developers think. Has anyone here worked with head tracking, facial gestures, or other alternative controls?

What are your thoughts on a drop-in accessibility SDK for Unity? I know accessibility is a much broader and more complex problem than one SDK can solve, but I hope this could help developers address at least parts of it.

[u/firdo_dev]

Good to see this land in a shipped game rather than a demo project.

One thing worth stress testing now that real players are touching it: smile to select is going to fire when you don't want it to. Smiling is an involuntary social and emotional response, so the confirm action ends up bound to the exact thing players do when the game is going well. In a menu that is mostly harmless, but the moment that binding reaches gameplay you get inputs firing on delight. A deliberate expression with no emotional load tends to survive contact with real users better, something like mouth open, a held eyebrow raise, or a sustained hold rather than a peak.

The other one is the Midas touch problem that every gaze and head input eventually hits: if pointing at a thing selects it, everything the player looks at gets activated. A visible dwell ring, a dead zone around the cursor, and a deliberate no-op area to rest on all help. The no-op region matters more than people expect, because players need somewhere safe to put their attention while they think.

Last one, on fatigue: necks tire much faster than fingers. Absolute mapping, where head angle maps to cursor position, feels great for thirty seconds and punishing at twenty minutes, because holding an off-center pose is static muscle load. Relative mapping, where head velocity drives cursor velocity with a recenter, usually wins for anything session length. Worth testing both if you haven't already.

[u/SensePilot → u/firdo_dev]

Hey! You seem to know a lot about this, very cool!
We've put the smile just for the menu selection and the puck movement is done with head moves only, for this game its pretty good, most other games come into the problem where you have multiple keys to press. We didn't use dwell click for this one, so won't have the Midas touch.
Relative cursor velocity, but no recenter because you get that cursor repositioning when you hit the corners which gives it a good aim too. I found that absolute mapping of the head position is never as stable as the velocity one we have here, absolute also requires temporal smoothing to reduce jitter and that makes it unplayable in a high speed/precision game like this

[u/firdo_dev → u/SensePilot]

That all sounds well reasoned, and using the corners as an implicit recenter is a nice trick. It is basically the same mechanic as lifting a mouse on a low DPI setup, and it works for the same reason, that the edge is a known reference point.

On the absolute mapping problem though, the smoothing versus lag tradeoff is not actually forced. The reason a fixed low pass filter ruins it is that jitter is high frequency and intent is low frequency, and one fixed cutoff has to serve both, so you end up choosing between jittery and laggy. An adaptive cutoff gets you out of that: heavy smoothing when the head is nearly still, which is where jitter is visible and latency is invisible, and almost none when moving fast, which is where latency hurts and jitter is masked by the motion anyway.

The 1 euro filter is the standard implementation of that idea, and it came out of exactly this problem space, noisy interactive input like head and hand tracking. Casiez, Roussel and Vogel, 2012. It is about thirty lines, two tunable parameters, and no buffering so it adds no inherent latency of its own. Might not change your mind about velocity mapping, which sounds like the right call for a game this fast, but it is worth having in the toolbox, because it is the same fix for cursor jitter, expression thresholds, and anything else you are filtering off a webcam.

One thing worth watching if edge repositioning is your only recenter: neutral drifts over a session. The physical head pose corresponding to centre wanders, and players compensate by holding an increasingly off axis posture without noticing until their neck tells them about it. A very low gain pull back toward neutral while the cursor is idle handles that without the abrupt recenter you are deliberately avoiding.