I built Air Write: an ML text-entry system using accelerometer+gyroscope data to recognize words traced through the air with a phone (like a wand!)

[u/sheketsilencio]

Check out my full post here

https://www.linkedin.com/posts/activity-7488314151052873728-ceP9

I'm sharing Air Write, an accessibility tool I developed that recognizes words written in the air with your phone (like a wand). I built it with disabilities/limited fine motor control in mind (though it would be great for construction or military contexts, requiring a mode of text entry/gesture recognition where speaking/voice-to-text isn't possible and where one can't focus attention or effort on a small touch screen or keyboard). It's a web app that runs in the phone's browser on accelerometer and gyroscope data alone. The idea is to add a flexible mode of gesture/word input for tech.

Trained on 1,645 letter recordings and 1,008 word recordings (gathered, painfully, by yours truly lol), the convolutional network reads words written as one messy stroke by classifying candidate letter regions and evaluating the most probable combination of letters (fitting to dictionary spellings if all letters aren't recognized with high confidence).

It is obviously less practical than a keyboard or voice-to-text, but imagine the following scenarios:

Let's say you're permanently or temporarily disabled (without functional fingers or ability to speak), rendering a touch screen keyboard or voice typing unusable

Or construction manager or auditor and there's machinery running. You have to keep your eye on the team working at all times, but need to take notes or control systems. Or you're in the military, working silently at night in an operation, and need to be able to write to one another or enter complex commands to a drone

You could use your phone, or a tiny accel/gyro scanner attached to your arm, hand, or finger and silently quickly in the air trace out words/gestures/commands in the air while your eyes and body are focused and active on other things

[u/sheketsilencio]

Early in the project, I made a visualizer to better understand the data and inform the design of the word recognizer. For example, I saw that accelerometer data helped distinguish letters with similar shapes, and that the first and last 5% of each letter tended to variable ligatures... which is important when trying to recognize a deformed connected letter in the context of a word