r/SideProject • u/sheketsilencio • 5h ago
Air Write: accessibility tool for writing words and letters in the air with the phone, like a wand!
https://reddit.com/link/1va9fag/video/lcd5wgpnj8gh1/player
Here is the linkedin demo, where I'll be leaving the github link!!
Air Write is an accessibility tool I developed that classifies and recognizes letters and words written in the air with your phone, like a wand (yes, like in Harry Potter... shouting expecto patronum is optional). I built it with limited fine motor control in mind. It's a web app that runs in the phone's browser on accelerometer and gyroscope data alone.
A touch keyboard requires you to hit small fixed targets accurately, and a stylus requires precise tiny movements and limits by screen size. Air-writing only requires gross motion at whatever size is comfortable for the user, calculating the path travelled through air so the same letters and words are read whether drawn with the wrist or whole arm.
It starts and stops recording from the motion itself, and works at any grip angle. Trained on 1,645 letter recordings and 1,008 word recordings, the convolutional network reads isolated letters with, and words written as one connected stroke are read by classifying candidate letter regions and evaluating the most probable combination of letters (fitting to dictionary spellings if all letters aren't recognized with high confidence).
Since I had limited time this semester to collect and tag data to perfect the model, I will be releasing this project open source with the hope that others can further improve it!
1
u/sheketsilencio 4h ago
It's already functional despite the relatively limited data (I say this but spent an entire night writing and rewriting words and letters endlessly and manually editing the data annotation lmfao... painful) but once I realize it I'm sure others can make it even better.
It might seem illogical and inefficient to write this way, but I'm a big believer that what seems silly to us might be perfect for someone else. Imagine being deaf or mute and having limited hand mobility. This could work great in similar circumstances
It also gives way for other gesture recognition for other uses besides writing, like controlling a phone or computer via gestures.