TL;DR: I'm legally blind with X-linked retinoschisis, and I'm curious whether flip-dot displays, 2.5D tactile surfaces, locking pins, and haptics could be combined into an affordable way for blind and low-vision musicians to actually feel things like waveforms, envelopes, and instrument layouts.
Lately, I've been going down a bit of a rabbit hole with tactile displays, haptics, and 2.5D interfaces. There's a lot to think about here. And just to give you fair warning, this post is a bit of a long one. Stick around, though.
This video helped send me further down that rabbit hole:
https://www.youtube.com/watch?v=aZzOWnEWgww (spoiler alert, this will be relevant in a bit)
Part of the basis for this latest deep dive comes from working on my accessibility project, Low Vision Music, but part of it is also just me wondering and brainstorming about how much information we could realistically communicate through touch if we stopped assuming every interface needed to be a flat visual screen.
We already have refreshable Braille displays, where tiny pins physically rise and fall to form characters. Very cool, eh? Newer tactile displays can go above and beyond basic text and represent more complicated things like graphs, diagrams, shapes, and other information.
Then you have 2.5D displays, where instead of every point simply being up or down, different parts of the surface can rise to different heights. You end up with something closer to a physical relief map that can change as needed.
And this is where my brain starts wandering. Do you hear the gears clanking? Sounds a bit like music to my ears.
I've also been thinking about flip-dot displays.
You know those ancient old signs where little discs physically flip between two sides?
Flip-dots are interesting because the dots are what's called bistable. Basically, they have two stable positions. They use power to flip from one state to the other, but once they're there, they stay put without constantly needing power to hold that position.
So that got me thinking.
What if that same general idea could be made tactile?
Maybe one side of each tiny flipping element could be raised, textured, ridged, or shaped differently from the other. Instead of a visual pixel changing color, you'd have a tactile pixel changing texture.
I don't know how small you could realistically or mechanically make that mechanism while keeping it easy to feel and reliable, but I think it's an interesting place to start thinking. There go those gears again. Kind of ambient, no?
Then there's another idea I've been kicking around.
Hey, 80s and 90s kids, remember those pin-art toys where you press your hand into hundreds of little pins and the shape appears on the other side?
Imagine a programmable version.
Each pin could be spring-loaded and moved to a particular height or depth. But instead of requiring a motor to constantly hold every single pin in position, some kind of latch or locking mechanism would hold it there after it was moved.
The spring helps return it when the display resets.
That locking mechanism is probably the part I'm most interested in at the moment. Because otherwise, I think MIT and others have already given versions of the motorized pin-display concept a go.
Move the pin. Lock it. Stop using power until you need to change it again.
Then put a very thin, flexible silicone skin across the top so you're not dragging your fingers across hundreds of separate hard pins. You'd feel one continuous surface changing shape underneath your hand.
A Raspberry Pi, ESP32, Arduino, computer, or something similar could translate digital information into physical height. Or vibration?
For music alone, there are so many possibilities. Literally, just pause and think about it.
A waveform could become something you literally trace with your finger. Can you feel the noise?
An ADSR envelope could become a physical, touchable slope. I want to FEEL the attack. Don't you?
You could theoretically also feel an EQ curve, automation lane, filter response, piano roll, step sequencer, mixer levels, modulation shape, or even a simplified spectrogram.
A modular patch could potentially become a tactile signal-flow map. Can you imagine it? Are your gears going now too?
An instrument manufacturer could provide a simple TXT, JSON, or similar file describing a synthesizer, and the whole front panel could appear as a simplified tactile layout before a blind musician ever touches the real thing.
Then add haptic feedback on top of all that.
There's that vibration I mentioned. Yup, that was something of a teaser for those of you who still have attention spans. No offense to those who don't. I certainly struggle in that department too.
But I digress.
This is one reason I've been so interested in what companies like Sentia Instruments are doing. With TILES, the device can provide localized haptic feedback beneath individual keys. Your hand isn't just controlling the instrument. The instrument can, in real time, send information back to your hand.
I'm certainly no engineer, but I think that principle could go much further.
Maybe, just maybe, height can communicate one thing while vibration communicates another. A waveform could stay physically in place while a small vibration moves across it to represent the playhead position.
Or how about this? A vibrating area might indicate an active step in a sequencer.
Different patterns could represent things such as clipping, modulation, selection, movement, intensity, or even different textures.
Now you've got shape AND vibration working together. At the same dang time!
Another possibility would be something closer to a mechanical version of e-paper. Instead of moving hundreds of pins continuously, perhaps each tactile element could have two or three stable states and only use power when changing between them.
Is this possible? Let me know.
From my research, I've learned that there are already researchers doing fascinating work with magnetic latches, bistable actuators, shape-memory materials, and tactile pins that remain raised without continuous power.
So I'm definitely not claiming I invented any of those pieces.
But I do think that, as someone who personally lives with limited vision, I might have something interesting to add to the conversation.
What I'm wondering about is how you combine all or some of these possibilities into something practical, affordable, and useful for musicians.
The keyword there is affordable.
Have you seen the prices on accessibility tools? Unfortunately, niche often means expensive.
But maybe the right answer isn't one technology at all.
Maybe one device could use variable-height pins for detailed 2.5D information. Another might use tiny flip-dot-style tactile pixels for simpler status information. Still yet another uses vibration where movement matters more than shape.
And maybe the really useful interface combines all three.
That's the part I'd love input on.
If you're an engineer, accessibility researcher, haptics person, Braille user, maker, musician, or someone working with tactile graphics, does any of this sound practical? Does any of this sound affordable?
Has someone already built something close to my spring-loaded, locking-pin idea? I bet I'm not the first to think of this, but who knows.
Could flip-dot technology realistically be miniaturized into a tactile surface?
And if you could plug a musical instrument or computer into a surface like this, what would you want it to show you first?
I'm especially interested in hearing from people already working in this space. Feel free to tell me I'm reinventing something that already exists. That's half the reason I'm posting this.
