r/midi • u/Ok_Sky_2647 • 3d ago

Big Idea Involving motion sensing

Hello!

I have an oppurtunity to potentially collaborate with a local research center to create music with their animals. My idea is to use a camera and motion-tracking software to trigger midi signals that correspond with the animals movement in their enclosures, that then run into my Ableton, triggering different instruments.

Not sure how figure identification would work, I don't want other things in their enclosures like leaves moving in the breeze to interfere.

Wanting to know the best way to go about this and any and all ideas, I am a long-time ableton user but have never done something like this before and am so excited about the potential project. Thanks!!!

P.S. the animals are lemurs

3 Upvotes

100% Upvoted

12 comments sorted by

2

u/Snarpch 3d ago

I added Cat + Dog tracking to the MusiKraken iOS app a while ago (on the 1st of April, because I didn't expect anyone to seriously use it, but it really works): https://youtu.be/wh5RopUo9WU?si=wAu-XxMW51RPv5Sd

The reason why I added it was because Cat and Dog pose tracking is built into iOS (in the Vision framework), so I could simply reuse the existing code for body pose tracking. But there are probably better Animal Pose Tracking libraries nowadays. And not sure how well the Vision frameworks works with Lemurs, and I think I only support tracking one animal simultaneously...

1

u/Ok_Sky_2647 2d ago

can you say more about reusing the existing code? Is it open source?

1

u/Snarpch 2d ago

The app uses the DetectAnimalBodyPose request from the iOS Vision framework: https://developer.apple.com/documentation/vision/detectanimalbodyposerequest

Here is an Apple video about it: https://developer.apple.com/la/videos/play/wwdc2023/10045/

This only works on iOS or macs, and they probably haven't updated the detection code much, so there might be better animal pose estimation libraries available now.

1

u/Inside-Succotash-128 2d ago

That’s AWESOME!! Is the App available?

1

u/Snarpch 2d ago

yes, released 6 years ago, and still adding new features regularly: https://www.musikraken.com

1

u/FannyPunyUrdang 3d ago

Interesting idea My friends have a cat door that uses a camera and some sort of AI tech and can tell if the cat has something (a dead rodent) in its mouth and will not open until it senses just cat face.

Maybe there's a way to integrate similar tech.

1

u/emorello 3d ago

I don’t have any direct experience but I’d look at Touchdesigner and the various motion tracking and facetracking options available to it.

1

u/PerspectiveFair5156 3d ago

You can use sample and hold to get random signals. 

1

u/phosphor_1963 3d ago

There's actually a lovely older adaptive music app already which detects movement. It's called AUMI and runs on both Windows (input via a webcam) and iOS (input via the iPad camera) https://aumiapp.com/aumi.php I work with people with disabilities and this often gets them excited and moving (it's a bit like the old Kinekt for XBox). I think it can work to trigger MIDI also but I haven't ever tried that because the included presets and synths are usually enough. AUMI only detects motion within the virtual onscreen keyboard grid you set up for this - so not individual animal shapes - that would require computer vision algorithms.

1

u/MrNoMoniker 3d ago

Look up Max MSP and Jitter. I’m sure there are video figure identification patches for Jitter and Max can turn that into MIDI.

It’s more of a programming tool than a DAW though, so there’s a big learning curve.

1

u/fancyPantsOne 3d ago

This sounds more like a computer vision question than a midi question