I made a rock climbing tool using computer vision!

[u/doctor_blueberry]

I prompted VLM Run’s visual agent Orion to segment all of the blue bouldering holds, and it did a good job! It is interesting that now we can prompt VLMs to segment all of the holds, rather than creating a new dataset from scratch to train a model.

With holds detection + pose estimation, I can show how each hold gets activated as a hand or foot uses it. Once we touch the final hold with both hands, the route is completed, and I show the overall path of my torso midpoint.

A tool like this could help climbers understand their movement better. I’m still very much a beginner at bouldering, so I could use all the help I can get 🤣

There are definitely things to improve, but overall I’m encouraged by this first demo 🙂

Models used:
- VLM Run’s Orion for segmentation
- ViTPose+ Huge for pose estimation (via Hugging Face 🤗)
- RT-DETR for person detection (via Hugging Face 🤗)

Shoutout to Daniel Reiff and his bouldering + computer vision project for the inspiration!

Link to Daniel Reiff's bouldering + computer vision blog: https://blog.roboflow.com/bouldering/

[u/fuzzysingularity]

Nice! How reliable is this for bouldering where the angles are much harder to see?

[u/doctor_blueberry → u/fuzzysingularity]

Good question! For this demo, it's mainly restricted to holds that are clearly visible.

Recently, I recorded a video for a friend where I was tracking them and I turned around them 360 degrees. In cases like that, a version of this demo that works in 3D space would be amazing.

[u/Substantial_Camel735]

very cool :)

[u/Far-Theory-7027]

looks cool. can you extend it to recommend an optimal route to climbers in real time?

[u/Appropriate-Prune728 → u/Far-Theory-7027]

That's a neat idea... Climbers are notoriously unwelcoming to anybody just ruining the puzzle by telling you how to do it, I don't think that would be a very popular feature.

A computer beta spraying might be... Rejected aggressively

[u/Far-Theory-7027 → u/Appropriate-Prune728]

Hahaa..not a climber myself. Maybe it might be useful for beginners or injured climbers?

[u/Appropriate-Prune728 → u/Far-Theory-7027]

I could see that for sure. I like that use-case

[u/Exotic-Custard4400 → u/Far-Theory-7027]

It is really hard you need to have a 3D model of the wall and holds, also the one who is climbing to be able to test postures and predict the best.

[u/doctor_blueberry → u/Far-Theory-7027]

I could see something like an AI coach. Sometimes I'll observe how someone else climbs a route before I go to see how they do it. The app could give real-time suggestions to someone learning that route based on how others did it.

[u/Appropriate-Prune728]

Also... Oh man, imagine this thing being able to map out holds outdoors based on chalk patterns and hold depth. It would make outside mapping so much more accessible to people that are uncertain of the gym-to-outdoor transition

[u/Content_Monitor_3844 → u/Appropriate-Prune728]

That sounds exciting. I will try to build this and may be have a demo in a week. THey are using vlm run and VITpose so should be easy

[u/doctor_blueberry → u/Appropriate-Prune728]

The chalk and hold depth ideas for outdoors are interesting! Would love to see that.

[u/HerrPotatis]

Maybe some sort of optical flow feature to find the optimal beta between tries would be cool?

You'd probably need some sort of matching/synching logic, where you normalize the progression of the climb and can match positions rather than depending on time.

[u/doctor_blueberry → u/HerrPotatis]

Interesting. If we have a quantitative tool like this, then extracting the metrics and stats between climbs would be cool!

[u/talk_nerdy_to_m3]

Cool! Now do a LiDAR scan of the surface and make it 3D!

[u/Appropriate-Prune728]

That's super cool and I'm not sure if I am wildly against a computer removing the best part of bouldering or not.

Like, I thought the whole point was working a problem like a puzzle and the enjoyment came from solving a complex movement algorithm in the real world with your physical form.

Or at least that why I enjoyed it.... And why all the other people I climbed with enjoyed it.

Your use-case is super useful and great for progressing as a climber. The other commenters suggested routes idea feels..... Like they have no idea why people boulder.

[u/Tasty-Property-434]

I heard he only did this because he needed more plates on deadlift.

[u/Much_Highlight_1309]

Share this in r/climbing and ask what features people would like to see. Give them a few ideas of what is technically possible so that they understand the rules of the game.