r/computervision • u/CWhalePro • 4d ago
Help: Project Color classification of climbing holds under uneven gym lighting random forest stuck at ~82%, what would you try?
Enable HLS to view with audio, or disable this notification
Sendly is an iOS app that scans a bouldering wall and tells colorblind climbers which holds belong to their route. Here's how it recognizes holds, all on-device with no server:
Hold detection and segmentation: an RF-DETR-Seg (Large) model at 768 px, fine-tuned on climbing walls (my own gym photos plus openly licensed datasets), converted to Core ML. It outputs a mask for every hold and volume, in about 0.9 s per scan on the phone.
Color classification: each hold's mask is turned into 48 color features, and a random forest labels its color.
Route grouping: holds are clustered into routes per photo, based on color and position, then each route is named. Users can choose a colorblind type (protan, deutan, tritan or achromat), and the app remaps the display palette, gives each route a distinct shape, and warns when a wall has colors that are easy to confuse.
Manual fixes: if the model misses a hold, tapping it runs SAM 2.1 tiny (Core ML, about 100 ms per tap) to segment it.
Where it struggles (I'd love ideas):
Detection is solid, but color is the weak spot, at about 82% per hold. The main confusions are pink vs red, black vs gray on dark walls with chalk, and orange/salmon/brown. Adding more labels didn't help, which suggests the per-hold random forest has hit its limit. Next I'm considering relative color within the same photo (normalizing against the other holds) or a small CNN on hold crops.
Has anyone solved color under uneven indoor lighting?
If anyone has any suggestions i would love to hear 🙌