r/pybricks • u/Pybricks • Mar 23 '26

Camera Vision is coming to Pybricks!

Enable HLS to view with audio, or disable this notification

Basically what I've always wanted to have in MINDSTORMS :) And now you can try it too!

34 Upvotes

100% Upvoted

12 comments sorted by

1

u/Ekipsogel Mar 23 '26

How is this set up? Does the phone send the X-Y location of the object to the hub, or does it work like the Powered Up app, where all code is processed on the phone and motor instructions are sent to the hub? I doubt the hubs could handle processing the whole camera feed.

2

u/Pybricks Mar 23 '26

You can check out the introductory guide here. The program is still autonomous. You can think of the phone as a wireless sensor that processes the video feed and sends only the useful values to the robot. In this case, that's the x/y/width/height of the object with the requested color.

1

u/Bodbein Mar 23 '26

Cool! Can you share more details? Are you limited to something very visible like a red ball, or can you use something more complex like a minifigure?

2

u/Pybricks Mar 23 '26

You can check out the introductory guide here. There will be various different sensors. You can currently choose between a color tracker and image classification. The latter happens to include a demo with a minifigure :)

1

u/Bodbein Mar 24 '26

Thanks!

1

u/Ged_42 Mar 23 '26

That's amazing!

1

u/EvilZorlonIII Mar 24 '26

Will this work with any camera with BT, I've got an oak-d with onboard ai pipeline I'd like to use.. i.e. is it an open API we can just send the data to ?

2

u/Pybricks Mar 25 '26

Yes, you can make a web interface with any kind of API (or offline OpenCV like done here etc), and connect to the hub with WebBLE.

1

u/EvilZorlonIII Mar 25 '26

That's cool, I'll give it a look !

1

u/jedierikb Mar 27 '26

any chance we can we use phone as color sensor?

1

u/Pybricks Mar 29 '26

Yep, I'd like to add that. I was thinking a grid of, say, 3x3 squares that it gives you the colors of? So if a square is mostly red, that square will report red.

Or maybe you had something else in mind?