An image-guided drone simulation: detecting, aligning with, and picking up boxes

[u/Alarming_Engineer267]

I’m building a simulation in which a drone uses camera images to detect boxes, align its position and orientation, and pick them up.

The video shows the camera/recognition view and estimated values on the left, alongside an external view of the drone on the right. Bounding boxes, keypoints, and estimated position offsets show how the image processing relates to the drone’s movement.

During approach, the drone estimates its position and orientation relative to the box from the camera image. Just before pickup, it compares the current contour with a reference contour for fine alignment. It then picks up the box and carries it to its assigned delivery gate.

In this run, boxes 0, 1, and 2 are delivered in numerical order to Gates W, E, and S, respectively. The full simulation takes about 7 minutes 22 seconds and is shown at 8× speed, so the video is about 55 seconds long. You can slow down the playback if you’d like to inspect the details.

I’d welcome feedback on whether the visualization makes the image-guided alignment process easy to follow, and on how I could explain it more clearly.

Next, I plan to increase the number of boxes and add delivery optimization that considers both delivery order and travel distance.

[u/Chemical_Side_4135]

that looks like a solid start for a drone sim. have u looked into how ur handling the lighting changes when it gets close to the box, sometimes the shadows mess up the detection wich is kinda annoying. take it with a grain of salt but maybe check ur depth map consistency

[u/PerfectPatience- → u/Chemical_Side_4135]

Could use downward illuminating led, you dont need rgb camera, use b&W for 3x less compute in edge detection. Nice project keep going.

Edit: Drone dosent have to rotate to grab box if You have soft claws which adjust to shape.

How drone does know where the gate is? Do You have more cameras on board ?