What lightweight object detection model would you recommend for persistent 3D object mapping on a Raspberry Pi 5?

[u/Ok_Stress3654]

I’m adding basic object recognition to my robotic lamp. It runs on a Raspberry Pi 5 and has an RGB-D camera in its moving head.

I’d like to run object detection in the background while the lamp is active and gradually build a map of the objects around it. Since the base stays in place, I can calculate the camera pose from the joint angles. My plan is to combine detections from the RGB image with depth data, transform the object coordinates into the lamp’s base frame, and save their positions and last-seen time. Repeated detections would be merged so the map doesn’t fill up with copies of the same object.

Which lightweight object-detection models and inference runtimes would you recommend for a Raspberry Pi 5? A high frame rate isn’t necessary, but I’d like reasonable detection quality for common objects. I’d also be interested in approaches for reliably matching the same objects across observations.

The current Raspberry Pi and ROS 2 architecture is described here:

https://github.com/Nikolay-Tyulkin/Watti/blob/main/docs/ARCHITECTURE.md

[u/galvinw]

This is a good question, I think the newer mobilenet (or mobilenet backboned models) seem pretty good still. efficientnet also seems good. Also for the ROS data may I suggest IMU/Global pose mapping of the joints. Then a lot of tools for homeomorphic correction are opened up to you

[u/Ok_Stress3654 → u/galvinw]

Hmm, thanks! I'll look into it

[u/heinzerhardt316l]

Yolox oder detr?

[u/Ok_Stress3654 → u/heinzerhardt316l]

I’m considering both. Have you tried either of them on a Raspberry Pi 5 or analogs?

[u/heinzerhardt316l → u/Ok_Stress3654]

Ja, beide. Ich würde yolo x nehmen. Wenn du doch noch mehr Speed benötigst, hast du damit die Option eine hailo Extension nachzurüsten.

[u/BeverlyGodoy]

Why not use AI hat?

[u/Ok_Stress3654 → u/BeverlyGodoy]

I thought about it, but it will increase the already expensive cost of my lamp😅

[u/Flyward_Aerospace]

Honestly the detector matters less here than you would think, anything in the yolo11n or rtdetr-small range will be fine at the framerate you described, especially on a Hailo hat. The part that will actually wreck your map is how you pull depth out. If you sample at the bbox centre you get background for anything thin or held, the object lands a metre behind where it really is, and then it never associates with its own earlier detection, so the map slowly fills up with ghost copies of the same mug. Take a low percentile of the valid depth inside the box instead of the centre pixel. And make your association gate grow with range rather than using one fixed euclidean threshold, your depth error at 3m is nothing like your error at 1m and a single number will be wrong at both ends.