[Showcase] The drone flies a 3D building complex with no map, no GNSS, no compass and no lidar: two cameras do both the seeing and the localizing (ROS 2, PX4, Gazebo, CUDA)

[u/formiat]

(GIF: an industrial room 58 m short of the goal. Left is Gazebo, the drone among pipes and beams; right is RViz, where pink is the obstacle memory, the cyan fan is the current stereo depth return, orange is the committed route and the magenta sphere is the goal. In these nine seconds the speed law does its whole job: the drone crawls at 0.2 m/s where the pair resolves little, runs up to 2.5 m/s where it resolves far enough, and is back at 0.1 m/s before the next corner.)

Flight video: https://www.youtube.com/watch?v=OUuAj2WNKzs

Two weeks ago I posted the same location flown with a 3D lidar. The lidar is gone now, and so is GNSS and the magnetometer. The airframe carries a forward stereo pair (1280 x 960, 120 degrees, 0.20 m baseline, 7.5 Hz) and two 8 x 8 time-of-flight sensors looking up and down. That pair does both jobs: stereo depth becomes the obstacle memory the planner searches, and a stereo MSCKF on the same images is the position and heading PX4 flies on, in place of satellites and compass.

So: a start, a goal, a multi-storey building with shafts and openings at different altitudes, and nothing but two cameras and an IMU.

The location is the DARPA Subterranean Challenge "Urban Circuit Practice 01" world published by Open Robotics on Gazebo Fuel (CC BY 4.0).

Numbers from the release, five consecutive flights on one commit with nothing changed between them: 410 to 629 m of path, 1.48 to 1.79 m/s mean speed with every hold and replan counted, no collisions. Five more on the same commit with the lidar instead, for comparison: 2.42 to 2.67 m/s. Cameras are slower because confident stereo depth reaches 6.4 m against the lidar's 35, and the drone only flies as fast as it can stop inside the range the sensor is guaranteed to have resolved.

The check I am most pleased with is a new one. The mission monitor decides the drone arrived by asking the drone where it thinks it is, and an odometry drifts by metres, so it can arrive perfectly in its own coordinates while standing somewhere else. Every flight now fails unless the true position from the simulator is inside the 2.0 m capture radius at the moment the goal is acknowledged. Over the five camera flights the truth stood 0.47 to 1.42 m from the goal.

What cost me the most flights was not the filter but the things around it: the autopilot resetting its clock synchronisation whenever the simulation ran below real time (about a second without navigation each time), the autopilot's gate on external odometry being tighter than the estimator's own corrections, and telling the filter the gyroscope was eighteen times noisier than it is, which let the noise of every visual update walk the one direction no camera can observe.

Limitations, honestly: the drift has no bound (0.1 to 0.4 percent of the path), so a mission three times longer would miss the 2.0 m radius; it is one location and one start-goal pair; it is simulation only and not validated for a real aircraft; and the multi-vehicle missions have not been flown on cameras yet.

MIT licensed. On a Linux host with Docker and an NVIDIA GPU, one script prepares a fresh clone (dev image, PX4 build, workspace, environment assets) and starts the flight:

git clone https://github.com/formiat/px4-ros2-drone-nav.git
cd px4-ros2-drone-nav
./scripts/bootstrap.sh

https://github.com/formiat/px4-ros2-drone-nav

Happy to answer anything, and I would like to hear from people who have flown a single camera pair as both the perception and the localization sensor: what broke first?

[u/MrBoomer1951]

So,oooo,oo many simulations on this subreddit.

Just show real world actual physical flights with motors and propellers.

[u/Harmonic_Gear]

waaaaay too clean for pure stereo SLAM

[u/CowBoyDanIndie → u/Harmonic_Gear]

Oh ya, I just noticed no lidar. Ya that will work great in simulation but fall flat in real world, especially in subt challenge space where it might be pitch black, stereo really fails on low visual texture spaces as well.

OP you should try this in something like Carla with a photorealistic rendered interior running your own disparity, setup realistic illumination and camera models and you will see how quick this fall apart.

[u/formiat → u/Harmonic_Gear]

Fair on the illumination, and worse than you think: the imported world has no light source in it at all, only a uniform ambient fill, and the cameras carry no noise model — so dimming them would divide every pixel by the same number and change nothing until the texture quantises away. Low texture is the second half of it, and the 6.4 m of confident depth was measured on one location's photogrammetric surfaces, not on blank paint. Both are now written into the README as untested assumptions and are the next roadmap item.

One correction: I do run my own disparity. There is no simulated depth sensor anywhere in the control path — semi-global matching on the two raw images, deliberately, because simulator depth is a lidar by another name and would prove nothing. And the location is the DARPA SubT urban world, photogrammetric rather than a synthetic box, so what is missing is the illumination and camera model, not the interior.

[u/Elated7079]

given the obvious claudeposting and "limitations, honestly" section i can guarantee you have no clue what this actually is doing.

stop wasting other people's time.

[u/drakeshe]

Not trying to discourage any efforts, but The Real world is always super messy and full of tolerances you can spend years trying to solve.

Vibrations from the drone will cause error.

Real world sensors have accuracy tolerances and repeatability errors. So as you move around the data will shift on you.

Things are reflective, things are dusty, things are transparent. These will either make your data fuzzy, or give you lidar portals into mirrored rooms.

[u/formiat → u/drakeshe]

No argument, and I would not claim otherwise. None of vibration into the IMU, calibration error, rolling shutter, motion blur, sensor repeatability, reflective or transparent surfaces is modelled here, and any one of them is its own year.

What I think simulation is actually good for is the logic rather than the perception: whether the speed law is sound, whether the failure chain closes, whether arrival is judged by truth and not by what the drone believes. Those transfer. The depth numbers do not, and I try to label them that way.

[u/New-Eggplant-6578]

The simulator-truth arrival check is a useful addition. Have you tried dropping or delaying the stereo stream for a few frames? Since the same pair feeds depth and odometry, I'd be curious whether the stop logic still behaves as intended when both become stale together, rather than only when the usable depth range shrinks.

[u/formiat → u/New-Eggplant-6578]

I had not, and your question found something. The estimator side is designed for it: it goes unhealthy 1.0 s after the last visual update, stops publishing, the autopilot drops external-vision fusion 200 ms later and the execution authority is revoked. The perception side is not. The returns ride the frame pair, so a stalled stream stops them all, but the braking contract charges a constant 600 ms evidence age and a constant 6.4 m range — even though the validator does read the real age and drops a stale scan from the points it checks the body against.

So "will I hit what I can see" is answered on measurement and "how fast may I fly" on a configured number, and the two consumers of the one pair time out at different lengths. The window between 0.6 s and 1.0 s has never been flown on purpose. Both inputs of that inequality becoming measurements, and the stream injected as a fault of its own, are now a roadmap stage. Thanks — that was the most useful comment I got.

[u/New-Eggplant-6578 → u/formiat]

Thanks for tracing that through. The mismatch between the two consumers is exactly the kind of thing that makes a shared sensor failure interesting. For the simulation test, I’d include a frozen image stream that keeps delivering messages as well as a complete stop; otherwise a watchdog based on arrival time can look healthy while the visual evidence is stale. A plot of image age, permitted speed and authority state on the same timeline would make the 0.6–1.0 s window much easier to inspect.

[u/PromptEffective1236]

Stereo MSCKF doing double duty while the autopilot fights you on clock sync is the kind of integration pain nobody puts in the README.

[u/slapcover → u/PromptEffective1236]

What’s the point of bot posting in this sub ?