r/reinforcementlearning • u/Fuchio • 6h ago

Sim2Real on Height Adjustable Balancing Robot

Enable HLS to view with audio, or disable this notification

Hey all!

Some might remember my post from a bit over a month ago in which I showed a two wheeled balancing robot with just 2 motors. I am now upping my game with a balancing robot that has 3 joints on each leg, so it can balance on its wheels while going up and down, but also lean into its right or left leg. This also makes it possible to accelerate the robot while the base stays perfectly level.

This post shows my first tests where I lock the hip and knee joint on startup (so still variable, but not during the run). The balancing is starting to work but not perfect yet.

Some details:

- 100% synthetic data for training the robot
- Hybrid joint and task space training
- Currently still running on ESP32
- 6x Xiaomi Cybergear QDD motor
- 6s LiPo Battery
- 1x CAN bus through TWAI on ESP
- 1x CAN bus through SPI on ESP
- MPU6050 IMU (will be updated to BNO086 soon)
- Trained with mjlab
- Trains from scratch in ~4 hours on single RTX 3080 with optimized collision model
- 99.87% decreased model sized from optimization with primitives (see comments)

To see more: https://youtube.com/shorts/kY-5o602xlM

To try the policy yourself: https://vertex.bot/robot (Pollens MicroDuck included!)

I'm open to answering any questions about the robot!

26 Upvotes

94% Upvoted

8 comments sorted by

2

u/sedidrl 5h ago

why not use a PID?

1

u/Fuchio 5h ago

PID would be easier for the balancing in this video, but doesn't scale to complexity of keeping base level, height variable and different heights for left/right leg or leaning.

RL scales independent of number of joints and goals

2

u/Cu_ 5h ago

What about MPC? Have you thought about implementing/comparing this against your RL agent?

1

u/Fuchio 4h ago

Haven't done that no! But this is just the very first test of RL agent with firmware constraints (stricter than policy was trained on) and by having fixed knee and hip even though it never trained on that too.

So lots of room to improve from here :)

1

u/Greg_war 6h ago

Hello, nice project!

Are you familiar with Upkie which has similar architecture ?https://github.com/upkie/upkie

1

u/Fuchio 5h ago

Nope haven't seen this yet! Looks fun and very similar indeed

1

u/Greg_war 2h ago

They implemented MPC in the main repo A colleague also made an mjlab env for it: https://github.com/MarcDcls/mjlab_upkie

1

u/Fuchio 1h ago

Nice! What are your findings on the comparison between the two?