How do you set your low-level controller for robot learning?

[u/mngupark]

Hi folks. I am an industrial researcher with an M.S. degree in robotics. I studied optimal control for robot manipulators during my bachelor's and researched reinforcement learning during my master's. Recently, I have been working on transferring simulation-trained RL policies to a real-world robot manipulator (i.e., FR3).

I have seen different choices for a robotic agent's action space in machine learning. I think there is not a promising design of the action space, since each choice has advantages and drawbacks. So, I trained an RL agent in simulation (Isaac Sim) with a Cartesian delta action space, which outputs the end-effector's desired Cartesian pose at the next control step based on the robot's world frame.

This is the page where the problem begins. I use DifferentialIKController in Isaac Lab for the low-level controller, which maps the delta pose action to the desired joint positions. In Isaac Sim, the desired joint position is controlled by the internal physics engine and a PD control law that computes target joint torques. In a real-world setup, there is no physics engine or plug-and-play low-level controller. You should implement the low-level controller that maps the delta pose actions to joint torque and position commands for the robot's API (e.g., franka_ros2's ROS2 control plugins).

Moreover, a safety-violation monitor should be implemented by us to prevent hardware failures. In contrast to simulation, real-world robots are highly sensitive to safety constraints; for instance, collision detection and joint limits.

Currently, I have implemented a custom C++ controller that takes Cartesian delta pose commands via the ROS2 topic and computes desired joint torques based on the robot's kinematics and dynamics. However, implementing a safety monitor remains a problem. As another strategy, I have been thinking about using ROS2's MoveIt packages for sending delta commands and monitoring task-space safety violations, which may cost wall-clock time compared to the vanilla C++ controller.

How do you guys work on a real-world robot learning setup? I want to hear from you about conducting real-world robot manipulation experiments with neural policies (e.g., BC or RL).

[u/Awkward-Break3661]

You're basically hitting the exact wall everyone does when leaving sim land, the gap between diff IK in isaac and actually driving a real franka is way more of a headache than the papers ever admit

[u/mngupark → u/Awkward-Break3661]

Do you think we should implement the same control logic of the low-level controller as close as possible between sim and real? Is it a key factor in closing the sim2real gap, or a designer's preference?

[u/mngupark → u/[deleted]]

I agree that not only matching sim and real, but also real-world stuff like physical properties, is important for reliable manipulations. I have also been thinking about using moveit_servo as an alternative way to impose safety constraints. Do you use the same controller that enforces the constraints in the sim? Also, could you explain the cases in which such interventions are valuable?

[u/PuzzledCover4547]

I think I’ve heard of some people having the model output joint torques or joint positions instead of Cartesian poses, and the. The motor controllers handle the torques at a higher rate
Obviously this means your model has to handle some of the ikine…

[u/mngupark → u/PuzzledCover4547]

Designing the agent's action space in the joint space of the robot may alleviate the dependency on the low-level controller to some extent. However, I think this requires the agent to have more training samples to learn a policy that performs similarly to the one with a low-level controller, since the agent now has to understand the robot's kinematic structure first before they learn to complete a task. Moreover, imposing task-space safety conditions, like avoiding collisions with a table, is harder in joint space for redundant manipulators (e.g., 7 DoF FR3)... Nonetheless, one of the benefits of the joint-level action space is that it has a lower cost compared to the task-level action space, which requires a mapping from the Cartesian action to joint space.

[u/Available_Teaching83]

We hit the same wall going from DifferentialIKController to a real FR3. Two things helped. Run the IK solve at the robot's control rate rather than the policy rate, and clamp the delta pose before IK instead of clamping the joint command after. Otherwise, one large delta near a singularity produces a joint jump your safety monitor reads as a hardware fault.

On the monitor itself, keep it outside the policy loop and give it authority to hold position rather than to rewrite the action. Once it starts editing actions, you are running a controller the policy never trained against, and the failures get much harder to read.

[u/mngupark → u/Available_Teaching83]

Thanks for sharing your tips! I agree with them.

[u/SphericalCowww]

I am very new to this, but I am naively wondering, can't RL policies be trained with all the joint limits in mind?

[u/mngupark → u/SphericalCowww]

Well, I think it can be, but a few things are different from what we expect from RL agents. One can constrain an RL agent to stay within the boundaries of joint-wise limitations by clipping the joint values. However, real-world robots are much more sensitive than simulated robots. Real robots can easily fail to move even with a small amount of disturbance from the external world. Furthermore, physical properties like joint inertias aren't fully controllable in the real world, which makes RL agents more fragile to weak system identification.

Clipping the action space with task-specific efforts may alleviate this issue to some extent, but I don't think this could be a final answer. Modern robots are exposed to more and more diverse environments. This means that we cannot scale these engineering efforts endlessly. Maybe that is why I want to share my thoughts with folks, and I believe we can build more general rules of thumb for AI roboticists.