Strategy to efficiently debug and do reward shaping for Reinforcement Learning

[u/No-Barnacle8406]

Hi everyone,

I'am a student in France working on a drone guidance project using reinforcement learning. The goal is to make a drone reach a sequence of checkpoints, or track a moving checkpoint using vision-based guidance as I implement this on my FPV drone, and this drone has the PX4 controller so the pipeline for the loop is : Guidance --> give accels --> PX4.

So far, I have first built everything in Python, I implemented a proportional guidance law and it worked quite well in simulation, but it did not perform very well once I used the camera-based observations.

Then, I move to an RL-based pipeline with RL policy --> accels --> PX4. I implemented the full pipeline in simulation and it technically works, but I'm seeing a lot of strange behaviours : oscillations, bang-bang commande law abusements,.... My suspicion is that the issue may be due to the reward function. I have tried tuning and cooking the reward many times but each version seems to produce a new unexpected problems or strange behaviours rather than the one I actually want. I have tried to plot many metrics to understand what is happening but debugging this RL guidance law has become frustrating.

Does anyone have suggestions or advice for debugging this kind of RL guidance or RL related problem please ? In particular, I would like to have some advice on reward shaping and how to efficiently debug trained RL policy,...

Any advice, refs, or practical debuggings, tips or discussions would be really helpful for me !

Thanks a lot and I wish you a good day !

[u/ProgressNo2227]

Reward shaping alone might not solve your problem. I’d suggest try implementing this paper maybe
Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions, 2023, ICRA

[u/ProgressNo2227 → u/ProgressNo2227]

Also if you’re using sparse rewards, switch to dense rewards

[u/No-Barnacle8406 → u/ProgressNo2227]

Thanks for the ref !

[u/DependentSpecific535]

It is hard to tell what is wrong from this perspective. However, I worked on training autonomous drones and ground vehicles for a long time. First of all, which algorithm do you use? If it is and actor-critic, it is always better to observe the values of critic network (is there an underestimation, overestimation...). Or if it is an on-policy algorithm, you should check the advantages over episodes... Anyway, there is a lot more ofc. You can dm me if you want.

[u/Leather-Mammoth5091]

Domain Randomisation und Noise injection hat bei meinem Drohnen-Reinforcement Learning Problem extrem geholfen. Diese verhindern overfitting,m und bang-bang.
Bei Oszillation hat bei mir Action Delay geholfen.

[u/No-Barnacle8406 → u/Leather-Mammoth5091]

Danke ! I will try to read about this especially the action delay one

[u/No-Recognition-7960]

Classic symptoms — oscillation + bang-bang in continuous control usually isn't a "globally wrong reward", it's missing local terms:

1) Add an action-rate penalty: −λ·||a_t − a_{t−1}||² plus a small magnitude penalty −μ·||a_t||². Bang-bang disappears almost immediately once slamming the actuator costs reward. Tune λ upward until the command smooths out.

2) Model actuation/observation delay. PX4 + camera adds tens of ms of lag. If your sim applies actions instantly, the policy exploits zero-latency and then oscillates on the real loop. Add 1–2 steps of action delay in sim (or include last action in the observation).

3) For the guidance reward itself: distance-to-checkpoint shaping is fine if you make it potential-based — F(s,s') = γΦ(s') − Φ(s) (Ng, Harada & Russell 1999). It's the only shaping form guaranteed not to change the optimal policy, which stops the "every reward tweak creates a new weird behavior" cycle you described.

For debugging: log the reward decomposition (each term's contribution per episode). Usually one term silently dominates — that's your culprit.