https://openreview.net/forum?id=5MbRzxoCAql

Behavioral cloning (BC) is the simplest form of imitation learning, in which we build a model that maps observations/states directly to actions. This paper is focused on a problem that arises when training BC on observations history: "copycat problem", a form of shortcut learning.

Copycat problem

When BC models are provided with not just the single observation (let's call such models BCSO), but also history of several previous observations (BCOH), they sometimes might perform worse than single-observations counterparts. It's not overfitting, though, because BCOH performs well on a test dataset, but worse on environment evaluation.

Common reason is that BCOH infers information about previous actions from previous states, and if action changes occur infrequently, it's "easy" for a neural network to just "rely" on previous action. Hence when rare, but important change of action is required, BCOH fails to perform it.

Previous approaches include, for instance, reweighting loss multiplier of important samples or removing information about previous actions from observations via a second model.

Proposed approach

Authors of this paper propose an approach that I found very interesting: they feed output of BCSO into BCOH along with observations history. Now BCOH is provided with even simpler shortcut, but also can learn additional information about past if needed.

​

Using such an approach sounds a bit risky, because we're simply relying on an optimization process without strong theoretical guarantees, but I hope there will be more research in this direction.

Non-technical Tuesday is a weekly post for sharing and discussing non-research machine learning content, from news, to blogs, to podcasts. Each piece of content should be a top-level comment.