Imitation Learning

Specifying a reward function for complex tasks is hard. Collecting millions of training examples with labels is expensive. There is a third path: show the robot what to do. Imitation learning (IL) is the family of techniques that extract a policy from observations of an expert performing a task — usually a human demonstrating through teleoperation or kinesthetic teaching. IL has enabled robots to fold laundry, perform surgical suturing, and cook meals from a handful of demonstrations. This lesson develops the main IL approaches, the mathematical reasons they can fail, and the modern methods that address those failures.

Behavioral Cloning: The Simplest Approach

The simplest formulation of imitation learning is behavioral cloning (BC). Treat the demonstration data as a supervised learning dataset: the inputs are states observed by the expert (camera images, joint positions), and the outputs are the actions the expert took in those states. Train a neural network policy pi_theta to minimize the prediction error on this dataset. Formal setup: let D = {(s1, a1), (s2, a2), ..., (sN, aN)} be the demonstration dataset where (si, ai) is a state-action pair from an expert trajectory. Behavioral cloning minimizes: L(theta) = (1/N) * sum_i L(pi_theta(si), ai) where L is a loss function appropriate for the action space (mean squared error for continuous joint commands, cross-entropy for discrete action choices). This is exactly supervised learning applied to a sequential decision problem. And it works remarkably well for many robot tasks, especially when the task is relatively short, the state space is well-covered by demonstrations, and the policy does not need to recover from unusual states. The BC approach has been implemented at large scale: the Open X-Embodiment dataset (Google, 2023) aggregated over 1 million robot demonstrations across 22 robot platforms, training a single generalist policy called RT-X. By training on enormous diverse datasets of demonstrations, such policies generalize across robots, objects, and tasks in ways that narrow task-specific training cannot.

DAgger: Learning from Corrective Feedback

The distributional shift problem motivates Dataset Aggregation (DAgger), a practical and theoretically grounded IL algorithm introduced by Ross, Gordon, and Bagnell in 2011. DAgger addresses compounding error by having the expert provide corrections on states the learned policy actually visits — not just states the expert would have visited. The algorithm proceeds iteratively: Iteration 1: Collect a set of demonstrations D1 from the expert. Train initial policy pi_1. Iteration 2: Run pi_1 in the environment. Record all states visited. Ask the expert to label the correct action for each visited state. Aggregate: D2 = D1 + {(s, a_expert) for all visited states}. Retrain on D2. Iteration k: Run pi_{k-1}, collect corrective labels from the expert on visited states, aggregate into growing dataset, retrain. The key insight is that the dataset now covers the distribution of states induced by the learned policy, not just the expert. Theoretically, DAgger reduces the compounding error from quadratic in task length (for vanilla BC) to linear — a significant improvement. The practical limitation of DAgger is that it requires an interactive expert during data collection: the expert must label states that arise from the robot's own behavior, which may look strange or dangerous. For tasks where the expert is a human teleoperator, this is feasible. For tasks where collecting additional corrections is expensive or risky, variants of DAgger using safety constraints or simulated experts are used.

Inverse Reinforcement Learning

Behavioral cloning treats the problem as pure supervised learning: copy the expert's actions. Inverse Reinforcement Learning (IRL) asks a deeper question: what reward function would explain the expert's behavior? The insight behind IRL: an expert demonstrating a complex task is implicitly optimizing something. If we can recover that reward function, we can use standard RL to train a policy that optimizes the same objective — with the benefit that the recovered reward function may generalize better than copied actions, and we can then optimize it in situations the expert never demonstrated. Formal setup: given a set of expert trajectories, IRL solves the inverse problem of finding R(s, a) such that the expert's behavior is near-optimal under R. This is an ill-posed problem (many reward functions can explain the same behavior), so IRL algorithms use various constraints or priors to regularize the solution. Maximum Entropy IRL (Ziebart et al., 2008) is the most influential formulation: find the reward function under which the expert trajectories have maximum entropy (are as random as possible) while still being optimal. This resolves the ambiguity by selecting the most uncertain explanation consistent with the observed behavior. Generative Adversarial Imitation Learning (GAIL, Ho and Ermon, 2016) combines IL with generative adversarial training: a discriminator learns to distinguish expert state-action pairs from policy-generated ones, and the policy is trained to fool the discriminator. This circumvents explicit reward function estimation and has been applied to humanoid locomotion and dexterous manipulation. In practice, GAIL and IRL-based methods tend to outperform behavioral cloning when demonstrations are limited, because they encode the intent behind behavior rather than its surface form.