Res-HIL: Human Corrections Speed Up Robot Manipulation Learning

A team led by researcher Mariia Iavorskaia has proposed Res-HIL, a human-guided residual reinforcement learning framework designed to make dexterous robot manipulation easier to teach, according to a paper posted on arXiv (cs.RO).

Most robot manipulation skills today are taught through imitation learning, where a robot watches human demonstrations and learns to copy them. The catch, as the paper points out, is that these policies often break down the moment a robot encounters something slightly different from its training data — and fixing that usually means collecting even more demonstrations, which takes considerable human time and effort.

An alternative approach, human-in-the-loop reinforcement learning, lets a person step in with corrective feedback while the robot trains online. But according to the paper, this method typically tries to learn an entire task policy from those interventions rather than simply polishing a policy the robot already has.

Res-HIL takes a middle path. It keeps a pretrained imitation policy frozen and instead learns a separate "residual" policy that layers small corrective actions on top of it. Whenever a human intervenes during training, that single moment generates two learning signals at once: direct supervision showing the residual policy exactly what correction to make, and reward shaping that retroactively adjusts how the robot's preceding autonomous behavior is scored. The framework also initializes the residual policy at zero, a design choice the researchers say helps stabilize and speed up the online learning process.

The team tested Res-HIL on five contact-rich manipulation tasks that combined high-precision movements with long-horizon sequences — the kind of tasks where small errors tend to compound over time. Starting from just 20 initial demonstrations, Res-HIL reportedly outperformed both a state-of-the-art full-policy human-in-the-loop reinforcement learning baseline and a residual fine-tuning approach that didn't use human guidance, on every one of the five tasks, after only ten minutes of online training. The paper also states that Res-HIL improved on the pretrained base policies it started from and beat imitation-only policies trained with five times as many demonstrations.

An ablation study included in the paper isolated the contribution of each design choice. Direct supervision of the residual policy turned out to be essential to overall performance, while the intervention-aware reward shaping component — though less critical on its own — substantially improved how efficiently the system learned.

The research adds to a growing body of work aimed at closing the gap between imitation learning's ease of use and reinforcement learning's ability to refine and correct behavior online. By focusing human effort on small, targeted corrections rather than full demonstrations or complete retraining, approaches like Res-HIL could lower the practical cost of deploying dexterous manipulation skills on real robots, particularly for contact-heavy tasks where precision matters. The full details of the tasks, robot hardware, and quantitative results are laid out in the arXiv preprint.