Can a robot adapt one demonstration to new situations?
Mixing evolutionary exploration with policy-gradient learning, guided by a single demonstration, let a simulated robot arm open doors in positions it was never shown, where copying the demonstration failed.
Source
Generalize Robot Learning From Demonstration to Variant Scenarios With Evolutionary Policy Gradient
Study at a glance
- Design
- Computational / modelling — New RL algorithm evaluated in simulation: OpenAI Gym continuous-control tasks with dense and sparse rewards (vs ES and DPPO), and a Sawyer door-opening task with one demonstration and changed door configurations (vs behaviour cloning and BC + DPPO).
- N
- No participant or dataset N; Gym curves averaged over six repeated runs and door-task success rates over five repeated runs, each scored on 100 test episodes.
- Population
- Simulated robots (Reacher, Hopper, HalfCheetah, Swimmer; Sawyer arm)
- Outcome
- Average cumulative reward; door-opening success rate in new configurations
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors built Evolutionary Policy Gradient (EPG): it creates a population of policies by adding noise to network weights, keeps the best-scoring ones, improves each with PPO, and merges them by training a child policy to imitate an ensemble of the elites, with an extra term pulling it toward the demonstration. They first tested exploration without demonstrations on OpenAI Gym robots, including versions modified to give only sparse rewards, against evolution strategies (ES) and distributed PPO. They then gave a simulated Sawyer arm one hand-engineered door-opening demonstration and tested it on doors with changed position, angle and handle placement.
What they found
On dense-reward tasks EPG matched or beat the baselines, notably escaping a local optimum that trapped PPO in Swimmer. With sparse rewards all methods did worse, but EPG was the most sample-efficient. In the door task, plain behaviour cloning failed in the new configurations, while EPG reached high success rates and generalised better than behaviour cloning followed by DPPO. Results are reported as curves in figures, without numerical tables in the text.
The limits
What it doesn't show
All experiments are in simulation, so nothing is shown about real robots, wear or safety. Results are reported mainly as learning curves over five or six repeats, with no statistical tests or numeric success rates in the text, making effect sizes hard to judge. The method is compared to a small set of baselines and only one manipulation task, and the demonstration-weight schedule must be tuned to how different the new environment is.
Key terms
- Learning from demonstration
- Teaching a robot a task by showing it examples of the desired behaviour rather than programming it.
- Behaviour cloning
- Supervised learning that maps observed states directly to the demonstrator's actions.
- Policy gradient (PPO)
- Reinforcement learning that adjusts a policy's parameters in the direction that increases expected reward; PPO limits how far each update moves.
- Evolutionary algorithm
- Optimisation by mutating a population of candidate solutions and keeping the fittest.
- Sparse reward
- A setting where the agent gets reward only rarely, such as on task completion, making exploration hard.
Flashcards
0 of 9 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
Which baseline failed to adapt to new door configurations?
Common questions
Why not just clone the demonstration?
A cloned policy only works in states like those it saw; when the door moved, behaviour cloning failed because the robot never explored the new situation.
What does the evolutionary part add over PPO alone?
Perturbing whole parameter vectors explores more broadly than action noise, helping escape local optima and find rewards when they are sparse.
How are elite policies combined?
Instead of averaging weights, a classifier picks which elite should act in each state, and a new child policy is trained to imitate that combined behaviour.
More on Robot learning