Can curiosity drive a humanoid robot to learn how its body moves?
Rewarding a real humanoid robot for improving its own model of how its movements turn out made it explore more efficiently than random or least-tried exploration, and led it to discover and probe a table on its own.
Source
Curiosity driven reinforcement learning for motion planning on humanoids
Study at a glance
- Design
- Computational / modelling — Two online experiments on a real iCub robot: one curious learner for an arm compared with random and least-tried exploration; three parallel curious learners for both arms and torso at a table.
- N
- No sample size in the usual sense: Experiment 1 used an arm model with 1296 state-actions and ran for 4000 transitions; Experiment 2 ran three learners for 18,000 transitions.
- Population
- A single physical iCub humanoid robot (arm and torso joints)
- Outcome
- Exploration coverage of states and state-actions, uniformity of visits, decay of intrinsic reward, emergent behaviour
Structured fields used in claim comparison tables when every cited study has a complete layer.
What they did
The authors divided the iCub robot's joint space into a grid of states, where each action aims to move to a neighbouring state, and learned the probability of where each action actually ends up. The only reward was intrinsic: how much a new observation changed the learned transition model, measured with KL divergence. In one experiment a single learner controlled four arm joints and was compared with random and least-tried exploration; in a second, three independent curious learners controlled the torso and both arms while the robot sat at a table.
What they found
The curious agent explored best: it reached novel states more aggressively and spread its visits and action tries more evenly than the random and least-tried agents. It learned that the robot's cable-driven mechanics make motions unreliable in places, which the authors argue makes plan-then-act planners unsuitable. With three learners, the arms and torso ended up coordinating to lean forward and touch the table repeatedly, because actions affected by the other learners stayed surprising longest.
The limits
What it doesn't show
All evidence comes from one robot with coarse state grids (three positions per joint), and comparisons are shown mainly through figures and qualitative observation rather than statistical tests or repeated runs. The 'table exploration' is an emergent behaviour described by the authors, not measured against a baseline. Learning took many hours on hardware, and the authors note that finer state spaces would make it slower still; the planner was not evaluated on external goal-reaching tasks.
Key terms
- Artificial curiosity
- An intrinsic reward that drives an agent towards situations where it is learning fastest, rather than towards an external goal.
- Intrinsic reward
- A reward generated by the agent's own learning process, such as surprise or model improvement, rather than by the task.
- KL divergence
- A measure of how much one probability distribution differs from another; here, how much a new observation changes the predicted outcome distribution.
- Markov decision process (MDP)
- A model of states, actions, transition probabilities and rewards used to plan or learn a policy.
- Probabilistic roadmap (PRM)
- A classic motion planner that samples configurations and connects them into a graph, then plans on it before acting.
Flashcards
0 of 10 answers reviewed
Research intelligence for this paper
See its role on concept claims, tensions it is part of, placement history, and related discoveries.
Quiz yourself
What drove the robot's exploration in these experiments?
Common questions
Why not just explore randomly?
Random exploration repeats some actions and neglects others. The curious agent focused on actions whose outcomes were still being learned, covering the space more evenly.
Why did the robot start touching the table?
Whether an arm hits the table depends on the torso, which the arm learner cannot see, so those actions had variable outcomes and stayed interesting longer, attracting repeated exploration.
Why split the robot into three learners?
A single learner over all twelve joints would have an enormous state space, making real-robot experiments far too slow.
More on Reinforcement learning