Scaling Up and Distilling Down

Language-Guided Robot Skill Acquisition

Huy Ha1, Pete Florence2, Shuran Song1,3
1Columbia University, 2Google DeepMind, 3Stanford University

www.cs.columbia.edu/~huy/scalingup/

Skill Learning

Skill Learning

a large set of reusable and robust skills

plan

novel scenarios and
new tasks
How can we scalably
acquire robot skills?

Behavior Cloning

BC-Z , Jang et al, CoRL 2021.

Diffusion Policy , Chi et al, RSS 2023.

✅ reliably produces robust robot skills

Behavior Cloning

✅ reliably produces robust robot skills

❌ reliance on human demonstration collection

Scalable
Skill
Learning

Reinforcement Learning

✅ automatic data collection and policy learning

Reinforcement Learning

✅ automatic data collection and policy learning

❌ exploration in sparse reward and long-horizon

Scalable
Skill
Learning

How can we get a
Unified Framework
for
Scalable Skill Learning?

Language-guided Robot Data Generation
&
Language-conditioned Robot Policy Learning

Scale up Language-labelled Robot Data Generation

Scale up Language-labelled Robot Data Generation

✅ rich manipulation skills

✅ flexibility to novel tasks & domains