CSC2626
Imitation Learning for Robotics
Florian Shkurti
Week 1: Behavioral Cloning vs. Imitation
New robotics faculty in CS
Jessica Burgner-Kahrs Animesh Garg Myself Igor Gilitschenski
Today’s agenda
• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
Administrivia
Administrivia
This is a graduate level course
Course website: http://www.cs.toronto.edu/~florian/courses/csc2626w21
Discussion forum + announcements: https://q.utoronto.ca (Quercus)
Request improvements anonymously: https://www.surveymonkey.com/r/LJJV5LY
Course-related emails should have CSC2626 in the subject
Prerequisites
Mandatory:
• Introductory machine learning (e.g. CSC411/ECE521 or equivalent)
• Basic linear algebra + multivariable calculus
• Intro to probability
• Programming skills in Python or C++ (enough to validate your ideas)
Recommended:
• Experience training neural networks or other function approximators
• Introductory concepts from reinforcement learning or control (e.g. value function/cost-to-go)
Prerequisites
Mandatory:
If you’re missing any of
• Introductory machine learning (e.g. CSC411/ECE521 or equivalent) these this is not the course
• Basic linear algebra + multivariable calculus for you.
• Intro to probability
You’re welcome to audit.
• Programming skills in Python or C++ (enough to validate your ideas)
Recommended: If you’re missing this we can
• Experience training neural networks or other function approximators organize tutorials to help you.
• Introductory concepts from reinforcement learning or control (e.g. value function/cost-to-go)
Grading
Two assignments: 50%
Course project: 50%
• Project proposal: 10%
• Midterm progress report: 5%
• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%
Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
Grading
Two assignments: 50% Individual submissions
Course project: 50%
• Project proposal: 10%
• Midterm progress report: 5%
• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%
Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
Grading
Two assignments: 50% Individual submissions
Course project: 50%
• Project proposal: 10%
• Midterm progress report: 5% Groups of 2-3
• Project presentation: 5%
• Final project report (6-8 pages) + code: 30%
Project guidelines
http://www.cs.toronto.edu/~florian/courses/csc2626w21/CSC2626_Project_Guidelines.pdf
Guiding principles for this course
Robots do not operate in a vacuum. They do not need to learn everything from scratch.
Guiding principles for this course
Robots do not operate in a vacuum. They do not need to learn everything from scratch.
Humans need to easily interact with robots and share our expertise with them.
Guiding principles for this course
Robots do not operate in a vacuum. They do not need to learn everything from scratch.
Humans need to easily interact with robots and share our expertise with them.
Robots need to learn from the behavior and experience of others, not just their own.
Main questions
How can robots incorporate others’
decisions into their own?
How can robots easily understand our
objectives from demonstrations?
How do we balance autonomous
control and human control in the
same system?
Main questions
Learning from demonstrations
How can robots incorporate others’ Apprenticeship learning
decisions into their own? Imitation learning
Reward/cost learning
How can robots easily understand our Task specification
objectives from demonstrations? Inverse reinforcement learning
Inverse optimal control
Inverse optimization
How do we balance autonomous
control and human control in the Shared or sliding autonomy
same system?
Applications
Any control problem where:
- writing down a dense cost function is difficult
- there is a hierarchy of decision-making processes
- our engineered solutions might not cover all cases
- unrestricted exploration during learning is slow or dangerous
https://www.youtube.com/watch?v=M8r0gmQXm1Y
Applications
Any control problem where:
- writing down a dense cost function is difficult
- there is a hierarchy of interacting decision-making processes
- our engineered solutions might not cover all cases
- unrestricted exploration during learning is slow or dangerous
https://www.youtube.com/watch?v=Q3LXJGha7Ws
Applications
Any control problem where:
- writing down a dense cost function is difficult
- there is a hierarchy of interacting decision-making processes
- our engineered solutions might not cover all cases
- unrestricted exploration during learning is slow or dangerous
https://www.youtube.com/watch?v=RjGe0GiiFzw
Applications
Any control problem where:
- writing down a dense cost function is difficult
- there is a hierarchy of interacting decision-making processes
- our engineered solutions might not cover all cases
- unrestricted exploration during learning is slow or dangerous Robot explorer
Applications
Any control problem where:
- writing down a dense cost function is difficult
- there is a hierarchy of interacting decision-making processes
- our engineered solutions might not cover all cases
- unrestricted exploration during learning is slow or dangerous
https://www.youtube.com/watch?v=0XdC1HUp-rU
Back to the future
https://www.youtube.com/watch?v=2KMAAmkz9go https://www.youtube.com/watch?v=ilP4aPDTBPE
Navlab 1 (1986-1989) Navlab 2 + ALVINN (Dean Pomerleau’s PhD thesis, 1989-1993)
30 x 32 pixels, 3 layer network, outputs steering command
~5 minutes of training per road type
ALVINN: architecture
https://drive.google.com/file/d/0Bz9namoRlUKMa0pJYzRGSFVwbm8/view
Dean Pomerleau’s PhD thesis
ALVINN: training set
Online updates via
backpropagation
Problems Identified by Pomerlau
Test distribution is different Catastrophic forgetting
from training distribution
(covariate shift)
(Partially) Addressing Covariate Shift
(Partially) Addressing Catastrophic Forgetting
1. Maintains a buffer of old (image, action) pairs
2. Experiments with different techniques to ensure diversity and avoid outliers
Behavioral Cloning = Supervised Learning
25 years later
https://www.youtube.com/watch?v=qhUvQiKec2U
How much has changed?
offline
End to End Learning for Self-Driving Cars, Bojarski et al, 2016
How much has changed?
“Our collected data is labeled with road type, weather condition, and the driver’s
activity (staying in a lane, switching lanes, turning, and so forth).”
End to End Learning for Self-Driving Cars, Bojarski et al, 2016
How much has changed?
How much has changed?
A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots, Giusti et al., 2016
https://www.youtube.com/watch?v=umRdt3zGgpU
How much has changed?
Not a lot for learning lane following with neural networks.
But, there are a few other beautiful ideas that do not involve end-to-end learning.
Visual Teach & Repeat
Human Operator or
Planning Algorithm
Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
Visual Teach & Repeat
Key Idea #1: Manifold Map
Build local maps relative to the
path. No global coordinate frame.
Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
Visual Teach & Repeat
Key Idea #1: Manifold Map Key Idea #2: Visual Odometry
Build local maps relative to the Given two consecutive images,
path. No global coordinate frame. how much has the camera
moved? Relative motion.
Visual Path Following on a Manifold in Unstructured Three-Dimensional Terrain, Furgale & Barfoot, 2010
Visual Teach & Repeat
https://www.youtube.com/watch?v=_ZdBfU4xJnQ https://www.youtube.com/watch?v=9dN0wwXDuqo
Centimeter-level precision in tracking the demonstrated path over kilometers-long trails.
Today’s agenda
• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
Back to Pomerleau
(Ross & Bagnell, 2010): How are we sure these errors are not due to
overfitting or underfitting?
1. Maybe the network was too small (underfitting)
2. Maybe the dataset was too small and the network overfit it
Steering commands
where s are image features
Test distribution is different
from training distribution
(covariate shift)
Back to Pomerleau
(Ross & Bagnell, 2010): How are we sure these errors are not due to
overfitting or underfitting?
1. Maybe the network was too small (underfitting)
2. Maybe the dataset was too small and the network overfit it
Steering commands
where s are image features
Test distribution is different
It was not 1: they showed that even a linear policy can work well.
from training distribution
It was not 2: their error on held-out data was close to training error.
(covariate shift)
Imitation learning Supervised learning
(Ross & Bagnell, 2010): IL is a sequential decision-making problem.
• Your actions affect future observations/data.
• This is not the case in supervised learning
Supervised Learning
Assumes train/test data are i.i.d.
If expected training error is
Expected test error after T decisions
Test distribution is different
from training distribution
(covariate shift) Errors are independent
Imitation learning Supervised learning
(Ross & Bagnell, 2010): IL is a sequential decision-making problem.
• Your actions affect future observations/data.
• This is not the case in supervised learning
Imitation Learning Supervised Learning
Train/test data are not i.i.d. Assumes train/test data are i.i.d.
If expected training error is If expected training error is
Expected test error after T decisions Expected test error after T decisions
is up to
Test distribution is different
from training distribution
(covariate shift) Errors compound Errors are independent
DAgger
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation
• Imitation learning as interactive supervision
• Aggregate training data from expert with test data from execution
DAgger
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation
• Imitation learning as interactive supervision
• Aggregate training data from expert with test data from execution
Imitation Learning via DAgger Supervised Learning
Train/test data are not i.i.d. Assumes train/test data are i.i.d.
If expected training error on aggr. dataset is If expected training error is
Expected test error after T decisions is Expected test error after T decisions
Errors do not compound Errors are independent
DAgger
Initial expert trajectories Supervised learning DAgger
https://www.youtube.com/watch?v=V00npNnWzSU
DAgger
DAgger
Q: Any drawbacks of using it in a robotics setting?
DAgger
https://www.youtube.com/watch?v=hNsP6-K3Hn4
Learning Monocular Reactive UAV Control in Cluttered Natural Environments, Ross et al, 2013
Today’s agenda
• Administrivia
• Topics covered by the course
• Behavioral cloning
• Imitation learning
• Quiz about background and interests
• (Time permitting) Query the expert only when policy is uncertain
DAgger: Assumptions for theoretical guarantees
(Ross & Gordon & Bagnell, 2011): DAgger, or Dataset Aggregation
• Imitation learning as interactive supervision
• Aggregate training data from expert with test data from execution
Imitation Learning via DAgger Supervised Learning
Train/test data are not i.i.d. Assumes train/test data are i.i.d.
If expected training error on aggr. dataset is If expected training error is
Expected test error after T decisions is Expected test error after T decisions
Strongly convex loss
No-regret online learner
Errors do not compound Errors are independent
Appendix: No-Regret Online Learners
Intuition: No matter what the distribution of input data, your online policy/classifier will do
asymptotically as well as the best-in-hindsight policy/classifier.
Policy has access to Policy has access to
data up to round i data up to round N
No-regret:
Appendix: Types of Uncertainty &
Query-Efficient Imitation
Let’s revisit the two main ideas from query-efficient imitation:
1. DropoutDAgger:
Keep an ensemble of learner policies, and only query the expert when they significantly disagree
2. SHIV, SafeDagger, MMD-IL:
(Roughly) Query expert only if input is too close to the decision boundary of the learner’s policy
Need to review a few concepts about different types of uncertainty.
Biased Coin
observations
Biased Coin
how biased is the coin?
Biased Coin
how biased is the coin?
Induces uncertainty in the model, or epistemic uncertainty,
which asymptotically goes to 0 with infinite observations
Biased Coin
Q: Even if you eventually discover the true model, can you predict if the next flip will be heads?
Biased Coin
Q: Even if you eventually discover the true model, can you predict if the next flip will be heads?
A: No, there is irreducible uncertainty / observation noise in the system. This is called aleatoric uncertainty.
Gaussian Process Regression
http://pyro.ai/examples/gp.html
Gaussian Process Regression
http://pyro.ai/examples/gp.html
Zero mean prior over functions
Noisy observations
Gaussian Process Regression
No matter how much data we get, this
observation noise will not go to zero
http://pyro.ai/examples/gp.html
Zero mean prior over functions
Noisy observations
Gaussian Process Regression
If we get data here we can reduce
model / epistemic uncertainty
http://pyro.ai/examples/gp.html
Zero mean prior over functions
Noisy observations
Gaussian Process Classification
Gaussian Processes for Machine Learning, chapter 2
Gaussian Process Classification vs SVM
Gaussian Processes for Machine Learning, chapter 2
GP handles uncertainty in f by averaging
while SVM considers only best f for classification.
Model Uncertainty in Neural Networks
Want
But easier to control network weights
Model Uncertainty in Neural Networks
Want
But easier to control network weights
How do we represent posterior over network weights?
How do we quickly sample from it?
Model Uncertainty in Neural Networks
Want
But easier to control network weights
How do we represent posterior over network weights?
How do we quickly sample from it?
Main ideas:
1. Use an ensemble of networks trained on different
copies of D (bootstrap method)
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)
3. Use MCMC to sample weights
Model Uncertainty in Neural Networks
Want
But easier to control network weights
How do we represent posterior over network weights?
How do we quickly sample from it?
Main ideas:
1. Use an ensemble of networks trained on different
copies of D (bootstrap method)
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)
3. Use MCMC to sample weights
Model Uncertainty in Neural Networks
Want
But easier to control network weights
How do we represent posterior over network weights?
How do we quickly sample from it?
Main ideas:
1. Use an ensemble of networks trained on different
copies of D (bootstrap method)
Variational inference
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)
3. Use MCMC to sample weights
Model Uncertainty in Neural Networks
Want
But easier to control network weights
How do we represent posterior over network weights?
How do we quickly sample from it?
Main ideas:
1. Use an ensemble of networks trained on different
copies of D (bootstrap method)
Variational inference
2. Use an approximate distribution over weights
(Dropout, Bayes by Backprop, …)
3. Use MCMC to sample weights
