STAT121 / AC209 / E-109
CS109 Data Science
Hanspeter Pfister
pfister@seas.harvard.edu
Joe Blitzstein
blitzstein@stat.harvard.edu
Verena Kaynig
vkaynig@seas.harvard.edu
Outline
What?
Why?
Who?
How?
Outline
What?
Why?
Who?
How?
Data Science
To gain insights into data through
computation, statistics, and visualization
A Data Scientist Is...
A data scientist is someone who knows more
statistics than a computer scientist and more
computer science than a statistician.
- Josh Blumenstock
Data Scientist = statistician + programmer +
coach + storyteller + artist
- Shlomo Aragmon
Nate Silver
Nate Silver won the election
Harvard Business Review
#natesilverfacts
http://techcrunch.com/2012/11/07/nate-silver-as-software/
Nate Silver on Pundits
Silver: Pundits are no
better than a coin toss.
Stewart: Do you foresee a
coin getting its own show?
The coin toss show?
http://www.thedailyshow.com/watch/wed-october-17-2012/nate-silver
Some Key Principles
use many data sources (the plural of anecdote is not data)
understand how the data were collected (sampling is essential)
weight the data thoughtfully (not all polls are equally good)
use statistical models (not just hacking around in Excel)
understand correlations (e.g., states that trend similarly)
think like a Bayesian, check like a frequentist (reconciliation)
have good communicationskills (What does a 60%
probability even mean? How can we visualize, validate, and
understand the conclusions?)
Netflix Prize
Netflix Prize Progress
HBR, Oct 2012
3 Years Later
We evaluated some of the new
methods offline but the additional
accuracy gains that we measured did
not seem to justify the engineering
effort needed to bring them into a
production environment.
Xavier Amatriain and Justin Basilico, 2012
Some Challenges
massive data (500k users, 20k movies, 100m ratings)
missing data (99% of data missing; not missing at
random)
extremely complicated set of factors that affect peoples
ratings of movies (actors, directors, genre, ...)
need to avoid overfitting (test data vs. training data)
curse of dimensionality (very high-dimensional
problem)
Kaggle
The Connectome
How is the mammalian brain wired?
~60 um3
600 GB
Courtesy of
Bobby Kasthuri.
Harvard
The Data Challenge
Pixel resolution: 3-5 nm; Slice thickness: 30-50 nm
1 mm : 40 Gpixels x 25,000 slices = ~1 PByte
3
Daniel Berger
Connectome Workflow
Cutting
Analysis
Proof Reading
Imaging
Visualization
Alignment &
Registration
Segmentation
Analysis
K. Al-Awami, et al.,
NeuroLines: A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity,
IEEE Transactions on Visualization and Computer Graphics, vol. 20, no. 12, pp. 2369-2378,
2014
Data Science
Computer
Science
Statistics
Domain Science
Drew Conway
Machine
Human
Human Cognition
Data Management
Data Mining
Machine Learning
Perception
Visualization
Business Intelligence
Statistics
Story Telling
Decision Making
Theory
Data Science
Inspired by Daniel Keim, Visual Analytics: Definition,
Process, and Challenges
Outline
What?
Why?
Who?
How?
The Age of Big Data
BBC, 2013
Big Data
Between the dawn of civilization and
2003, we only created five exabytes of
information; now were creating that
amount every two days.
Eric Schmidt, Google (and others)
http://onesecond.designly.com/
travers808,Visual.ly
Jim Gray, Microsoft
By 2018, the US could face a shortage
of up to 190,000 workers with analytical
skills
McKinsey Global Institute
The sexy job in the next 10 years will
be statisticians. Data Scientists?
Hal Varian, Prof. Emeritus UC Berkeley
Chief Economist, Google
Hal Varian Explains...
The ability to take data to be able to
understand it, to process it, to extract
value from it, to visualize it, to
communicate it's going to be a hugely
important skill in the next decades, not
only at the professional level but even at
the educational level for elementary school
kids, for high school kids, for college kids.
Because now we really do have essentially
free and ubiquitous data. Hal Varian
Ask an interesting
question.
What is the scientific goal?
What would you do if you had all the data?
What do you want to predict or estimate?
Get the data.
How were the data sampled?
Which data are relevant?
Are there privacy issues?
Explore the data.
Plot the data.
Are there anomalies?
Are there patterns?
Model the data.
Build a model.
Fit the model.
Validate the model.
Communicate and
visualize the results.
What did we learn?
Do the results make sense?
Can we tell a story?
IPython Notebooks
http://nbviewer.ipython.org/
Outline
What?
Why?
Who?
How?
Hanspeter Pfister
An Wang Professor of Computer Science, SEAS
Director, Institute for Applied Computational Science
pfister@seas.harvard.edu / @hpfister
Joe Blitzstein
Professor of the Practice in Statistics,
Co-Director of Undergraduate Studies in Statistics
blitz@fas.harvard.edu, twitter @stat110, SC 714
Verena Kaynig-Fittkau
Lecturer and research scientist at IACS
vkaynig@seas.harvard.edu, NW B164
Rahul Dave
Head TF and Lecturer at IACS
rahuldave@gmail.com, NW B164
CS 109 Staff
Andrew Reece
Antonio Coppola
Austen Novis
Brian Feeny
Dana Katzenelson
Giri Gopalan
Irma Nomani
Jacob Dorabialski
Joseph Song
Kathy Li
Lawrence Kim
Leandra King
Luis Campos
Marcus Way
Michael Ma
Michael Packer
Nelson Santos
Richard Kim
Rick Wei-Jong Lee
Sail Wu
Stephen Klosterman
Xintao Qiu
Yingzhuo (Diana) Zhang
Yuhao Zhu
About You
Outline
What?
Why?
Who?
How?
CS109 Key Facets
data munging/scraping/sampling/cleaningin order to get an
informative, manageable data set;
data storage and management in order to be able to access
data quickly and reliably during subsequent analysis;
exploratory data analysisto generate hypotheses and
intuition about the data;
predictionbased on statistical tools such as regression,
classification, and clustering; and
communicationof results through visualization, stories, and
interpretable summaries.
Act I: Predictions
Data Collection, Munging, and Storage
Exploratory Data Analysis (EDA)
Classification & Regression
Cross Validation
Dimensionality Reduction
Effective Communication & Writing
Act II: Recommendations
Support Vector Machines
Decision Trees & Random Forests
Bagging & Boosting
Machine Learning Best Practices
MapReduce, Amazons EC2, and Spark
Act III: Clustering & Text
Bayesian Thinking & Naive Bayes
Text Analysis: LDA & Topic Modeling
Clustering
Effective Presentations
Deep Learning
Guest Lecture: Experimental Design
cs109.org
Concepts...
Lectures
...and Skills
Sections
Sections
Introduce tools & skills; available as lab
notebooks and videos
Mandatory, except for DCE students
First (group) section this Friday!
10am-12pm in MD G115
Regular sections first week as office hours
to get help with Python, Git, and HW0
Section Schedule (TBD)
Monday
Tuesday
9:00 AM
Wednesday
Thursday
Friday
Rahul, NW-B150
10:00 AM
Leandra
Ima
Steve
Luis
11:00 AM
NW-B150
NW-B150
NW-B150
NW-B150
12:00 PM
1:00 PM
Diana
2:00 PM
NW-B150
Lecture
Lawrence
Lecture
NW-B150
3:00 PM
Joseph
NW-B103
NW-B150
NW-B103
Michael Packer
4:00 PM
NW-B150
5:00 PM
Michael Ma
Antonio
Sail, Nelson
NW-B150
6:00 PM
Austen, Dana
NW-B150 and B166
Richard
7:00 PM
NW-B150 and B166
Kathi
NW-B150
8:00 PM
NW-B150
NW-B150
Homework
Real-World focus
Scrape and wrangle messy data
Apply sophisticated statistical analysis
Visualize and communicate results
Election data, music charts,
recommendations, etc.
Programming
xkcd
Piazza
Sign up by next Friday (HW0)
Announcements posted here
Questions, feedback, discussions, etc.
Help each other!
Grades
No exams!
50% Homework
40% Projects (3-4 person teams)
10% Participation (Piazza & Sections)
10 point scale, holistic grading
Projects
Policies
HWs due on Thursdays, 11:59 pm EST
6 late days for HW (no questions asked)
Cannot submit HW later than 2 days
Regrading requests within 7 days in writing
Grade may improve or go down
Collaboration Policy
Work you turn in must be your own
Projects are a 3-4 person team effort
With project group peer assessment
Acknowledge all help and code you used
Harvard Honor Code
Is this course for me ???
Prerequisites
Programming experience
CS50 and/or C, C++, Java, Python, etc.
Basic statistical knowledge
STAT100, ideally STAT110
Willingness to learn new software & tools
This can be time consuming
You will need to read online documentation
Be Patient
Be Flexible
Be Constructive
http://davidzinger.wordpress.com/2007/05/page/2/
Next Steps
HW 0, mandatory, needs to be submitted!
Good test of your basic skills
Installation of several Python frameworks
Complete the survey by tomorrow! Needed to be
able to submit HW 0
Not graded, do it as soon as possible
Read syllabus carefully
Important Links
Create a github account at http://github.com
Then fill in our survey at http://goo.gl/forms/bJwajS8zO8
HW 0 document at https://github.com/cs109/2015lab1/
blob/master/hw0.ipynb
Week 1 notebooks at https://github.com/cs109/2015lab1
HW repositories will be created for you on github. See
HW 0 for details.
