Modern Machine Learning in Python

Heikki Huttunen
University Lecturer, Dr.Eng.
Heikki.Huttunen@tut.fi

Department of Signal Processing

Tampere University of Technology, Finland

August 14, 2015

Outline

• Motivation
• Why machine learning?
• Machine learning basics
• Why Python?
• Review of widely used classifiers and their implementation in
Python
• Examples
• Data driven design of a classifier for ovarian cancer detection
• ICANN MEG Mind Reading Challenge 2011
• IEEE MLSP Birds 2013 Competition
• DecMeg 2014 Decoding the Human Brain Competition
• Hands-on session (TC407)
• Slides available at http://www.cs.tut.fi/~hehu/MMLP.pdf
Introduction

• Machine learning has become an

important tool for multitude of
scientific disciplines.
• Training based approaches are rapidly
substituting traditional manually
engineered pipelines.
• ”Training based” = we show examples
of what is interesting and hope the
machine learns to do it for us
• A few modern research topics:
• Image recognition (what is in this
image and where?)
• Speech recognition (what do I say?)
Price et al., ”Highly accurate two-gene
• Medicine (data-driven diagnosis) classifier for differentiating gastrointestinal
stromal tumors and leiomyosarcomas,” PNAS
2007.
 Why Python?

• Python is becoming increasingly central tool

for data science.
• This was not always the case: 10 years ago
everyone was using Matlab.

Python
• However, due to licensing issues and heavy
development of Python, scientific Python
started to gain its user base.

numpy
• Python’s strength is in its variability and
huge community.

sklearn
• There are 2 versions: Python 2.7 and 3.4.
We’ll use the former one as it’s better

matplotlib
supported by most packages interesting to us.
 Alternatives to Python in Science
Python vs. R
Python vs. Matlab
• R is #1 workhorse for statistics
• Matlab is #1 workhorse for
and data analysis.
linear algebra.
• Popularity: http://www.
• Matlab is professionally
kdnuggets.com/polls/2015/
maintained product.
r-vs-python.html
• Some Matlab’s toolboxes are
• R is great for specific data
great (Image Processing tb).
analysis and visualization needs.
Some are obsolete (Neural
Network tb). • However, Python interfaces to
other domains ranging from
• New versions 2 twice a year.
deep neural networks (Caffe,
Amount of novelty varies.
Theano) and image analysis
• Matlab is expensive for (OpenCV) to even a fullblown
non-educational users. webserver (Django)
Essential Modules

• numpy: The matrix / numerical analysis layer at the bottom

• scipy: Scientific computing utilities (linalg, FFT, signal/image
processing...)
• sklearn: Machine learning (our focus here)
• matplotlib: Plotting and visualization
• opencv: Computer vision
• pandas: Data analysis
• caffe, theano, minerva: Deep neural networks
• spyder: The front end (Scientific PYthon Development
EnviRonment)
Where to get Python?

• It is possible to construct your custom Python environment by

installing individual modules.
• Alternatively, one may install a full distributions, such as
• Anaconda http://www.continuum.io/ ← my favorite
• Enthought Canopy http://www.enthought.com/
• Pythonxy http://code.google.com/p/pythonxy/
• ...or in linux:
# apt-get install python
# apt-get install python-numpy
# apt-get install python-sklearn
# apt-get install python-matplotlib
# apt-get install spyder
 The Language

• Python was designed to be a highly

readable language.
• Python uses whitespace to delimit
program blocks. First you hate it, later
you love it.
• All used modules are imported using
an import declaration.
• The members within the module are
referred using the dot, e.g.,:
np.cos([1,2,3])
• Interpreted language. Also interactive
with IPython extensions.
Classification
• Many machine learning problems can be posed as
classification tasks.
• Most classification tasks can be posed as a problem of
partitioning a vector space into disjoint regions.
• These problems consist of following components:

Samples: x1 , x2 , . . . , xN ∈ RP
Class labels: y1 , y2 , . . . , yN ∈ {1, 2, . . . , C }
Classifier: F (x) : RP 7→ {1, 2, . . . , C }

• Now, the task is to find the function F that maps the samples
most accurately to their corresponding labels.
• For example: Find the function F that minimizes the number
of erroneous predictions, i.e., the cases in which F (xk ) 6= yk .
Classification Example

• For example, consider the 2-dimensional

dataset on the right. 2

• The data consists of 200 blue crosses and

0
200 red circles.
• Based on these data, what would be a 2 Which class?
good partition of the 2D space into ”red”
4
and ”blue” regions?
• What kind of boundaries are allowed 6
between the regions?
• Straight lines? 8
• Continuous curved boundaries?
• Boundary without any restriction? 10
2 1 0 1 2 3 4 5 6

• Note: In 2-D this can be solved manually,

but not in 4000-D.
Different Classifiers

• In the following we will take a non-mathematical overview of

the following widely used classifiers.
• Nearest Neighbor classifier
• Linear classifiers (with linear boundary)
• The support vector machine (with nonlinear boundary)
• Ensemble classifiers: Random Forest
Nearest Neighbor Classifier
2 Nearest Neighbor Classifier

2 Classified as RED

• Probably the most natural idea is to

4
forget the boundary approach and just see
what kind of samples are nearby. 6

• This is the idea behind the Nearest

8
neighbor classifier: Just copy the class
label of the most similar training sample 10
2 1 0 1 2 3 4 5 6
to the unknown test sample.
K-Nearest Neighbor Classifier
2 5-Nearest Neighbor Classifier

• The problem with Nearest Neighbor is its 2 Classified as BLUE

fragility to changes in the training data.
4
• The classification boundary may change a
lot by moving only one training sample. 6

• The robustness can be increased by taking

8
a majority vote of more nearby samples.
• The K-Nearest neighbor classifier 10
2 1 0 1 2 3 4 5 6
selects the most frequent class label
among the K nearest training samples.
Other Problems with the Nearest Neighbor

• Nearest neighbor is prone to overlearning: Especially the 1-NN

forms local regions around every isolated data point.
• It is highly unlikely that these represent the general trend of
the whole population.
• Moreover, the training step extremely fast while the
classification step becomes extremely slow (consider training
data with billion high-dimensional samples).
• Therefore, more compact representations are preferred:
Training time is usually not critical while execution time is.
Linear Classifiers
2 A Linear Classifier

• A linear classifier learns a linear decision

0
boundary between classes.
• In mathematical terms, the classification 2 Classified as BLUE
rule can be written as
4
(
Class 1, if wT x<b
F (x) = T
6
Class 2, if w x ≥ b
8
where the weights w are learned from the
data. 10
2 1 0 1 2 3 4 5 6

• The expression wT x = k wk xk
P
essentially transforms the
multidimensional data x to a real number,
which is then compared to a threshold b.
 Flavors of Linear Classifiers

There exists many algorithms for learning the weights w, including:

• Linear Discriminant • Support Vector • Logistic Regression
Analysis (LDA) Machine (SVM) (LR)
2 Linear Discriminant Analysis 2 Support Vector Machine 2 Logistic Regression

0 0 0

2 Classified as BLUE 2 Classified as RED 2 Classified as RED

4 4 4

6 6 6

8 8 8

10 10 10
2 1 0 1 2 3 4 5 6 2 1 0 1 2 3 4 5 6 2 1 0 1 2 3 4 5 6
Details
• The LDA:
• The oldest of the three: Fisher, 1935.
• ”Find the projection that maximizes class separation”, i.e., pull
the classes as far from each other as possible.
• Closed form solution, fast to train.
• The SVM:
• Vapnik and Chervonenkis, 1963.
• ”Maximize the margin between classes”.
• Slowest of the three, performs well with sparse
high-dimensional data.
• Logistic Regression (a.k.a. Generalized Linear Model):
• History traces back to 1700’s, but proper formulation and
efficient training algorithm by Nelder and Wedderburn in 1972.
• Statistical algorithm: ”Maximize the likelihood of the data”.
• Outputs also class probability. Has been extended to
automatic feature selection.
The Class Probabilities of Logistic Regression

2 Logistic Regression Probabilities

Classified as RED
2 p = 60.6 %

10
2 1 0 1 2 3 4 5 6
 Nonlinear SVM
2 Support Vector Machine with Kernel Trick

• The Support Vector Machine can be

0
extended to nonlinear class boundaries.
• This is based on the kernel trick that 2 Classified as RED
implicitly maps the data to a higher
4
dimensional space.
• The same effect is achieved by appending 6
”fake” dimensions to the feature vector; e.g.,
8

x = (x1 , x2 ) =⇒ x0 = (x1 , x2 , x12 , x22 )

10
2 1 0 1 2 3 4 5 6
The classifier is then trained in 4-D.
• The kernel trick is a lot faster than the
explicit map, and allows mappings to even
∞-dimensional spaces!
Random Forest

• Random forest (RF) is popular tree based

classifier, proposed by Breiman in 1993. Temp < 5

• The RF is based on a collection of Yes No

decision trees.
• A decision tree is a straightforward Snow? Raining?
if-then-else-like diagram that combines Yes No Yes No
individual attributes into a decision.
• Decision trees are easily trained to learn
Winter Fall Fall Sunny?
the data.
Yes No
• For example, the tree on the right
predicts the time of year from the Summer Spring

following measured attributes:

{Temperature, Snow, Rain, Sunny}.
Random Forest
Snow?

• The problem of decision trees is that they

overlearn the data, i.e., the training data No Yes
is memorized with a poor ability to
generalize. Raining? Winter
• With no restrictions, the DT will have one
root-leaf-path per sample, which is No Yes
essentially same as nearest neighbor.
Sunny? Fall
• RF avoids this by training many
”imperfect” trees. No Yes
• Each tree is trained with a subset of
samples and a subset of features, i.e., Spring Summer
some of the attributes are hidden from
the training. Decision Tree without the
Temperature attribute.
Random Forest
2 Random Forest

2 Classified as BLUE

• After training many trees each with

4
different samples and features, the RF
predicts the class by taking the majority 6

vote: what is the most frequent label.

8
• The number of trees varies from a few
dozen to few thousand—default in 10
2 1 0 1 2 3 4 5 6
Python is 10.
Random Forest
2 Random Forest Class Probabilities

• The collection of trees gives a natural way

0
of estimating class probabilities: Just use
the proportion of trees voting for each Classified as BLUE
2 p = 66.0 %
class.
• RF’s also includes a method for assessing 4

feature importances.
6
Relative Importance of Attributes
Horizontal axis 8

26.8%
10
2 1 0 1 2 3 4 5 6

73.2%

Vertical axis
Generalization
• The important thing is how well the classifier works with
unseen data.
• An overfitted classifier memorizes the training data and does
not generalize.
2 1-NN / Training Data: Acc = 100.0 % 2 1-NN / Test Data: Acc = 88.2 %

0 0

2 2

4 4

6 6

8 8

10 10
2 1 0 1 2 3 4 5 6 2 1 0 1 2 3 4 5 6
 Overfitting
3 Example time series
2
1
0
• Generalization is also related to overfitting. 1

• On the right, a polynomial model is fitted to 2

3
0.0 0.2 0.4 0.6 0.8 1.0
a time series to minimize the error between
3 7'th order polynomial fit
the samples (red) and the model (green). 2

• As the order of the polynomial increases, the 1

0
model starts to follow the data very faithfully. 1
2
• Low-order models do not have enough 3
0.0 0.2 0.4 0.6 0.8 1.0
expression power. 25'th order polynomial fit
3
• High-order models are over-fitting to noise 2
1
and become ”unstable” with crazy values 0
near the boundaries. 1
2
3
0.0 0.2 0.4 0.6 0.8 1.0
 Regularization
3 Example time series
2
1
0
1
• Regularization adds a penalty term to the 2
3
fitting error. 0.0 0.2 0.4 0.6 0.8 1.0

3 7'th order regularized polynomial fit

• The model is encouraged to use small 2
coefficients. 1
0
• Large coefficients are expensive, so the model 1

can afford to fit only to the major trends. 2

3
0.0 0.2 0.4 0.6 0.8 1.0
• On the right, the high order model has a 25'th order regularized polynomial fit
3
good expression power, but does still not 2

follow the noise patterns. 1

0
1
2
3
0.0 0.2 0.4 0.6 0.8 1.0
 Sparsity
1.0 1e13 Unregularized model: 24 nonzeros
0.8
• Regularization also enables the design of 0.6
0.4
sparse classifiers. 0.2
0.0
0.2
• In this case, the penalty term is designed 0.4
0.6
0.8
such that it favors zero coefficients. 0 5 10
Order
15 20 25

20
`2 penalty: 24 nonzeros
• A zero coefficient for a linear operator is 15
10
equivalent to discarding the corresponding 5
0
feature altogether. 5
10
• The plots illustrate the model coefficients 15
0 5 10 15 20 25
Order
without regularization, with traditional 6
`1 penalty: 8 nonzeros
4
regularization and sparse regularization. 2
0
• The importance of sparsity is twofold: The 2
4
model can be used for feature selection, but 6
8
often also generalizes better. 0 5 10
Order
15 20 25
 Example: Classifier Design for Ovarian Cancer
Detection

35 Proteomic Mass Spectrum

Healthy
• We will study an example of Cancer
30
classifying proteomic fingerprints of
25
mass-spectra measured from ovarian
cancer patients and healthy controls.a 20

• 121 cancer patients and 95 healthy

15
controls.
10
• Mass spectra consists of 4000 features
(already downsampled at the 5

preprocessing step).
0
a
Conrads, Thomas P., et al. ”High-resolution serum proteomic
5
features for ovarian cancer detection.” Endocrine-Related Cancer (2004). 0 500 1000 1500 2000 2500 3000 3500 4000
 Reading the Data

• The data is in csv (comma separated Code:

values) format, where each line
contains 4000 MS measurements for
one person.
• The attached code reads each line and
stores the 4000 measurements into a
list.
• The class labels are read in a similar
manner.
• Alternatively, we could have read using
Result:
numpy’s loadtxt function, or the
Pandas module;
http://pandas.pydata.org/
 Training a Classifier
Code:
• The attached code trains and applies a
Logistic Regression classifier for the
ovarian cancer problem, with
essentially 3 lines of code:
from sklearn.linear_model \
import LogisticRegression Result:
0.8 LR coefficients (error: 0.00 %)
clf = LogisticRegression()
0.6
clf.fit(X, y) 0.4
prediction = clf.predict(X) 0.2
0.0
• Note that the above prediction is 0.2
0.4
100 % accurate; an unrealistic result
0.6
due to testing on the (rather small) 0.8
0 500 1000 1500 2000 2500 3000 3500 4000
training set.
 Cross-validation

• The generalization ability of a classifier needs

to be tested with unseen samples.
• In practice this can be done by splitting the
Testing Training
data into separate training and test sets.
Training Training
• A standard approach is to use K -fold
Training Training
cross-validation:
Training Training
• Split the training data to K parts (called
Training Testing
folds)
• Use each fold for testing exactly once and
train with the other folds
• The error estimate is the mean of the K
estimates
 Cross-validation in sklearn
Code:

• Cross-validation is extremely elegant in

sklearn.
• Essentially 1 line of code is required:

from sklearn.cross_validation \
import cross_val_score
scores = cross_val_score(clf, X, y)

• The cross-validator receives the classifier

and the data and returns a list of scores Result:
for each test fold.
• The function can receive additional
parameters that modify the number of
folds, possible stratification (balancing the
classes in folds), etc.
 Cross-validating a set of Classifiers
Code:

• A nice property of sklearn is that each

predictor conforms to the same
interface (i.e., each implements
.fit(), and .predict() methods).
• Thus, we can examine a list of them in
a for loop easily.
• It seems that the linear classifiers
(Logistic Regression and Linear SVM)
are the best performers for this case. Result:
 Regularization
Code:

• Now that linear classifiers seem to be

suitable for this data, we will see how
regularization affects the results.
• Remember: Regularization penalizes
for large coefficients and thus prevents
overlearning. Moreover, `1
regularization makes the coefficients
sparse. Result:
100

• The code on the right calculates the 90

LinSVM-L1 [C = 100]:
CV-10 error for a range of 80 Accuracy = 99.09 %
Accuracy / % 70 Sparse Dense
regularization parameters.
60 LogReg-L1
• The resulting accuracies are plotted at LinSVM-L1
50 LogReg-L2
LinSVM-L2
the bottom. 40 -3
10 10-2 10-1 100 101 102 103 104 105 106 107
Regularization parameter
 Analysis of the Classifier

Coefficients (3.4 % nonzero)

• The `1 regularized Linear SVM was the 2.0
1.5
most accurate classifier. 1.0
0.5
• It is defined by its coefficients w. 0.0
0.5
• The nonzero coefficients correspond to 1.0

the peaks of the mass spectra. 1.5

2.0
0 500 1000 1500 2000 2500 3000 3500 4000
• The decision rule is very simple
35 Proteomic Mass Spectrum
( Healthy
30
P Cancer
cancer, if wk xk ≥ b 25
prediction = P 20
healthy, if wk xk < b 15
10

with wk the coefficients, xk the mass 5

0
spectrum and b the threshold (which 5
0 500 1000 1500 2000 2500 3000 3500 4000
happens to be b = 0 this time).
Recent Developments

• One of the most exiting new research directions is Deep

Learning, which studies neural networks with many layers.
• The methodology has set new records in various areas that are
abundant with data, e.g., speech recognition, image
recognition, etc.
• The default language of most available tools is Python.
• Under the hood, the engines are typically implemented in
C++ or CUDA, that links to the Python front end.
Deep Learning Tools

• The four most popular Deep learning toolboxes are (in the
order of decreasing popularity):
• Caffe: http://caffe.berkeleyvision.org/ from Berkeley
• Torch: http://torch.ch/ from NYU (used by Facebook)
• pylearn2: http://deeplearning.net/ from U. Montreal
• Minerva https://github.com/dmlc/minerva from New
York, Peking Univ.
• All except Torch can be used via a Python front end (Torch
uses Lua language).
Applications

• Application example 1: ICANN MEG Mind Reading

Challenge, June 2011
• Application example 2: IEEE MLSP 2012 Birds
competition, Sept. 2013
• Application example 3: DecMeg2014: Decoding the Human
Brain, July 2014
ICANN MEG Mind Reading Challenge I
• We participated in the Mind reading challenge from MEG
data organized in the ICANN 2011 conference. 1 2 3

1
H. Huttunen et al., ”Regularized logistic regression for mind reading with parallel validation,” Proc.
ICANN/PASCAL2 Challenge: MEG Mind-Reading. Aalto University publication series, Espoo, June 2011.
2
H. Huttunen et al., ”MEG Mind Reading: Strategies for Feature Selection,” in Proc. of The Federated
Computer Science Event 2012, Helsinki, May 2012.
3
H. Huttunen et al., ”Mind Reading with Regularized Multinomial Logistic Regression,” Machine Vision and
Applications, Aug. 2013
ICANN MEG Mind Reading Challenge Data

• The competition data consists of MEG recordings while

watching five different movie categories.
• The data contains measurements of 204 planar gradiometer
channels at 200Hz rate, segmented into samples of one
second length.
• The task was to design and implement a classifier that takes
as an input the MEG signals and produces the predicted class
label.
• The training data with annotations had 727 one-second
samples, and the secret test data 653 unlabeled samples.
• 50 samples of the training data were special, because they
were recorded on the same day as the test data.
The Curse of Dimensionality

• Since each measurement is a time series, it cannot be fed to

classifier directly.
• Instead, we calculated a number of common quantities.
• In all, our pool of features consists of 11 × 204 = 2244
features. This means 2244 × 5 = 11220 parameters.
−11
x 10
2

−2

−4
20 40 60 80 100 120 140 160 180 200

−11
x 10
2

−2

−4
0 20 40 60 80 100 120 140 160 180 200
Where the Selected Features are Located?

• As a side product we gain insight on which areas of the brain

were used for prediction.
IEEE MLSP2013 Birds Competition

• The task of the participants was to train an algorithm to

recognize bird sounds recorded at the wildlife.
• The recordings consisted of bird sounds from C = 19 species,
and the sounds were overlapping (with up to 6 birds in a
single recording).
• There were altogether 645 ten-second audio clips with sample
rate Fs = 16 kHz.
• Half of the samples (Ntrain = 323) were used for model
training, and half (Ntest = 322) of the data was provided
without annotation.
IEEE MLSP2013 Birds Competition
• The task of the participants was to train an algorithm to
recognize bird sounds recorded at the wildlife.
• The recordings consisted of bird sounds from C = 19 species,
and the sounds were overlapping (with up to 6 birds in a
single recording).
• There were altogether 645 ten-second audio clips.
• Half of the samples were used for model training, and the
participants should predict the labels of the other half.
• We participated in the challenge, and our final submission
placed 7th among the 77 teams.
IEEE MLSP2013 Birds — Our Method

• Our approach mixed six prediction models by averaging the

predicted class probabilities together.
• Most fruitful model used dictionary based Bag-of-Words
features.
• Learn a dictionary of typical patches in the spectrograms
(including the non-annotated ones).
• Count the occurrences of each dictionary atom in each
spectrogram.
IEEE MLSP2013 Birds — Our Method

• Examples of patch histograms are shown below.

• Left: two bird species present, Right: no birds present,
Bottom: a single bird present.
• One can spot the noise encoding patches (1, 4, 6, 7, 11 . . .),
which are active in all histograms
2500 2500

2000 2000

1500 1500
Count

Count
1000 1000

500 500

0 0
0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50
Atom index Atom index
2500

2000

1500
Count

1000

500

0
0 5 10 15 20 25 30 35 40 45 50
Atom index
 DecMeg2014 Competition
0
Face shown

50

• In the 2014 DecMeg competition, the 100

Sensor Number
task was to predict whether test persons 150

were shown a face or a non-face using 200

MEG recordings of the brain. 250

300
• Sequences were 1.5 second long 0.00 0.25 0.50 0.75 1.00
Time / s
1.25 1.50

measurements with 306 channels; face 0

Non-face shown
shown at 0.5 s - 1.5 s. 50

• In total, 16 training and 7 test subjects; 100

Sensor Number
approximately 600 datapoints from each 150

subject. 200

• Intersubject variation is significant. 250

300
0.00 0.25 0.50 0.75 1.00 1.25 1.50
Time / s
Our Model

• Our is a hierarchical combination of two layers of classifiers:

• First layer consists of Logistic Regression classifiers mapping
individual sensors or individual timepoints to class probabilities.
• The second layer is a 1000-tree Random Forest with LR
predictions as inputs.
• The idea of stacking classifiers originates to Wolpert’s stacked
generalization (1992), and was inspired by our first attempts
to approach the problem using a deep convolutional net.
LR
10
Time

20 LR
30

...
50 100 150 200 250 300
Sensor
LR

LR ... LR LR LR

RF Decision
 Feature Importances
0.06

0.05

0.04

Feature Importance
31 timepoints
0.03

• The reliability of each 1st layer classifier 0.02

306 sensors

0.01

can be seen from the feature importances 0

0 50 100 150 200 250 300 350

of the Random Forest. View Index

• The full time series from individual

sensors seem a lot more informative than
snapshots of the whole brain.
• Eventually our method was successful
placing us on the 2nd place of 267 teams.
• The implementation is 100 % sklearn:
https://github.com/mahehu/decmeg
Summary

• The significance of Python as a language of scientific

computing has exploded.
• Machine learning has been one of the key areas in this
development.
• This has been coined by the open-access attitude of the
community: licensing tends to be very free.
• One of the key benefits of
scikit-learn is the
uniform API: It’s easy to test
a wide range of algorithms
with a short piece of code.
• Deep learning is a growing
trend—powered by Python.