1

M. R. Ahmadzadeh
Isfahan University of Technology
Ahmadzadeh@cc.iut.ac.ir

M. R. Ahmadzadeh
Isfahan University of Technology
 Textbooks
3

 Introduction to Machine Learning - Ethem Alpaydin

 Pattern Recognition and Machine Learning, Bishop.
 Machine Learning, Mitchell, Tom.
 The Elements of Statistical Learning, Hastie, T., R. Tibshirani, and J.
H. Friedman.
 Foundations of Machine Learning by Mehryar Mohri, Afshin
Rostamizadeh and Ameet Talwalkar
 Machine Learning: A Probabilistic Perspective, by Kevin P. Murphy.
 Introduction to Data Mining by Tan, Steinbach and Kumar
 Pattern Classification (2nd ed.) by Richard O. Duda, Peter E. Hart
and David G. Stork
 Pattern Recognition, 4th Ed., Theodoridis and Koutroumbas
 Grading Criteria
4

 Midterm Exam ≈ 25%

 HW, Comp. Assignments and projects: ≈ 30%
 Final exam ≈ 45%

 Course Website:
 http://ivut.iut.ac.ir or http://elearning.iut.ac.ir/
 Email: Ahmadzadeh@cc.iut.ac.ir
 EBooks …
 Contents
5

 1 Introduction 1
 2 Supervised Learning 21
 3 Bayesian Decision Theory 49
 4 Parametric Methods 65
 5 Multivariate Methods 93
 6 Dimensionality Reduction 115
 7 Clustering 161
 8 Nonparametric Methods 185
 9 Decision Trees 213
  10 Linear Discrimination 239
 11 Multilayer Perceptrons 267
6

 12 Local Models 317

 13 Kernel Machines 349
 14 Graphical Models 387
 15 Hidden Markov Models 417
 16 Bayesian Estimation 445
 17 Combining Multiple Learners 487
 18 Reinforcement Learning 517
 19 Design and Analysis of ML Experiments 547
 A Probability 593
 Lecture Slides for
INTRODUCTION
TO
MACHINE
LEARNING
3RD EDITION
ETHEM ALPAYDIN
© The MIT Press, 2014
CHAPTER 1:
alpaydin@boun.edu.tr
http://www.cmpe.boun.edu.tr/~ethem/i2ml3e INTRODUCTION
 Big Data
8

 Widespread use of personal computers and

wireless communication leads to “big data”
 We are both producers and consumers of data
 Data is not random, it has structure, e.g., customer
behavior
 We need “big theory” to extract that structure from
data for
(a) Understanding the process
(b) Making predictions for the future
 Why “Learn”?
9

 Machine learning is programming computers to optimize

a performance criterion using example data or past
experience.
 There is no need to “learn” to calculate payroll
 Learning is used when:
 Human expertise does not exist (navigating on Mars),
 Humans are unable to explain their expertise (speech
recognition)
 Solution changes in time (routing on a computer network)
 Solution needs to be adapted to particular cases (user
biometrics)
 What We Talk About When We Talk
10
About “Learning”
 Learning general models from a data of particular
examples
 Data is cheap and abundant (data warehouses,
data marts); knowledge is expensive and scarce.
 Example in retail: Customer transactions to consumer
behavior:
People who bought “Blink” also bought “Outliers”
(www.amazon.com)
 Build a model that is a good and useful
approximation to the data.
 Data Mining
11

 Retail: Market basket analysis, Customer

relationship management (CRM)
 Finance: Credit scoring, fraud detection
 Manufacturing: Control, robotics, troubleshooting
 Medicine: Medical diagnosis
 Telecommunications: Spam filters, intrusion detection
 Bioinformatics: Motifs, alignment
 Web mining: Search engines
 ...
 What is Machine Learning?
12

 Optimize a performance criterion using example

data or past experience.
 Role of Statistics: Inference from a sample
 Role of Computer science: Efficient algorithms to
 Solve the optimization problem
 Representing and evaluating the model for inference
 Machine Learning vs Pattern
13
Recognition
 Pattern Recognition: automatic discovery of regularities in data
and the use of these regularities to take actions – classifying
the data into different categories. Example: handwritten
recognition. Input: a vector x of pixel values. Output: A digit
from 0 to 9.
 Machine Learning: a large set of input vectors x1 ,... , xN , or a
training set is used to tune the parameters of an adaptive
model. The category of an input vector is expressed using a
target vector t. The result of a machine learning algorithm: y(x)
where the output y is encoded as the target vectors.
 Applications
14

 Association
 Supervised Learning
 Classification

 Regression

 Unsupervised Learning
 Reinforcement Learning
 Learning Associations
15

 Basket analysis:
P (Y | X ) probability that somebody who buys X
also buys Y where X and Y are products/services.

Example: P ( Chips | Yogurt ) = 0.7

 Classification
16

 Example: Credit
scoring
 Differentiating
between low-risk and
high-risk customers
from their income and
savings

Discriminant: IF income > θ1 AND savings > θ2

THEN low-risk ELSE high-risk
 Classification: Applications
17

 Face recognition: Pose, lighting, occlusion (glasses,

beard), make-up, hair style
 Character recognition: Different handwriting styles.
 Speech recognition: Temporal dependency.
 Medical diagnosis: From symptoms to illnesses
 Biometrics: Recognition/authentication using physical
and/or behavioral characteristics: Face, iris,
signature, etc
 Outlier/novelty detection:
 Face Recognition
18

Training examples of a person

Test images

ORL dataset,
AT&T Laboratories, Cambridge UK
 A classic example of a task that requires machine learning:
It is very hard to say what makes a 2
19
Regression

 Example: Price of a
used car
y = wx+w0
 x : car attributes
y : price
y = g (x | q )
g ( ) model,
q parameters

20
 Regression Applications
21

 Navigating a car: Angle of the steering

 Kinematics of a robot arm
(x,y) α1= g1(x,y)
α2= g2(x,y)
α2

α1

 Response surface design

 Supervised Learning: Uses
22

 Prediction of future cases: Use the rule to predict

the output for future inputs
 Knowledge extraction: The rule is easy to
understand
 Compression: The rule is simpler than the data it
explains
 Outlier detection: Exceptions that are not covered
by the rule, e.g., fraud
 Unsupervised Learning
23

 Learning “what normally happens”

 No output
 Clustering: Grouping similar instances
 Example applications
 Customer segmentation in customer relationship management
(CRM)
 Image compression: Color quantization

 Bioinformatics: Learning motifs

 Reinforcement Learning
24

 Learning a policy: A sequence of outputs

 No supervised output but delayed reward
 Credit assignment problem
 Game playing
 Robot in a maze
 Multiple agents, partial observability, ...
 Resources: Datasets - Journals
25

 UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html

 Statlib: http://lib.stat.cmu.edu/

 Journal of Machine Learning Research www.jmlr.org

 Machine Learning
 Neural Computation
 Neural Networks
 IEEE Trans on Neural Networks and Learning Systems
 IEEE Trans on Pattern Analysis and Machine Intelligence
 Journals on Statistics/Data Mining/Signal Processing
/Natural Language Processing/ Bioinformatics/ ...
 Resources: Conferences
26

 International Conference on Machine Learning (ICML)

 European Conference on Machine Learning (ECML)
 Neural Information Processing Systems (NIPS)
 Uncertainty in Artificial Intelligence (UAI)
 Computational Learning Theory (COLT)
 International Conference on Artificial Neural Networks
(ICANN)
 International Conference on AI & Statistics (AISTATS)
 International Conference on Pattern Recognition (ICPR)
 ...