Classification
CLASSIFICATION
TYPES
Binary Classification Multi-class Classification
Binary classification is the task of Multiclass classification is the task
classifying the elements of a given of classifying the elements of a
set into two groups on the basis of given set into more than two
a classification rule. groups on the basis of a
classification rule.
BMSCE - ME | PA G E
MCL - Python 2
Classification
• Can you separate red class
from blue class ?
BMSCE - ME
MCL - Python
| PA G E 3
Linear Boundary
• Straight line for two dimensions.
BMSCE - ME
MCL - Python
| PA G E 4
Linear Boundary
• Straight line for two dimensions.
• Plane for three dimensions.
• Hyperplane for higher dimension
BMSCE - ME
MCL - Python
| PA G E 5
Confusion Matrix Accuracy
Predicted
𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑎𝑔𝑒𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑐𝑙𝑎𝑠𝑠𝑖𝑓𝑖𝑒𝑑
𝑑𝑎𝑡𝑎𝑝𝑜𝑖𝑛𝑡𝑠
Positive Negative ¿ ( 𝑇𝑃 +𝑇𝑁 )
A c c u+𝐹𝑃
(𝑇𝑃+𝐹𝑁 r a c +𝑇𝑁
y )
Positive a (TP) b (FN)
Actual Confusion Matrix
Negative c (FP) d (TN)
Other error metrics
• Precision
Sensitivity Specificity • Re c a l l
• F score
a / (a + c) d/ (b + d) • ROC cur ve
Algorithm
Construct a frequency table for the target and
select its most frequent value.
For classification this is Baseline Model
Disease
Yes No
9 6
0.6 0.4
ZeroR Method
BMSCE - ME
MCL - Python
| PA G E 7
Blood
Pressure
Classification
Target
variable is
discrete
Target : 0/1
BMSCE - ME
MCL - Python
| PA G E 11
Can you fit a
Linear regression
model ?
BMSCE - ME
MCL - Python
| PA G E 12
BMSCE - ME
MCL - Python
| PA G E 13
BMSCE - ME
MCL - Python
| PA G E 14
BMSCE - ME
MCL - Python
| PA G E 15
BMSCE - ME
MCL - Python
| PA G E 16
BMSCE - ME
MCL - Python
| PA G E 17
BMSCE - ME
MCL - Python
| PA G E 18
BMSCE - ME
MCL - Python
| PA G E 19
Algorithm
Model the log-odds ratio as a linear function of
independent variables and then convert the log odds ratio
to probability using the sigmoid (logistic) function
Logistic Function
Logistic
Regression BMSCE - ME | PA G E 20
MCL - Python
Algorithm
C a l c u l a te t h e p o s te r i o r p ro ba b i l i t y, P ( A | B ) ,
from P(A), P(B), and P(B|A). Naive Bayes
classifier assume that the effect of the
value of a predictor ( x) on a given class
(c) is independent of the values of other
predictors.
Naive Bayes
BMSCE - ME
MCL - Python
| PA G E 22
Algorithm
C a l c u l a te t h e p o s te r i o r p ro ba b i l i t y, P ( A | B ) ,
from P(A), P(B), and P(B|A). Naive Bayes
classifier assume that the effect of the
value of a predictor ( x) on a given class
(c) is independent of the values of other
predictors.
P(A|D) P(A)
P(D)
Naive Bayes
BMSCE - ME | PA G E
P(D|Alco & S & Age) = P(A|D) * P(S|D)MCL
* P(Age|D)
- Python 23
* P(D)
Naive Bayes
PROS
• Very easy and fast
• Can be used for multiclass prediction
• Performs well with categorical features
• If features are independent NB gives
superior predictions
CONS
• Features are not independent in most real
life examples
• Issue with category that was not found in
training data
• Assumption that the numerical features
follow normal distribution
P(Y | X) = P(X1 | Y) * P( X2 | Y)…. P(Xn|Y) * P(Y)
BMSCE - ME
MCL - Python
| PA G E 24
Algorithm
SVM performs classification by coming up with a
hyperplane that maximizes the separation margin
between the two classes. The vectors that support
the hyperplane are support vectors
Plot all the data rows as a point in N-Dimensional
space
N dimensions refer to N features
So the point will have feature as the value for a
particular coordinate
Support Vector Come up with a hyperplane that can separate these
Machines (SVM) points in the best way possible into different classes
in that N-dimensions BMSCE - ME | PA G E
25
MCL - Python
Which one would you select ?
Why ?
The black line
► Because it is classifying the points
accurately with the highest margin
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 26
Which one would you select ?
Why ?
I would again select the black line
► Because it is classifying the points
accurately with the highest margin
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 27
Maximum Margin Classifier
► Classifies with the maximum margin
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 28
What will you do in this case?
Can we calculate some other feature from X
and Y and then try to separate this ?
How about Z = X2 + Y2
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 29
What will you do in this case?
Can we calculate some other feature from F1
and F2 and then try to separate this ?
How about Z = F12 + F22
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 30
What will you do in this case?
Can we calculate some other feature from F1
and F2 and then try to separate this ?
How about Z = F12 + F22
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 31
What will you do in this case?
Can we calculate some other feature from F1
and F2 and then try to separate this ?
How about Z = F12 + F22
Decision Boundary
BMSCE - ME
MCL - Python
| PA G E 32
Support Vector
Machines
PROS
Works very well for small and clean
datasets
Works well with clear separation margins
Very effective in high dimensional space
Kernels give more flexibility
CONS
Large data sets require lot of training
time and eventually won’t perform well
Can’t do a good job when noisy data is
given ( overlapping classes)
BMSCE - ME
MCL - Python
| PA G E 33
Algorithm
Decision tree uses Entropy and Information Gain to
construct a tree. Top-down, greedy search through
the space of possible branches with no backtracking
Entropy for 1 attribute
Entropy for 1 attribute
Decision Trees
BMSCE - ME
MCL - Python
| PA G E 34
BMSCE - ME
MCL - Python
| PA G E 35
BMSCE - ME
MCL - Python
| PA G E 36
Steps in Decision Trees
Step
1 Calculate entropy of the target variable
Step Calculate entropy for each branch (split by
2 various features)
Step
3 Calculate Gain for each of the above splits
Step Choose attribute with the largest
4 information gain as the decision node
Step Check if the entropy is zero, else continue
5 further
Step Run recursively on the all branches, until all
6 data is classified (Branch entropy == 0)
BMSCE - ME
MCL - Python
| PA G E 37
Decision Trees
PROS
Implicitly perform feature selection
Discover Nonlinear relationships
Not affected by outliers
Easy to interpret and explain
Rules generated which can be shared
easily
CONS
Decision Trees do not work well if you
have smooth boundaries
Super attributes will give higher info gain
Missing values are ignored
BMSCE - ME
MCL - Python
| PA G E 38
Bias-Variance
Tradeoff
BIAS :
How well the model fits the data
Simpler models: Stable (low variance) but they VARIANCE :
don't get close to the truth (high bias).
How much the model changes based on
Complex models: More prone to being over fit changes in the inputs
(high variance) but they are expressive enough to get
close to the truth (low bias).
BMSCE - ME
MCL - Python
| PA G E 39
Decision Trees +
Bagging
If the number of cases in the training set is N, sample
01 N cases at random - but with replacement, from the
original data. This sample will be the training set for
growing the tree.
02 If there are M input variables, a number m<<M is
specified such that at each node, m variables are
selected at random out of the M and the best split on
Random Forest these m is used to split the node.
The value of m is held constant during the forest
03 growing. BMSCE - ME | PA G E 40
MCL - Python
BAGGING
• It is also called as bootstrap aggregating
• Bagging tries to combine predictions from
multiple similar learners trained on
different datasets by averaging their
predictions.
• It reduces variance and helps to avoid
overfitting
• Mostly used with decision trees
BMSCE - ME
MCL - Python
| PA G E 41
Features of
Random Forests
Implicitly gives estimates of what
01 Unexcelled in accuracy among
current algorithms.
04 variables are important in the 07 Generated forests can be
saved for future use on other
classification. data.
It has an effective method for
02 Runs efficiently on
large data sets.
05 estimating missing data and maintains 08 It offers an experimental
method for detecting variable
accuracy when a large proportion of
interactions
the data are missing.
03 06 09
Can handle thousands of input It has methods for balancing error
Uses OOB for error
variables without variable in class population unbalanced
calculation
deletion. data sets.
BMSCE - ME
MCL - Python
| PA G E 42
