Just Give me the Codes
Lecture 5: Data Preprocessing II
GOALS
• Normality (multivariate/bivariate/univariate distribution)
• Outlier detection and removal
Recap & Step 25
From last lecture:
Created a new df ‘Norway’
From ‘Norway’ created yet another df ‘selection’ (3
numerical variables)
This lecture (Step 25)
Warnings are a nuisance
Follow Step 25 to view dimensions of ‘selection’ and to
suppress ‘warnings’
Place a hashtag before filter.warnings() should you
select to read the warnings
Normality – The Assumption of Normality
Data needs to follow a normal distribution for many statistical tests
Referred to as the Assumption of Normality
critical for sample sizes <30
Choose appropriate statistical test for your sample size
Example: Sample size 23
Royston test for multivariate normality
Fail to reject the null at the 5% level (data follows a multivariate normal distribution)
Henze-Zirkler test for multivariate normality
Reject the null at the 5% level (data does not follow a multivariate normal distribution)
Refer to the links at the end of the lecture for more information on normality tests
Step 26 – Install and import Pingouin
Python limited with multivariate normality tests
pingouin package
Univariate, bivariate, multivariate
Follow Step 26 to install and import the
penguin package
Refer to links at the end of the lecture for more
information on pingouin
Step 27 – Shapiro-Wilk normality test
The null hypothesis for the Shapiro-Wilk
normality test states that the data is normally
distributed
The alternative hypothesis for the Shapiro-Wilk
normality test that the data is not normally
distributed.
Follow Step 27 to determine the normality of
each variable
All 3 numerical variables have univariate
normal distributions at significance level 0.05,
however it is good practice to visualize your
dataset to diagnose any deviation from
multivariate normality (when testing for
multivariate normality)
Step 28 – Visually inspect TFR_foreign
Step 28
One could
additionally place
a k after -o to
account for a
black marker
For example
plt.plot(a, fit, ‘-ok’)
Step 29 – Visually inspect TFR_native
Step 29
Step 30 – Visually inspect Overall_TFR
Step 30
Step 31 – Skewness & Kurtosis
Measuring skewness:
skewness = 0 : normally distributed (symmetrical
distribution)
skewness > 0 : longer right tail; mass of distribution is
concentrated on the left of the figure
skewness < 0 : longer left tail; mass of distribution is
concentrated on the right of the figure
Measuring kurtosis:
kurtosis = 0: normal distribution
Kurtosis > 0: distribution’s tails are larger than for a normal
distribution
Kurtosis < 0: distribution’s tails are smaller than for a normal
distribution
Results from Step 31 show TFR_foreign to be moderately
skewed, whilst TFR_native and Overall_TFR are fairly
symmetrical. TFR_foreign is heavy-tailed whilst TFR_native is
light-tailed. Overall_TFR has a kurtosis value consistent with a
normal distribution.
Step 32: Multivariate normal distribution
The null hypothesis for the Henze-Zirkler multivariate normality test states that the data follows
multivariate normal distribution
The alternative hypothesis for the Henze-Zirkler multivariate normality test states that the data does
not follow a multivariate normal distribution.
Dataset (‘selection’) does NOT have a multivariate normal distribution at significance level 0.05
We will try removing one of the variables (establish bivariate normality between each variable pair)
Steps 33-34: Bivariate normal distribution
TFR_foreign and Overall_TFR:
DO NOT satisfy the bivariate normality
assumption at the significance level
0.05.
TFR_foreign and TFR_native:
Satisfy the bivariate normality
assumption at the significance level
0.05.
TFR_native and Overall_TFR:
Satisfy the bivariate normality
assumption at the significance level
0.05.
In case you weren’t aware:
Bivariate = 2 variables
Multivariate = 2 or more variables
Steps 35-36: IQR and outliers
Follow Steps 35-36
to determine IQR
for each column
and accordingly,
number of outliers
per column
Step 37 – Position of outliers
Follow Step 37 to
view position of
outliers
Remember,
index=0 is position 1
Therefore outliers
at position 13 and
23
Steps 35-37
checked for
univariate outliers
Step 38 – Concat() & import seaborn
Concatenating in this direction:
x→y
y→z
x→z
Steps 39-41: PairGrid
Bivariate relationships:
g = TFR_foreign & TFR_native
h = TFR_native & Overall_TFR
i = TFR_foreign & Overall_TFR
Steps 39-41: PairGrid plots
Step 42 – Pearson’s correlation coefficient
pairwise_corr() function is part of the pingouin package
Pearson’s correlation coefficient hypothesis:
NULL: No linear relationship exists
ALTERNATIVE: A linear relationship does exist
There IS a significant linear relationship between TFR_native and Overall_TFR
There is NO significant linear relationship between TFR_foreign & Overall_TFR
There is NO significant linear relationship between TFR_foreign & TFR_native
What does all this mean?
TFR_foreign cause of outliers
Deleting outliers would reduce dataset by 20%
Deleting, imputing or transforming
No guarantee outliers deleted first instance
Faced with deleting or imputing new outliers
Originality of dataset reduced even further
No universal method for outlier detection and removal; choice comes
with experience
Steps 43-47: If the 5 outliers were
deleted…..
Steps 48-52 – Imputing outliers with the
median & new MVN test
Note: a p-value
>0.05 for ‘impute’
and ‘no_outliers’
datasets does not
infer no outliers
outlier tests need
to be conducted
again
End of Lecture 5
Well done! You have gained intermediate skills in Data Preprocessing!
Where to go from here? Lecture 6 of course! But things to consider:
Read up on normality tests
Read up on Pingouin
A great place to start:
Link to Pingouin: https://pingouin-stats.org/index.html
Pingouin univariate normality: https://pingouin-stats.org/generated/pingouin.normality.html
Pingouin multivariate normality: https://pingouin-
stats.org/generated/pingouin.multivariate_normality.html
Link to article on Normality tests: https://www.nrc.gov/docs/ML1714/ML17143A100.pdf
