CCST9047: The Age
of Big Data
Lecture 3
Things to reminder
‣ We only have 4 tutorials in the weeks of:
‣ 2.17 - 2.21 (next week), 2.24 - 2.28, 3.24 - 3.28, 3.31 - 4.4
Weekday Start time End Venue
THU 10:30 time
11:20 HW311
THU 11:30 12:20 HW311
THU 13:30 14:20 MB226
THU 14:30 15:20 MB226 If you haven’t registered, please
THU 15:30 16:20 MB226 do it asap!
THU 16:30 17:20 MB226
THU 17:30 18:20 MB226
FRI 10:30 11:20 MB154
FRI 11:30 12:20 MB127
FRI 13:30 14:20 MB127
Things to reminder
‣ The detailed description of the nal project will be released by this evening.
fi
Recap of Previous Lecture
Volume
‣ Daily applications
‣ Web data
‣ Social Network data
‣ Image and Video
Big Data
‣ Text data
‣ Sensor data
‣ Medical data Velcolity Variety
Recap of Previous Lecture
‣ Key points:
‣ How data is generated and collected in daily applications.
‣ How these data can enable better services?
‣ How data is formalized and organized with various data types.
‣ Properties of di erent data.
‣ Simple calculations of the data size.
Then, after collecting data, how to organize then and do some simple analysis?
ff
Things Will be Covered
‣ Data transformations
‣ Data visualizations
‣ Simple data statistics
Data Transformation
‣ Data are the results of deliberate human intervention
‣ Data vary across domains
‣ Di erent domains have di erent form of data
‣ Data vary within domains
‣ Data within the domain have di erent values.
The raw data is complex, and we need some about of preparation!
ff
ff
ff
Data Preparation
‣ Deliverables from big data
‣ Building prediction tools
‣ Building decision-making models
‣ Generate gures and reports
Lot of things need to do before preparing the deliverables.
fi
Data Preparation Python
‣ After collecting the data, we should import pandas as pd
‣
url = 'https://raw.githubusercontent.com/nytimes/
Data is typically organized as a table covid-19-data/master/us-states.csv'
Data = pd.read_csv(url)
‣ Understand what the variables are
‣ Manage column types Columns
‣ Handle missing values
Rows
Data Table and Type
‣ What do tables mean?
‣ Collection of di erent variables
‣ Collections of multiple observations
Di erent variables
Observation
ff
ff
Data Table and Type
‣ What do columns mean?
‣ Di erent features
‣ Di erent aspects of the data point
Di erent Features
ff
ff
ff
Data Table and Type
‣ How were the data collected?
‣ Province_state and country_region are collected by human
‣ Last_update time is collected by the clock
‣ Lat and Long_ are collected by GPS
‣ Con rmed and deaths are collected by human
Column Data type Description
Province_State String State name
Country_Region String Country name
Last_update Datetime Time of update
Lat Float Value of latitude
Long_ Float Value of longtitude
Con rmed Int Number of con rmed cases
Death Int Number of deaths
fi
fi
fi
Features and Observations
‣ Features represent the values collected in di erent manners
‣ Collected by di erent sensors
‣ Collected at di erent locations
‣ Collected for di erent objects
‣ Collected at di erent times
‣ …
‣ Observations represent the data point containing the corresponding features
‣ Can be di erent at di erent times
‣ Can be di erent for di erent individuals
‣ …
https://en.wikipedia.org/wiki/Moore%27s_law
ff
ff
ff
ff
ff
ff
ff
ff
ff
One Single Observation
‣Covid data
observation
• Observation
• Each observation is a
collection of feature values.
• Di erent observations refer to
di erent states and the
corresponding information
ff
ff
Managing Data Types
‣ Data has di erent “types”
‣ Numerica, categorial, dates, integers, string
‣ We need to manage some of them to make their information clearer
‣ Numerical methods cannot directly handle string-type data
Column Data type Description
Province_State String State name
Country_Region String Country name
Last_update String Time of update
Lat Float Value of latitude
Long_ Float Value of longtitude
Con rmed Int Number of con rmed cases
Death Int Number of deaths
fi
fi
ff
Managing Data Types: Dates
‣ The update time is stored as string
‣ To better make use of it, we need to convert the string to date time
String Datetime
“2023-01-23 04:31:38” → datetime(2023, 1, 23, 04, 31, 38)
One single date time string can be used to derive a number of new
features, e.g., year, month, day, …
Managing Data Types: Categorical
‣ Categorical type is general in many dataset
‣ categorical variable is a variable that can take on one of a limited, and
usually xed, number of possible values/groups
Province_State is a categorical variable as it
can only be one of the states or small
islands in the US.
fi
Managing Data Types: Categorical
‣ Managing categorical variables
‣ The number of groups may be large, merge some values into one group.
{Alabama, Alaska, American Samoa,
Arizona, Arkansas, California} → {Arizona,
California, and others}.
7 groups reduce to 3 groups.
Managing Data Types: Categorical
‣ Managing categorical variables
‣ A single categorical might encode multiple pieces of information, need
to decompose them to better organize information.
Location Province_State Country
California, US California US
Arizona, US Arizona US
Guangdong, China Guangdong China
British Columbia, Canada British Columbia Canada
Decompose categorical variable by using more features.
Managing Data Types: Categorical
‣ Managing categorical variables
‣ Converting categorical variables to numerical values
Using integers, ps code maps
state to numbers
fi
Managing Data Types: Categorical
‣ Managing categorical variables
‣ Converting categorical variables to numerical values
One-hot encoding
Correct?
• Each variable is represented
TRUE [1, 0, 0]
as an one-hot vector.
TRUE
FALSE [0, 1, 0] • Dimension/number of entries
FALSE of the vector is the number of
Don’t know Don’t know [0, 0, 1] categorical groups.
• One entry being 1 and
remaining being 0.
Handling Missing Values
‣ Missing value occurs frequently
‣ Caused by human error, sensor failures, data collection constraints
‣ If we can choose not to use it, it is typically marked as NA (or NaN).
‣ We can consider removing the entire rows with missing values
NaN NaN
Handling Missing Values
‣ Missing value occurs frequently
‣ If we need to use it for building models, we may need imputation
‣ Using the average from the observed cases
‣ Predict the missing value by prior knowledge (e.g., too large, too small)
‣ Predict the missing value using simple rules (mean, mode, etc.)
NaN The deaths of American
Samoa will be very
likely smaller than 100.
Handling Missing Values
‣ Missing value occurs frequently
‣ If we need to use it for building models, we may need imputation
‣ There are many more powerful methods for handling missing value by
using more data points.
‣ many modern machine learning models allow masking the missing
value, but implicit complete it during the model training.
Data Visualization
‣ Visualization can help you get a qualitative understand about the data
‣ Visualization can help you know what happens, good or bad
‣ See if the visualization matches your understanding
‣ Visualization can also help you understand the data from di erent
perspective
‣ Correlations between di erent features
‣ Trend of the data
ff
ff
Visualization Encodings
‣ Visualization represents data using graphical marks
‣ Di erent attributes of the marks encode data variables
‣ Marks allows us to make comparisons
‣ We can further reason about the data based on the visualization
x,y positions Di erent size Di erent color Di erent shape
Reasoning: the value will keep increasing in the future
ff
ff
ff
ff
Visualization Encodings
‣ Visualization represents data using graphical marks
‣ Some encodings are better for some variables types
‣ Size is better for continuous, not good for categorical
‣ For categorical, it would be better to use position
‣ Some encodings are easier to perceive
‣ Color is better than shape
‣ But do not use two many colors
Different Visualization
‣ Given a dataset, we have di erent visualization methods
‣ Using curve to see the trend import matplotlib.pyplot as plt
data_ca = data[data['state']=='California']
daily_cases = np.diff(data_ca['cases'].to_numpy())
plt.plot(daily_cases)
Daily covid cases in California
What can we conclude and reason about?
• There are 4 waves.
• The second wave is stronger than
the rst one.
• The daily cases will likely to
decrease in the future days.
Each point can be viewed as a data pair (x, y)
fi
ff
Different Visualization
‣ Given a dataset, we have di erent visualization methods
‣ Using curve to make comparisons
Daily covid cumulative cases in
California and Texas What can we conclude and reason about?
• California has more cases than
Texas.
• The di erence is become larger.
• The cumulative cases are still
increasing with a steady rate.
ff
ff
Different Visualization
‣ Given a dataset, we have di erent visualization methods
‣ Using pie chart to highlight the ratio of di erent components
Cumulative cases pie chart
What can we conclude and reason about?
• California cases take 12% of the all
cases.
• There are 4 states taking over 5% of
all cases.
• Can help us understand which
component takes the largest ratio.
ff
ff
Different Visualization
‣ Given a dataset, we have di erent visualization methods
‣ Using bar chart to highlight the comparison
Daily cases comparison at di erent date What can we conclude and reason about?
• 2020-08-12 has signi cantly more
cases than 2020-03-15.
• 2021-02-28 has fewer cases than
2020-08-12.
• Cases in Aug. 2020 will be more than
those in May 2020.
fi
ff
ff
Different Visualization
‣ Given a large number of observations, we can
‣ Use histogram to visualize the distribution density
‣ Count within bin ranges
Histogram of the daily cases What can we see and reason about?
• Mostly the daily cases are less than
20000.
• There are some days with cases
greater than 30000.
• Daily cases are mostly concentrating
around 2000-4000.
Different Visualization
‣ Given a large number of observations, we can
‣ Use histogram to visualize the distribution density
‣ Count within bin ranges
‣ Histogram of the daily cases
Normalizing to distribution: make total area is 1.
• Only the y-axis changes
• The shape is the same!
Different Visualization
‣ Given a large number of observations, we can
‣ Use scatter plot to visualize the distribution
‣ Also infer the relationships between di erent features
Distribution of cases and deaths What can we see and reason about?
• There are some outliers.
• Larger daily cases typically imply
larger daily deaths.
ff
Different Visualization
‣ Visualizations are now easy to generate using Python
Some details will be discussed in the tutorial!
Simple Data Stats
‣ Simple data statistics help summarize the property of data
‣ Statistics provide a quantitative understanding
‣ Statistics for a single variable
‣ mean, median
‣ variance, standard deviation
‣ Statistics for two variables
‣ Covariance
‣ Correlation
Statistics for Single Variables: Location Measures
‣ Given a large number of observations, we aim to understand
‣ What’s the typical value of them
‣ What’s the center of them
Daily Covid deaths in Aug. 2021 in California
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43
89 92 112 62 45 33 68 140 115 122 71 70 47 70 124]
Statistics for Single Variables: Location Measures
‣ Location measures: mean and median
‣ Mean denotes the average of samples
‣ Median denotes the sample at the center of the distribution,
i.e.,# samples < median = # samples > median
Given a set of values x1, x2, …, xn, the mean Given a set of values x1, x2, …, xn, the mode
is calculated using the following formula is calculated using the following formula
n • First sort the values from smallest to largest:
1
x(1), x(2), …, x(n)
n∑
x̄ = mean(x) = xi
i=1
• Then pick the middle values:
• If n is odd, pick x(n+1)/2
• If n is even, pick [xn/2 + xn/2+1]/2.
Statistics for Single Variables: Location Measures
‣ Location measures: mean and median
‣ Mean denotes the average of samples
‣ Median denotes the sample at the center of the distribution,
i.e.,# samples < median = # samples > median
Daily Covid deaths in Aug. 2021 in California
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43 89
92 112 62 45 33 68 140 115 122 71 70 47 70 124]
[-357 -5 10 13 23 31 33 34 39 43 45 45 47 51 52 54
Sorted sequence
62 63 68 70 70 71 89 92 93 112 115 121 122 124 140]
Mean = 50.6, median = 54
Statistics for Single Variables: Location Measures
‣ Location measures: mean and median
‣ Change the samples lead to the change of mean and median
‣ Median is more stable after removing a small number of outliers
Daily Covid deaths in Aug. 2021 in California Removing unreasonable samples
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43
89 92 112 62 45 33 68 140 115 122 71 70 47 70 124]
[10 13 23 31 33 34 39 43 45 45 47 51 52 54 62 63
Sorted sequence
68 70 70 71 89 92 93 112 115 121 122 124 140]
Mean = 66.6, median = 62
Mathematical Description of Mean and Median
‣ Find the center of a number of samples
‣ De nition of centers
‣ How to nd them
Mathematically, center is de ned as the point such that the average
distance to this point is minimized.
n
1
x n∑
x̄ = arg min dist(x, xi)
i=1
fi
fi
fi
Mathematical Description of Mean and Median
‣ Find the center of a number of samples
‣ De nition of centers
‣ Distance metric can be absolute deviation: | xi − xj |
‣ Distance metric can be squared deviation: (xi − xj) 2
Mean minimizes the average Median minimizes the average
distance using squared deviation distance using absolute deviation
n n
1 2 1
x n∑ x n∑
x̄ = arg min (x − xi) x̄ = arg min | x − xi |
i=1 i=1
fi
Visualization of the mean and median
‣ For di erent distribution, median and mean have di erent behaviors
As the data distribution become more balanced, mean and median will be closer.
ff
ff
Spread of the data
‣ How to characterize the spread of the data
Mean is the center of the data, spread characterizes the mean of deviation of a
data to the center.
Statistics for Single Variables: Spread Measures
‣ Given a large number of observations, the spread also shows
‣ The uncertainty among them
‣ The variation of the values around the mean
Daily Covid deaths in Aug. 2021 in California
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43
89 92 112 62 45 33 68 140 115 122 71 70 47 70 124]
Variance and Standard Deviation
‣ Spread measures: variance and standard deviation
‣ The average distance to the mean
‣ Recall 1 n
2
n∑
x̄ = arg min (x − xi)
x
i=1 When x̄ is estimated, we typically
‣ Variance is then denoted as use the following formula
n
1 n 1 2
n−1∑
2 Var = (x̄ − xi)
n∑
Var = (x̄ − xi)
i=1 i=1
‣ Standard deviation is the squared root of variance 1 n
2
n−1∑
std = (x̄ − xi)
n
Std has the same unit 1 2 i=1
∑
std = (x̄ − xi)
as the sample. n i=1
Variance and Standard Deviation
‣ Calculations
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43
89 92 112 62 45 33 68 140 115 122 71 70 47 70 124]
Variance = 6852, std = 83
[ 121 -5 93 54 39 13 45 10 51 -357 52 63 34 31 23 43
89 92 112 62 45 33 68 140 115 122 71 70 47 70 124]
Variance = 1232, std = 35
Variance and standard deviation are even more sensitive to the outliers.
Statistics for Two Variables
‣ Given a large number of observations of two features, we aim to
understand
‣ What’s the correlation between them
Daily Covid deaths in Sept. 2021 in California
[ 144 91 93 15 73 153 176 164 87 50 113 110 225 178 123 86 72 52
86 158 130 178 30 16 161 68 148 145 109 18]
Daily Covid cases in Sept. 2021 in California
[12022 9727 10551 7054 11141 9061 11053 9861 9154 7392 16056 8746 8760
12292 10053 8263 7068 8696 6669 7557 9726 7495 1555 1179 19950 5305 7143
9241 8033 1363]
Statistics for Two Variables
• Deaths and cases are positively
correlated.
• How to handle the correlation?
Covariance
‣ Covariance: mean of the multiplication of deviations
n
1
n∑
cov(x, y) = (xi − x̄)(yi − ȳ)
i=1
‣ Consider extreme cases:
‣ What happens if xi = yi?
‣ What happens if xi = − yi ?
‣ What are the units of the covariance?
Correlation
‣ Correlation
‣ Covariance depends on the scaling of the variables
‣ But correlation does not
cov(x, y)
corr(x, y) =
std(x)std(y)
‣ Consider extreme cases:
‣ What happens if xi = yi?
‣ What happens if xi = − yi ?
‣ What is the unit of the correlation? [-1, 1]
Covariance and Correlations
‣ Calculations
[ 144 91 93 15 73 153 176 164 87 50 113 110 225 178 123 86 72 52
86 158 130 178 30 16 161 68 148 145 109 18]
[12022 9727 10551 7054 11141 9061 11053 9861 9154 7392 16056 8746 8760
12292 10053 8263 7068 8696 6669 7557 9726 7495 1555 1179 19950 5305 7143
9241 8033 1363]
Covariance = 115786, correlation = 0.55
Hard to interpret Easier to interpret
Correlation is more informative than covariance.
Summary
‣ Data transformations
‣ Features, observations, data type, tidy data
‣ Data visualizations
‣ Markers, plots, comparisons
‣ Simple Statistics
‣ Mean, median, variance, correlation
