Data quality issue
Incorrect rows
Summary rows
Extra rows
Missing Column Names
Fix rows and columns Inconsistent column names
Unnecessary columns
Columns containing Multiple data values
No Unique Identifier
Misaligned columns
Disguised Missing values
Missing Values Significant number of Missing values in a row/column
Partial missing values
Non-standard units
Values with varying Scales
Standardise Numbers
Over-precision
Remove outliers
Extra characters
Different cases of same words
Standardise Text
Non-standard formats
Encoding Issues
Incorrect data types
Correct values not in list
Fix Invalid Values
Wrong structure
Correct values beyond range
Validate internal rules
Duplicate data
Filter Data
Extra/Unnecessary rows
Filter Data
Columns not relevant to analysis
Dispersed data
Examples
Header rows, footer rows
Total, subtotal rows
Column numbers, indicators, blank rows
Column names as blanks, NA, XX etc.
X1, X2,C4 which give no information about the column
Unidentified columns, irrelevant columns, blank columns
E.g. address columns containing city, state, country
E.g. Multiple cities with same name in a column
Shifted columns
blank strings, "NA", "XX", "999" etc
Missing time zone, century etc
Convert lbs to kgs, miles/hr to km/hr
A column containing marks in subjects, with some subject
marks out of 50 and others out of 100
4.5312341 kgs, 9.323252 meters
Abnormally High and Low values
Common prefix/suffix, leading/trailing/multiple spaces
Uppercase, lowercase, Title Case, Sentence case, etc
23/10/16 to 2016/10/20
“Modi, Narendra" to “Narendra Modi"
CP1252 instead of UTF-8
Number stored as a string: "12,300"
Date stored
String storedasasaastring:
number:"2013-Aug"
PIN Code "110001" stored as
110001
Non-existent country, PIN code
Phone number with over 10 digits
Temperature less than -273° C (0° K)
Gross sales > Net sales
Date of delivery > Date of ordering
If Title is "Mr" then Gender is "M"
Identical rows, rows where some columns are identical
Rows that are not required in the analysis. E.g if
observations before or after a particular date only are
required for analysis, other rows become unnecessary
Columns that are not needed for analysis e.g. Personal
Detail columns such as Address, phone column in a
dataset for
Parts of data required for analysis stored in different files
or part of different datasets
How to resolve
Delete
Delete
Delete
Add the column names
Add column names that give some information
about the data
Delete
Split columns into components
Combine columns to create unique identifiers
e.g. combine City with the State
Align these columns
Set values as missing values
Delete rows, columns
Fill the missing values with the correct value
Standardise the observations so all of them
have the same consistent units
Make the scale common. E.g. a percentage scale
Standardise precision for better presentation of
data. 4.5312341 kgs couldbe presented as 4.53
kgs
Correct if by mistake else Remove
Remove the extra characters
Standadise the case/bring to a common case
Correct the format/Standardise format for
better readability in R
Encode unicode properly
Convert to Correct data type
Delete the invalid values, treat as Missing
Deduplicate Data/ Remove duplicated data
Filter rows to keep only the relevant data.
Filter columns-Pick columns relevant to analysis
Bring the data together, Group by required keys,
aggregate the rest
