My Internship Document
My Internship Document
On
“DATA SCIENCE”
Submitted By
SHAIK SHAHID(Y21IT117)
NOVEMBER-2023
BONAFIDE CERTIFICATE
This is to certify that this internship report “DATA SCIENCE USING PYTHON”
is the Bonafede work of “SHAIK SHAHID (Y21IT117)” who have carried out the
work under my supervision and submitted in partial fulfillment for the award of
INTERNSHIP (IT-353) during the year 2023-2024.
ii
INDEX
Sr. Topic Page No.
No.
1 Title 1
2 Index 3
3 Training Certificate 4
4 Declaration 6
5 Acknowledgement 7
6 About Training 8
7 About Internshala 8
8 Objectives 8
9 Data Science 8-10
10 My Learnings 10-25
11 Final Project 26-32
12 Reason for choosing Data Science 33
13 Learning Outcome 34
14 Scope in Data Science 35
15 Results 36
iii
The successful completion of any task would be incomplete without proper
suggestions, guidance, and environment. The combination of these three factors acts like
backbone to our internship “DATA SCIENEC USING PYTHON”.
iv
I would like to express our gratitude to the Management of R.V.R&J.C.
COLLEGE OF ENGINEERING for providing us with a pleasant environment and
excellent lab facility.
I regard my sincere thanks to our Principal, Dr. Kolla Srinivas, for providing
support and a stimulating environment.
I would like to express our special thanks to our guide Smt. B. Manasa,
Assistant Professor who helped us in doing the internship successfully.
SHAIK SHAHID
(Y21IT117)
v
DECLARATION
I hereby certify that the work which is being presented in the report entitled
“Data Science” in fulfilment of the requirement for completion of industrial
training in Department of
SHAIK SHAHID
Y21IT117
vi
ACKNOWLEDGEMENT
The work in this report is an outcome of continuous work over a period and drew
intellectual support from Internshala and other sources. I would like to articulate
our profound gratitude and indebtedness to Internshala helped us in completion of
the training. I am thankful to Internshala Training Associates for teaching and
assisting me in making the training successful.
SHAIK SHAHID
Y21IT117
1. ABOUT TRAINING
3. OBJECTIVES
To explore, sort and analyse mega data from various sources to take advantage of them and
reach conclusions to optimize business processes and for decision support.
Examples include machine maintenance or (predictive maintenance), in the fields of marketing
and sales with sales forecasting based on weather.
4. DATA SCIENCE
Data Science as a multi-disciplinary subject that uses mathematics, statistics, and computer
science to study and evaluate data. The key objective of Data Science is to extract valuable
information for use in strategic decision making, product development, trend analysis, and
forecasting.
Data Science concepts and processes are mostly derived from data engineering, statistics,
programming, social engineering, data warehousing, machine learning, and natural language
processing. The key techniques in use are data mining, big data analysis, data extraction and
data retrieval.
Data science is the field of study that combines domain expertise, programming skills, and
knowledge of mathematics and statistics to extract meaningful insights from data. Data science
practitioners apply machine learning algorithms to numbers, text, images, video, audio, and
more to produce artificial intelligence (AI) systems to perform tasks that ordinarily require
human intelligence. In turn, these systems generate insights which analysts and business users
can translate into tangible business value.
1. The first step of this process is setting a research goal. The main purpose here is making
sure all the stakeholders understand the what, how, and why of the project.
viii
2. The second phase is data retrieval. You want to have data available for analysis, so this
step includes finding suitable data and getting access to the data from the data owner.
The result is data in its raw form, which probably needs polishing and transformation
before it becomes usable.
3. Now that you have the raw data, it’s time to prepare it. This includes transforming the data
from a raw form into data that’s directly usable in your models. To achieve this, you’ll
detect and correct different kinds of errors in the data, combine data from different data
sources, and transform it. If you have successfully completed this step, you can progress
to data visualization and modeling.
4. The fourth step is data exploration. The goal of this step is to gain a deep understanding
of the data. You’ll look for patterns, correlations, and deviations based on visual and
descriptive techniques. The insights you gain from this phase will enable you to start
modeling.
5. Finally, we get to the sexiest part: model building (often referred to as “data modeling”
throughout this book). It is now that you attempt to gain the insights or make the
predictions stated in your project charter. Now is the time to bring out the heavy guns,
but remember research has taught us that often (but not always) a combination of simple
models tends to outperform one complicated model. If you’ve done this phase right,
you’re almost done.
6. The last step of the data science model is presenting your results and automating the
analysis, if needed. One goal of a project is to change a process and/or make better
decisions. You may still need to convince the business that your findings will indeed
change the business process as expected. This is where you can shine in your influencer
role. The importance of this step is more apparent in projects on a strategic and tactical
level. Certain projects require you to perform the business process over and over again,
so automating the project will save time.
ix
5. MY LEARNINGS
1) INTRODUCTION TO DATA SCIENCE
• Overview & Terminologies in Data Science •
Applications of Data Science
➢ Unfamiliar detection (fraud, disease, etc.)
➢ Automation and decision-making (credit worthiness, etc.)
➢ Classifications (classifying emails as “important” or “junk”)
➢ Forecasting (sales, revenue, etc.)
➢ Pattern detection (weather patterns, financial market patterns, etc.)
➢ Recognition (facial, voice, text, etc.)
➢ Recommendations (based on learned preferences, recommendation engines can
refer you to movies, restaurants and books you may like)
x
• Capture: data entry, signal reception, data extraction
• Maintain: Data cleansing, data staging, data processing.
• Process: Data mining, clustering/classification, data modelling
• Communicate: Data reporting, data visualization
• Analyse: Predictive analysis, regression
The field of bringing insights from data using scientific techniques is called data science.
Applications
xi
Spectrum of Business Analysis
Big Data
What is likely to
Complexity happen?
Predictive Analysis
What’s happening
now?
Dashboards
Why did it
happen?
Detective Analysis
What happened?
Reporting
xii
Big Data
Stage where complexity of handling data gets beyond the traditional system.
Can be caused because of volume, variety or velocity of data. Use specific tools to
analyse such scale data.
Recommendation System
Example-In Amazon recommendations are different for different users according to their past
search.
• Social Media
1. Recommendation Engine
2. Ad placement
3. Sentiment Analysis
• How google and other search engines know what are the more relevant results for our
search query?
1. Apply ML and Data Science
2. Fraud Detection
3. AD placement
xiii
Python Introduction
Python is an interpreted, high-level, general-purpose programming language. It has efficient
high-level data structures and a simple but effective approach to object-oriented programming.
Python’s elegant syntax and dynamic typing, together with its interpreted nature, make it an
ideal language for scripting and rapid application development in many areas on most
platforms.
Why Python???
4. Extensive Packages.
• UNDERSTANDING OPERATORS:
Theory of operators: - Operators are symbolic representation of Mathematical tasks.
• VARIABLES AND DATATYPES:
Variables are named bounded to objects. Data types in python are int (Integer), Float,
Boolean and strings.
• CONDITIONAL STATEMENTS:
If-else statements (Single condition)
If- elif- else statements (Multiple Condition)
• LOOPING CONSTRUCTS:
For loop
xiv
• FUNCTIONS:
Functions are re-usable piece of code. Created for solving specific problem.
Two types: Built-in functions and User- defined functions.
Functions cannot be reused in python.
• DATA STRUCTURES:
LISTS: A list is an ordered data structure with elements separated by comma and
Statistics
Descriptive Statistic
Mode
It is a number which occurs most frequently in the data series.
It is robust and is not generally affected much by addition of couple of
new values. Code import pandas as pd data=pd.read_csv( "Mode.csv")
//reads data from csv file
data.head() //print first five lines
mode_data=data['Subject'].mode() //to take mode of
subject column print(mode_data) Mean
import pandas as pd data=pd.read_csv( "mean.csv")
//reads data from csv file
data.head() //print first five lines
mean_data=data[Overallmarks].mean() //to take mode of subject column
print(mean_data)
xv
Median
Absolute central value of data set.
import pandas as pd data=pd.read_csv( "data.csv")
//reads data from csv file
data.head() //print first five lines
median_data=data[Overallmarks].median() //to take mode of subject column
print(median_data)
Types of variables
• Continous – Which takes continuous numeric values. Eg-marks
• Categorial-Which have discrete values. Eg- Gender
• Ordinal – Ordered categorial variables. Eg- Teacher feedback
• Nominal – Unorderd categorial variable. Eg- Gender
Outliers
Any value which will fall outside the range of the data is termed as a outlier. Eg- 9700 instead
of 97.
Reasons of Outliers
• Types-During collection. Eg-adding extra zero by mistake.
• Measurement Error-Outliers in data due to measurement operator being faulty.
• Intentional Error-Errors which are induced intentionally. Eg-claiming smaller amount of
alcohol consumed then actual.
• Legit Outlier—These are values which are not actually errors but in data due to
legitimate reasons.
Eg - a CEO’s salary might actually be high as compared to other employees.
Interquartile Range (IQR)
Is difference between third and first quartile from last. It is robust to outliers.
xvi
Histograms
Histograms depict the underlying frequency of a set of discrete or continuous data that are
measured on an interval scale.
import pandas as pd
histogram=pd.read_csv(histogram.csv)
import matplotlib.pyplot as plt
%matplot inline plt.hist(x= 'Overall
Marks',data=histogram) plt.show()
Inferential Statistics
Inferential statistics allows to make inferences about the population from the sample data.
Hypothesis Testing
Hypothesis testing is a kind of statistical inference that involves asking a question, collecting
data, and then examining what the data tells us about how to proceed. The hypothesis to be
tested is called the null hypothesis and given the symbol Ho. We test the null hypothesis
against an alternative hypothesis, which is given the symbol Ha.
T Tests
When we have just a sample not population statistics.
Use sample standard deviation to estimate population standard deviation.
T test is more prone to errors, because we just have samples.
xvii
Z Score
The distance in terms of number of standard deviations, the observed value is away from
mean, is standard score or z score.
distribution.
Predictive Modelling
Making use of past data and attributes we predict future using this
data. Eg-
Past Horror Movies
Future Unwatched Horror Movies
xviii
Predicting stock price movement
1. Analysing past stock prices.
Types
1. Supervised Learning
Supervised learning is a type algorithm that uses a known dataset (called the training
dataset) to make predictions. The training dataset includes input data and response
values.
• Regression-which have continuous possible values. Eg-Marks
• Classification-which have only two values. Eg-Cancer prediction is either 0 or 1.
2. Unsupervised Learning
Unsupervised learning is the training of machine using information that is neither
classified nor. Here the task of machine is to group unsorted information according to
similarities, patterns and differences without any prior training of data.
• Clustering: A clustering problem is where you want to discover the inherent
groupings in the data, such as grouping customers by purchasing behaviour.
• Association: An association rule learning problem is where you want to discover
rules that describe large portions of your data, such as people that buy X also tend
to buy Y.
2. Hypothesis Generation
3. Data Extraction/Collection
5. Predictive Modelling
6. Model Development/Implementation
xix
Problem Definition
Identify the right problem statement, ideally formulate the problem mathematically.
Hypothesis Generation
List down all possible variables, which might influence problem objective. These variables
should be free from personal bias and preferences.
Quality of model is directly proportional to quality of hypothesis.
Data Extraction/Collection
Collect data from different sources and combine those for exploration and model building.
While looking at data we might come across new hypothesis.
Data Exploration and Transformation
Data extraction is a process that involves retrieval of data from various sources for further data
processing or data storage.
Steps of Data
Extraction • Reading
the data Eg- From csv
file
• Variable identification
• Univariate Analysis
• Bivariate Analysis
• Missing value treatment
• Outlier treatment
• Variable Transformation
Variable Treatment
It is the process of identifying whether variable is
1. Independent or dependent variable
xx
2. Continuous or categorical variable
Bivariate Analysis
• When two variables are studied together for their empirical relationship.
• When you want to see whether the two variables are associated with each other.
• It helps in prediction and detecting anomalies.
Missing Value Treatment
Reasons of missing value
1. Non-response – Eg-when you collect data on people’s income and many choose not to
answer. 2. Error in data collection. Eg- Faculty data
3. Error in data reading.
Types
1. MCAR (Missing completely at random): Missing values have no relation to the variable in
exist and the variables other than the variables in which missing values exist.
3. MNAR (Missing not at random): Missing values have relation to the variable in which
xxi
2. Isnull()
Output will we in True or False
Different methods to deal with missing values
1. Imputation
Outlier Treatment
Reasons of Outliers
1. Data entry Errors
2. Measurement Errors
3. Processing Errors
Types of Outlier
Univariate
Analysing only one variable for outlier.
Eg – In box plot of height and weight.
Weight will we analysed for outlier
Bivariate
Analysing both variables for outlier.
Eg- In scatter plot graph of height and weight. Both will we analysed.
Identifying Outlier
xxii
Graphical Method
• Box Plot
Scatter Plot
Formula Method
Using Box Plot
< Q1 - 1.5 * IQR or > Q3+1.5 * IQR
Where IQR= Q3 – Q1
Q3=Value of 3rd quartile
Q1=Value of 1st quartile
Treating Outlier
1. Deleting observations
2. Transforming and binning values
3. Imputing outliers like missing values
xxiii
4. Treat them as separate Variable Transformation Is the process by which-
1. We replace a variable with some function of that variable. Eg – Replacing a
variable x with its log.
2. We change the distribution or relationship of a variable with others. Used to
–
1. Change the scale of a variable
Common methods of Variable Transformation – Logarithm, Square root, Cube root, Binning,
etc.
Model Building
It is a process to create a mathematical model for estimating / predicting the future based on
past data.
Eg-
A retail wants to know the default behaviour of its credit card customers. They want to predict
the probability of default for each customer in next three months.
• Probability of default would lie between 0 and 1.
• Assume every customer has a 10% default rate.
Probability of default for each customer in next 3 months=0.1
It moves the probability towards one of the extremes based on attributes of past information.
A customer with volatile income is more likely (closer to) to default.
A customer with healthy credit history for last years has low chances of default (closer to 0).
1. Algorithm Selection
2. Training Model
3. Prediction / Scoring
xxiv
Algorithm Selection
Example-
Algorithms
• Logistic Regression
• Decision Tree
• Random Forest
Training Model
xxv
• Train
Past data (known dependent variable).
Used to train model.
• Test
Future data (unknown dependent
variable) Used to score.
Prediction / Scoring
It is the process to estimate/predict dependent variable of train data set by applying model
rules.
We apply training learning to test data set for prediction/estimation.
10
0
0 1 2 3 4 5 6 7 8 9
Logistic Regression
Logistic regression is a statistical model that in its basic form uses a logistic function to model a
binary dependent variable, although many more complex extensions exist.
K-means clustering is a type of unsupervised learning, which is used when you have unlabelled
data (i.e., data without defined categories or groups). The goal of this algorithm is to find
groups in the data, with the number of groups represented by the variable K. The algorithm
xxvii
works iteratively to assign each data point to one of K groups based on the features that are
provided. Data points are clustered based on feature similarity.
6. FINAL PROJECT
PREDICTING IF CUSTOMER BUYS TERM DEPOSIT
Problem Statement:
Your client is a retail banking institution. Term deposits are a major source of income for a bank.
A term deposit is a cash investment held at a financial institution. Your money is invested for
an agreed rate of interest over a fixed amount of time, or term. The bank has various outreach
xxviii
plans to sell term deposits to their customers such as email marketing, advertisements,
telephonic marketing and digital marketing.
Telephonic marketing campaigns still remain one of the most effective ways to reach out to
people. However, they require huge investment as large call centers are hired to actually
execute these campaigns. Hence, it is crucial to identify the customers most likely to convert
beforehand so that they can be specifically targeted via call.
You are provided with the client data such as: age of the client, their job type, their marital
status, etc. Along with the client data, you are also provided with the information of the call such
as the duration of the call, day and month of the call, etc. Given this information, your task is to
predict if the client will subscribe to term deposit.
Data Dictionary: -
xxix
Prerequisites:
We have the following files:
• train.csv: This dataset will be used to train the model. This file contains all the client and
call details as well as the target variable “subscribed”.
• test.csv: The trained model will be used to predict whether a new set of clients will
subscribe the term deposit or not for this dataset. TEST.csv file: -
xxx
TRAIN.csv file: -
xxxi
Problem Description
Provided with following files: train.csv and test.csv.
Use train.csv dataset to train the model. This file contains all the client and call details as well
as the target variable “subscribed”. Then use the trained model to predict whether a new set
of clients will subscribe the term deposit.
xxxii
xxxiii
xxxiv
xxxv
Reason for choosing data science
Data Science has become a revolutionary technology that everyone seems to talk about.
Hailed as the ‘sexiest job of the 21st century’. Data Science is a buzzword with very few people
knowing about the technology in its true sense.
While many people wish to become Data Scientists, it is essential to weigh the pros and cons of
data science and give out a real picture. In this article, we will discuss these points in detail and
provide you with the necessary insights about Data Science.
Advantages: - 1.
It’s in Demand
2. Abundance of Positions
Disadvantages: -
1. Mastering Data Science is near to impossible
xxxvi
Learning Outcome
After completing the training, I am able to:
• Develop relevant programming abilities.
• Demonstrate proficiency with statistical analysis of data.
• Develop the skill to build and assess data-based model.
• Execute statistical analysis with professional statistical software.
• Demonstrate skill in data management.
• Apply data science concepts and methods to solve problem in real-world contexts and
will communicate these solutions effectively.
xxxvii
7. SCOPE IN DATA SCIENCE FIELD
Few factors that point out to data science’s future, demonstrating compelling reasons why it is
crucial to today’s business needs are listed below:
xxxviii
Data is generated by everyone on a daily basis with and without our notice. The interaction we
have with data daily will only keep increasing as time passes. In addition, the amount of data
existing in the world will increase at lightning speed. As data production will be on the rise, the
demand for data scientists will be crucial to help enterprises use and manage it well.
8. RESULTS
In this complete 6 weeks training I successfully learnt about DATA SCIENCE. Also, now I’m
able to perform data analysis using python. I also attempted various quizzes and assignments
provided for periodic evaluation during 6 weeks and completed this training with 100% score in
Final Test.
xxxix
9. TRAINING CERTIFICATE
***
xl
xli