###QUIZ 2 Instructions
- Fill all the incomplete functions. Strictly follow the function specs.
- Do not copy or plagiarize. IIPE,VIZAG has a very strict policy against
plagiarism
Download and read file
### Download data from google drive. You need not mess with this code.
import requests
def download_file_from_google_drive(id, destination):
URL = "https://docs.google.com/uc?export=download"
session = requests.Session()
response = session.get(URL, params = { 'id' : id }, stream = True)
token = get_confirm_token(response)
if token:
params = { 'id' : id, 'confirm' : token }
response = session.get(URL, params = params, stream = True)
save_response_content(response, destination)
def get_confirm_token(response):
for key, value in response.cookies.items():
if key.startswith('download_warning'):
return value
return None
def save_response_content(response, destination):
CHUNK_SIZE = 32768
with open(destination, "wb") as f:
for chunk in response.iter_content(CHUNK_SIZE):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
if __name__ == "__main__":
file_id = '1e_Azf9zGvSWsDhM9PP2sfMNKC72-iWAK'
destination = 'data.txt'
download_file_from_google_drive(file_id, destination)
with open('data.txt', 'r') as f:
data_raw = f.readlines()
1. Data preparation
Now the entire data is stored in the list data_raw.
Every line in the file is a different element of the list.
First let us look at the first five elements of the list.
1.1
Write a function that returns first five elements of the list if length of list is greater than or equal
to 5 and None value otherwise.
def first_five_in_list(l):
"""
Inputs:
l: Python list
Outputs:
l_5 : python list, first five elements of list if length of list greater
than 5; None otherwise
"""
### Your code here
return l_5
1.2
def remove_trailing_newlines(s):
"""
Function that removes all trailing newlines at the end of it
Inputs:
s : string
Outputs:
s_clean : string, string s but without newline characters at the end
"""
### Write your code here
return s_clean
If we apply remove_trailing_newlines to first element of data_ra
w, we get
You can see that the newline at the end has disappeared.
1.3
But we now we need to apply this function to the whole list.
Write a function named mapl, that takes two arguments - a function on elements of type t and a
list l of elements of type t and applies the function over all elements of the list l and returns them
as a list.
def mapl(f, l):
"""
Function that applies f over all elements of l
Inputs:
f : function, f takes elements of type t1 and returns elements of type
t2
l : list, list of elements of type t1
Ouptuts:
f_l : list, list of elements of type t2 obtained by applying f over ea
ch element of l
"""
### Write your code here
return f_l
Now we can use mapl to apply remove_trailing_newlines to all lines in data_raw
data_clean = mapl(remove_trailing_newlines, data_raw)
First five elements of data_clean look like this:
This is a dataset of text messages. We have to classify this into spam or ham. Ham means non-
spam relevant text messages. More details can be found here -
http://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection
You can see that each line starts by specifying whether the message is ham or spam and then there
is a tab character, \t followed by actual text message.
Now we need to split the lines to extract the two components - data label (ham or spam) and data
sample (the text message).
1.4
Write a function split_at_s that takes two strings - text and s.
It splits the string text into two parts at the first occurence of s.
Then it wraps both parts in a tuple and returns it.
def split_at_s(text, s):
"""Function that splits string text into two parts at the first occurenc
e of string s
Inputs:
text: string, string to be split
s : string, string of length 1 at which to split
Outputs:
split_text: tuple of size 2, contains text split in two (do not includ
e the string s at which split occurs in any of the split parts)
"""
### Write your code here
return split_text
Python has a very handy feature used to define short functions called lambda expressions. This is
from official python docs
Use lambda expressions and split_at_s to write a function, split_at_tab that takes only one
argument - text and splits at the first occurence of '\t' character. (If you can't understand
lambda expressions, just define the function in the ususal way)
### Write your code here
1.5
Now apply split_at_tab function over the elements of list data_clean and assign it to
variable named data_clean2.
#### Write your code here
After splitting at '\t' character, one data point looks like
this -
Now let us remove the punctuations in an sms.
import string
def remove_punctuations_and_lower(text):
"""Function that removes punctuations in a text
Inputs:
text: string
Outputs:
text_wo_punctuations
"""
return (text.translate(str.maketrans("","", string.punctuation))).lower(
)
1.6
Now use the function remove_punctuations to remove punctuations from the text part of all of
the tuples in data_clean2 and assign it to a variable named dataset
### Write your code here
First 5 elements of dataset look like this now.
Now let us count number of occurences of ham and spam in our dataset.
1.7
Write a function counter that takes two arguments -
a list l of elements of type t
a function f:t→u (means f takes an argument of type t and returns values of type u)
Counter returns a dictionary whose keys are u1,u2,…etc - unique values of type u obtained by
applying f over elements of l.
The values corresponding to the keys are the the number of times a particular key say u1 is
obtained when we apply f over elements of l
def counter(l, f):
"""
Function that returns a dictionary of counts of unique values obtained b
y applying f over elements of l
Inputs:
l: list; list of elements of type t
f: function; f takes arguments of type t and returns values of type u
Outputs:
count_dict: dictionary; keys are elements of type u, values are ints
"""
### Write your code here
return count_dict
1.8
Write a function named aux_func that can be passed to counter along with the list dataset to
get a dictionary containing counts of ham and spam
#### Write your code here
The counts of ham and spam as we can see are {'ham': 4827, 'spam': 747}
Now let us split our dataset into training and test sets. We'll first shuffle the elements of the
dataset, then we'll use 80% of data for training and 20% for testing.
1.9
Write a function that takes a list, randomly shuffles it and then returns it.
Hint: Use the random library of python - https://docs.python.org/3/library/random.html
Double-click (or enter) to edit
def random_shuffle(l):
"""Function that returns a randomly shuffled list
Inputs:
l: list
Outputs:
l_shuffled: list, contains same elements as l but randomly shuffled
"""
### Write your code here
return l_shuffled
1.10
Now split the shuffled list. Take 80% (4459) samples and assign them to a variable called
data_train . Put the rest in a variable called data_test
### Write your code here
2.Data Modeling
We shall use Naive Bayes for modelling our classifier. You can read about Naive Bayes from
here (https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes). But you
don't actually need to read it, because we are going to move step by step in building this
classifier.
First we need to find the probabilities P(wi|C)
We read P(A|B) as probability of event A, given event B.
P(wi|C) is probability that word wi occurs in the sms given that the sms belongs to class C
where C can be either spam or ham .
But we will be finding P~(wi|C) which is the smoothed probability function to take care of
words with 0 probabilities that may cause problems.
P~(wi|C)=Number of occurences of wi in all samples of class C+1Total number of words in all samples of class
C + Vocabulary size
2.1
Find the vocabulary - list of unique words in all smses of data_train and assign it to the
variable vocab
### Write your code here
2.2
For every word wi in vocab, find the count (total number of occurences) of wi in all smses of
type spam. Put these counts in a dictionary and assign it to a variable named dict_spam where
key is the word wi and value is the count.
In a similar way, create a variable called dict_ham which contains counts of each word in
vocabulary in smses of type ham. (This is only w.r.t samples in data_train)
### Write your code here
2.3
For every word wi in vocab, find the smoothed probability P~(wi| spam ) and put in a
dictionary named dict_prob_spam. In a similar way, define the dictionary dict_prob_ham
which contains smoothed probabilities P~(wi| ham )
### Write your code here
3. Prediction
We need to test our model on data_test . For each sample of data_test, prediction procedure
is as follows:
For all words common to the sample and vocabulary, find spam_score and ham_score
If spam_score is higher than ham_score, then we predict the sample to be spam and vice
versa.
spam_score = P(spam)∗P~(w1| spam )∗P~(w2| spam )∗… where w1,w2,… are
words which occur both in the test sms and vocabulary.
Similary, ham_score = P(ham)∗P~(w1| ham )∗P~(w2| ham )∗… where w1,w2,…
are words which occur both in the test sms and vocabulary.
Here P(spam)=Number of samples of type spam in training setTotal number of samples in training set
Similarly, P(ham)=Number of samples of type ham in training setTotal number of samples in training
set
(Note: The above is prediction procedure for a single sample in data_test)
Write a function predict which does this.
3.1
def predict(text, dict_prob_spam, dict_prob_ham, data_train):
"""Function which predicts the label of the sms
Inputs:
text: string, sms
dict_prob_spam: dictionary, contains dict_prob_spam as defined above
dict_prob_spam: dictionary, contains dict_prob_ham as defined above
data_train: list, list of tuples of type(label, sms), contains trainin
g dataset
Outputs:
prediction: string, one of two strings - either 'spam' or 'ham'
"""
### Write your code here
return prediction
3.2
Now find accuracy of the model. Apply function predict to all the samples in data_test.
accuracy=number of correct predictionssize of test set
Write the function accuracy which applies predict to all samples in data_test and returns
accuracy
def accuracy(data_test, dict_prob_spam, dict_prob_ham, data_train):
"""Function which finds accuracy of model
Inputs:
data_test: list, contains tuples of data (label, sms)
dict_prob_spam: dictionary, contains dict_prob_spam as defined above
dict_prob_spam: dictionary, contains dict_prob_ham as defined above
data_train: list, list of tuples of type(label, sms), contains trainin
g dataset
Outputs:
accuracy: float, value of accuracy
"""
### Write your code here
return accuracy
