Of Data Monsters

Seinfeld RNN

Tue, 01 Sep 2015 12:00:00 +0000

Char-rnn generated Seinfeld Episode

This post explores character-level multi-layer Recurrent Neural Network as language models. It uses Andrej Karpathy’s char-rnn library char-rnn. He has a detailed blog post on it, explaining various experiments with these RNNs. There have been some interesting experiments based on it, from generating cooking recipes, bible rewrite and Obama speeches to rap and music composition. I wondered how well will it generate sample scripts of Seinfeld episodes if I train the model on all 180 episode scripts. The char-rnn code is pretty straight forward to use and Andrej does a really good job explaining the tuning parameters.

Results were pretty interesting.

Seed “JERRY”

JERRY: Hearing anyone amazing. (They both sit down, but on one.) DOCTOR: I explain the TV Marya Henungician. JERRY: I mean she put some guy down and stor it like that. Uh, could you get over the point? GEORGE: I think he's good, and find out worth me. GEORGE: No, no. GEORGE: (eating a TV, Newman spans the steps.) DARRYL: I have catched apartment roments again. It's just convanted. I forgot about the bus KEVIN: Now for your keys. HELEN: So, shes thinking of a damn assumber!! JERRY: Wow, I have holded jokes. JOE: What are you gonna do? He's going to like that? (scene ends) \[setting: Jerry's Apartment\] (Jerry's car pets of George's trip, Allison Poppie's guest. Kramer tells Jerry he puts his head on.) PETERMAN: I'm a old man! Now, please? 7, no. ELAINE: George, what do you call it in the Wendy?" ELAINE: "...embhe.. JERRY: Hey, hey, what kind of rubber guy came down we have, chotge GEORGE: No yesale. JERRY: (Opens the door) Is it Tive Vemma. so, how can I tell you? GEORGE: No Th supposed to be the stoundation. It's committee! No one will be a drop here. Talk like An idea. JERRY: Not been on it? HELEN: No, Mister Costanza, what do you mean, they were still an about forget the pilot. (Kramer walks over to George's sarcastical suit for the next bar. George goes with Manya) KRAMER: Well, Im not taking this turkey now. \[Jerry's Apartment\] (I think I join us with her here slips where the corlandlas burns most of the gang. GEORGE: About it... (Difficults exit) JERRY: Ogh one of those feight, fish and out, I wasn't like, all they got a week, and that I got uh.. Aho. CLERK: I have no place struggle... (Talking with her broke gesture to George. He's got a wacket on the door.) GEORGE: Hi. No how were you? GEORGE: Police. She wanted a friend of the bus now. That's so nice for a tree but the bucket should be pretty now. JERRY: Why did you go out and I'd go? It was now, but then she keeps slept with a piano in the chicken

Seed “ELAINE”

ELAINE: George, how much did you know? I can't do this. GEORGE: So what did she think? JERRY: You know, much so much dishouse? FRANK: We got a little nuty in the building. Hey, the can's kneephap on it. Here we go. I'll put it on. It means so would it have done the steams. [Scene moves around desperate with sitting beside a handicap room watching theatro. All right and she's George and George in the next bar.] KRAMER: Others People hear problems, Just a little because he can work off tried clisic. (goes to motion) Wigness Get for are doing charatter? JERRY: (still not staying how putting your seats and George, getting on his shirt, with making back their hole.) KRAMER: Hey, ah! It's honest! GEORGE: Yight. JERRY: Oh, my god! FELDMAN: Thats all me to splat around the intercam on when you bit it? DOCTOR: . . . JERRY: I would love you. JERRY: They've met me again! JERRY: So you would be the ones with things of this is like a homeer overlightevies. (Jerry and Kramer grab on, George pulls his hand to the while.) What punk? JERRY: What's your day in the Prusor sold, so tell him why you want to leave? ELAINE: Look, How did you do that? NEWMAN: Oh, hi when what, I beat with you. Blood, min have sex, it would be not more. We found any party to meet her. CLERK: Maybe you can talk to him. ELAINE: Party? Dad, they could, and, unfortunately, the friend, I'm going to the New York Bookstore on the couch, sell your new couch. JERRY: Hated the doctor?!! (Clicks through his finger back) KRAMER: Well, I mean, I told you!...uh... Could you dump up my mail test? BICKES: Ya.. (Kramer opens the door.) JERRY: Come on, of course. I'm going to be fun. KRAMER: Yeah. JERRY: Yeah... KRAMER: You got a thing? Mailragh... Paul? [Jerry and George at Dleas and Jerry's apartment] Notice: "THE TEACHER MOUTH AT, THE CULATS THE THE PASTERING TURCING MACHINE, GEORGE tHEBERBS CAR AND THE SCENE DOING INCORTES GEORGE's BOHT AND HAS GOT ON OFFICE INSIDE CORTIAND AIRING FOR. INT.

Seed “KRAMER”
KRAMER: Chapper apple. All right okay. I can't believe it. I don't think anyone wish me. GEORGE: All right, then no little degrees. KRAMER: Hello, Mr. Stranger. ELAINE: What time is it? KRAMER: Oh, Jerry, what am I gonna do? KRAMER: (disgusted) That's not your line. Thats the - why did you be some kind of short? DENISE: No, no, I'm sorry of course this can make me join spot on you. GEORGE: Please, respect, you know, rubbing little sense. Can you tell me that I can borrow it? JERRY: Would you know what the thing is? What are you doing here? GEORGE: Good for you when I was too writing uh zunk that can. (Newman hangs up) KRAMER: Yea you know, what shouldn't I do? JERRY: Jerry, I'm on the chair. Mother's doesn't come back. JERRY: Oh, no. Would you do me a favour? MORTY: Kramer JERRY: I saw Herbans and Mary Anne, for Lena? JERRY: Not from Vovan. Outta the meal, she ofceced this guy. A bunkie liet with some bananas. I gotta get them any good exvocton for face store company. JERRY: He turned me on in? ELAINE: I, uh...petulaction... SUSAN: hot'd it go? KRAMER: Yeah, but George is thinking. ELAINE: I told you to see you. JERRY: I still say with us, very nice. I'm all agreed. Why is there all all this? I mean it's not here when youre really there. GEORGE: I will, anything at the Alarike and Kxardittour Jerry. JERRY: People who use halition. GEORGE: Excuse me, you half is just here.. And why don't you pay him out of the hall? FATHER-PRIEST: Into Marisa Tomei. That's not me. JERRY: Stop it! Hair, don't worry, I've took my tacked. Everything is welcome against uh.. (Kramer presses the kitchen from his window, grabs Jerry and George.) JERRY: Sanly Just centals! Don't you see?" JERRY: What, are you through? KRAMER: (pointing) Oh. (They make back posters With the distraction as they're dry) GEORGE: This is March. JERRY: Take his phone here! Go ahead. ELAINE: Who says to him? KRAMER: I can't even make dinner. It's a fish. I want to tell ya where the feeling should get me together.

Seed “GEORGE”
GEORGE: Elaine, how much did you know? I can't do this. GEORGE: So what did she think? JERRY: You know, much so much dishouse? FRANK: We got a little nuty in the building. Hey, the can's kneephap on it. Here we go. I'll put it on. It means so would it have done the steams. [Scene moves around desperate with sitting beside a handicap room watching theatro. All right and she's George and George in the next bar.] KRAMER: Others People hear problems, Just a little because he can work off tried clisic. (goes to motion) Wigness Get for are doing charatter? JERRY: (still not staying how putting your seats and George, getting on his shirt, with making back their hole.) KRAMER: Hey, ah! It's honest! GEORGE: Yight. JERRY: Oh, my god! FELDMAN: Thats all me to splat around the intercam on when you bit it? DOCTOR: . . . JERRY: I would love you. JERRY: They've met me again! JERRY: So you would be the ones with things of this is like a homeer overlightevies. (Jerry and Kramer grab on, George pulls his hand to the while.) What punk? JERRY: What's your day in the Prusor sold, so tell him why you want to leave? ELAINE: Look, How did you do that? NEWMAN: Oh, hi when what, I beat with you. Blood, min have sex, it would be not more. We found any party to meet her. CLERK: Maybe you can talk to him. ELAINE: Party? Dad, they could, and, unfortunately, the friend, I'm going to the New York Bookstore on the couch, sell your new couch. JERRY: Hated the doctor?!! (Clicks through his finger back) KRAMER: Well, I mean, I told you!...uh... Could you dump up my mail test? BICKES: Ya.. (Kramer opens the door.) JERRY: Come on, of course. I'm going to be fun. KRAMER: Yeah. JERRY: Yeah... KRAMER: You got a thing? Mailragh... Paul? [Jerry and George at Dleas and Jerry's apartment] Notice: "THE TEACHER MOUTH AT, THE CULATS THE THE PASTERING TURCING MACHINE, GEORGE tHEBERBS CAR AND THE SCENE DOING INCORTES GEORGE's BOHT AND HAS GOT ON OFFICE INSIDE CORTIAND AIRING FOR. INT.

Pretty interesting results given the fact that the model works at character level without knowing any english grammar rules!

Training params I used on 4mb of Seinfeld scripts
th train.lua -data_dir data/seinfeld -rnn_size 700 -num_layers 3 -dropout 0.5 -batch_size 40 -seq_length 40

Titanic with Julia

Mon, 10 Aug 2015 12:00:00 +0000

Julia on Titanic

This is an introduction to Data Analysis and Decision Trees using Julia. In this tutorial we will explore how to tackle Kaggle’s Titanic competition using Julia and Machine Learning. This tutorial is adopted from the Kaggle R tutorial on Machine Learning on Datacamp In case you’re new to Julia, you can read more about its awesomeness on julialang.org.
Again, the point of this tutorial is not to teach machine learning but to provide a starting point to get your hands dirty with Julia code. The benchmark numbers on the Julia website look pretty impressive. So get ready to embrace Julia with a warm hug!

Let’s get started. We will mostly be using 3 main packages from Julia ecosystem

We start with loading the dataset from the Titanic Competition from kaggle. We will use readtable for that and inspect the first few data points with head() on the loaded DataFrame. We will use the already split train and test sets from DataCamp training set: http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/train.csv test set: http://s3.amazonaws.com/assets.datacamp.com/course/Kaggle/test.csv I have them downloaded in the data directory.

using DataFrames


train = readtable("data/train.csv")
head(train)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Ticket	Fare	Cabin	Embarked
1	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	A/5 21171	7.25	NA	S
2	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Thayer)	female	38.0	1	PC 17599	71.2833	C85	C
3	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	STON/O2. 3101282	7.925	NA	S
4	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	113803	53.1	C123	S
5	5	0	3	Allen, Mr. William Henry	male	35.0	0	373450	8.05	NA	S
6	6	0	3	Moran, Mr. James	male	NA	0	330877	8.4583	NA	Q

test = readtable("data/test.csv")
head(test)

	PassengerId	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
1	892	3	Kelly, Mr. James	male	34.5	0	0	330911	7.8292	NA	Q
2	893	3	Wilkes, Mrs. James (Ellen Needs)	female	47.0	1	0	363272	7.0	NA	S
3	894	2	Myles, Mr. Thomas Francis	male	62.0	0	0	240276	9.6875	NA	Q
4	895	3	Wirz, Mr. Albert	male	27.0	0	0	315154	8.6625	NA	S
5	896	3	Hirvonen, Mrs. Alexander (Helga E Lindqvist)	female	22.0	1	1	3101298	12.2875	NA	S
6	897	3	Svensson, Mr. Johan Cervin	male	14.0	0	0	7538	9.225	NA	S

size(train, 1)




891

Let’s take a closer look at our datasets. describe() helps us to summarize the entire dataset.

describe(train)

PassengerId
Min      1.0
1st Qu.  223.5
Median   446.0
Mean     446.0
3rd Qu.  668.5
Max      891.0
NAs      0
NA%      0.0%

Survived
Min      0.0
1st Qu.  0.0
Median   0.0
Mean     0.3838383838383838
3rd Qu.  1.0
Max      1.0
NAs      0
NA%      0.0%

Pclass
Min      1.0
1st Qu.  2.0
Median   3.0
Mean     2.308641975308642
3rd Qu.  3.0
Max      3.0
NAs      0
NA%      0.0%

Name
Length  891
Type    UTF8String
NAs     0
NA%     0.0%
Unique  891

Sex
Length  891
Type    UTF8String
NAs     0
NA%     0.0%
Unique  2

Age
Min      0.42
1st Qu.  20.125
Median   28.0
Mean     29.69911764705882
3rd Qu.  38.0
Max      80.0
NAs      177
NA%      19.87%

SibSp
Min      0.0
1st Qu.  0.0
Median   0.0
Mean     0.5230078563411896
3rd Qu.  1.0
Max      8.0
NAs      0
NA%      0.0%

Parch
Min      0.0
1st Qu.  0.0
Median   0.0
Mean     0.38159371492704824
3rd Qu.  0.0
Max      6.0
NAs      0
NA%      0.0%

Ticket
Length  891
Type    UTF8String
NAs     0
NA%     0.0%
Unique  681

Fare
Min      0.0
1st Qu.  7.9104
Median   14.4542
Mean     32.20420796857464
3rd Qu.  31.0
Max      512.3292
NAs      0
NA%      0.0%

Cabin
Length  891
Type    UTF8String
NAs     687
NA%     77.1%
Unique  148

Embarked
Length  891
Type    UTF8String
NAs     2
NA%     0.22%
Unique  4

If you want to just check the datatypes of the columns you can use eltypes()

eltypes(train)




12-element Array{Type{T<:Top},1}:
 Int64     
 Int64     
 Int64     
 UTF8String
 UTF8String
 Float64   
 Int64     
 Int64     
 UTF8String
 Float64   
 UTF8String
 UTF8String

Data Analysis

Let’s get our hand real dirty now. We want to check how many people survived the disaster. Its good to know how the distribution looks like on the classification classes.

counts() will give you a frequency table, but it does not tell you how the split is. It is the equivalent of table() in R.

counts(train[:Survived])




2-element Array{Int64,1}:
 549
 342

countmap() for rescue! Countmap gives you a dictionary of value => frequency

countmap(train[:Survived])




Dict{Union(NAtype,Int64),Int64} with 2 entries:
  0 => 549
  1 => 342

If you want proportions like prop.table() in R, you can use proportions() or proportionmap()

proportions(train[:Survived])




2-element Array{Float64,1}:
 0.616162
 0.383838




proportionmap(train[:Survived])




Dict{Union(NAtype,Int64),Float64} with 2 entries:
  0 => 0.6161616161616161
  1 => 0.3838383838383838

Counts does not work for categorical variables, so you can use countmap() there.

countmap(train[:Sex])

Dict{Union(NAtype,UTF8String),Int64} with 2 entries:
  "male"   => 577
  "female" => 314

Now that we know the split between male and female population on Titanic and the proportions that survived, let check if the sex of a person had an impact on the chances of surviving.
We will examine this using a stacked histogram of Sex vs Survived using Gadfly package.

using Gadfly


plot(train, x="Sex", y="Survived", color="Survived", Geom.histogram(position=:stack), Scale.color_discrete_manual("red","green"))

Its pretty clear that more females survived over males.
So let’s predict Survived=1 if Sex=female else Survived=0. With that you have your first predictive model ready!!

But let’s not stop there, we have lot more dimensions to the data. Let’s see if we can make use of those to enhance the model. We have Age as one of the columns, using which we can create another dimension to the data - a variable indicating whether the person was a child or not.

train[:Child] = 1
train[isna(train[:Age]), :Child] = 1 #this can be avoided. Its just an explicit way to demonstrate the rule
train[train[:Age] .< 18, :Child] = 1 #same applies for this, as all Child fields are 1 by default except when Age >= 18
train[train[:Age] .>= 18, :Child] = 0
head(train)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Ticket	Fare	Cabin	Embarked	Child
1	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	A/5 21171	7.25	NA	S	0
2	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Thayer)	female	38.0	1	PC 17599	71.2833	C85	C	0
3	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	STON/O2. 3101282	7.925	NA	S	0
4	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	113803	53.1	C123	S	0
5	5	0	3	Allen, Mr. William Henry	male	35.0	0	373450	8.05	NA	S	0
6	6	0	3	Moran, Mr. James	male	NA	0	330877	8.4583	NA	Q	1

Now that we have our Child indicator variable, let’s try to plot the survival rate of children on Titanic.

plot(train, x="Child", y="Survived", color="Survived", Geom.histogram(position=:stack), Scale.color_discrete_manual("red","green"))

We see the number of Children survived is lower than the number of victims.

Let try to combine the gender and child variables to see how that impacts the survival rate. We go back to our friendly bug Gadfly to construct a stacked histogram with subplots.

plot(train, xgroup="Child", x="Sex", y="Survived", color="Survived", Geom.subplot_grid(Geom.histogram(position=:stack)), Scale.color_discrete_manual("red","green"))

These kind of visual analysis is a powerful tool to perform the initial data mining before diving into any deeper machine learning algorithms for predictions.

We clearly see that Age and Gender influence the survival. But we cannot generate all combinations to create a ifelse model for prediction. Enter DecisionTrees. Decision trees do exactly what we did with Gender but at a larger scale where all dimensions (or features) are invovled in forming the decision tree.
In our case the decision we need to make is Survived or Not.

Before doing that we need to handle the missing data in our feature variables. There are multiple ways to handle missing data and we wont go deeper into them. We will use a simple technique of filling the missing value with

Mean value if the variable is numeric.
Max frequency categorical variable value for categorical variables.

From describe() above we have seen that Embarked and Age have some missing values (NA)

countmap(train[:Embarked])

Dict{Union(NAtype,UTF8String),Int64} with 4 entries:
  "Q" => 77
  "S" => 644
  "C" => 168
  NA  => 2

train[isna(train[:Embarked]), :Embarked] = "S"

"S"

meanAge = mean(train[!isna(train[:Age]), :Age])

29.69911764705882

train[isna(train[:Age]), :Age] = meanAge

29.69911764705882

Now that we knocked out the NAs out of our way, let’s roll up our sleeves and grow a decision tree! We will be using the DecisionTree julia package.
Features form the predictors and labels are the response variables for the decision tree. We will start with building the Arrays for features and labels.

using DecisionTree

train[:Age] = float64(train[:Age])
train[:Fare] = float64(train[:Fare])
features = array(train[:,[:Pclass, :Sex, :Age, :SibSp, :Parch, :Fare, :Embarked, :Child]])
labels=array(train[:Survived])


891-element Array{Int64,1}:
 0
 1
 1
 1
 0
 0
 0
 0
 1
 1
 1
 1
 0
 ⋮
 1
 1
 0
 0
 0
 0
 0
 0
 1
 0
 1
 0


features

891x8 Array{Any,2}:
 3  "male"    22.0     1  0   7.25    "S"  0
 1  "female"  38.0     1  0  71.2833  "C"  0
 3  "female"  26.0     0  0   7.925   "S"  0
 1  "female"  35.0     1  0  53.1     "S"  0
 3  "male"    35.0     0  0   8.05    "S"  0
 3  "male"    29.6991  0  0   8.4583  "Q"  1
 1  "male"    54.0     0  0  51.8625  "S"  0
 3  "male"     2.0     3  1  21.075   "S"  1
 3  "female"  27.0     0  2  11.1333  "S"  0
 2  "female"  14.0     1  0  30.0708  "C"  1
 3  "female"   4.0     1  1  16.7     "S"  1
 1  "female"  58.0     0  0  26.55    "S"  0
 3  "male"    20.0     0  0   8.05    "S"  0
 ⋮                            ⋮             
 1  "female"  56.0     0  1  83.1583  "C"  0
 2  "female"  25.0     0  1  26.0     "S"  0
 3  "male"    33.0     0  0   7.8958  "S"  0
 3  "female"  22.0     0  0  10.5167  "S"  0
 2  "male"    28.0     0  0  10.5     "S"  0
 3  "male"    25.0     0  0   7.05    "S"  0
 3  "female"  39.0     0  5  29.125   "Q"  0
 2  "male"    27.0     0  0  13.0     "S"  0
 1  "female"  19.0     0  0  30.0     "S"  0
 3  "female"  29.6991  1  2  23.45    "S"  1
 1  "male"    26.0     0  0  30.0     "C"  0
 3  "male"    32.0     0  0   7.75    "Q"  0

Stumps

A stump is the simplest decision tree, with only one split and two classification nodes (leaves). It tries to generate a single split with most predictive power. This is performed by splitting the dataset into 2 subset and returning the split with highest information gain. We will use the build_stumps api from DecisionTree and feed it with the above generated labels and features.

stump = build_stump(labels, features)
print_tree(stump)

Feature 2, Threshold male
L-> 1 : 233/314
R-> 0 : 468/577

The stump picks feature 2, Sex to make the split with threshold being “male”. The interpretation being - all males data points to the right leaf and all female datapoints go to the left leaf. Hence the 314 on L and 577 on R - number of female and male datapoints (respectively) in train set.
L -> 1 implies Survived=1 is on left branch
R -> 0 implies Survived=0 is on right branch
This creates a basic 1-rules tree with deciding factor being gender of the person.

Let’s verify the split by predicting on the training data.

predictions = apply_tree(stump, features)
confusion_matrix(labels, predictions)

2x2 Array{Int64,2}:
 468   81
 109  233

Classes:  {0,1}
Matrix:   
Accuracy: 0.7867564534231201
Kappa:    0.5421129503407983

Since we predict male who survived and females who didnt survive wrongly we get a miclassification count of 81 + 109. This gives use an accuracy of 1-190/891 = 0.7867

Decision Trees

Now that we have our 1-level decision stump, lets bring it all together and build our first multi-split decision tree. We can use build_tree() which recursively splits the dataset till we get pure leaves which contain samples of only 1 class.

model = build_tree(labels, features)
print_tree(model, 5)
length(model)

Feature 2, Threshold male
L-> Feature 1, Threshold 3
    L-> Feature 6, Threshold 29.0
        L-> Feature 6, Threshold 28.7125
            L-> Feature 3, Threshold 24.0
                L-> 1 : 15/15
                R-> 
            R-> 0 : 1/1
        R-> Feature 3, Threshold 3.0
            L-> 0 : 1/1
            R-> Feature 5, Threshold 2
                L-> 1 : 84/84
                R-> 
    R-> Feature 6, Threshold 23.45
        L-> Feature 8, Threshold 1
            L-> Feature 3, Threshold 37.0
                L-> 
                R-> 
            R-> Feature 6, Threshold 15.5
                L-> 
                R-> 
        R-> Feature 5, Threshold 1
            L-> 1 : 1/1
            R-> Feature 6, Threshold 31.3875
                L-> 0 : 15/15
                R-> 
R-> Feature 6, Threshold 26.2875
    L-> Feature 3, Threshold 15.0
        L-> Feature 4, Threshold 3
            L-> Feature 3, Threshold 11.0
                L-> 1 : 12/12
                R-> 
            R-> 0 : 1/1
        R-> Feature 7, Threshold Q
            L-> Feature 6, Threshold 15.2458
                L-> 
                R-> 
            R-> Feature 6, Threshold 13.5
                L-> 
                R-> 
    R-> Feature 4, Threshold 3
        L-> Feature 3, Threshold 16.0
            L-> 1 : 7/7
            R-> Feature 6, Threshold 26.55
                L-> 1 : 4/4
                R-> 
        R-> Feature 3, Threshold 4.0
            L-> Feature 3, Threshold 3.0
                L-> 0 : 4/4
                R-> 1 : 1/1
            R-> 0 : 18/18





200

We get a large tree so we use the depth parameter of print_tree() to pretty print to a depth of 5.
We can count the number of leaves of our tree using length() on the tree model. Length of our tree is 200.

Although we have a nice looking decision tree ready, we have a bunch of problems - * our classifier is complex because of the tree size * our model overfits on the training data in an attempt to improve accuracy on each bucket. Notice

Solution - Tree Pruning

From wikipedia/Pruning_decision_tree

“Pruning is a technique in machine learning that reduces the size of decision trees by removing sections of the tree that provide little power to classify instances. Pruning reduces the complexity of the final classifier, and hence improves predictive accuracy by the reduction of overfitting.”

# prune tree: merge leaves having >= 70% combined purity (default: 100%)
model = prune_tree(model, 0.7)
# pretty print of the tree, to a depth of 5 nodes (optional)
print_tree(model, 5)
length(model)

Feature 2, Threshold male
L-> Feature 1, Threshold 3
    L-> Feature 6, Threshold 29.0
        L-> Feature 6, Threshold 28.7125
            L-> Feature 3, Threshold 24.0
                L-> 1 : 15/15
                R-> 
            R-> 0 : 1/1
        R-> 1 : 98/100
    R-> Feature 6, Threshold 23.45
        L-> Feature 8, Threshold 1
            L-> Feature 3, Threshold 37.0
                L-> 
                R-> 0 : 6/7
            R-> Feature 6, Threshold 15.5
                L-> 
                R-> 1 : 14/16
        R-> 0 : 24/27
R-> Feature 6, Threshold 26.2875
    L-> Feature 3, Threshold 15.0
        L-> Feature 4, Threshold 3
            L-> Feature 3, Threshold 11.0
                L-> 1 : 12/12
                R-> 
            R-> 0 : 1/1
        R-> Feature 7, Threshold Q
            L-> Feature 6, Threshold 15.2458
                L-> 
                R-> 1 : 4/6
            R-> Feature 6, Threshold 13.5
                L-> 
                R-> 0 : 62/64
    R-> Feature 4, Threshold 3
        L-> Feature 3, Threshold 16.0
            L-> 1 : 7/7
            R-> Feature 6, Threshold 26.55
                L-> 1 : 4/4
                R-> 
        R-> 0 : 22/23





125

Pruning is a bottom-up process which merges leaves into one leaf in a repeated fashion.
The merge criteria is based on the pruning parameter (0.7 in the example above). The pruning parameter is a purity threshold. If the combined purity of two adjacent leaves is greater than the pruning threshold then the two leaves are merged into one.

Length of our pruned tree is reduced to 125 from 200.

We can further reduce the number of leaves by reducing the threshold. But how do you know what is a good threshold ?
There are multiple techniques to determine that one of them being grid search where you try multiple thresholds, plot the number of trees with the threshold values to compare against each other.
One way to do this is using n-fold cross validation.
tl;dr - Cross validation breaks the dataset into train and test sets for measuring accuracy. N-fold CV does this by randomly partitioning the train set into N equal subsets and each subset is used once as a test set while others are used as train sets. Using this we get N prediction performance numbers.

# run n-fold cross validation for pruned tree,
purities = linspace(0.1, 1.0, 5)
accuracies = zeros(length(purities))

for i in 1:length(purities)
    accuracies[i] = mean(nfoldCV_tree(labels, features, purities[i], 5))
end

plot(x=purities, y=accuracies, Geom.point, Geom.line, Guide.xlabel("Threshold"), Guide.ylabel("Accuracy"), Guide.title("Pruning Comparison"))


Fold 1


2x2 Array{Int64,2}:
 111  0
  67  0



2x2 Array{Int64,2}:
 118  0
  60  0



Classes:  {0,1}
Matrix:   
Accuracy: 0.6235955056179775
Kappa:    0.0

Fold 2
Classes:  {0,1}
Matrix:   
Accuracy: 0.6629213483146067
Kappa:    0.0

Fold 


2x2 Array{Int64,2}:
 107  0
  71  0


3
Classes:  {0,1}
Matrix:   
Accuracy: 0.601123595505618
Kappa:    0.0

Fold 


2x2 Array{Int64,2}:
 111  0
  67  0



2x2 Array{Int64,2}:
 101  0
  77  0


4
Classes:  {0,1}
Matrix:   
Accuracy: 0.6235955056179775
Kappa:    0.0

Fold 5
Classes:  {0,1}
Matrix:   
Accuracy: 0.5674157303370787
Kappa:    0.0

Mean Accuracy: 0.6157303370786517

Fold 


2x2 Array{Int64,2}:
 110  0
  68  0



2x2 Array{Int64,2}:
 111  0
  67  0


1
Classes:  {0,1}
Matrix:   
Accuracy: 0.6179775280898876
Kappa:    0.0

Fold 2
Classes:  {0,1}
Matrix:   
Accuracy: 0.6235955056179775
Kappa:    0.0

Fold 


2x2 Array{Int64,2}:
 114  0
  64  0



2x2 Array{Int64,2}:
 107  0
  71  0


3
Classes:  {0,1}
Matrix:   
Accuracy: 0.6404494382022472
Kappa:    0.0

Fold 4
Classes:  {0,1}
Matrix:   
Accuracy: 0.601123595505618
Kappa:    0.0

Fold 


2x2 Array{Int64,2}:
 107  0
  71  0



2x2 Array{Int64,2}:
 87  20
 14  57


5
Classes:  {0,1}
Matrix:   
Accuracy: 0.601123595505618
Kappa:    0.0

Mean Accuracy: 0.6168539325842697

Fold 1
Classes:  {0,1}
Matrix:   
Accuracy: 0.8089887640449438
Kappa:    0.6072680077871512

Fold 


2x2 Array{Int64,2}:
 88  21
 22  47


2
Classes:  {0,1}
Matrix:   
Accuracy: 0.7584269662921348
Kappa:    0.4898013598186908

Fold 


2x2 Array{Int64,2}:
 98  16
 21  43



2x2 Array{Int64,2}:
 93  15
 23  47


3
Classes:  {0,1}
Matrix:   
Accuracy: 0.7921348314606742
Kappa:    0.5407892901966255

Fold 4
Classes:  {0,1}
Matrix:   
Accuracy: 0.7865168539325843
Kappa:    0.5434665226781858

Fold 


2x2 Array{Int64,2}:
 91  20
 17  50



2x2 Array{Int64,2}:
 95  17
 30  36


5
Classes:  {0,1}
Matrix:   
Accuracy: 0.7921348314606742
Kappa:    0.5611088897774225

Mean Accuracy: 0.7876404494382022

Fold 1
Classes:  {0,1}
Matrix:   
Accuracy: 0.7359550561797753
Kappa:    0.4102636402086564

Fold 


2x2 Array{Int64,2}:
 87  14
 28  49


2
Classes:  {0,1}
Matrix:   
Accuracy: 0.7640449438202247
Kappa:    0.5087396504139834

Fold 


2x2 Array{Int64,2}:
 80  23
 15  60



2x2 Array{Int64,2}:
 105  16
  10  47


3
Classes:  {0,1}
Matrix:   
Accuracy: 0.7865168539325843
Kappa:    0.5684573178512186

Fold 4
Classes:  {0,1}
Matrix:   
Accuracy: 0.8539325842696629
Kappa:    0.6735787840315982

Fold 


2x2 Array{Int64,2}:
 80  31
 16  51



2x2 Array{Int64,2}:
 95  23
 17  43


5
Classes:  {0,1}
Matrix:   
Accuracy: 0.7359550561797753
Kappa:    0.461439423200721

Mean Accuracy: 0.7752808988764045

Fold 1
Classes:  {0,1}
Matrix:   
Accuracy: 0.7752808988764045
Kappa:    0.5092362834298318

Fold 


2x2 Array{Int64,2}:
 91  20
 21  46



2x2 Array{Int64,2}:
 99  11
 18  50


2
Classes:  {0,1}
Matrix:   
Accuracy: 0.7696629213483146
Kappa:    0.5078894133513149

Fold 3
Classes:  {0,1}
Matrix:   
Accuracy: 0.8370786516853933
Kappa:    0.6480294558843585

Fold 


2x2 Array{Int64,2}:
 87  19
 21  51



2x2 Array{Int64,2}:
 77  26
 26  49


4
Classes:  {0,1}
Matrix:   
Accuracy: 0.7752808988764045
Kappa:    0.5314556462226901

Fold 5
Classes:  {0,1}
Matrix:   
Accuracy: 0.7078651685393258
Kappa:    0.4009061488673138

Mean Accuracy: 0.7730337078651685

It seems we get the highest accuracy is at a threshold just above 0.5

Conclusion

We saw how easy it is to get your hands dirty in Julia waters. Decision trees are pretty effective but they are slower to train on large datasets, but once trained the predictions are fast - just a walk down the tree.

Download files multi-threaded in R

Wed, 22 Jul 2015 12:00:00 +0000

I had a csv with 5 columns with one of the columns having image urls (picture_url) and one of them having the image id (id). And I had 100,000 of these.

Task was to download all the images with image-id as the image name.

I tried regular download.file() in R but that soon turned out to be an epic fail.

Hello doParallel!

image.df <- read.table("images.csv", 
                      header = TRUE, 
                      sep = ",", 
                      colClasses = c(rep("character", 5)))

You can find the number of cores on linix using lscpu lscpu

require(doParallel)
registerDoParallel(cores=24)

Business end of things

foreach(i=1:nrow(image.df)) %dopar% try(download.file(image.df[i,'picture_url'], 
                                                      destfile=paste(image.df[i,'id'], 
                                                                    ".jpg", 
                                                                    sep="")
                                                      )
                                        )

I am pretty sure there are other (and probably even better) ways to achieve this, but this gets the job done pretty nicely for me.

Dataframes

Sun, 23 Nov 2014 12:00:00 +0000

I have been playing a lot lately with Julia. In the past I have used python and R for my data analysis tasks. I always tend to have a cheatsheet in some form for the basic dataframe operations in pandas and R. Its hard to make that context switch in the syntax from one language to another.

With Julia, my brain was just not ready to accept another set of syntax. So I thought a quick reference comparing the basic dataframe manipulation syntax for all 3 languages would be nice. Most of my data tasks start with some ETL tasks on dataframes. So this is more of a reference for my use in future but can be useful to others too who face the same situation.

I took this awesome tutorial by Greg Reda on “Working with DataFrames” and tried to port the example to R and Julia.

All the code from this post can be found at this github repo dataframes-compare.

We start with the MovieLens 100k dataset.

This data set consists of:

100,000 ratings (1-5) from 943 users on 1682 movies.
Each user has rated at least 20 movies.
Simple demographic info for the users (age, gender, occupation, zip)

Data import

Lets start with loading the data into our dataframe. The methods to read the data from a flat file are pretty simple and straight forward in all 3, the only tricky part is when adding column names in Julia. But there are ways to get around.

Julia

using DataFrames
u_col_names=[symbol("user_id"), symbol("age"), symbol("sex"), symbol("occupation"), symbol("zip_code")]

Another way to do the same without adding each entry as a symbol. Thanks to Jubobs from this stackoverflow post for the suggestion.

col_names=["user_id", "age", "sex", "occupation", "zip_code"]
u_col_names=map(symbol, col_names)
users = DataFrames.readtable("data/ml-100k/u.user", separator='|', header=false, names=u_col_names)

Edit: Probably the easiest way to do this is (as pointed out by lot of people on HN and in the comments section here)

u_col_names=[:user_id,  :age,  :sex,  :occupation,  :zip_code]
users = DataFrames.readtable("data/ml-100k/u.user", separator='|', header=false, names=u_col_names)

There is no way right now to load selective columns from a file. The next IO version will hopefully have that mechanism. Stackoverflow question selecting-columns-while-importing-data-with-dataframes-readtable

Python

import pandas as pd

u_col_names = ['user_id', 'age', 'sex', 'occuptation', 'zip_code']
users = pd.read_csv('data/ml-100k/u.user', sep = '|', names=u_col_names)

Let’s only load the first five columns of the file with usecols

m_col_names = ['movie_id', 'title', 'release_date', 'video_release_date', 'imdb_url']
movies = pd.read_csv('data/ml-100k/u.item', sep='|', names=m_col_names, usecols=range(5))

R

u_col_names <- c('user_id', 'age', 'sex', 'occupation', 'zip_code')
users <- read.csv('data/ml-100k/u.user', sep='|', col.names=u_col_names, header=FALSE)

Loading the first 5 columns only

m_col_names = c('movie_id', 'title', 'release_date', 'video_release_date', 'imdb_url')
movies = read.table('data/ml-100k/u.item', sep='|', colClasses=c("integer", "character", "factor", "factor", "character", rep("NULL", 19)), quote="")

Stackoverflow question only-read-limited-number-of-columns-in-r also quotes in strings cause importing errors so you need quote=”” Cannot specify col.names in read.table as we are skipping 19 columns. R will complain with “more columns than column names”

colnames(movies) <- m_col_names

If you notice closely R and python accept string literals in single quotes ‘ but Julia treats is as a character literal. I personally feel thats how it should be if I think about how C/C++ treats string literals.

Sanity check

Now that we have the data loaded in our dataframes, its usually good practice to see the classes/types of each column. We will also try to get the summary statistics for the columns of our dataframe.

Julia

eltypes(users)

	5-element Array{Type{T<:Top},1}:
	 Int64     
	 Int64     
	 UTF8String
	 UTF8String
	 UTF8String


describe(users)

	user_id
	Min      1.0
	1st Qu.  236.5
	Median   472.0
	Mean     472.0
	3rd Qu.  707.5
	Max      943.0
	NAs      0
	NA%      0.0%
	
	age
	Min      7.0
	1st Qu.  25.0
	Median   31.0
	Mean     34.05196182396607
	3rd Qu.  43.0
	Max      73.0
	NAs      0
	NA%      0.0%
	
	sex
	Length  943
	Type    UTF8String
	NAs     0
	NA%     0.0%
	Unique  2
	
	occupation
	Length  943
	Type    UTF8String
	NAs     0
	NA%     0.0%
	Unique  21
	
	zip_code
	Length  943
	Type    UTF8String
	NAs     0
	NA%     0.0%
	Unique  795

Python

type(users)

	pandas.core.frame.DataFrame

users.dtypes

	user_id         int64
	age             int64
	sex            object
	occuptation    object
	zip_code       object
	dtype: object

users.describe

		user_id		age
	count	943.000000	943.000000
	mean	472.000000	34.051962
	std	272.364951	12.192740
	min	1.000000	7.000000
	25%	236.500000	25.000000
	50%	472.000000	31.000000
	75%	707.500000	43.000000
	max	943.000000	73.000000

R

class(users)

	[1] "data.frame"

lapply(users, class)

	$user_id
	[1] "integer"
	
	$age
	[1] "integer"
	
	$sex
	[1] "factor"
	
	$occupation
	[1] "factor"
	
	$zip_code
	[1] "factor"
	
str(users)

	'data.frame':	943 obs. of  5 variables:
	'data.frame':	943 obs. of  5 variables:
	 $ user_id   : int  1 2 3 4 5 6 7 8 9 10 ...
	 $ age       : int  24 53 23 24 33 42 57 36 29 53 ...
	 $ sex       : Factor w/ 2 levels "F","M": 2 1 2 2 1 2 2 2 2 2 ...
	 $ occupation: Factor w/ 21 levels "administrator",..: 20 14 21 20 14 7 1 1 19 10 ...
	 $ zip_code  : Factor w/ 795 levels "00000","01002",..: 623 690 271 332 134 759 652 49 2 643 ...
	 $ user_id   : int  1 2 3 4 5 6 7 8 9 10 ...
	 $ age       : int  24 53 23 24 33 42 57 36 29 53 ...
	 $ sex       : Factor w/ 2 levels "F","M": 2 1 2 2 1 2 2 2 2 2 ...
	 $ occupation: Factor w/ 21 levels "administrator",..: 20 14 21 20 14 7 1 1 19 10 ...
	 $ zip_code  : Factor w/ 795 levels "00000","01002",..: 623 690 271 332 134 759 652 49 2 643 ...

summary(users)

	    user_id           age        sex             occupation     zip_code  
	 Min.   :  2.0   Min.   : 7.00   F:273   student      :196   55414  :  9  
	 1st Qu.:237.2   1st Qu.:25.00   M:669   other        :105   55105  :  6  
	 Median :472.5   Median :31.00           educator     : 95   10003  :  5  
	 Mean   :472.5   Mean   :34.06           administrator: 79   20009  :  5  
	 3rd Qu.:707.8   3rd Qu.:43.00           engineer     : 67   55337  :  5  
	 Max.   :943.0   Max.   :73.00           programmer   : 66   27514  :  4  
	                                         (Other)      :334   (Other):908  
	    user_id           age        sex             occupation     zip_code  
	 Min.   :  2.0   Min.   : 7.00   F:273   student      :196   55414  :  9  
	 1st Qu.:237.2   1st Qu.:25.00   M:669   other        :105   55105  :  6  
	 Median :472.5   Median :31.00           educator     : 95   10003  :  5  
	 Mean   :472.5   Mean   :34.06           administrator: 79   20009  :  5  
	 3rd Qu.:707.8   3rd Qu.:43.00           engineer     : 67   55337  :  5  
	 Max.   :943.0   Max.   :73.00           programmer   : 66   27514  :  4  
	                                         (Other)      :334   (Other):908

We can see that the ratio is around 1:2.5 between female and male users and most of them (if you exclude “other”) are students. Minimum age is 7 and it goes till 73 as the max.

Subsetting

Lets try to query our data now and filter it with some conditions.

Head/Tail

We will warmup with some top/bottom values to get a feel of the data we have at hand.

Julia

head(users)

	6x5 DataFrame
	| Row | user_id | age | sex | occupation   | zip_code |
	|-----|---------|-----|-----|--------------|----------|
	| 1   | 1       | 24  | "M" | "technician" | "85711"  |
	| 2   | 2       | 53  | "F" | "other"      | "94043"  |
	| 3   | 3       | 23  | "M" | "writer"     | "32067"  |
	| 4   | 4       | 24  | "M" | "technician" | "43537"  |
	| 5   | 5       | 33  | "F" | "other"      | "15213"  |
	| 6   | 6       | 42  | "M" | "executive"  | "98101"  |

tail(users)

	6x5 DataFrame
	| Row | user_id | age | sex | occupation      | zip_code |
	|-----|---------|-----|-----|-----------------|----------|
	| 1   | 938     | 38  | "F" | "technician"    | "55038"  |
	| 2   | 939     | 26  | "F" | "student"       | "33319"  |
	| 3   | 940     | 32  | "M" | "administrator" | "02215"  |
	| 4   | 941     | 20  | "M" | "student"       | "97229"  |
	| 5   | 942     | 48  | "F" | "librarian"     | "78209"  |
	| 6   | 943     | 22  | "M" | "student"       | "77841"  |

If you want to select a custom number of rows in head or tail you can pass it as a param

head(users, 3)

	3x5 DataFrame
	| Row | user_id | age | sex | occupation   | zip_code |
	|-----|---------|-----|-----|--------------|----------|
	| 1   | 1       | 24  | "M" | "technician" | "85711"  |
	| 2   | 2       | 53  | "F" | "other"      | "94043"  |
	| 3   | 3       | 23  | "M" | "writer"     | "32067"  |

I wont paste the output of these for python and R as they are similar to what Julia shows..

Python

users.head()
users.tail()
users.head(3)

R

head(movies)
tail(movies)
head(movies, n=3)

Row subset

Lets try to get all the rows from 50th row to the 55th row

Julia

users[50:55,:]

	6x5 DataFrame
	| Row | user_id | age | sex | occupation   | zip_code |
	|-----|---------|-----|-----|--------------|----------|
	| 1   | 50      | 21  | "M" | "writer"     | "52245"  |
	| 2   | 51      | 28  | "M" | "educator"   | "16509"  |
	| 3   | 52      | 18  | "F" | "student"    | "55105"  |
	| 4   | 53      | 26  | "M" | "programmer" | "55414"  |
	| 5   | 54      | 22  | "M" | "executive"  | "66315"  |
	| 6   | 55      | 37  | "M" | "programmer" | "01331"  |

Python

users[49:55]

R

users[50:55,]

Notice that Julia and R have 1-based indexing whereas python has 0-based indexing. Also python slicing a:b is a(included) and b(excluded) so you have to do 49:55 to get rows 50-55 in python.

Column subset

You can select a single column by column name.

Julia

head(users[:occupation])

	6-element DataArray{UTF8String,1}:
	 "technician"
	 "other"     
	 "writer"    
	 "technician"
	 "other"     
	 "executive"

Python

users['occupation'].head()

R

head(users$occupation)

Multiple column selection works by passing a vector of column names

Julia

head(users[:,[:occupation, :sex, :age]])

	6x3 DataFrame
	| Row | occupation   | sex | age |
	|-----|--------------|-----|-----|
	| 1   | "technician" | "M" | 24  |
	| 2   | "other"      | "F" | 53  |
	| 3   | "writer"     | "M" | 23  |
	| 4   | "technician" | "M" | 24  |
	| 5   | "other"      | "F" | 33  |
	| 6   | "executive"  | "M" | 42  |

Python

users[['occupation', 'sex', 'age']].head()

R

head(users[,c('occupation', 'sex', 'age')])

Query / Conditional subset

Subsetting a dataframe based on querying a column for a condition can be achieved like this -

Julia

users[users[:occupation] .== "writer", :]
	45x5 DataFrame
	| Row | user_id | age | sex | occupation | zip_code |
	|-----|---------|-----|-----|------------|----------|
	| 1   | 3       | 23  | "M" | "writer"   | "32067"  |
	| 2   | 21      | 26  | "M" | "writer"   | "30068"  |
	| 3   | 22      | 25  | "M" | "writer"   | "40206"  |
	| 4   | 28      | 32  | "M" | "writer"   | "55369"  |
	| 5   | 50      | 21  | "M" | "writer"   | "52245"  |
	⋮
	| 43  | 853     | 49  | "M" | "writer"   | "40515"  |
	| 44  | 896     | 28  | "M" | "writer"   | "91505"  |
	| 45  | 911     | 37  | "F" | "writer"   | "53210"  |

Python

users[users.occupation == 'writer']

There is another way in python, by using the query construct of the dataframe.

users.query('occupation=="writer"')

R

users[users$occupation == 'writer',]

Notice the subtle changes in all these examples. For example in the last query subsetting, for python you dont need to specify “select all columns” by adding a ‘ ,” ‘ as you do in Julia or a ‘, ‘ as you do in R.

I will try to write a followup post on Joins on Dataframes in these 3.

Discuss this post on hacker news.

Hello World!

Fri, 21 Nov 2014 12:00:00 +0000

Well, whats the point to live if you are a coder and not have a Hello World! start to your blog. Interestingly this happens to be the World Hello Day!