This was a nice paper illustrating and explaining CART (classification and regression trees).
089-2013 Using
Classification and Regression Trees (CART) in SAS® Enterprise Miner™ for
Applications in Public Health
"They (CARTs) are typically model free in their implementation.
Howbeit, a model based statistic is sometimes used for a splitting criterion.
The main idea of a classification tree
is a statistician’s version of the popular twenty questions game. Several
questions are asked with the aim of answering
a particular research question at hand. However, they are advantageous because
of their non -parametric and non- linear nature. They do not make any
distribution assumptions and treat the data generation process as unknown and
do not require a functional form for the predictors. They also do not assume additivity of the predictors which
allows them to identify complex interactions. Tree methods are probably one of
the most easily interpreted statistical techniques. They can be followed
with little or no understanding of Statistics and to a certain extent follow the decision
process that humans use to make decisions. In this regard, they are
conceptually simple yet present a powerful
analysis (Hastie et al 2009)."
Showing posts with label Decision Trees. Show all posts
Showing posts with label Decision Trees. Show all posts
Wednesday, April 10, 2013
Monday, April 8, 2013
Using Advanced Analytics to Recruit Students for Improved Retention & Graduation
Forthcoming:SAS Global Forum (April 28-May 1 2013)
Paper 044-2013
A Data Driven Analytic
Strategy for Increasing Yield and Retention at Western Kentucky University
Using SAS Enterprise BI and SAS Enterprise Miner
Matt
Bogard, Western Kentucky University, Bowling Green, KY
ABSTRACT
As
many universities face the constraints of declining enrollment demographics, pressure
from state governments for increased student success, as well as declining
revenues, the costs of utilizing anecdotal
evidence and intuition based on ‘gut’ feelings to make time and resource
allocation decisions become significant. This paper describes how we are using
SAS® Enterprise
Miner to develop a model to score university students based on their
probability of enrollment and retention early in the enrollment funnel so that
staff and administrators can work to recruit
students that not only have an average or better chance of enrolling but also
succeeding once they enroll. Incorporating these results into SAS® EBI will allow us to deliver
easy-to-understand results to university personnel.
PDF TEXT available at Proceedings of the SAS® Global Forum 2013 Conference
Conference: http://support.sas.com/events/sasglobalforum/2013/index.html
Conference: http://support.sas.com/events/sasglobalforum/2013/index.html
The correct bibliographic citation for this publication is as follows:
SAS Institute Inc. 2013.Proceedings of the SAS® Global Forum 2013 Conference. Cary, NC:
SAS Institute Inc. 2013.Proceedings of the SAS® Global Forum 2013 Conference. Cary, NC:
Wednesday, March 6, 2013
Decision Trees and Gradient Boosting
Decision Trees
Decision tree algorithms search through the input space and find values of the input variables (split values) that maximize the differences in the target value between groups created by the split. The final model is characterized by the split values for each explanatory variable and creates a set of rules for classifying new cases.
Gradient Boosting
Boosting algorithms are ensemble methods that make predictions based on the average results of a series of weak learners. Gradient boosting involves fitting a series of trees, with each successive tree being fit to a resampled training set that is weighted according to the classification accuracy of the previously fit tree. The original training data is resampled several times and the combined series of trees form a single predictive model. This differs from other ensemble methods using trees, such as random forests. Random forests are a modified type of bootstrap aggregation or bagging estimator (Freidman et al,2009). With random forests, we get a predictor that is an average of a series of trees grown on a bootstrap sample of the training data with only a random subset of the available inputs from the training data used to fit each tree (De Ville, 2006). Gradient boosting can perform similarly to random forests and boosting may tend to dominate bagging methods in many applications. (Freidman et al,2009).
References:
Friedman, Jerome H. (2001), Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189-1232. Available at http://stat.stanford.
Hasti, Tibshirani and Friedman. (2009)Elements of Statistical Learning: Data Mining,Inference, and Prediction. Second Edition. Springer-Verlag.
DeVille, Barry. (2006). Decision Trees for Business Intelligence and Data Mining Using SAS® Enterprise Miner. SAS® Institute.
Sunday, January 29, 2012
Decision Tree Basics in SAS and R
Assume we were going to use a decision tree to predict ‘green’ vs. ‘’red” cases (see below- note this plot of the data was actually created in R).
We want to use 2 variables say X1 and X2 to make a prediction of ‘green’ or ‘red’. In its simplest form, the decision tree algorithm searches through the values of X1 and X2 and finds the values that do the ‘best’ job of ‘splitting’ the cases. For each possible value of X, the algorithm performs a chi-square test. The values that create the best ‘split’ (in SAS Enterprise Miner this is based on a metric called ‘worth’ which is a function of the p-value associated with the chi-square test [worth = -log(p-value)]) are chosen. (See below)
After all of the ‘best’ splits are determined, these values then become rules for distinguishing between cases (in this case ‘green’ vs. ‘red’) What you end up with in the end is a set of ‘rectangles’ defined by a set of ‘rules’ that can be visualized by a ‘tree’ diagram (see visualization from R below).
(see visualization from SAS Enterpise Miner Below)
But, in SAS Enterprise Miner, the algorithm then goes on to look at new data (validation data) and assesses how well the splitting rules do in terms of predicting or classifying new cases. If it finds that a ‘smaller’ tree with fewer variables or fewer splits or rules or ‘branches’ does a better job, it prunes or trims the tree and removes them from the analysis. In the end, you get a final set of rules that can then be applied algorithmically to predict new cases based on observed values of X1 and X2 (or whatever variables are used by the tree).
In SAS Enterprise Miner, with each split created by the decision tree, a metric for importance based on the Gini impurity index (which is a measure of variability or impurity for categorical data) is calculated. This measures how well the tree distinguishes between cases (again ‘green’ vs. ‘red’ or ‘retained’ vs. ‘non-retained’ in our model) and ultimately how well the model explains whatever it is we are trying to predict. Overall, if splits based on values of X2 ‘reduce impurity’ more so than splits based on values of X1 , then X2 would be considered the ‘best’ or ‘most’ predictive variable.
The rpart algorithm used in R may differ in some of the specific details, but as discussed in The Elements of Statistical Learning (Hastie, Tibshirani and Friedman (2008)) all decision trees pretty much work the same, fitting the model by recursively partitioning the feature space into rectangular subsets.
R code for the Decision Tree and Visualizations Above:
# *------------------------------------------------------------------ # | PROGRAM NAME: R_tree_basic # | DATE:4/26/11 # | CREATED BY: Matt Bogard # | PROJECT FILE:P:\R Code References\Data Mining_R # *---------------------------------------------------------------- # | PURPOSE: demo of basic decision tree mechanics # | # *------------------------------------------------------------------ rm(list=ls()) # get rid of any existing data ls() # view open data sets setwd('/Users/wkuuser/Desktop/R Data Sets') # mac setwd("P:\\R Code References\\R Data") # windows library(rpart) # install rpart decision tree library # *------------------------------------------------------------------ # | get data # *----------------------------------------------------------------- dat1 <- read.csv("basicTree.csv", na.strings=c(".", "NA", "", "?"), encoding="UTF-8") plot( dat1$x2, dat1$x1, col = dat1$class) # plot data space # fit decision tree (r <- rpart(class ~ x1 + x2, data = dat1)) plot(r) text(r) library(rattle) # data mining package drawTreeNodes(r) # for more detailed tree plot supported by rattle
Sunday, May 1, 2011
Decision Tree Mechanics with R and SAS
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
Created by Pretty R at inside-R.org
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
(click to enlarge)
# *------------------------------------------------------------------ # | PROGRAM NAME: R_tree_basic # | DATE:4/26/11 # | CREATED BY: Matt Bogard # | PROJECT FILE:P:\R Code References\Data Mining_R # *---------------------------------------------------------------- # | PURPOSE: demo of basic decision tree mechanics # | # *------------------------------------------------------------------ rm(list=ls()) # get rid of any existing data ls() # view open data sets setwd('/Users/wkuuser/Desktop/R Data Sets') # mac setwd("P:\\R Code References\\R Data") # windows library(rpart) # install rpart decision tree library # *------------------------------------------------------------------ # | get data # *----------------------------------------------------------------- dat1 <- read.csv("basicTree.csv", na.strings=c(".", "NA", "", "?"), encoding="UTF-8") plot( dat1$x2, dat1$x1, col = dat1$class) # plot data space # fit decision tree (r <- rpart(class ~ x1 + x2, data = dat1)) plot(r) text(r) library(rattle) # data mining package drawTreeNodes(r) # for more detailed tree plot supported by rattle # *------------------------------------------------------------------ # | # | # | chi square test - 1st split # | # | # *----------------------------------------------------------------- # create a categorical for the first cutoff for x2 > 6.5 dat1$cutoff <- (ifelse (dat1$x2 >= 6.5, "x2 >=6.5 ", "x2 < 6.5")) # library(MASS) # required for cross tabulation tab1 <- table(dat1$cutoff,dat1$class) # cross tabulation print(tab1) Xsq <-chisq.test(tab1, correct = FALSE)# chi-squared test for independence print(Xsq) print(Xsq$exp) # print expected values # *------------------------------------------------------------------ # | chi square test - 1st split - choose an arbitrarily higher and # | lower split value and compare to optimal split chosen by tree # *----------------------------------------------------------------- # higher x2 split value dat1$h1 <- (ifelse (dat1$x2 >= 7.5, "x2 >=7.5 ", "x2 < 7.5")) tab2 <- table(dat1$h1,dat1$class) # cross tabulation print(tab2) Xsq <-chisq.test(tab2, correct = FALSE)# chi-squared test for independence print(Xsq) # chi square value is lower # lower x2 split value dat1$l1 <- (ifelse (dat1$x2 >= 5.5, "x2 >=5.5 ", "x2 < 5.5")) tab3 <- table(dat1$l1,dat1$class) # cross tabulation print(tab3) Xsq <-chisq.test(tab3, correct = FALSE)# chi-squared test for independence print(Xsq) # chi square value is lower # look at current dat1 data set summary dim(dat1) names(dat1) # *------------------------------------------------------------------ # | # | # | chi square test - 2nd split # | # | # *----------------------------------------------------------------- # *------------------------------------------------------------------ # | get data # *----------------------------------------------------------------- # to get the data in the 2nd split we have to first subset or partition the # data where x2 >= 6.5, hence the partition in the 'recursive partitioning' # algorithm used by decision trees) dat2 <- dat1[dat1$x2 >= 6.5,] dim(dat2) # N = 44 which is correct, recall print(tab1) # *------------------------------------------------------------------ # | create a categorical for the second cutoff for x1 >= 4.5 # *----------------------------------------------------------------- dat2$cutoff <- (ifelse (dat2$x1 >= 4.5, "x1 >=4.5 ", "x1 < 4.5")) # *------------------------------------------------------------------ # | cross tab & chi square test # *----------------------------------------------------------------- tab4 <- table(dat2$cutoff,dat2$class) # cross tabulation print(tab4) Xsq <-chisq.test(tab4, correct = FALSE)# chi-squared test for independence print(Xsq) # X-squared = 44, df = 1, p-value = 3.284e-11 # *------------------------------------------------------------------ # | chi square test - 2nd split - choose an arbitrarily higher and # | lower split value and compare to optimal split chosen by tree # *----------------------------------------------------------------- # higher x1 split value dat2$h2 <- (ifelse (dat2$x1 >= 5.5, "x1 >=5.5 ", "x1 < 5.5")) tab5 <- table(dat2$h2,dat2$class) # cross tabulation print(tab5) Xsq <-chisq.test(tab5, correct = FALSE)# chi-squared test for independence print(Xsq) # X-squared is lower # lower x1 split value dat2$l2 <- (ifelse (dat2$x1 >= 3.5, "x1 >=3.5 ", "x1 < 3.5")) tab6 <- table(dat2$l2,dat2$class) # cross tabulation print(tab6) Xsq <-chisq.test(tab6, correct = FALSE)# chi-squared test for independence print(Xsq) # X-squared is lower # look at data in dat2 dim(dat2) names(dat2) # note dat2 inherits the variables h1 and l1 (cutoffs for x2) from dat1, but they are not # relevant to our analysis in the 2nd split (which focuses on cutoffs for x1) # *------------------------------------------------------------------ # | # | export data sets to SAS to repeat analysis using a SAS data set, # | base SAS, and Enterprise Miner # | # | # *----------------------------------------------------------------- # export data sets to SAS to repeat analysis using a SAS data set, # base SAS, and Enterprise Miner library(foreign) # export data and SAS code for analyzing dat1 write.foreign(dat1, "dat1.txt", "dat1_read.sas", package="SAS") # export data and SAS code for analyzing dat2 write.foreign(dat2, "dat2.txt", "basicTreedat2_read.sas", package="SAS")
Subscribe to:
Posts (Atom)