Question 2.
The files credit_card_data.txt (without headers) and credit_card_data-headers.txt
(with headers) contain a dataset with 654 data points, 6 continuous and 4 binary predictor variables. It
has anonymized credit card applications with a binary response variable (last column) indicating if the
application was positive or negative. The dataset is the “Credit Approval Data Set” from the UCI Machine
Learning Repository (https://archive.ics.uci.edu/ml/datasets/Credit+Approval) without the categorical
variables and without data points that have missing values.
1. Using the support vector machine function ksvm contained in the R package kernlab, find a
good classifier for this data. Show the equation of your classifier, and how well it classifies the
data points in the full data set. (Don’t worry about test/validation data yet; we’ll cover that
topic soon.)
Notes on ksvm
You can use scaled=TRUE to get ksvm to scale the data as part of calculating a
The term λ we used in the SVM lesson to trade off the two components of correctness
classifier.
and margin is called C in ksvm. One of the challenges of this homework is to find a
value of C that works well; for many values of C, almost all predictions will be “yes” or
almost all predictions will be “no”.
ksvm does not directly return the coefficients a0 and a1…am. Instead, you need to do the
last step of the calculation yourself. Here’s an example of the steps to take (assuming
your data is stored in a matrix called data):1
# call ksvm. Vanilladot is a simple linear kernel.
model <- ksvm(data[,1:10],data[,11],type=”C-
svc”,kernel=”vanilladot”,C=100,scaled=TRUE)
# calculate a1…am
a <- colSums(model@xmatrix[[1]] * model@coef[[1]])
a
# calculate a0
a0 <- –model@b
a0
# see what the model predicts
pred <- predict(model,data[,1:10])
pred
# see what fraction of the model’s predictions match the
actual classification
sum(pred == data[,11]) / nrow(data)
Hint: You might want to view the predictions your model makes; if C is too large or too small,
they’ll almost all be the same (all zero or all one) and the predictive value of the model will be
poor. Even finding the right order of magnitude for C might take a little trial-and-error.
1
I know I said I wouldn’t give you exact R code to copy, because I want you to learn for yourself. In general, that’s
definitely true – but in this case, because it’s your first R assignment and because the ksvm function leaves you in
the middle of a mathematical calculation that we haven’t gotten into in this course, I’m giving you the code.
Note: If you get the error “Error in vanilladot(length = 4, lambda = 0.5) :
unused arguments (length = 4, lambda = 0.5)”, it means you need to convert
data into matrix format:
model <-
ksvm(as.matrix(data[,1:10]),as.factor(data[,11]),type=”C-
svc”,kernel=”vanilladot”,C=100,scaled=TRUE)
SOLUTION:
There are multiple possible answers. See file solution 2.2-1.R for the R code for one answer.
Please note that a good solution doesn’t have to try both of the possibilities in the code; they’re both
shown to help you learn, but they’re not necessary.
One possible linear classifier you can use, for scaled data z, is
-0.0010065348z1 - 0.0011729048z2 - 0.0016261967z3 + 0.0030064203z4 + 1.0049405641z5 -
0.0028259432z6 + 0.0002600295z7 - 0.0005349551z8 - 0.0012283758z9 + 0.1063633995z10 + 0.08158492
= 0.
It predicts 565 points (about 86.4%) correctly. (Note that this is its performance on the training data; as
you saw in Module 3, that’s not a reliable estimate of its true predictive ability.) This quality of linear
classifier can be found for a wide range of values of C (from 0.01 to 1000, and beyond). Using unscaled
data, it’s a lot harder to find a C that does this well.
2. You are welcome, but not required, to try other (nonlinear) kernels as well; we’re not covering
them in this course, but they can sometimes be useful and might provide better predictions than
vanilladot.
It’s also possible to find a better nonlinear classifier using a different kernel; kudos to those of you who
went even deeper and tried this!
3. Using the k-nearest-neighbors classification function kknn contained in the R kknn package,
suggest a good value of k, and show how well it classifies that data points in the full data set.
Don’t forget to scale the data (scale=TRUE in kknn).
Notes on kknn
You need to be a little careful. If you give it the whole data set to find the closest points
to i, it’ll use i itself (which is in the data set) as one of the nearest neighbors. A helpful
feature of R is the index –i, which means “all indices except i”. For example, data[-
i,] is all the data except for the ith data point. For our data file where the first 10
columns are predictors and the 11th column is the response, data[-i,11] is the
response for all but the ith data point, and data[-i,1:10] are the predictors for all
but the ith data point.
(There are other, easier ways to get around this problem, but I want you to get
practice doing some basic data manipulation and extraction, and maybe some looping
too.)
Note that kknn will read the responses as continuous, and return the fraction of the k
closest responses that are 1 (rather than the most common response, 1 or 0).
SOLUTION:
Here’s one possible solution. See file solution 2.2-3.R for code. Please note that a good solution
doesn’t have to try all of the possibilities in the code.
As detailed in the code, we observe maximum accuracy for k=12 and k=15. (Again, as above, we’re
reporting performance on the training data, which is generally not good practice, as you saw in Module
3.) A summary of the number of correct predictions for different values of k is shown below (using
scaled data).
Value of k Correct Percent correct
(scaled data) predictions predictions
1-4 533 81.50%
6 553 84.56%
7,9 554 84.71%
8 555 84.86%
10,19-20 556 85.02%
5,11,13-14,16-18 557 85.17%
12,15 558 85.32%
As the table shows, the key (in the training data) is to use k ≥ 5; smaller values of k are significantly
inferior. Although k=12 and k=15 look slightly better than the rest, it’s not a statistically significant
difference.
Note that if we’re just looking at fraction of correct predictions, it might be easy to get caught
up in finding the very highest amount we can find. Don’t lose sight of the fact that these
differences might just be 1 data point out of 654 – which is not statistically significant.
We could do the same using unscaled data, by changing one word in the R code; replace
model=kknn(V11~V1+V2+V3+V4+V5+V6+V7+V8+V9+V10,data[-i,],data[i,],k=X,
scale = TRUE)
with
model=kknn(V11~V1+V2+V3+V4+V5+V6+V7+V8+V9+V10,data[-i,],data[i,],k=X,
scale = FALSE)
Using unscaled data, the results are significantly worse.
Value of k Correct Percent correct
(unscaled data) predictions predictions
1-4 434 66.36%
10 443 67.74%
11 445 68.04%
12 447 68.35%
9,13 449 68.65%
14-15 450 68.81%
5,20 452 69.11%
7-8,16-19 453 69.27%
6 455 69.57%
