1. Use the information provided below and compute the following.

A regression model is trained

to predict defaulter applications with 1 lakh data points. The model is tested on out-of-sample
100 data points. The following classification matrix is obtained at Thresholding level of 0.5.
Here, 0’s are considered as non-defaulters and 1’s are defaulters. The model is evaluated on its
ability to accurately predict (i.e., classify) the observations from the test dataset.

Actual/Predicted Predicted =0 Predicted =1

Actual =0 30 10
Actual =1 20 40

What is the sensitivity - ability to correctly classify defaulters (1’s) as 1’s – of the model

(a) 40-50%
(b) 50-60%
(c) 60-70%
(d) 70-80%

Hint: Sensitivity = True Positives /(True Positives+False Negatives)= 40/(60)*100=66.67%

2. Use the information provided below and compute the following. A regression model is trained
to predict defaulter applications with 1 lakh data points. The model is tested on out-of-sample
100 data points. The following classification matrix is obtained at Thresholding level of 0.5.
Here, 0’s are considered as non-defaulters and 1’s are defaulters. The model is evaluated on its
ability to accurately predict (i.e., classify) the observations from the test dataset.

Actual/Predicted Predicted =0 Predicted =1

Actual =0 30 10
Actual =1 20 40

What is the specificity - ability to correctly classify non-defaulters (0’s) as 0’s – of the model

(a) 40-50%
 (b) 50-60%
(c) 60-70%
(d) 70-80%

Hint: Specificity = True Negatives /(False Positives+True Negatives)= 30/(40)*100=75.0%

3. Use the information provided below and compute the following. A regression model is trained
to predict defaulter applications with 1 lakh data points. The model is tested on out-of-sample
100 data points. The following classification matrix is obtained at Thresholding level of 0.5.
Here, 0’s are considered as non-defaulters and 1’s are defaulters. The model is evaluated on its
ability to accurately predict (i.e., classify) the observations from the test dataset.

Actual/Predicted Predicted =0 Predicted =1

Actual =0 30 10
Actual =1 20 40

If thresholding level is increased, the following correctly reflects its impact on model
performance.

(a) Sensitivity increases and Specificity increases

(b) Sensitivity increases and Specificity decreases
(c) Sensitivity decreases and Specificity increases
(d) Sensitivity decreases and Specificity decreases

Hint: Increasing the thresholding level decreases the sensitivity and increases the specificity

4. Use the information provided below and compute the following. A regression model is trained
to predict defaulter applications with 1 lakh data points. The model is tested on out-of-sample
100 data points. The following classification matrix is obtained at Thresholding level of 0.5.
Here, 0’s are considered as non-defaulters and 1’s are defaulters. The model is evaluated on its
ability to accurately predict (i.e., classify) the observations from the test dataset.
 Actual/Predicted Predicted =0 Predicted =1
Actual =0 30 10
Actual =1 20 40

What is the overall accuracy of the model (Count R-square)

(a) 45-55%
(b) 55-65%
(c) 65-75%
(d) 75-85%

Hint: 𝑂𝑣𝑒𝑟𝑎𝑙𝑙 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 (Count R^2)=(𝑇𝑁+𝑇𝑃)/𝑁= (30+40)/100=70%

5. A classification model uses simply coin tossing game with a fair coin as model of prediction.
Head’s are counted as 0’s and Tail’s as 1’s. What would be the sensitivity and specificity of the
model

(a) Sensitivity =20-40%, Specificity =40-60%

(b) Sensitivity =40-60%, Specificity =40-60%
(c) Sensitivity =40-60%, Specificity =20-40%
(d) Sensitivity =20-40%, Specificity =20-40%

Hint: A coin tossing game (with fair coin) has an accuracy of 50%, so 50% of 0’s and 1’s will be
accurately classified and vice-versa 50% of them will be inaccurately classified.

6. A classification model uses simply coin tossing game with a fair coin as model of prediction.
Head’s are counted as 0’s and Tail’s as 1’s. Under the receiver operating characteristic (ROC)
curve, what would be the area under the curve (AUC) of the model

a) 45-55%
b) 55-65%
c) 65-75%
d) 75-85%
Hint: A coin tossing game (with fair coin) has an accuracy of 50%, so the area under the curve or
the performance of the model is 50%.

7. For a Thresholding value of Tau =1, the following is the correct value

(a) Sensitivity =1, Specificity =1

(b) Sensitivity =0, Specificity =0
(c) Sensitivity =1, Specificity =0
(d) Sensitivity =0, Specificity =1

Hint: For Tau =1, all the 1’s will be classified as 0’s, hence sensitivity =0. And all the 0’s will be
classified as 0’s, hence specificity is 1.

8. For a Thresholding value of Tau =0, the following is the correct value

a) Sensitivity =1, Specificity =1

b) Sensitivity =0, Specificity =0
c) Sensitivity =1, Specificity =0
d) Sensitivity =0, Specificity =1

Hint: For Tau =0, all the 1’s will be classified as 1’s, hence sensitivity =1. And all the 0’s will be
classified as 1’s, hence specificity is 0.
9. As a data scientist, you realize that conventional R-square measure is not a very appropriate
goodness-of-fit indicator for classification algorithms. You teammate suggests using 𝑃𝑠𝑒𝑢𝑑𝑜 −
LLF
𝑅 2 = 1 − LLF . Here LLF is loglikelihood function, that is log of joint probability of observing
0
the data. LLF0 is loglikelihood function for the naïve model. For a good classification model,
you expect the joint probability of observing the data to be very high and low for a poor naïve
model. If you believe that your trained model is very poor and only as good as naïve model,
your 𝑃𝑠𝑒𝑢𝑑𝑜 − 𝑅 2 values can be

a) 0-10%
b) 10-20%
c) 20-30%
 d) 30-40%

Hint: If your model is as good as Naïve model, the LLF will be equal to LLF0 and 𝑃𝑠𝑒𝑢𝑑𝑜 − 𝑅 2 =
0.

10. As a data scientist, you realize that conventional R-square measure is not a very appropriate
goodness-of-fit indicator for classification algorithms. You teammate suggests using 𝑃𝑠𝑒𝑢𝑑𝑜 −
LLF
𝑅2 = 1 − . Here LLF is loglikelihood function, that is log of joint probability of observing
LLF0
the data. LLF0 is loglikelihood function for the naïve model. For a good classification model,
you expect the joint probability of observing the data to be very high and low for a poor naïve
model. If you believe that your trained model is far better than naïve model, your 𝑃𝑠𝑒𝑢𝑑𝑜 − 𝑅 2
values can be as high as.

a) 90-100%
b) 80-90%
c) 70-80%
d) 60-70%

Hint: If the model is too good as compared to naïve model (LLF0), the joint probabilities
approach 1, and LLF will be close to 0. Then 𝑃𝑠𝑒𝑢𝑑𝑜 − 𝑅 2 will be close to 1.