Azure Machine Learning
© Jitesh Khurkhuriya
Create an AzureML workspace Create models by using AzureML Designer
Run training scripts in an AzureML workspace
Manage data objects in an AzureML workspace
Set up AzureML Run Experiments Generate metrics from an experiment run
Manage experiment compute contexts
Workspace and Train Models
Automate the model training process
DP-100
Create production compute targets Use Automated ML to create optimal models
Deploy and Optimize and
Deploy a model as a service Consume Models Manage Models Use Hyperdrive to tune hyperparameters
Create a pipeline for batch inferencing Use model explainers to interpret models
Publish a designer pipeline as a web service Manage models
© Jitesh Khurkhuriya
Typical Machine Learning Workflow
Data Processing Build and Train Model Deploy and Monitor
Data Ingestion Data Validation Model Training
Model Deployment
Cross Validation
Feature Engineering Inferences/Predictions
Hyperparameter Tuning
Feature Selection Monitor Results
Model Selection
© Jitesh Khurkhuriya
Typical Machine Learning Workflow
© Jitesh Khurkhuriya
Typical Machine Learning Workflow
Model Building and
Data Processing Hyperparameter Tuning Evaluate and Select
Selection
• ID • Logistic Regression • L1 and L2 Select Best model
• Credit Given • Boosted Decision Tree • Tolerance based on the goal
• Gender • Decision/Random Forest • Number of trees
• Education • Support Vector Machines • Max Depth
• Marital Status • Deep Learning • Splits per node
• ….. • Min Samples per node
• Default?
• Which Features to • Which Algorithm to • How to tune the • Compare the goals
select? Select? parameters?
• How to deal with
missing values?
© Jitesh Khurkhuriya
Azure AutoML
Bring Dataset Rank Run Accuracy
Run 1 Data Processing + Algorithm + Tunning 62%
1 Run 5 92%
Run 2 Data Processing + Algorithm + Tunning 82% 2 Run 3 91%
3 Run 2 82%
Run 3 Data Processing + Algorithm + Tunning 91%
Define Goal 4 Run 4 74%
Run 4 Data Processing + Algorithm + Tunning 74% 5 Run 1 62%
Run 5 Data Processing + Algorithm + Tunning 92%
Constraints
© Jitesh Khurkhuriya
Note on norm_micro_recall
© Jitesh Khurkhuriya
Recall
Predicted
False True
False 253 115 𝑇𝑃 46
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual
𝑇𝑃 + 𝐹𝑁 46 + 66
True 66 46
© Jitesh Khurkhuriya
Recall
Predicted
No Yes
No 253 115 𝑇𝑃 46
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual
𝑇𝑃 + 𝐹𝑁 46 + 66
Yes 66 46
© Jitesh Khurkhuriya
Recall
Predicted
C1 C2
C1 253 115 𝑇𝑃 46
𝑅𝑒𝑐𝑎𝑙𝑙 = = = 0.4107
Actual
𝑇𝑃 + 𝐹𝑁 46 + 66
C2 66 46
© Jitesh Khurkhuriya
Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual
253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46
© Jitesh Khurkhuriya
Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual
253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46
𝑅𝑒𝑐𝑎𝑙𝑙𝐶1 + 𝑅𝑒𝑐𝑎𝑙𝑙𝐶2 0.4107 + 0.6875
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = = = 0.5491
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑎𝑠𝑠𝑒𝑠 2
© Jitesh Khurkhuriya
Normalized Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual
253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5491
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 − 𝑅 1
𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 𝑅= 𝐶 → 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑙𝑎𝑠𝑠𝑒𝑠
1 −𝑅 𝐶
© Jitesh Khurkhuriya
Normalized Macro Recall
𝐶𝑜𝑟𝑟𝑒𝑐𝑡 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
Predicted 𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑡𝑢𝑎𝑙 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑡𝑖𝑜𝑛𝑠 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑙𝑎𝑠𝑠
C1 C2
46
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = = 0.4107
C1 253 115 46 + 66
Actual
253
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = = 0.6875
253 + 115
C2 66 46
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5491
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 − 𝑅 0.5491 − 0.5
𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = = = 0.0982
1 −𝑅 1 − 0.5
© Jitesh Khurkhuriya
Normalized Macro Recall
Predicted Predicted
C1 C2 C1 C2
C1 368 0 C1 368 0
Actual
Actual
C2 112 0 C2 0 112
368 368
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = =1 𝑅𝑒𝑐𝑎𝑙𝑙(𝐶1) = =1
368 + 0 368 + 0
0 112
𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = =0 𝑅𝑒𝑐𝑎𝑙𝑙(𝐶2) = =1
0 + 112 112 + 0
𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 0.5 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = 1
0.5 − 0.5 1 − 0.5
𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = =0 𝑛𝑜𝑟𝑚 𝑚𝑎𝑐𝑟𝑜 𝑟𝑒𝑐𝑎𝑙𝑙 = =1
1 − 0.5 1 − 0.5
© Jitesh Khurkhuriya
Azure Machine Learning
Thank You..!!
© Jitesh Khurkhuriya
