DL Experiments
with Experiment Manager App
(DL) Experiments play important roles in papers
Doing DL Experiments
We need …
• Keep track of training parameters
• Reuse training data across multiple networks
• Analyze and compare results
Experiment Manager App
The Experiment Manager app enables you to create deep learning
experiments to train networks under various initial conditions and
compare the results. For example, you can use deep learning
experiments to:
• Sweep through a range of hyperparameter values or use Bayesian
optimization to find optimal training options. Bayesian optimization
requires Statistics and Machine Learning Toolbox™.
• Use the built-in function trainNetwork or define your own custom
training function.
• Compare the results of using different data sets or test different deep
network architectures.
Doing DL Experiments
We need …
• Keep track of training parameters
• Reuse training data across multiple networks
• Analyze and compare results
Experiment Manager App
Experiment Manager organizes your experiments and results in a project.
• You can store several experiments in the same project.
• Each experiment contains a set of results for each time that you run
the experiment.
• Each set of results consists of one or more trials corresponding to a
different combination of hyperparameters.
By default, Experiment Manager runs one trial at a time. If you have Parallel
Computing Toolbox™, you can configure your experiment to run multiple
trials simultaneously. Running an experiment in parallel allows you to use
MATLAB® while the training is in progress.
Open the Experiment Manager App
• MATLAB Toolstrip: On the Apps tab, under Machine Learning and
Deep Learning, click the app icon.
• MATLAB command prompt: Enter experimentManager.
Experiment Manager Use cases
• Create a Deep Learning Experiment for Classification
• Create a Deep Learning Experiment for Regression
• Tune Experiment Hyperparameters by Using Bayesian Optimization
Create a Deep Learning Experiment for Classification
cd(setupExample('nnet/ExperimentManagerClassificationExample'));
setupExpMgr('MerchandiseClassificationProject');
Create a Deep Learning Experiment for Classification
The Description field contains a textual description of the experiment. For this example, the
description is:
Merchandise image classification using:
- an untrained network (default) or a pretrained network (googlenet)
- various solvers for training networks (sgdm, rmsprop, or adam)
The Hyperparameters section specifies the strategy (Exhaustive Sweep) and hyperparameter values
to use for the experiment. When you run the experiment, Experiment Manager trains the network
using every combination of hyperparameter values specified in the hyperparameter table. This
example uses two hyperparameters:
Network specifies the network to train. The options include "default" (the default network provided
by the experiment template for image classification) and "googlenet" (a pretrained GoogLeNet
network with modified layers for transfer learning).
Solver indicates the algorithm used to train the network. The options include "sgdm" (stochastic
gradient descent with momentum), "rmsprop" (root mean square propagation), and "adam"
(adaptive moment estimation).
Create a Deep Learning Experiment for Classification
The Setup Function configures the training data, network architecture, and training options for the
experiment. To inspect the setup function, under Setup Function, click Edit. The setup function opens in
MATLAB Editor.
The input to the setup function is a structure with fields from the hyperparameter table. The setup
function returns three outputs that you use to train a network for image classification problems. The
setup function has three sections.
Load Training Data defines image datastores containing the training and validation data. This example
loads images from the file MerchData.zip. This small data set contains 75 images of MathWorks
merchandise, belonging to five different classes. The images are of size 227-by-227-by-3. For more
information on this data set, see Image Data Sets.
Define Network Architecture defines the architecture for a convolutional neural network for deep
learning classification. In this example, the choice of network to train depends on the value of the
hyperparameter Network.
Specify Training Options defines a trainingOptions object for the experiment. The example trains the
network for 8 epochs using the algorithm specified by the Solver entry in the hyperparameter table.
Create a Deep Learning Experiment for Classification
When you run the experiment, Experiment Manager trains the network defined by the setup function
six times. Each trial uses a different combination of hyperparameter values. By default, Experiment
Manager runs one trial at a time.
Create a Deep Learning Experiment for Classification
While the experiment is running, click Training Plot to display the training plot and track the progress of
each trial.
Create a Deep Learning Experiment for Classification
To find the best result for your experiment, sort the table of results by validation accuracy.
• Point to the Validation Accuracy column.
• Click the triangle icon.
• Select Sort in Descending Order.
The trial with the highest validation accuracy appears at the top of the results table.
Create a Deep Learning Experiment for Classification
To display the confusion matrix for this trial, select the top row in the results table and click Confusion
Matrix.
Create a Deep Learning Experiment for Regression
cd(setupExample('nnet/ExperimentManagerClassificationExample'));
setupExpMgr('MerchandiseClassificationProject');
Create a Deep Learning Experiment for Regression
The Hyperparameters section specifies the strategy (Exhaustive Sweep) and
hyperparameter values to use for the experiment. When you run the
experiment, Experiment Manager trains the network using every combination
of hyperparameter values specified in the hyperparameter table. This example
uses two hyperparameters:
Probability sets the probability of the dropout layer in the neural network. By
default, the values for this hyperparameter are specified as [0.1 0.2].
Filters indicates the number of filters used by the first convolution layer in the
neural network. In the subsequent convolution layers, the number of filters is a
multiple of this value. By default, the values of this hyperparameter are
specified as [4 6 8].
Create a Deep Learning Experiment for Regression
The Setup Function configures the training data, network architecture, and training options for the
experiment. To inspect the setup function, under Setup Function, click Edit. The setup function
opens in MATLAB Editor.
The input to the setup function is a structure with fields from the hyperparameter table. The setup
function returns four outputs that you use to train a network for image regression problems. The
setup function has three sections.
Load Training Data defines the training and validation data for the experiment as 4-D arrays. The
training and validation data sets each contain 5000 images of digits from 0 to 9. The regression
values correspond to the angles of rotation of the digits.
Define Network Architecture defines the architecture for a convolutional neural network for
regression.
Specify Training Options defines a trainingOptions object for the experiment. The example trains
the network for 30 epochs. The learning rate is initially 0.001 and drops by a factor of 0.1 after 20
epochs. The software trains the network on the training data and calculates the root mean squared
error (RMSE) and loss on the validation data at regular intervals during training. The validation data
is not used to update the network weights.
Create a Deep Learning Experiment for Regression
The Metrics section specifies optional functions that evaluate the results of the experiment.
Experiment Manager evaluates these functions each time it finishes training the network. To inspect
a metric function, select the name of the metric function and click Edit. The metric function opens in
MATLAB Editor.
This example includes a metric function Accuracy that determines the percentage of angle
predictions within an acceptable error margin from the true angles. By default, the function uses a
threshold of 10 degrees.
Create a Deep Learning Experiment for Regression
When you run the experiment, Experiment Manager trains the network defined by the setup
function six times. Each trial uses a different combination of hyperparameter values. By default,
Experiment Manager runs one trial at a time.
Create a Deep Learning Experiment for Regression
To test the performance of an individual trial, export the trained network and display a box plot of
the residuals for each digit class.
Select the trial with the highest accuracy.
On the Experiment Manager toolstrip, click Export.
In the dialog window, enter the name of a workspace variable for the exported network. The default
name is trainedNetwork.
Use the exported network as the input to the function plotResiduals. For instance, in the MATLAB
Command Window, enter:
plotResiduals(trainedNetwork)
The function creates a residual box plot for each digit. The digit classes with highest accuracy have a
mean close to zero and little variance.
Create a Deep Learning Experiment for Regression
Tune Experiment Hyperparameters by Using Bayesian Optimization
cd(setupExample('nnet/ExpMgrBayesOptExample'));
setupExpMgr('CIFARBayesianOptimizationProject');
Tune Experiment Hyperparameters by Using Bayesian Optimization
This example shows how to use Bayesian optimization in Experiment
Manager to find optimal network hyperparameters and training
options for convolutional neural networks.
Bayesian optimization provides an alternative strategy to sweeping
hyperparameters in an experiment. You specify a range of values for
each hyperparameter and select a metric to optimize, and Experiment
Manager searches for a combination of hyperparameters that
optimizes your selected metric. Bayesian optimization requires
Statistics and Machine Learning Toolbox.
Tune Experiment Hyperparameters by Using Bayesian Optimization
In this example, you train a network to classify images from the CIFAR-
10 data set. The experiment uses Bayesian optimization to find the
combination of hyperparameters that minimizes a custom metric
function. The hyperparameters include options of the training
algorithm, as well as parameters of the network architecture itself. The
custom metric function determines the classification error on a
randomly chosen test set.
Tune Experiment Hyperparameters by Using Bayesian Optimization
This example uses these hyperparameters:
SectionDepth — This parameter controls the depth of the network. The total
number of layers in the network is 9*SectionDepth+7. In the experiment setup
function, the number of convolutional filters in each layer is proportional to
1/sqrt(SectionDepth), so the number of parameters and the required amount of
computation for each iteration are roughly the same for different section depths.
InitialLearnRate — The best learning rate can depend on your data as well as the
network you are training.
Momentum — Stochastic gradient descent momentum adds inertia to the
parameter updates by having the current update contain a contribution
proportional to the update in the previous iteration. The inertial effect results in
smoother parameter updates and a reduction of the noise inherent to stochastic
gradient descent.
L2Regularization — Use L2 regularization to prevent overfitting. Search the space
of regularization strength to find a good value. Data augmentation and batch
normalization also help regularize the network.
Tune Experiment Hyperparameters by Using Bayesian Optimization
Under Bayesian Optimization Options, you can specify the duration of
the experiment by entering the maximum time (in seconds) and the
maximum number of trials to run. To best utilize the power of Bayesian
optimization, you should perform at least 30 objective function
evaluations.
The Setup Function configures the training data, network architecture,
and training options for the experiment. To inspect the setup function,
under Setup Function, click Edit. The setup function opens in MATLAB®
Editor.
Tune Experiment Hyperparameters by Using Bayesian Optimization
In this example, the setup function has three sections.
• Load Training Data downloads and extracts images and labels from
the CIFAR-10 data set. The data set is about 175 MB. Depending on
your internet connection, the download process can take some time.
For the training data, this example creates an
augmentedImageDatastore by applying random translations and
horizontal reflections. Data augmentation helps prevent the network
from overfitting and memorizing the exact details of the training
images. To enable network validation, the example uses 5000 images
with no augmentation. For more information on this data set, see
Image Data Sets.
Tune Experiment Hyperparameters by Using Bayesian Optimization
In this example, the setup function has three sections.
• Define Network Architecture defines the architecture for a convolutional
neural network for deep learning classification. In this example, the
network to train has three sections, each with SectionDepth identical
convolutional layers. Each convolutional layer is followed by a batch
normalization layer and a ReLU layer. The convolutional layers have added
padding so that their spatial output size is always the same as the input
size. Between sections, max pooling layers downsample the spatial
dimensions by a factor of two.
Tune Experiment Hyperparameters by Using Bayesian Optimization
In this example, the setup function has three sections.
• Specify Training Options defines a trainingOptions object for the
experiment using the values for the training options 'InitialLearnRate',
'Momentum', and 'L2Regularization' generated by the Bayesian
optimization algorithm. The example trains the network for a fixed
number of epochs, validating once per epoch and lowering the learning
rate by a factor of 10 during the last epochs to reduce the noise of the
parameter updates and allow the network parameters to settle down
closer to a minimum of the loss function.
Tune Experiment Hyperparameters by Using Bayesian Optimization
The Metrics section specifies optional functions that evaluate the results of
the experiment. Experiment Manager evaluates these functions each time
it finishes training the network. To inspect a metric function, select the
name of the metric function and click Edit. The metric function opens in
MATLAB Editor.
This example includes the custom metric function ErrorRate. This function
selects 5000 test images and labels at random, evaluates the trained
network on these images, and calculates the proportion of images that the
network misclassifies.
Tune Experiment Hyperparameters by Using Bayesian Optimization
function metricOutput = ErrorRate(trialInfo)
datadir = tempdir;
[~,~,XTest,YTest] = loadCIFARData(datadir);
idx = randperm(numel(YTest),5000);
XTest = XTest(:,:,:,idx);
YTest = YTest(idx);
YPredicted = classify(trialInfo.trainedNetwork,XTest);
metricOutput = 1 - mean(YPredicted == YTest);
end
The Optimize and Direction fields indicate the metric that the Bayesian
optimization algorithm uses as an objective function. For this experiment,
Experiment Manager seeks to minimize the value of the ErrorRate metric.
Tune Experiment Hyperparameters by Using Bayesian Optimization
When you run the experiment, Experiment Manager searches for
the best combination of hyperparameters with respect to the
chosen metric. Each trial in the experiment uses a new
combination of hyperparameter values based on the results of
the previous trials. By default, Experiment Manager runs one trial
at a time.
Tune Experiment Hyperparameters by Using Bayesian Optimization
cd(setupExample('nnet/ExpMgrBayesOptExample'));
setupExpMgr('CIFARBayesianOptimizationProject');
Tune Experiment Hyperparameters by Using Bayesian Optimization
cd(setupExample('nnet/ExpMgrBayesOptExample'));
setupExpMgr('CIFARBayesianOptimizationProject');
