Simple Python tools for plotting learning curves of neural network models trained with the Keras, Ultralytics YOLO or scikit-learn framework in a single figure in an easy-to-understand format.

Currently the lcurvetools package provides three basic functions: lcurves, history_concatenate and lcurves_by_MLP_estimator.

NOTE: All of the plotting examples below are for interactive Python mode in Jupyter-like environments. If you are in non-interactive mode you may need to explicitly call plt.show() after calling the lcurves or lcurves_by_MLP_estimator function to display the window with built plots on your screen.

• Installation
• The lcurves function to plot learning curves based on model training history dictionaries
  • Plotting learning curves of one model based on one dictionary
    • Usage scheme
    • Typical appearance of the output figure
    • Customizing appearance of the output figure
  • Plotting learning curves based on a list of dictionaries with fitting histories of several models
    • Usage scheme
  • Plotting learning curves based on a list of dictionaries with independent refitting histories of one model
    • Usage scheme
• The history_concatenate function to concatenate two History.history dictionaries
  • Usage scheme
• The lcurves_by_MLP_estimator function to plot learning curves of the scikit-learn MLP estimator
  • Usage scheme
  • Typical appearance of the output figure
  • Customizing appearance of the output figure

The easiest way to install the lcurvetools package for the first time is using pip:

pip install lcurvetools

To update a previously installed package to the latest version, use the command:

pip install lcurvetools --upgrade

The lcurves function and its alias lcurves_by_history plot learning curves based on a dictionary or a list of dictionaries with model training histories. Each dictionary with a history should contain keys with training and validation values of losses and metrics, as well as learning rate values at successive epochs, for example: {"loss": [0.5, 0.3, 0.2], "val_loss": [0.6, 0.4, 0.25], "accuracy": [0.7, 0.8, 0.85], "val_accuracy": [0.65, 0.75, 0.8], "lr": [0.01, 0.01, 0.001]}. This training history format is adopted in the Keras library and is also widely used when working with other libraries.

Neural network model training with Keras is performed using the fit method. The method returns the History object with the history attribute which is dictionary and contains keys with training and validation values of losses and metrics, as well as learning rate values at successive epochs. The lcurves function and its alias lcurves_by_history use the History.history dictionary to plot the learning curves as the dependences of the above values on the epoch index.

Import the keras module and the lcurves function:

import keras
from lcurvetools import lcurves

1. Create, compile and fit the keras model:

   model = keras.Model(...) # or keras.Sequential(...)
   model.compile(...)
   hist = model.fit(...)

2. Use hist.history dictionary to plot the learning curves as the dependences of values of all keys in the dictionary on an epoch index with automatic recognition of keys of losses, metrics and learning rate:

   lcurves(hist.history);

The appearance of the output figure depends on the list of keys in the hist.history dictionary, which is determined by the parameters of the compile and fit methods of the model. For example, for a typical usage of these methods, the list of keys would be ['loss', 'accuracy', 'val_loss', 'val_accuracy'] and the output figure will contain 2 subplots with loss and metrics vertical axes and might look like this:

model.compile(loss="categorical_crossentropy", metrics=["accuracy"])
hist = model.fit(x_train, y_train, validation_split=0.1, epochs=50)
lcurves(hist.history);

Of course, if the metrics parameter of the compile method is not specified, then the output figure will not contain a metric subplot.

Minimum values of loss curves and the best values of metric curves are marked by points. The best values of metrics are determined based on the specified optimization mode for each metric (see optimization_modes parameter). By default, the optimization modes are determined automatically based on metric name with the lcurves.utils.get_mode_by_metric_name function.

Usage of callbacks for the fit method can add new keys to the hist.history dictionary. For example, the ReduceLROnPlateau callback adds the lr key with learning rate values for successive epochs. In this case the output figure will contain additional subplot with learning rate vertical axis in a logarithmic scale and might look like this:

hist = model.fit(x_train, y_train, validation_split=0.1, epochs=50,
    callbacks=[keras.callbacks.ReduceLROnPlateau()],
)
lcurves(hist.history);

The lcurves function has optional parameters to customize the appearance of the output figure. For example, the epoch_range_to_scale option allows to specify the epoch index range within which the subplots of the losses and metrics are scaled.

• If epoch_range_to_scale is a list or a tuple of two int values, then they specify the epoch index limits of the scaling range in the form [start, stop), i.e. as for slice and range objects.
• If epoch_range_to_scale is an int value, then it specifies the lower epoch index start of the scaling range, and the losses and metrics subplots are scaled by epochs with indices from start to the last.

So, you can exclude the first 5 epochs from the scaling range as follows:

lcurves(hist.history, epoch_range_to_scale=5);

For a description of other optional parameters of the lcurves function to customize the appearance of the output figure, see its docstring.

The lcurves function returns a numpy array or a list of the matplotlib.axes.Axes objects corresponded to the built subplots from top to bottom. So, you can use the methods of these objects to customize the appearance of the output figure.

axs = lcurves(history, epoch_range_to_scale=6)
axs[0].tick_params(axis="x", labeltop=True)
axs[-1].set_xlabel('number of passed epochs')
axs[-1].legend().remove()

1. Create, compile and fit the several keras models:

   model_1 = keras.Model(...) # or keras.Sequential(...)
   model_1.compile(...)
   hist_1 = model.fit(...)
   
   model_2 = keras.Model(...) # or keras.Sequential(...)
   model_2.compile(...)
   hist_2 = model.fit(...)
   
   <...>

2. Use a list of dictionaries [hist_1.history, hist_2.history, ...] to plot all learning curves of all models in a single figure:

   lcurves([hist_1.history, hist_2.history, ...]);

1. Organize a loop of multiple independent retraining of the model and create a list of dictionaries with fitting histories:

   histories = []
   for i in range(5):
       model = keras.Model(...) # or keras.Sequential(...)
       model.compile(...)
       hist = model.fit(...)
       histories.append(hist.history)

2. Use the histories dictionary list to plot all learning curves of the model in a single figure:

   lcurves(histories);

Note: The ability to plot all of a model's learning curves in a single figure is useful for k-fold cross-validation analysis.

This function is useful for combining histories of a model fitting with two or more consecutive runs into a single history to plot full learning curves.

• Import the keras module and the history_concatenate, lcurves_by_history function:

import keras
from lcurvetools import history_concatenate, lcurves

• Create, compile and fit the keras model:

model = keras.Model(...) # or keras.Sequential(...)
model.compile(...)
hist1 = model.fit(...)

• Compile as needed and fit using possibly other parameter values:

model.compile(...) # optional
hist2 = model.fit(...)

• Concatenate the .history dictionaries into one:

full_history = history_concatenate(hist1.history, hist2.history)

• Use full_history dictionary to plot full learning curves:

lcurves(full_history);

The scikit-learn library provides 2 classes for building multi-layer perceptron (MLP) models of classification and regression: MLPClassifier and MLPRegressor. After creation and fitting of these MLP estimators with using early_stopping=True the MLPClassifier and MLPRegressor objects have the loss_curve_ and validation_scores_ attributes with train loss and validation score values at successive epochs. The lcurves_by_MLP_estimator function uses the loss_curve_ and validation_scores_ attributes to plot the learning curves as the dependences of the above values on the epoch index.

• Import the MLPClassifier (or MLPRegressor) class and the lcurves_by_MLP_estimator function:

from sklearn.neural_network import MLPClassifier
from lcurvetools import lcurves_by_MLP_estimator

• Create and fit the scikit-learn MLP estimator:

clf = MLPClassifier(..., early_stopping=True)
clf.fit(...)

• Use clf object with loss_curve_ and validation_scores_ attributes to plot the learning curves as the dependences of loss and validation score values on epoch index:

lcurves_by_MLP_estimator(clf);

The lcurves_by_MLP_estimator function with default value of the parameter on_separate_subplots=False shows the learning curves of loss and validation score on one plot with two vertical axes scaled independently. Loss values are plotted on the left axis and validation score values are plotted on the right axis. The output figure might look like this:

Note: the minimum value of loss curve and the maximum value of validation score curve are marked by points.

The lcurves_by_MLP_estimator function has optional parameters to customize the appearance of the output figure. For example,the lcurves_by_MLP_estimator function with on_separate_subplots=True shows the learning curves of loss and validation score on two separated subplots:

lcurves_by_MLP_estimator(clf, on_separate_subplots=True);

And the epoch_range_to_scale option allows to specify the epoch index range within which the subplots of the losses and metrics are scaled (see details about this option in the docstring of the lcurves_by_MLP_estimator function).

lcurves_by_MLP_estimator(clf, epoch_range_to_scale=10);

For a description of other optional parameters of the lcurves_by_MLP_estimator function to customize the appearance of the output figure, see its docstring.

The lcurves_by_MLP_estimator function returns a numpy array or a list of the matplotlib.axes.Axes objects of an output figure (see additional details in the Returns section of the lcurves_by_MLP_estimator function docstring). So, you can use the methods of these objects to customize the appearance of the output figure.

axs = lcurves_by_MLP_estimator(clf, epoch_range_to_scale=11)
axs[0].grid(axis='y', visible=False)
axs[1].grid(axis='y', visible=False)
axs[0].set_xlabel('number of passed epochs')
axs[1].set_ylabel('validation accuracy');