Visualize the Keras training process

Keras provides a Callback interface to track the results of each step in the training process, including each batch and each epoch. Although named “Callback function,” the actual way to extend this functionality is to inherit from the keras.callbacks.Callback class, which provides two properties related to the model training process:

• params: Compile the parameters set when the model;
• model: Model object.

This interface enables real-time visualization of the variation in the error size of each batch and epoch iteration in the FIT process. Take “Neural Networks and Deep Learning – Chap3 Improving the Way Neural Networks Learn” as an example, suppose we want to train a simplest Neural network:

This neural network with only one neuron has two parameters to be trained, one weight W and one bias B. Suppose that the data to be trained is only (1, 0). Here compare the learning effects of the two cost functions, MSE and Cross Entropy.

First build this model:

from keras import Sequential, initializers, optimizers
from keras.layers import Activation, Dense

import numpy as np

def viz_keras_fit(w, b, runtime_plot=False, loss="mean_squared_error", act="sigmoid"):
    d = DrawCallback(runtime_plot=runtime_plot)
    
    # Initialize parameters
    w = initializers.Constant([w])
    b = initializers.Constant([b])

    x_train, y_train = np.array([1]), np.array([0])
    
    model = Sequential()
    model.add(Dense(1, 
        activation=act,
        input_shape=(1,),
        kernel_initializer=w,
        bias_initializer=b))

    # Learning Rate = 0.15
    sgd = optimizers.SGD(lr=0.15)
    model.compile(optimizer=sgd, loss=loss)

    model.fit(x = x_train,
        y = y_train,
        epochs=150,
        verbose=0,
        callbacks=[d]) # Callback List
    return d
Copy the code

The initial parameters are still (2, 2) replaced by Cross Entropy as the loss function:

Although real-time visualization is achieved, mapping may take longer than an epoch, so it would be better to record the loss of each step before mapping:

Observing the learning of the model in real time can help us select loss function, activation function, model structure and hyperparameters at the initial stage. Here is the implementation of the DrawCallback:

import pylab as pl
from IPython import display
from keras.callbacks import Callback

class DrawCallback(Callback):
    def __init__(self, runtime_plot=True):
        super().__init__()
        self.init_loss = None
        self.runtime_plot = runtime_plot
        
        self.xdata = []
        self.ydata = []
    def _plot(self, epoch=None):
        epochs = self.params.get("epochs")
        pl.ylim(0, int(self.init_loss*2))
        pl.xlim(0, epochs)
    
        pl.plot(self.xdata, self.ydata)
        pl.xlabel('Epoch {}/{}'.format(epoch or epochs, epochs))
        pl.ylabel('Loss {:.4f}'.format(self.ydata[- 1]))
        
    def _runtime_plot(self, epoch):
        self._plot(epoch)
        
        display.clear_output(wait=True)
        display.display(pl.gcf())
        pl.gcf().clear()
        
    def plot(self):
        self._plot()
        pl.show()
    
    def on_epoch_end(self, epoch, logs = None):
        logs = logs or {}
        loss = logs.get("loss")
        if self.init_loss is None:
            self.init_loss = loss
        self.xdata.append(epoch)
        self.ydata.append(loss)
        if self.runtime_plot:
            self._runtime_plot(epoch)
Copy the code

---

Notion Note: Visualize the Keras training process