Keras is a great deep learning framework that is as easy to use as SkLearn.

However, because it is so abstract, it is not as convenient as PyTorch in customizing some attributes of the model.

Today write a problem encountered in the process of using, and the solution.

How to save the model with the highest F1-score on Val Data

Metrics supported by Keras does not include F1-scores, but in classification problems, F1-scores is an important evaluation index.

How to calculate F1 Macro in Keras?

I mean this is misleading to a lot of people, how did he solve this problem? As follows:

from keras import backend as K

def f1(y_true, y_pred):
    def recall(y_true, y_pred):
        """Recall metric.

        Only computes a batch-wise average of recall.

        Computes the recall, a metric for multi-label classification of
        how many relevant items are selected.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

    def precision(y_true, y_pred):
        """Precision metric.

        Only computes a batch-wise average of precision.

        Computes the precision, a metric for multi-label classification of
        how many selected items are relevant.
        """
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision
    precision = precision(y_true, y_pred)
    recall = recall(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))


model.compile(loss='binary_crossentropy',
          optimizer= "adam",
          metrics=[f1])Copy the code

Then add “val_f1” to earlyStop’s monitor,

early = EarlyStopping(monitor="val_f1", mode="max", patience=4)Copy the code

It seems that we have solved the problem of how to get the best val_F1 model, but in fact f1 is not what we want… Because it is Only computes a batch-wise average of recall. There is a big difference between batch f1 and EPOCH f1!!

The solution

The solution is simple: Keras provides a powerful Callback class that can be inherited from a Callback class

Take the dichotomy problem for example,

from sklearn.metrics import f1_score, recall_score, precision_score
from keras.callbacks import Callback

def boolMap(arr):
    if arr > 0.5:
        return 1
    else:
        return 0


class Metrics(Callback):
    def __init__(self, filepath):
        self.file_path = filepath

    def on_train_begin(self, logs=None):
        self.val_f1s = []
        self.best_val_f1 = 0
        self.val_recalls = []
        self.val_precisions = []

    def on_epoch_end(self, epoch, logs=None):
        val_predict = list(map(boolMap, self.model.predict([self.validation_data[0], self.validation_data[1]])))
        val_targ = self.validation_data[2]
        _val_f1 = f1_score(val_targ, val_predict)
        _val_recall = recall_score(val_targ, val_predict)
        _val_precision = precision_score(val_targ, val_predict)
        self.val_f1s.append(_val_f1)
        self.val_recalls.append(_val_recall)
        self.val_precisions.append(_val_precision)
        print(_val_f1, _val_precision, _val_recall)
        print("max f1")
        print(max(self.val_f1s))
        if _val_f1 > self.best_val_f1:
            self.model.save_weights(self.file_path, overwrite=True)
            self.best_val_f1 = _val_f1
            print("best f1: {}".format(self.best_val_f1))
        else:
            print("val f1: {}, but not the best f1".format(_val_f1))
        returnCopy the code

It’s easy to use,

metrics = Metrics(file_path)
callbacks_list = [metrics]Copy the code

Just add the callback_list to the model training process.