Deep learning evaluation index summary and code implementation

Classification task

Confusion matrix

The confusion matrix is the classification result of the statistical classification model, that is, the number of samples classified into the right class and wrong class are counted, and the result is displayed in a table, which is the confusion matrix.

Preliminary understanding of confusion matrix, when taking dichotomous confusion matrix as an introduction, multi-classification confusion matrix is based on dichotomous as an extension!

For the dichotomous problem, category 1 is called Positive, category 2 is called Negative, the correct prediction of classifier is denoted as True, and the wrong prediction is denoted as False. The four basic terms are combined to form the four basic elements of the confusion matrix, as follows:

TP (True Positive) : a True case, which is predicted to be Positive by the model but actually Positive (predicted by the model to be category 1, but actually category 1)
FP (False Positive) : False Positive example, Positive example predicted by the model, but actually negative example (category 1 predicted by the model, category 2 actually)
FN (False Negative) : False Negative example, Negative example predicted by model, positive example actually (category 2 predicted by model, category 1 actually)
TN (True Negative) : True Negative example, the model predicted as Negative example, actually is Negative example (model predicted as category 2, actually category 2)

The evaluation index

import numpy as np
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score, multilabel_confusion_matrix
np.seterr(divide='ignore',invalid='ignore')

""" ConfusionMetric Mertric P N P TP FN N FP TN """
class ClassifyMetric(object) :
    def __init__(self, numClass, labels=None) :
        self.labels = labels
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,)*2)
        
    def genConfusionMatrix(self, y_true, y_pred) :
    	  return confusion_matrix(y_true, y_pred, labels=self.labels)
      
    def addBatch(self, y_true, y_pred) :
        assert  np.array(y_true).shape == np.array(y_pred).shape
        self.confusionMatrix += self.genConfusionMatrix(y_true, y_pred)
 
    def reset(self) :
        self.confusionMatrix = np.zeros((self.numClass, self.numClass))

    def accuracy(self) :
    	  accuracy = np.diag(self.confusionMatrix).sum() / self.confusionMatrix.sum(a)return accuracy

    def precision(self) :
    	  precision = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=0)
    	  return np.nan_to_num(precision)

    def recall(self) :
    	  recall = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=1)
    	  return recall

    def f1_score(self) :
    	  precision = self.precision()
    	  recall = self.recall()
    	  f1_score = 2 * (precision*recall) / (precision+recall)
    	  return np.nan_to_num(f1_score)

if __name__ == '__main__':
    y_true = ["cat"."ant"."cat"."cat"."ant"."bird"]
    y_pred = ["ant"."ant"."cat"."cat"."ant"."cat"]
    metric = ClassifyMetric(3["ant"."bird"."cat"])
    metric.addBatch(y_true, y_pred)
    acc = metric.accuracy()
    precision = metric.precision()
    recall = metric.recall()
    f1Score = metric.f1_score()
    print('acc is : %f' % acc)
    print('precision is :', precision)
    print('recall is :', recall)
    print('f1_score is :', f1Score)

    print('\n')
    # Compare with sklearn

    metric = confusion_matrix(y_true, y_pred, labels=["ant"."bird"."cat"])
    accuracy_score1 = accuracy_score(y_true, y_pred)
    print("accuracy_score", accuracy_score1)
    precision_score1 = precision_score(y_true, y_pred, average=None, zero_division=0)
    print("precision_score", precision_score1)
    recall_score1 = recall_score(y_true, y_pred, average=None, zero_division=0)
    print("recall_score", recall_score1)
    f1_score1 = f1_score(y_true, y_pred, average=None)
    print("f1_score", f1_score1)
Copy the code

Semantic segmentation

Semantic segmentation is a pixel-level classification, and its commonly used evaluation indicators are:

Pixel Accuracy (PA),
Class Pixel Accuray (CPA),
Category Mean Pixel Accuracy (MPA),
Intersection over Union, IoU
Mean Intersection over Union, MIoU
Frequency Weighted Intersection over Union (FWIoU)

The calculation is based on the Confusion Matrix. Therefore, a basic knowledge of obfuscation matrix is of great benefit to the understanding of the six commonly used evaluation indicators.

Confusion matrix

The focus of confusion matrix in semantic segmentation is not on categories, but on pixels, to judge whether a pixel is correctly predicted.

The calculation formula of semantic segmentation confusion matrix is as follows:

def generate_matrix(label_true, label_pred, n_class) :
    mask = (label_true >= 0) & (label_true < n_class)
    label = n_class * label_true[mask] + label_pred[mask]
    confusionMatrix = np.bincount(label, minlength=n_class**2).reshape(n_class, n_class)
    return confusionMatrix
Copy the code

Parsing the code in detail:

label_true = np.array([
	[0.1.1],
	[2.1.0],
	[2.2.1]])
label_pred = np.array([
	[0.2.0],
	[2.1.0],
	[1.2.1]])

n = 3
mask = (label_true >= 0) & (label_true < n)
print(mask, '\n')
This is to ensure that the tag is correct (each element of the tag is within [0, n_class)). The mask marked correctly is an array of all true values [[true true true] [true true true] [true true true]] ""

a = label_true[mask].astype(int)
print(a, '\n')
# flatten label_true [0 1 1 2 1 0 2 2 1]

b = label_pred[mask]
print(b, '\n')
# flatten label_pred [0 2 0 2 1 0 1 2 1]

c = n * a + b
print(c, '\n')
# [0 5 3 8 4 0 7 8 4] 

d = np.bincount(c, minlength=n**2)
print(d, '\n')
# [2 0 0 1 2 1 0 1 2] 

""" -minlength =n_class ** 2 The length of the output vector is n_class * n_class. - According to np.bincout, each value of an element in C is the index of an element in D +1, that is to say, the value of an element in C is actually the index corresponding to D, and the element in D is the number of occurrences of an element in C, such as D [4], which represents element 4 in C. D [4] = 2; d[4] = 2; That's a times n plus b.

D = d.reshape(n,n)
print(D, '\n')
0 0 Is 0 0 converting vector D into a 3 by 3 matrix through n (n, n) [2 0 0] [1 2 1] [0 1 2] then the value of D[I,j] is the original value of D[I *n+j], the value of D[I,j] represents the number of times I *n+j appears in C, c = a*n + b, so it can be seen that I corresponds to a,j corresponds to B, And at the same position of A and B, they exactly represent the real category and the predicted category, that is, D[I,j] represents the number of all pixels with the predicted result as category J and the actual label as category I. D[1,1]= D[1 *3+1]= D[4]=2, indicating that the number of all pixels with prediction category 1 and actual label 1 is 2. "" "
Copy the code

The evaluation index

import numpy as np
import itertools
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

""" ConfusionMetric Mertric P N P TP FN N FP TN """
class SegmentationMetric(object) :
    def __init__(self, numClass) :
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,)*2)
        
    def genConfusionMatrix(self, imgLabel, imgPredict) :
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass**2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix
      
    def addBatch(self, imgLabel, imgPredict) :
        assert  imgLabel.shape == imgPredict.shape
        self.confusionMatrix += self.genConfusionMatrix(imgLabel, imgPredict)
 
    def reset(self) :
        self.confusionMatrix = np.zeros((self.numClass, self.numClass))

    def plot_confusion_matrix(self, classes, normalize=False, title='Confusion matrix', cmap=plt.cm.Blues) :
        cm = self.confusionMatrix
        if normalize:
            cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
            print("Normalized confusion matrix")
        else:
            print('Confusion matrix, without normalization')
     
        plt.imshow(cm, interpolation='nearest', cmap=cmap)
        plt.title(title)
        plt.colorbar()
        tick_marks = np.arange(len(classes))
        plt.xticks(tick_marks, classes, rotation=45)
        plt.yticks(tick_marks, classes)
     
        thresh = cm.max(a) /2.
        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
            num = '{:.2f}'.format(cm[i, j]) if normalize else int(cm[i, j])
            plt.text(j, i, num,
                     verticalalignment='center',
                     horizontalalignment="center",
                     color="white" if num > thresh else "black")
     
        plt.tight_layout()
        plt.ylabel('True label')
        plt.xlabel('Predicted label')
        plt.show()

    # Evaluation index
      
    def pixelAccuracy(self) :
        # return all class overall pixel accuracy
        # PA = acc = (TP + TN) / (TP + TN + FP + TN)
        acc = np.diag(self.confusionMatrix).sum() /  self.confusionMatrix.sum(a)return acc
 
    def classPixelAccuracy(self) :
        # return each category pixel accuracy(A more accurate way to call it precision)
        # acc = (TP) / TP + FP
        classAcc = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=0)
        return classAcc # return a list value, e.g. [0.90, 0.80, 0.96], representing the prediction accuracy of categories 1, 2, and 3
 
    def meanPixelAccuracy(self) :
        classAcc = self.classPixelAccuracy()
        meanAcc = np.nanmean(classAcc) # np.nanmean mean, nan means encounter nan type, its value is 0
        return meanAcc Nanmean ([0.90, 0.80, 0.96, nan, nan]) = (0.90 + 0.80 + 0.96) / 3 = 0.89
      
    def intersectionOverUnion(self) :
      	# Intersection = TP Union = TP + FP + FN
        # IoU = TP / (TP + FP + FN)
        intersection = np.diag(self.confusionMatrix) # take the value of the diagonal element and return the list
        union = np.sum(self.confusionMatrix, axis=1) + np.sum(self.confusionMatrix, axis=0) - np.diag(self.confusionMatrix) # axis = 1 denotes the value of the obfuscation matrix row, which returns the list; Axis = 0 means to take the value of the obfuscation matrix column and return the list
        IoU = intersection / union  # return a list of IoU values for each category
        return IoU
      
    def meanIntersectionOverUnion(self) :
        IoU = self.intersectionOverUnion()
        mIoU = np.nanmean(IoU) Find the average of each category of IoU
        return mIoU
 
    def frequencyWeightedIntersectionOverUnion(self) :
        # FWIOU = [(TP+FN)/(TP+FP+TN+FN)]*[TP/(TP+FP+FN)]
        freq = np.sum(self.confusionMatrix, axis=1) / np.sum(self.confusionMatrix)
        iu = self.intersectionOverUnion()
        FWIoU = (freq[freq > 0] * iu[freq > 0]).sum(a)return FWIoU


if __name__ == '__main__':
    label_true = np.array([0.1.1.2.1.0.2.2.1]) # Can be directly replaced with a labeled picture
    label_pred = np.array([0.2.0.2.1.0.1.2.1]) # Can be directly replaced with a prediction picture
    metric = SegmentationMetric(3) # 3 indicates that there are three categories. Fill in as many categories as possible
    metric.addBatch(label_true, label_pred)
    print(metric.confusionMatrix)
    pa = metric.pixelAccuracy()
    cpa = metric.classPixelAccuracy()
    mpa = metric.meanPixelAccuracy()
    IoU = metric.intersectionOverUnion()
    mIoU = metric.meanIntersectionOverUnion()
    FWIoU = metric.frequencyWeightedIntersectionOverUnion()
    print('pa is : %f' % pa)
    print('cpa is :', cpa)
    print('mpa is : %f' % mpa)
    print('IoU is :', IoU)
    print('mIoU is : %f' % mIoU)
    print('FWIoU is : %f' % FWIoU)
    # metric.plot_confusion_matrix(classes=['background', 'cat', 'dog'])

    # contrast sklearn
    metric = confusion_matrix(label_true, label_pred)
    print(metric)
Copy the code

Deep learning evaluation index summary and code implementation

Deep learning evaluation index summary and code implementation

Classification task

Confusion matrix

The evaluation index

Semantic segmentation

Confusion matrix

The evaluation index

Related Posts

Minute homemade face recognition (how to quickly identify the favorite sister ~)

RepPointsV2: more monitoring, stronger performance | NIPS 2020

Seeing the future of AI: Today’s platform is tomorrow’s ecology