Deep Learning 006- Graft – Use Keras transfer learning to improve performance

(Python libraries and versions used in this article: Python 3.6, Numpy 1.14, Scikit-Learn 0.19, Matplotlib 2.2, Keras 2.1.6, Tensorflow 1.9.0)

An article in our own definition model to solve the problem of the binary classification, after 20 rounds of training to get about 74% accuracy, on the one hand is the cause of our epoch is so small, on the other hand is because the model is too simple, simple structure, so can’t be too complicated things, so how to improve forecast accuracy? One effective method is transfer learning.

The essence of transfer learning is graft: applying the weights of network structures and weights from other large data sets (such as ImageNet, etc.) to a new project, such as the cat/dog binary problem here. Of course, there are definitely two categories of cat and dog in ImageNet, so it can be said that the small data set here is a subset of ImageNet. However, for other data sets that have no relationship with ImageNet at all, transfer learning also has certain effect. Of course, for data sets with poor correlation between the two data sets, Using transfer learning may not work very well.

The specific approach is: A mature network structure (for example, VGG16 here) and parameters are used to remove all its full link layers, and only the convolutional layers are retained. These convolutional layers can be regarded as feature extractors of images (the features obtained are called bottleneck features), while the full link layer is a classifier, which can effectively classify the features of these images. For the new project, the number of categories we want to classify is not ImageNet’s 1000 classes, but like 2 classes here. Therefore, classifiers are useless to us. We need to create and train our own classifiers. The VGG16 network structure is as follows:

Conv blocks 1-5 are both convolution layer and pooling layer, which constitute the feature extractor of the image, while Flatten and Dense constitute the classifier.

Here, we transfer the structure and parameters of Conv Block 1-5 to assemble our own classifier.

During training, we can establish image data stream as in my last blog, import image data stream into VGG16 model to extract features, and then send these features into the self-defined classifier for training and optimize the parameters of the self-defined classifier. However, the training speed in this way is very slow. Here, we used the convolution layer of VGG16 to uniformly extract the features of all pictures, save these features, and then directly load features for training. Loading numbers is much faster than loading pictures, so the training is also much faster.

My main reference for this post is: Keras series-Image Multi-classification Training and Fine-tuning with Bottleneck Features (III), This blog post is also a reference to Building powerful image classification models using very little data, but I found that the codes in these two blog posts could not run in many places. The main reason was probably the Keras or Tensorflow upgrade, so I made some necessary changes.


1. Prepare data sets

First, use the pre-trained model VGG16 to extract the features of train set and test set images, and then save these features, which are actually numpy.ndarray, so they can be saved as numbers, and then load these numbers for training.

# The training set and test set here are not the train set and test set of the original picture, but the features extracted from the picture by VGG16, which constitute the new train set and test set
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
from keras import applications
def save_bottlebeck_features(a):
    datagen = ImageDataGenerator(rescale=1. / 255) # No image enhancement required

    # build the VGG16 network
    model = applications.VGG16(include_top=False, weights='imagenet') 
    # use the weights of imagenet as the initial weights of VGG16. Since it is only feature extraction, only the previous convolution layer is taken without the need for DenseLayer, so include_top=False

    generator = datagen.flow_from_directory( Generate train set
        train_data_dir,
        target_size=(IMG_W, IMG_H),
        batch_size=batch_size,
        class_mode=None, 
        shuffle=False) The value of # must be False; otherwise, it will not correspond to the following label.
    bottleneck_features_train = model.predict_generator(
        generator, train_samples_num // batch_size) # If 32, the division gives 62, discarding the decimals, and thus getting 1,984 samples
    np.save('E:\PyProjects\DataSet\FireAI\DeepLearning\FireAI006/bottleneck_features_train.npy', bottleneck_features_train)
    print('bottleneck features of train set is saved.')

    generator = datagen.flow_from_directory(
        val_data_dir,
        target_size=(IMG_W, IMG_H),
        batch_size=batch_size,
        class_mode=None,
        shuffle=False)
    bottleneck_features_validation = model.predict_generator(
        generator, val_samples_num // batch_size)
    np.save('E:\PyProjects\DataSet\FireAI\DeepLearning\FireAI006/bottleneck_features_val.npy',bottleneck_features_validation)
    print('bottleneck features of test set is saved.')
Copy the code

The trainset features are saved in the E: PyProjects\DataSet\FireAI\DeepLearning\FireAI006/ Library library. And the characteristics of the test set are also saved to.. / bottleneck_features_val. Npy.


2. Model and train

Obviously, we don’t need to extract the various features of the image here, the previous VGG16 has done that for us, so we just need to classify these features, so we can just build a classifier model.

A simple dichotomy model was established with Keras as follows:

def my_model(a):
    A custom model that is merely equivalent to a classifier, containing only the full connection layer, can be used to classify extracted features:
    # Structure of the model
    model = Sequential()
    model.add(Flatten(input_shape=train_data.shape[1:))Flatten all data
    model.add(Dense(256, activation='relu')) # 256 fully connected units
    model.add(Dropout(0.5)) # dropout regular
    model.add(Dense(1, activation='sigmoid')) # The model defined here is only the following full connection layer, which needs to be customized because it is special to this project

    # Model configuration
    model.compile(optimizer='rmsprop',
                  loss='binary_crossentropy', metrics=['accuracy']) Optimizer, etc

    return model
Copy the code

Although the model has been established, we need to train the parameters in it, using the features just extracted from VGG16 for training:

# Only need to train classifier model, do not need to train feature extractor
train_data = np.load('E:\PyProjects\DataSet\FireAI\DeepLearning\FireAI006/bottleneck_features_train.npy') # Load the vGG16-NOTOP feature for all images in the training photo set
train_labels = np.array(
    [0] * int((train_samples_num / 2)) + [1] * int((train_samples_num / 2)))
# label is 1000 CAT and 1000 dog. Here, VGG16 features are extracted in order, so [0] represents CAT and 1 represents dog

validation_data = np.load('E:\PyProjects\DataSet\FireAI\DeepLearning\FireAI006/bottleneck_features_val.npy')
validation_labels = np.array(
    [0] * int((val_samples_num / 2)) + [1] * int((val_samples_num / 2)))

# Build classifier model
clf_model=my_model()
history_ft = clf_model.fit(train_data, train_labels,
              epochs=epochs,
              batch_size=batch_size,
              validation_data=(validation_data, validation_labels))
Copy the code

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — – — – a — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Train on 2000 samples, Validate on 800 samples of Epoch 1/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 6 s 3 ms/step – loss: 0.8426 acc: 0.7455 – val_loss: 0.4280 – val_acc: 0.8063 Epoch 2/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 5 s 3 ms/step – loss: 0.3928 – ACC: 0.8365 – val_loss: 0.3078 – val_ACC: 0.8675 Epoch 3/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 5 s 3 ms/step – loss: 0.3144 acc: 0.8720 – val_loss: 0.4106 – val_acc: 0.8588

.

Epoch 18/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 5 s 3 ms/step – loss: 0.0479 acc: 0.9830 – val_loss: 0.5380 – val_acc: 0.9025 Epoch 19/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 5 s 3 ms/step – loss: 0.0600 acc: 0.9775-VAL_loss: 0.5357 – val_ACC: 0.8988 Epoch 20/20 2000/2000 [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] – 5 s 3 ms/step – loss: 0.0551 acc: 0.9810 – val_loss: 0.6057 – val_acc: 0.8825

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — – — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — –

Plot loss and ACC in the training process as follows:

Obviously, after the 5th epoch, train set and test set were separated significantly, indicating that there was a strong over-fitting later, but the accuracy rate of test set was still about 90%.

It can be seen that compared with the network structure of three-layer convolution layer + two-layer full connection layer defined by ourselves in the last article, the method of VGG16 network structure has higher accuracy and requires less time for training.

Note that we are not training any of the parameters in VGG16, but only the parameters in our own classifier model.

# # # # # # # # # # # # # # # # # # # # # # # # small * * * * * * * * * * and # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

1. Transfer learning is to use the existing model and its parameters to extract the features of pictures, and then build its own classifier to classify these features.

2. Here we are not training the structure and parameters of the existing model, only the custom classifier. If we want to train the parameters of the existing model, it is the category of fine-tune

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #


Note: This part of the code has been uploaded to (my Github), welcome to download.