Article source can be in wechat public number “01 binary” background reply “image retrieval” to obtain.

preface

In our previous article, “Image Retrieval Series — Detecting Image Similarity with Python,” we introduced a very common algorithm in image retrieval, the perceptive hashing algorithm. This is a very simple and fast algorithm that generates a specific “fingerprint” for each image and then uses a similarity measure to determine how close the two images are.

However, with the rise of deep learning, the development of image field has been greatly promoted. In terms of feature extraction, neural network has irreplaceable advantages. In the last article, we also introduced that image retrieval is often based on the comparison of features of images, to see how much feature matching degree, so as to retrieve images with high similarity. And detection image features, VGG16 has unique advantages.

Next, this paper will use a simple case to realize a small tool for image retrieval based on deep learning.

The preparatory work

As usual, let’s prepare the tools we will use this time:

  • IDE: Pycharm
  • Python: 3.7
  • Packages: Keras + TensorFlow + Pillow + Numpy

keras

Keras is a high-level neural network API written in pure Python and based on Tensorflow, Theano, and CNTK backends. To put it simply, KerAS is another encapsulation of frameworks such as TF, making it easier to use.

Based on vgg16 network to extract the image feature VGG, as we all know, the network has been widely used in the field of image, following many level deeper, wider network model is based on the extension, VGG network can be a very good image to extract the useful characteristics of the implementation is based on Keras, extraction is the last layer of the convolution.

Train of thought

The main idea is based on CVPR2015 paper “Deep Learning of Binary Hash Codes for Fast Image Retrieval” to achieve a content-based Image Retrieval system under massive data. Simply put is the characteristics of each image extraction of image database (general form for the eigenvector), stored in a database, for retrieving images, extracting the same feature vector, and then and the vector and the distance vector in the database (similarity calculation), find out some of the most close to the eigenvector, its corresponding image is the search results. As shown below:

The user request and pre-processing part is mainly the Web server should do, not discussed here, next we will focus on the implementation of the red line part.

In field

Extracting image features

Keras provides a demo of feature extraction using VGG16 in its Chinese documentation

from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np

model = VGG16(weights='imagenet', include_top=False)

img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224.224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

features = model.predict(x)
Copy the code

Here we need to make a simple change to encapsulate it into a class for later invocation. As shown below:

Considering the length, many comments have been deleted on the code pictures in this paper. If you want to know the detailed comment information, you can reply to “Image Retrieval” in the background of wechat public account “01 binary” to obtain the source code. The same below

Save the features and their corresponding file names as h5 files

What is an H5 file

H5 files are the 5th generation version of Hierarchical Data Format (HDF5) for storing and organizing large-scale Data.

H5 simplifies the file structure into two main object types:

  1. Dataset is a multi-dimensional array of the same type of data

  2. A group is a container structure that can contain data sets and other groups. If different types of data sets are stored in a file, the management of these data sets uses groups

For intuitive understanding, we can refer to our file system, different files in different directories:

Directory is the group in hdF5 file, which describes the classification information of DataSet, and manages and divides various DataSet effectively through group. A file is the dataset in an HDF5 file, representing the specific data

Here is the relationship between dataset and group:

In Python, we usually use the H5PY library to operate on.h5 files.

Extract image features from data set and save them in H5 file

We’ll name a database folder under the project root as the data set and write a method to get the images in the folder:

def get_imlist(path):
    return [os.path.join(path, f) for f in os.listdir(path) if f.endswith('.jpg')]
Copy the code

And then we can read the data one by one and extract the features and save them in a file. The diagram below:

At this point, we’re almost done training the model.

Select a test image to test the retrieval effect

After the above operations, we have saved the features of all the images in the data set into the model, and the rest is to extract the features of the images to be tested, and then compare the similarity between the vectors (cosine similarity) with the features in the feature set one by one, and then return them to the user in order of similarity.

Tips: Python representations of various degrees of similarity can be calculated by referring to Python Numpy

Taking a bag as the test picture, the output result is as follows:

PyCharm allows you to easily view the images generated by Matplotlib. The first one is the test image and the next three are the search images. You can see that the results are quite good.

Tips: If you want to use Resnet or Densenet to extract features, you just need to make changes to the above code, removing comments and modifying parts of the code. See the source code.

The last

So far we have used deep learning to achieve a small tool for image retrieval, how to combine it with web/ APP is not the scope of this article, interested in download the source code to change, you can also scan the qr code below wechat public number “01 binary” to contact me.

reference

  1. Deep Learning and Computer Vision (11) fast Image Retrieval System based on Deep Learning
  2. Massive Image Retrieval System Based on VGG-16
  3. Based on deep learning to achieve graph search function
  4. Python implementation of various similarity calculations
  5. Application Application
  6. Python Numpy calculates all kinds of distances
  7. H5 File Introduction