0. Write first

Earlier we introduced the implementation of the FM model, and also mentioned its influence in the recommendation area. In this paper, we will talk about DeepFM, a recommendation model combining neural network and FM. Other models with related ideas will be introduced successively in the following articles.

Personal experience:

  1. Combined with MLP’s high-order feature combination ability and FM’s low-order feature crossover ability
  2. DNN architecture representation of FM model
  3. Parallel structure design of model

Thesis Address:

Arxiv.org/pdf/1703.04…

Thesis Code:

Github.com/ChenglongCh…

1. The background

The FM model has proven to be effective in industry recommendation scenarios. In the current situation of deep learning neural network in full swing, is there a method that can skillfully combine the ability of FM model to deal with low-order feature crossover and the characteristics of DNN model to play the advantages of both in the recommendation scene? The answer is yes, Harbin Institute of Technology and Huawei put forward a recommendation model DeepFM with parallel structure, which supports this feature well. The following will introduce the overall model design from DeepFM model architecture, FM module and DNN module.

2. Model architecture

The model structure of DeepFM is shown in figure 1.

The whole model is divided into two parts, FM-part and DNN-part. The FM part uses neural network to realize the expression form of traditional FM model, and the Dnn part is a multi-layer neural network. Finally, the output of the two parts is summed up and the recommended result is obtained by Softmax.

3. FM part

The FM module is used to embed the traditional FM model in DeepFM. The FM neural network model is shown in the figure below.

First look at the input layer and embedding layer of FM module. The input layer and embedding layer of FM module are no different from the current mainstream practice, which can be divided into two steps:

  1. The category features are individually thermal coded, and then a layer of embedding
  2. Continuous features direct embedding

In embedding layer, the embedding dimension of each feature field has the same meaning as the vector dimension K in the traditional FM model. After embedding, enter FM computing layer. The calculation method of the FM computing layer here is completely consistent with the traditional FM model, as shown in the following formula. The first-order feature term weighted sum the neurons of the model input layer, and the second-order crossover feature term crosses each vector of the embedding layer. At this time, the weight of each vector is 1. Then, the first order and second order feature terms are summed up to perfectly realize the calculation process of FM.

4. DNN part

The DNN part of DeepFM still adopts MLP multi-layer neural network, as shown in the figure below.

The input layer and embedding layer of DNN module are completely consistent with FM, which will not be described here. DNN module splicing each vector in embedding layer horizontally and then feeding it into multi-layer neural network to realize the high order combination of features. Finally, the output of FM module and DNN module are summed and passed through softmax layer to obtain the final output result of the model.

5. To summarize

As a classic case of deep learning applied to recommendation scenarios, DeepFM integrated the advantages of FM model and MLP model, designed FM module to realize the second-order crossover of features, and combined with multi-layer neural network to improve the higher-order crossover of features of the model.