preface

This semester is almost over. My main work in this semester is to develop recommendation algorithm model for an e-commerce company as an intern. Another thing is to finish the thesis proposal, and the final direction is also set on the recommendation system based on graph neural network. During this period of time, I have also reviewed a lot of literature and blogs. Let’s briefly talk about the application of representational learning rerecommendation system that I know. The level is limited, not in place, please understand.

What is representational learning?

There may be more or less different definitions of representational learning in different articles. At the beginning, I thought representational learning was a very abstract word, but with the increase of the number of articles I read, I have my own understanding. In general, in the process of machine learning, we often need to put specificThe targetAccording to specificrepresentationforsaid. For example, in the field of natural language processing (NLP), a word can be represented as a vector in a Euclidean space, where the vectors of near-synonyms should also be closer together. For a graph (network), a node can also be represented as a vector, and closely related nodes are closer to each other in Euclidean space. Learning goal specific representationprocessIt can be called representational learning. Of course, the above is only a simple example, there may be some differences in the actual business.

Why do we need representational learning?

First of all, representation learning is not absolutely necessary, such as in traditional recommendation systems, but matrix decomposition can also play a certain role. However, one advantage of using representation learning is that it can transform some hard-to-use data (such as text, topology nodes) into vectors that can be used by more algorithms (such as deep neural networks). In addition, in the process of representation learning, in order to make the representation vector (embedding vector) better represent the represented target, we often need to use some methods to capture the features of these targets, so representation learning also plays a role of feature extraction in some scenarios. In addition, sometimes representational learning can also play a role in reducing dimensions, which I won’t go into here.

Deep neural Network based representation learning

Because deep learning itself has a strong ability of representation, and deep learning has achieved good results in multi-task in recent years, the recommendation system based on deep learning has attracted the attention of many researchers and enterprises, and has become mature.

– Twin towersAs shown in the figure, it is not too difficult to understand the twin towers model. User information is input on the left side, and commodity information is input on the right side. With the strong representation ability of deep learning, we can obtain the embedded representation of user and commodity respectivelyu(x),v(y), for the top layer of the twin towers, after get embedded said, only need to use a < u (x), v (y) > embedded computing the distance vector, then the x, y actual situation (whether there is interaction) as label began to supervised learning, after the training, we can get two users can be generated and goods embedded depth of vector neural network (within DNN), In practical application, the two-tower model is often used for recall operation. The user embedding behind the left user tower needs to be calculated in real time when the request comes, and the user embedding behind the right item tower is expected to be calculated well when it is trained. In this way, the retrieval of the nearest vector becomes a problem in industrial application. There are many mature and efficient tools for retrieving nearest vectors, such as Faiss, Dirty travel, Elasticsearch, and more. [1, 2]

It is worth mentioning that Ali’s order gathering algorithm [3] based on SDNE is similar to the twin tower model. But the towers switched from DNN to DEEP autoencoders (DAE), which themselves are great tools for representation learning.

  • Neural Collaborative Filtering [4]

Deep collaborative filtering is a paper proposed in 2017. The idea is that user and item are input to a embedding layer respectively to get a preliminary embedding representation, and then applied to GMF and MLP layers respectively. GMF can be simply understood as a width model and MLP as a depth model. Combined with the Wide&deep model proposed by Google in 2016 [5], it is actually easy to understand the idea of deep collaborative filtering.

  • In addition, there are some classic papers, such as YouTubeDNN[6], A series of attention-based papers by Ali, DIN, DIEN, etc.

Representation learning based on Graph neural Network

Although traditional deep learning methods have achieved great success in extracting the features of Euclidean spatial data, in fact, data in many application scenarios are generated from non-Euclidean space, and the performance of traditional deep learning methods in processing non-Euclidean spatial data is still unsatisfactory. For example, in the field of e-commerce, the interaction record between users and goods constitutes a graph, and in social media, the user’s social network is also a graph. Figure complexity make the depth of the existing learning algorithm when dealing with faces huge challenges, because the graph topology is irregular, each figure the number of nodes is different, the each node in the graph may have a different number of adjacent nodes, lead to some important operation (such as convolution) and cannot be directly used in figure. In recent years, however, in many factors of success, the researchers drew on convolution, loop network and depth of the ideas of the automatic encoder, define and design the neural network structure of figure data for processing, put forward neural network, and found that its nodes in network structure information extraction and characterization of has the very good effect. Especially in the past two years, more and more researchers began to apply graph neural network to recommendation system. Here we mainly talk about two algorithms to illustrate the basic route of some recommendation systems based on book network.

  • Convolutional Collaborative Filtering with Multi-component Graph [9]

This paper is a paper published in AAAI2020. Although it is called graph convolution collaborative filtering, it feels more like graph attention collaborative filtering, and many graph attention mechanisms are applied in this paper. The author first constructs the interaction of user goods as a bipartite graph, and then learns the embedded representation of user and commodity respectively (this embedded representation represents the interest of user and commodity) through the graph attention mechanism. Multiple embedded representations are learned in multiple projection Spaces, and the weight of the embedded representations in different projection Spaces is learned again by using the attention mechanism, and then these embedded representations are weighted and aggregated to obtain the final embedded representation. After obtaining the final embedded representations of the user’s goods, these embedded representations can be put into a DNN for training, and the monitoring label is still whether the user has interacted with the goods.

  • Session-based Recommendation with Graph Neural Networks

This paper is a in figure session recommended classic papers used on neural network, it could be divided into a session, the user’s interaction history and establish session figure individually, then in the session figure used in figure embedded neural network learning objects and session said that this method has a good effect on dynamic capture user interest, but no consider user information.

The last

In general, the application of representation learning in recommendation system is worth studying. It can not only be used to do quick recall, but also to do sorting (fine sorting). In the fast recall stage, the embedding vector can be trained first, and then the recall set can be obtained with the help of efficient retrieval tools such as FAISS. In the sorting stage, embed vectors can be put into DNN for further training, or more complex models can be designed. In addition, with the popularity of graph neural network in the past two years, as well as its excellent performance in network representation, its application in recommendation system has become a hot spot, but the application of graph neural network recommendation system in industry still needs further exploration.

It is worth mentioning that, through this period of study, I found that in actual industrial production, there may be no need for particularly complex models. Efficiency, realizability and engineerability are a key consideration of practitioners.

Happy New Year, huh!

reference

  1. Youtube2019 Twin Towers recall paper intensive reading
  2. Recommendation system Embedding vector recall in immediate engineering practice
  3. Ali order algorithm for the first time open! Analysis of packaged Goods mining system based on Graph Embedding
  4. He X, Liao L, Zhang H, et al. Neural collaborative filtering[C]//Proceedings of the 26th international conference on world wide web. 2017: 173-182.
  5. Cheng H T, Koc L, Harmsen J, et al. Wide & deep learning for recommender systems[C]//Proceedings of the 1st workshop on deep learning for recommender systems. 2016: 7-10.
  6. Covington P, Adams J, Sargin E. Deep neural networks for youtube recommendations[C]//Proceedings of the 10th ACM conference on recommender systems. 2016: 191-198.
  7. Zhou G, Zhu X, Song C, et al. Deep interest network for click-through rate prediction[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1059-1068.
  8. Zhou G, Mou N, Fan Y, et al. Deep interest evolution network for click-through rate prediction[C]//Proceedings of the AAAI conference on artificial intelligence. 2019, 33: 5941-5948.
  9. Wang X, Wang R, Shi C, et al. Multi-component graph convolutional collaborative filtering[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 6267-6274.
  10. Wu S, Tang Y, Zhu Y, et al. Session-based recommendation with graph neural networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33: 346-353.