ElasticSearch7+Spark build high matching search service + 1000 thousand recommendation system

Chapter 1 Course Guide [Finally Met You]

This chapter comprehensively describes the project business background, architecture design concept and the core technical capabilities needed for the review search recommendation course.

1-1 Course guide and Trial

1-2 *** Pre-School Required reading *** (Guide for learning and problem Solving)

1-3 Do you really ask questions?

Chapter 2 Project Design [Perfect execution process from project requirements to technical solutions]

This chapter describes how to simulate the whole process of review, search and recommendation project from business requirement BRD to product requirement PRD, and finally to technical solution selection, architecture design and technology implementation in the real situation.

2-1 Project design — Business requirements

2-2 Project design — Technical decomposition & module design trial

Stage 2-3 summary

2-4 [Stage Summary] Summary of project design

2-5 project design exercises after class

Chapter 3 Project Infrastructure [Basic Capabilities of Business System]

In this chapter, SpringBoot plus Mybatis framework is used to complete the capacity building of basic projects, and general return value, exception handling, page request resources and other processing methods are added to improve the basic capacity building of the business system.

3-1 Introduction to development tools

3-2 Springboot Web Construction

3-3 Service and Mybatis Access (part 1)

3-4 Service and Mybatis Access (Part 2)

3-5 General return and exception handling

3-6 Page request resource processing

3-7 [Stage Summary] Summary of project foundation construction

3-8 “Thinking hard, Consolidating Learning Achievements” Foundation Building exercises after class

Chapter 4 User, Operation and Merchant Capacity Building of basic Services

In this chapter, the business system capability of the project is divided into c-end user layer and operation background management layer, and the unified authority management for user-oriented login and registration service and operation configuration background is constructed, as well as a set of procedures for the entry system of comment merchants.

4-1 User Service Construction (I)

4-2 User Service Construction (Part 2)

4-3 User module front-end page access

4-4 Introduction to Metronic template of operation background

4-5 Operation Background Administrator Module (I)

4-6 Operation Background Administrator Module (Part 2)

4-7 Operation background home page module

4-8 Process for Merchants to enter (1)

4-9 Process for Merchants to enter (2)

4-10 Merchant Settlement Process (3)

4-11 Process for Merchants to enter (4)

4-12 [Stage Summary] Summary of user, operation and merchant capacity building of basic services

4-13 [Thinking hard, Consolidating learning Results] User, operation and merchant capacity building exercises for basic services

Chapter 5 Categories of basic Services and Capacity building of stores

In this chapter, the operation background capacity will be expanded, category service and shop opening ability will be added to the operation background, and all basic business capacity construction of the review will be completed.

5-1 Category Management Service (I)

5-2 Category Management Services (Middle)

5-3 Category Management Services (Part 2)

5-4 Store Management Service (I)

5-5 Store Management Service (Middle)

5-6 Store Management Services (Part 2)

5-7 [Stage Summary] Categories of basic services and summary of store capacity building

5-8 [Thinking diligently, Consolidating Learning Achievements] Basic service categories, store capacity building exercises

Chapter 6 Review Store Search Recommendation 1.0 Era [Initial Experience of Search Business]

In this chapter, the ability to use business system and database, the construction of keyword search secondary polymerization with multiple conditions screening filtering search capabilities, and combining the LBS geographic fence through distance control search rankings, at the same time will give preference to the distance and evaluation good merchant stores recommended to the user, to complete the review store search ability to recommend framework 1.0. .

6-1 stores recommended V1.0 access (above) trial

6-2 Store Recommended V1.0 Access (Medium)

6-3 V1.0 Access Recommended for Stores (Part 2)

6-4 Store search V1.0 access (1)

6-5 Store search V1.0 access (2)

6-6 Store search V1.0 access (3)

6-7 Store search V1.0 access (4)

6-8 [Phase Summary] Search recommended V1.0 architecture summary

6-9 [Thinking hard, Consolidating learning Achievements] Search recommended V1.0 structure exercises

Chapter 7: ElasticSearch7

In this chapter, we introduce ElasticSearch7 by discovering flaws in the search 1.0 architecture, explain the fundamentals of efficient and scalable distributed search engines, and introduce the basic syntax of es.

7-1 Objectives of this chapter

7-2 Nature and Principle of Search (I)

7-3 Nature and Principle of Search (2)

7-4 Setup of ElasticSearch and Kibana environment

7-5 Principles of Distributed Search Engines (I)

7-6 Principles of Distributed Search Engines (Middle)

7-7 Principles of Distributed Search Engines (Part 2)

Basic ES Grammar (PART 1)

Basic ES Grammar (middle)

Basic ES Grammar (part 2)

7-11 ES Advanced query syntax

7-12 Indicates the ES field type

7-13 [Stage Summary] Principles of ES Foundation summary & Environment

7-14 [Stage summary] ES Basic syntax index & basic query

7-15 [Stage summary] ES Basic grammar advanced basic query &field type

7-16 【 Thinking diligently, consolidating learning Achievements 】ES Basic Exercises

Chapter 8 ElasticSearch7 advanced level

In this chapter, advanced data import, advanced query statement, multi-field query, filtering sort, scoring principle of TF/IDF BM25 and the realization of custom score will be studied in depth based on open source data TMDB. Finally, closed-loop learning of ES will be conducted.

8-1 TMDB is introduced

TMDB Index Creation (PART 1)

TMDB Index Creation (Part 2)

8-4 TMDB document import

8-5 Advanced Query Statements (1)

8-6 Advanced Query Statements (2)

8-7 Advanced Query Statements (3)

8-8 Advanced Query Statements (4)

8-9 Advanced Query Statements (5)

8-10 Advanced Query Statements (6)

8-11 Advanced Multi-field Query (1)

8-12 Advanced Multi-field Query (2)

8-13 Advanced Multi-Field Query (3)

8-14 Advanced Multi-field Query (4)

8-15 Filtering and Sorting

Custom Score Calculation (I)

8-17 Custom Score Calculation (Ii)

8-18 [Phase Summary] Construction test of ADVANCED ES

[Stage summary] ES advanced multi-field query

8-20 [Stage Summary] Custom ranking of ES advanced order

8-21 [Thinking hard, Consolidating learning Results] ES Advanced thinking questions after class

Chapter 9 Back to review, Data Access in Search 2.0 Architecture

In this chapter, you will learn and access the IK Chinese word divider, and build full and incremental indexes for review searches using logstash-input- JDBC.

9-1 Construction of IK Word Segmentation (1)

9-2 IK Word Segmentation Construction (2)

9-3 IK Word Segmentation Construction (3)

9-4 Construction of IK Word Segmentation (4)

9-5 Construction of IK Word Segmentation (5)

Store Index Construction (I)

9-7 Store Index Construction (Ii)

Logstash – Input – JDBC Build Index (1)

Logstash – Input – JDBC Build Index (2)

Logstash – Input – JDBC Build Index (3)

9-11 Logstash – Input – JDBC Build Index (4)

9-12 [Phase Summary] Upgrade data access -IK Chinese word segmentation with ES tool

9-13 [Phase Summary] Upgrade data access – index building using ES tool

9-14. Upgrading data access using the ES tool – Exercises

Chapter 10 Back to Reviews, Search Access in search 2.0 Architecture

In this chapter, the application capability of the search access layer is upgraded. The original 1.0 search architecture based on the service database is changed to ElasicSearch7 based on the search engine and access the application through RestApi.

10-1 Store Search (1)

10-2 Store Search (2)

10-3 Store Search (3)

10-4 Store Search (4)

10-5 Java Search Access (1)

10-6 Java Search Access (2)

10-7 Java Search Access (3)

10-8 Java Search Access (4)

10-9 Java Search Access (5)

10-10 Java Search Access (6)

10-11 Java Search Access (7)

10-12 Remaining issues

10-13 [Phase Summary] Basic debugging of search access in search 2.0 architecture

10-14 [Phase Summary] Java access for Search Access in the Search 2.0 architecture

10-15 [Thinking hard, Strengthening learning Outcomes] Search access in search 2.0 architecture

Chapter 11 Relevance Transformation of Review Search Progression

In this chapter, the search algorithm is optimized by exploring the shortcomings of search 2.0 and combining with multi-dimensional technologies such as customized word segmentation, synonym extension and partof speech correlation remodeling.

11-1 Extended Thesaurus for Customized Word segmentation (PART 1)

11-2 Extended Thesaurus for Customized Word segmentation (Part 2)

11-3 Synonym extension

11-4 Correlation remodeling (I)

11-5 Correlation remodeling (Middle)

11-6 Correlation remodeling (II)

11-7 [Stage summary] Summary of relevance transformation of comment search progression

11-8 [Diligent thinking, solid Learning Results] Review the relevance of advanced search exercises

Chapter 12: Quasi-real-time Index for Evaluating Search Progress

In this chapter, the construction of quasi-real-time indexing capability is completed by exploring the deficiencies in search 2.0 and learning the customized Canal middleware.

Advanced Construction of Canal Index (1)

Advanced Construction of Canal Index (2)

Advanced Construction of Canal Index (3)

Advanced Construction of Canal Index (4)

Advanced Construction of Canal Index (5)

Advanced Construction of Canal Index (6)

12-7 Custom Canal Access (1)

12-8 Custom Canal Access (2)

12-9 Canal Custom Access (3)

12-10 Custom Canal Access (4)

12-11 Search course summary

12-12 [Phase Summary] Capacity building of quasi-real-time index

12-13 [Thinking, Consolidating learning Achievements] Quasi-real-time index construction exercises

Chapter 13 Review recommendation 2.0 Architecture advancement [Spark MLLib based machine learning recommendation algorithm]

In this chapter, by exploring the deficiencies in the review recommendation 1.0 architecture, the recommendation system will be reformed, and the hierarchical recommendation model with personalized recall algorithm and personalized sorting algorithm will be defined to complete the design of the recommendation architecture with thousands of faces.

13-1 Principles of Recommendation (I)

13-2 Principles of Recommendation (Part 2)

13-3 Recommended system architecture

13-4 Explanation of the principle of personalized ALS recall algorithm

13-5 Explanation of lr algorithm principle of personalized sorting

13-6 Description of the Spark principle

Realization of ALS Algorithm (PART 1)

Realization of ALS Algorithm (Middle)

Realization of ALS Algorithm (Part 2)

Implementation of LR Algorithm (PART 1)

Lr Algorithm Implementation (Middle)

Lr Algorithm Implementation (Part 2)

13-13 Comments recommended Access (I)

13-14 Comments recommended Access (Ii)

13-15 Course Summary

13-16 [Stage Summary] Summary of the recommendation system

Chapter 14 [Supplementary chapter] CTR estimation model algorithm

When the seller makes a search and recommendation, it will also analyze the user’s click on the product and purchase behavior and make recommendations based on this. CTR estimation models are used in this scenario. This chapter focuses on the random forest algorithm and GBDT algorithm in the CTR estimation model, and the implementation and testing of GBDT.

14-1 Introduction to the CTR prediction model

14-2 Explanation of CTR Estimation Model Algorithm (I)

14-3 Explanation of CTR Prediction Model Algorithm (Middle)

14-4 Explanation of CTR Estimation Model Algorithm (II)

14-5 CTR prediction model random forest algorithm

14-6 CTR estimation model GBDT algorithm

14-7 Realization of click rate prediction model

14-8 Click rate estimation ABtest

This course is over