ElasticSearch, or ES for short, is a distributed, RESTful search and analytics engine that solves the growing number of combinational search applications. At the heart of the Elastic Stack, it stores your data centrally, helping you find what you expect and what you don’t expect.

1, ES basic knowledge

ES is an integration tool that integrates the search engine Lucense.

A picture is worth a thousand words:

2. Complete data collection and index establishment

Our data source comes from MySql, and it reads data from MySql logs, and we’ve talked about it a couple of times, so let’s review it.

2.1 Setting MySql to support Binlog

Modify the /etc/my.conf configuration and add the following:

server-id=123
log-bin=mysql-bin
binlog_format=ROW
Copy the code

Restart the MySql service.

2.2 Added StreamSet support for Mysql

I hate Streamset, why not integrate it lol.

Install Mysql Jdbc driver and BinaryLog driver: driver download: cdn.mysql.com//Downloads/… Streamsets-datacollector /streamsets-libs /streamsets-datacollector/streamsets-libs /streamsets-datacollector/streamsets-libs Streamsets -datacollector-jdbc-lib and streamsets-datacollector-mysql-binlog-lib copy mysql JDBC driver to these two directories respectively. Restart the StreamSet service. Jar: chmod -r 777 mysql-connector-java-8.0.22.jar: chmod -r 777 mysql-connector-java-8.0.22.jar

3. Draw flow charts

One of the tricksIn order to avoid errors in the ES target, we can use the upsert method (with updates, without inserts).

One of the bugs: An error occurs when you start and stop pipeline tasks repeatedly.

Pipeline Status: RUNNING_ERROR: MYSQL_006 - MySql server error: 
A slave with the same server_uuid/server_id as this slave has connected to the master;
 the first event 'mysql-bin.000001' at 278098179,
  the last event read from './mysql-bin.000001' at 278117580,
  the last byte read from './mysql-bin.000001' at 278117580.
Copy the code

We simply change the configuration server ID of the mysql binary log. It is possible that the connection may not be closed during frequent startup and shutdown.

3.1 Detailed Configuration Parameters

See CSDN for details below.

4. Operation and observation

Well, the big face slapping didn’t happen, so everything went well!Index management already has the index we need.

5 Data Panel

Use Kibana to create a query index, and then observe the data in the data search, as follows:

Further optimizations may be tailored to demand, but most of the work has already been done.

Doesn’t it feel good not to write code?

6, summary

Huangdi neijing has a cloud: spring in March, this is hair Chen, heaven and earth are born, all things with glory, sleep at night to get up early, wide step in the court, be hair slow form, to make chi sheng, born but not kill, to but not take, appreciation and not punishment, this spring should be, the way to keep alive. Against the liver, summer for cold change, less in the elderly.

Since the ancient scripture said so, brothers and sisters, in accordance with the season, don’t you quickly click “like”, one key three even what? Raise liver protect liver, praise that line!