Reasons for using MaxWell

Since a recent project needed to synchronize data from the subsystem to the main system and needed to add some additional information to the data transferred from the subsystem, it was considered possible to use some components to synchronize data from the database. Ali’s Canal was the first to be sought, but there were some problems in practical use, such as frequently not receiving data changes. After searching for a long time, I did not find the specific reason (I still do not know why, maybe I have a configuration problem), so I turned to MaxWell.

MaxWell’s introduction

MaxWell is an application that can read the MySQL binary log binlog in real time and generate JSON messages that are sent as producers to Kafka, Kinesis, RabbitMQ, Redis, Google Cloud Pub/Sub, files, and other platforms. Its common application scenarios include ETL, cache maintenance, table level DML metrics collection, increment to search engine, data partition migration, database cutting binlog rollback scheme, etc.

The environment

Maxwell is developed in Java, so install Java first.

Configure the mysql

Conf file (Windows: my.ini), open bin-log and change the format of bin-log

log-bin=master
binlog_format=row
Copy the code

Download the MaxWell

MaxWell Download link

Configuration MaxWell

Unzip and copy Maxwell’s config

CD maxwell cp config. Properites. Example config. The properites modify the configuration file vim config. The properitesCopy the code

Configuration file Contents

# tl; Dr Config log_level=info # Host =localhost user=maxwell password=maxwell # debug log_level= debug Rabbitmq_host =127.0.0.1 rabbitmq_port=5672 rabbitmq_user=admin rabbitmq_pass=admin rabbitmq_virtual_host=/ rabbitmq_exchange=maxwell rabbitmq_exchange_type=direct rabbitmq_exchange_durable=true rabbitmq_exchange_autodelete=false rabbitmq_routing_key_template=%db%.%table% rabbitmq_message_persistent=false rabbitmq_declare_exchange=trueCopy the code

run

Once configured, open Maxwell, starting mysql and RabiitMQ first

Linux

CD./bin #config is the configuration file to use./maxwell --config=.. /config.properites --daemon # After the daemon is used, you can see information about the data in the logs directory. tail -f .. /logs/MaxwellDaemon.outCopy the code

Windows

java -Dfile.encoding=UTF-8 -Dlog4j.shutdownCallbackRegistry=com.djdch.log4j.StaticShutdownCallbackRegistry -cp .. /lib/* com.zendesk.maxwell.Maxwell --config=.. /config.propertiesCopy the code

Data synchronization

After MaxWell drops the data to MQ, MQ allocates it to the corresponding queue via routing_key, connects to MQ using PHP to consume the data, and then synchronizes the data between databases.

The end of the

The actual operation effect is not bad, there is no problem, generally speaking, it is quite good.