Personal feeling

Recently, I have been working on databus Binlog synchronization. I wrote a smooth article on databus setup and installation before. At that time, the project was just launched, and I did some preliminary research, but I didn’t expect so many problems in the later stage.

I was at home today deploying the first databus service to the company server and started synchronizing the test database data, finally getting out of my local development environment. That’s a little paragraph (there are still a few thorny issues to be resolved).

The following is my work review of these days:

  1. Fail fast

    First I went to Databus and cloned the databus source code, went to the Example directory, and figured out how to run it. In fact, the Databus documentation is relatively small, so you can only run the examples and fit your own requirements into them.

    If the documentation is not very detailed during the technical research, there are few online examples, and there is no relevant community, you can only continue to fail quickly, and slowly you will understand it (it is better to output a document of your own understanding to deepen your understanding).

    Check out Github to focus on issues, or do a search to see if someone else has posted anything about the issue that you can use as a reference to familiarize yourself with the process. A second look at the official technical documentation with the question in mind may be more rewarding

  2. Ask questions

    I also encountered a lot of syntax hurdles while writing the consumer block, such as how Java can design classes better, thread safety, connection pooling, versioning, Gradle build, etc.

    Because there is no relevant basis, of course, the first is to use the search engine to see if there is a solution, and then also sent to the Java group to ask questions, at the beginning there are people to answer my questions, and then basically no one. And then one of my friends blocked me.

    You can’t rely on a group of people, and unless you’re asking questions that people are interested in answering, you’re mostly on your own.

  3. Quick charge

    If you don’t know some basic knowledge, there are plenty of resources on the Internet for you to learn. Add some books you need to know to your bookshelf in advance, save some blog articles on your mobile phone, and read them in detail during subway time. In the process of reading, I calm down and think about whether there is a problem with my current design implementation and whether there is a better way to optimize it.

    When the problem to be solved is not solved, I usually do not feel so good after work or dinner. If a problem is not improved after trying for a long time, I still suggest leaving the station for a while.

    I usually send the error log to my phone, either on the subway or at home with a bite to eat. B: well… Maybe it is not a big problem, but sometimes it is some small details, most of which are ignored by the anxiety of solving the problem at that time.

  4. Don’t ignore the people around you

    If you have problems that you feel are difficult to solve or that you don’t have the experience to solve, you can ask your colleagues around you. It’s better to do research together than alone.

    More communication, active communication with various departments, professional knowledge in different fields can make you understand the essence of things, good interpersonal relationship can make you happier at work (I still need to improve this aspect)


Here are a few of the problems I encountered at work:

1. When we first launched binlog synchronization, it did not start from 00001 (we only kept it for 15 days).

Change maxScn configuration (1234 is the current number of binlog file positions)

1234 * (1<< 32) + 56789 = 5299989700053 (0x4D20000DDD5)
Copy the code

show master status; You can view the binlog information for the master library

Run the show binary logs command to view binlog information

SHOW BINLOG EVENTS [IN ‘log_name’] [FROM pos] [LIMIT [offset,] row_count] Query specific event information


Determination of SCN: The value consists of 64bits. The higher 32 bits indicate the number of the binlog file. Low 32-bit on behalf of the event in binlog file offset, such as in mysql – bin. 000001 file offset for 4 SCN is expressed as (1 < < 32) | 4 = 4294967300



2. 5.6 Post-binlog CRC32 Problem

show global variables like 'binlog_checksum'
show global variables like 'innodb_checksums';
Copy the code

Change to binlog_checksum=null, crC32 problem is not resolved

Crc32 databus/can be subprojects. Gradle externalDependency/’ or ‘=’ com. Google: open – replicator: 1.0.7 ‘instead ExternalDependency/’ or ‘=’ com. Zendesk: open – replicator: 1.4.4 ‘


3. Max_allowed_packet problem (preferably consistent)

show variables like  '%max_allow%';
set global max_allowed_packet = 32*1024*1024
Copy the code



4. Reset binlog(use with caution, it may cause master-slave problems)

reset master;
Copy the code



5. Databus restarts and deletes the location detection point, otherwise the previous binlog is not synchronized

client >  databus2-checkpointsCopy the code

6. Timestamp issues:

timestamp
datetime
Copy the code
5.7 TIMESTAMP 0000-00-00 00:00:00 Problemset session sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
Copy the code

Synchronizing these two time types is cumbersome, and there are compatibility issues with synchronizing 5.6 to 5.7. As for the explanation of this event field, I published a separate article before, you can have a look if you are interested.


This is far from over. The work has just begun. A new technology system, from ignorance to implementation and then to stable online operation, will step on many pits or even thunder, the reality requires that the problem must be strangled in the early stage. Share your code with others to review, share your ideas with others, and read more mature design solutions.



There are more potholes to tread.. I hope you keep paying attention.