Rsync + inotify implements real-time bidirectional automatic file synchronization

For more technical articles, check out: github.com/yongxinz/te…

Meanwhile, welcome to follow my wechat official account AlwaysBeta, more exciting content waiting for you.

Introduction to the

With the continuous expansion of the application system scale, data security and reliability also put forward better requirements, Rsync in the high-end business system has gradually exposed many deficiencies.

First, rsync synchronizes data by scanning all files for comparison and differential transmission. If you have millions or even tens of millions of files, it’s time-consuming to scan all of them, and very few of them are changing, which is very inefficient.

Second, rsync cannot monitor and synchronize data in real time. Although rsync can be triggered by the Linux daemon process, the time difference between the two triggered actions must occur. As a result, data on the server and client may be inconsistent, and data cannot be fully recovered in the event of an application failure.

Based on the above two cases, rsync+inotify can be used to solve the problem, and real-time data synchronization can be achieved.

Inotify is a powerful, fine-grained, asynchronous mechanism for controlling file system events. The Linux kernel has added inotify support since 2.6.13. Inotify monitors the addition, deletion, modification, and movement of files in a file system. Using this kernel interface, third-party software can monitor changes to files in a file system, and inotify tools is the software that does that. After the first full synchronization using Rsync, inotify can monitor the source directory in real time. Whenever a file changes or new files are generated, it will be synchronized to the target directory immediately. It is very efficient and practical.

rsync

The installation

yum -y install rsync
Copy the code

Source code installation is not introduced here.

Commonly used parameters

-v: displays detailed synchronization information. -A: archive mode, equivalent to -rlptGod. -r: indicates a recursive directory"modify time"-g: keep file ownership group -o: Keep file owner -d: Keep device files and special files like --devices --specials -z: compress and transfer data before sending -h: keep hard link -n: test run, Do not make any changes -P sameas--partial --progress --partial: support resumable --progress: show transfer progress --delete: if the source file disappears, Target files can also be deleted. --delete-excluded: specify files to be deleted at the destination. --delete-after: By default, rsync cleans up files at the destination before starting data synchronization. If you use this option, rsync will synchronize data first and then delete files that need to be cleaned up. --exclude=PATTERN: exclude files that match PATTERN --exclude-from-e SSH: Indicates that SSH is used for encrypted tunnel transmissionCopy the code

Deploy using

• Server A: 192.168.0.1
• Server B: 192.168.0.2

With two Linux servers, we can assume that A is the server and B is the client.

1. Server configuration:

Modify the configuration file /etc/rsyncd.conf on the server as follows:

# rsync daemon user
uid = www
# Group to run the rsync daemon
gid = www
If chroot is set to yes, you must use root permission and cannot back up link files outside the path
use chroot = yes
# read-only
read only = no
# only write
write only = no
# set white list, you can specify the IP segment (172.18.50.1/255.255.255.0), each IP segments separated by Spaces
hosts allow = 192.168. 02.
hosts deny = *
# Maximum number of client connections allowed
max connections = 4
Welcome file path, optional
motd file = /etc/rsyncd.motd
Pid file path
pid file = /var/run/rsyncd.pid
# Log file transfer
transfer logging = yes
Log file format
log format = %t %a %m %f %b
# specify a log file
log file = /var/log/rsync.log
Delete some files or directories out of sync
exclude = lost+found/
# set timeout
timeout = 900
ignore nonreadable = yes
# Set files that do not need to be compressed
dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2

# module, you can configure more than one
[sync_file]
The root directory of the module, sync directory, pay attention to permissions
path = /home/test
Is it allowed to list module contents
list = no
# ignore error
ignore errors
# add a comment
comment = ftp export area
The user name that the module validates can be separated by Spaces or commas
auth users = sync
The module authentication password file can be placed in the global configuration
secrets file = /etc/rsyncd.secrets
Copy the code

Secrets file: /etc/rsyncd.secrets

### rsyncd. Secrets file configuration
# Username: password
sync:123456
Copy the code

Edit the /etc/rsyncd.motd file as follows:

### rsyncd.motd file configuration
++++++++++++++++++
sync zhang : rsync start
++++++++++++++++++
Copy the code

To set file permissions, do not skip this step:

chmod 600 /etc/rsyncd.secrets
Copy the code

Activation:

rsync --daemon --config=/etc/rsyncd.conf
Copy the code

Add boot autostart:

echo 'rsync --daemon --config=/etc/rsyncd.conf' >> /etc/rc.d/rc.local
Copy the code

2. Client configuration:

Create the password file /etc/rsyncd.pass and write the password as follows:

### configure the rsyncd.pass file
123456
Copy the code

To set file permissions, do not skip this step:

chmod 600 /etc/rsyncd.pass
Copy the code

Now you can execute commands on the client side to synchronize the files.

Synchronize data from the server => client:

rsync -avzP --delete sync@192.168. 01.::sync_file /home/test --password-file=/etc/rsyncd.pass
Copy the code

Synchronize data from the client to the server:

rsync -avzP --delete /home/test sync@192.168. 01.::sync_file --password-file=/etc/rsyncd.pass
Copy the code

So far, rsync has been configured. If you want to achieve bidirectional synchronization, you only need to configure B as the server and A as the client, and start corresponding services respectively.

Next, inotify monitors file changes for real-time synchronization.

inotify

The installation

yum install -y inotify-tools
Copy the code

Commonly used parameters

Inotifywait parameter description:

-m, - monitor: keeps the event listening state# Important parameters-r, - recursive: queries directories recursively# Important parameters-q, - quiet: Displays information about only monitoring events# Important parameters-t, - timeout: indicates the timeout period. - timefmt: indicates the output format of the specified time# Important parameters- format: Specifies the time output format# Important parameters-e, - event: indicates events such as delete, add, and change# Important parameters
Copy the code

Inotifywait events

Attrib: changes in the properties of a file or directory close_write: changes in the real file# Important parametersClose_nowrite: a file or directory is closed or closed after being opened in read-only mode Close: a file or directory is closed, either in read or write mode Open: a file or directory is opened moveD_to: a file or directory is moved to moved_from: a file or directory is moved from move: Move a file or directory to a monitor directory# Important parametersCreate: Creates a file or directory under the monitored directory# Important parametersDelete: deletes files or directories under the monitored directory# Important parametersDelete_self: the file or directory is deleted and the directory itself is deleted. Unmount: the file system is unmountedCopy the code

Common commands

1. Create events

inotifywait -mrq  /data --timefmt "%d-%m-%y %H:%M" --format "%T %w%f Event message: %e" -e create
Copy the code

2. Delete events

inotifywait -mrq  /data --timefmt "%d-%m-%y %H:%M" --format "%T %w%f Event message: %e" -e delete
Copy the code

3. Modify events

inotifywait -mrq  /data --timefmt "%d-%m-%y %H:%M" --format "%T %w%f Event message: %e" -e close_write
Copy the code

The script to monitor

#! /bin/bash
Path=/home/testServer = 192.168.0.2 User = the sync module = sync_filemonitor() {
  /usr/bin/inotifywait -mrq --format '%w%f' -e create,close_write,delete The $1 | while read line; do
    if [ -f $line ]; then
      rsync -avz $line --delete ${User}@${Server}: :${module} --password-file=/etc/rsyncd.pass
    else
      cd The $1 &&
        rsync -avz ./ --delete ${User}@${Server}: :${module} --password-file=/etc/rsyncd.pass
    fi
  done
}

monitor $Path;

Copy the code

Directly start the script in the background, you can monitor the file changes, so as to achieve file synchronization between servers.

So what if you want to synchronize multiple directories? The only solution I can think of is to write multiple shell scripts, each responsible for a directory, but I always feel that this approach is not very good. Any suggestions?

Reference article:

www.jianshu.com/p/bd3ae9d80… www.mengzhaoxu.xyz/2018/12/24/… www.cnblogs.com/bigberg/p/7… Cloud.tencent.com/developer/a… Blog.csdn.net/chenghuikai… www.devopssec.cn/2018/08/23/…