Milvus is a Keepalived based high availability solution

Milvus and Keepalived

Milvus as a leading performance, easy-to-learn and easy-to-use vector search engine, the project has been widely concerned since the open source, hundreds of developers joined Milvus community began to test, use, contribute documents, code and so on. As more and more users deploy Milvus in production systems, product High Availability becomes increasingly important.

Keepalived is a Linux high availability component based on Virtual Router Redundancy Protocol (VRRP). The virtual route redundancy protocol can be regarded as a protocol to realize the high availability of routers. That is, N routers providing the same functions form a router group. In this router group, there is a master host and multiple backup hosts. In this case, you need to select a standby device as the host based on the VRRP priority. VRRP is used to ensure high availability between the active and standby nodes.

Based on the architecture design of Milvus computing and storage separation, you can easily realize the high availability of Milvus services by using a variety of mature high availability components. This article introduces the High availability solution of Milvus based on Keepalived. The overall architecture is shown in the figure below:

High availability solution implementation

A system with high availability generally has two or more nodes, which can be divided into active nodes and standby nodes. When a node is unavailable or unable to process a customer’s request, the service responding to the request will be transferred to another available node in a timely manner. This is transparent to the client, and the customer does not need to care about the specific provision of the resources used, and the system will automatically complete.

The Milvus high availability solution described in this article will be set up on two computers, one as the host (IP: 192.168.1.85) and the other as the standby (IP: 192.168.1.38). The active and standby nodes must be connected to a reliable shared storage for storing data in Milvus.

Install Milvus

In the Lord for machine installation Milvus respectively, refer to https://milvus.io/docs/zh-CN/userguide/install_milvus/.

Install Keepalived and its dependencies

Run the following command on the active and standby nodes to modify the system network:

# vim /etc/sysctl.confCopy the code

Delete the # before net.ipv4.ip_forward=1, add net.ipv4.ip_nonlocal_bind=1 to the next line, save and exit.

Enter the command for it to take effect:

# sysctl -pCopy the code

Install Keepalived and its dependencies on the active and standby machines:

# apt-get install libssl-dev openssl libpopt-dev# apt-get install keepalivedCopy the code

Configuration Keepalived

Keepalived is configured on the active and standby servers. Provide a virtual routing address, which is set to 192.168.1.104 for this article.

# vim/etc/keepalived/keepalived. Conf host configuration file is as follows:! Configuration File for keepAlivedGlobal_defs {router_id sol01 # Active/standby route ID}vrrp_script chk_milvus {script Interval 2 weight-20}vrrp_instance VI_SERVER {state MASTER # "/etc/keepalived/chk_milvus.sh Virtual_router_id 51 specifies the VRRP group name. The Settings must be the same on the active and standby hosts. Priority 110 # Auth_type PASS auth_pass 1111} virtual_ipAddress {# Virtual IP address, 192.168.1.104/24 / track_script {chk_milvus}} / track_script {chk_milvus} Configuration File for keepAlivedGlobal_defs {router_id sol02 # Active/standby route ID}vrrp_instance VI_SERVER {state BACKUP # Virtual_router_id 51 specifies the VRRP group name. The Settings on the active and standby VMS must be the same. Priority 91 # (1-254) Auth_type PASS auth_pass 1111} virtual_ipAddress {# Virtual IP address, The active and standby servers must be identical 192.168.1.104/24 # Note that the subnet mask is configured} notify_master "/etc/keepalived/start_docker.sh master" # start the standby milvus server Notify_backup "/etc/keepalived/stop_docker.sh backup"Copy the code

Starting Keepalived

# service keepalived start Check keepalived status # service keepalived startCopy the code

test

After following the above steps, you can implement a basic Milvus high availability solution.

If the Milvus service of the host is enabled, the client can connect to the Milvus service of the host through the virtual address 192.168.1.104. 10 million 128 dimensional vector data were used in the test. The 10 million data were first imported into the table, which could be used for query and other operations.

Then, manually power off the host or stop the Milvus service on the host to simulate the abnormal situation on the host. After the shutdown, run the IP a command on the standby server to see that the virtual address 192.168.1.104 has been transferred to the standby server. The docker ps command also shows that the Milvus service on the standby server has started automatically. The Milvus service that the client connects to through 192.168.1.104 is actually running on the standby machine, and the whole switching process can be completed in a few seconds. In this case, the client can continue to perform Milvus operations on the standby server.

conclusion

This article introduces a basic Milvus high availability solution to give you some ideas, Concrete structures, steps and the related script please refer to https://github.com/milvus-io/bootcamp/tree/master/solutions/Milvus_HA. You are welcome to join the Milvus open source community and explore more perfect high availability solutions according to specific scenarios.