I have done a load balancing experiment in an article using Nginx+Redis to achieve session sharing load balancing. Its main architecture is as follows:

Debian1 is used as a scheduling server to distribute requests, that is, users access debian1, and debain1 sends requests to the application server according to certain policies: debian2 or debain3, or even more debain4, 5, 6……

Status and the data can be placed in an external distributed cache and distributed database services, this application service itself is a stateless, so the machine is very easy to increase or decrease, the application of high availability is guaranteed (for a high availability with state switch with increase not only pay attention to the machine, also note the backup redundancy and data consistency, etc.). But what was overlooked was that the high availability of the scheduling server debian1 itself was not taken into account as a single point of problem.

The first idea of high availability is dual-system hot backup, which automatically switches over in the event of a failure, so we will add a standby machine to debian1, debain1′. Based on my knowledge, I now divide the solutions into two types: high availability that is aware of the client and high availability that is transparent to the client, and select an example to experiment with.

Note: the following ha implementations use dual-system hot backup. For convenience, the scheduling server debian1 is referred to as the host, and the standby machine debian1′ is referred to as the standby machine.

The client has perceived high availability

The client has high availability awareness, that is, the client needs to cooperate with the client, the client itself to confirm server changes and switch access targets. For example, both the host and the standby host are registered in ZooKeeper (or other similar registries such as Redis). The client listens to the server information in ZooKeeper and switches to the standby host if the host is offline.

A pseudo ZooKeeper cluster is created

Firstly, a pseudo-Cluster of ZooKeeper with 3 nodes was built on the local computer. Download version 3.5.4-beta from the official website, unzip it, and then make three copies. Do the following for each copy:

  • Go to conf and create a configuration file zoo.cfg. The code is as follows:

    InitLimit =10 syncLimit=5 clientPort=2181 (different for each node: 2181,3181,4181) tickTime=2000 dataDir=E:/ zooKeeper-3.5.4 -1/data (different for each node: 3.5.4-2,3.5.4-3) dataLogDir=E:/zookeeper-3.5.4-1/datalog (different for each node, *.*::2888:3888 (LAN IP of the experimental machine or direct localhost) Server.2 =192.168.*.*::4888:5888 Server. 3 = 192.168. *. * : : 6888-7888Copy the code

  • Create dataDir and dataLogDir above, and in the dataDir directory you must create myID files that write different integer ids, namely x of server.x above, such as 1

  • Go to the bin directory and add set ZOOCFG=.. before calling in zkserver. CMD. /conf/zoo. CFG and start with it.

By the way, I will use my previous project CHKV for code development, because NameNode or DataNode in this project can also use ZooKeeper to achieve high availability. Welcome to improve this project with me and make progress together.

Schedule server development

The scheduling server registers itself with ZooKeeper and provides services to clients. We use the Curator framework to interact with ZooKeeper, with special attention to versioning issues.

The main code is as follows:

public static void main(String... arg) throws Exception { thisNode = ManagementFactory.getRuntimeMXBean().getName(); logger.debug("my pid: {}",thisNode); CuratorFramework Curator = CuratorFrameworkFactory.builder ().connectString(CONNECT_ADDR) .connectionTimeoutms (CONNECTION_TIMEOUT)// Connection creation timeout. SessionTimeoutMs (SESSION_TIMEOUT)// Session timeout. RetryPolicy (policy) .build(); curator.start(); Boolean result = becomeMaster(becomeMaster); if (result){ logger.info("Successfully Became Master"); }else { logger.info("Failed to Became Master"); } // Listen on NodeCache cache = new NodeCache(MASTER_NODE_PATH,false); cache.getListenable().addListener(()->{ ChildData data = cache.getCurrentData(); if (data ! = null){ String path = data.getPath(); Stat stat = data.getStat(); String dataString = new String(data.getData()); logger.debug("masterNode info, path:{},data:{},stat,{}",path,dataString,stat); }else { logger.info("masterNode is down, try to become Master"); if (becomeMaster(curator)){ logger.info("Successfully tried to Became Master"); }else { logger.info("Failed to try to Became Master"); }}}); cache.start(true); } // Verify master private static Boolean confirm(CuratorFramework Curator) throws Exception {masterNode = new String(curator.getData().forPath(MASTER_NODE_PATH)); logger.info("masterNode: {}",masterNode); return thisNode.equals(masterNode); } // becomeMaster private static Boolean becomeMaster(CuratorFramework curator) throws Exception {String path= ""; try { path = curator.create() .creatingParentContainersIfNeeded() .withMode(CreateMode.EPHEMERAL) .forPath(MASTER_NODE_PATH,thisNode.getBytes()); logger.debug(path); }catch (Exception e){ logger.error(e.getMessage()); } return MASTER_NODE_PATH.equals(path); }Copy the code

The full code is on GitHub.

Client development

The client listens to the scheduling server change events and sends application requests to ZooKeeper. In fact, the application server can also use this code to listen for changes to the scheduling server.

The main code is as follows:

public static void main(String... arg) throws Exception { CuratorFramework curator = CuratorFrameworkFactory .builder() .connectString(CONNECT_ADDR) .connectionTimeoutMs(CONNECTION_TIMEOUT) .sessionTimeoutMs(SESSION_TIMEOUT) .retryPolicy(policy) .build(); curator.start(); NodeCache cache = new NodeCache(curator, MASTER_NODE_PATH,false); cache.getListenable().addListener(()->{ ChildData data = cache.getCurrentData(); if (data ! = null){ String path = data.getPath(); Stat stat = data.getStat(); String dataString = new String(data.getData()); logger.debug("masterNode info, path:{},data:{},stat,{}",path,dataString,stat); masterInfo = dataString; }else { logger.info("masterNode is down, waiting"); }}); cache.start(true); MasterInfo = new String(MASTER_NODE_PATH); // Get the host, block wait try {masterInfo = new String(master_getData ().forpath (MASTER_NODE_PATH)); }catch (Exception e){ logger.error("no masterInfo"); masterInfo = null; } while (masterInfo==null); logger.info("masterInfo:{}",masterInfo); }Copy the code

The full code is on GitHub.

High availability transparent to clients

High availability is transparent to the client, that is, the client doesn’t need to do any work, the server switches don’t switch the client doesn’t know and doesn’t care. There are two methods to implement this. One is that the client accesses the host using the domain name, and then the monitoring host goes offline and reallocates the domain name to the standby host. Of course, this switchover will cost time, depending on the defined DNS cache time. The second method is that the client accesses the host through IP address and assigns the external IP address (or virtual IP address) to the standby host through IP drift technology after the host goes offline. In this way, timely switchover can be achieved.

Keepalived is often used to implement IP drift in practical environments.

The keepalived Solution for LVS and its official website were used for reference

First host, for machine to install keepalived, then configure host/etc/keepalived/keepalived conf:

Vrrp_instance VI_1 {state MASTER # MASTER = MASTER; BACKUP = standby Virtual_router_id 51 priority 100 advert_int 1 # Auth_type PASS auth_pass 1111} virtual_ipAddress {# Virtual IP address, 10.23.8.80}} virtual_server 10.23.8.80 80 {# Persistence_timeout 600 delay_loop 6 Lb_ALgo WLC # Persistence_TIMEOUT 600 Protocol TCP real_server 172.18.1.11 80 {# backend application server weight 100 # node weight TCP_CHECK {connect_timeout 3 # 3 timeout}} Real_server 172.18.1.12 80 {# Backend application server weight 100 TCP_CHECK {connect_timeout 3}} real_server 172.18.1.13 80 {# backend Application server weight 100 TCP_CHECK {connect_timeout 3}}Copy the code

Configure the standby/etc/keepalived/keepalived conf, similar to the host, but the state is backup, and lower weight can be:

vrrp_instance VI_1 { state BACKUP interface eth1 virtual_router_id 51 priority 90 advert_int 1 authentication { Auth_type PASS auth_pass 1111} virtual_ipaddress {10.23.8.80}}Copy the code

reflection

To put it plainly, the former is realized in the application layer, and the latter is realized in the transmission layer, so we can think that each layer of the computer network can actually do load balancing and high availability.

Check out the original article, from MageekChiu