System architecture design: Smooth release and ABTesting, do you do both?

Author: WengZhiHua Reference: https://www.cnblogs.com/wzh2010/

Introduction to smooth publishing

background

There is no mature smooth release scheme for the cloud office related system of the unit, so every release is a direct release, and the change of DLL files or configuration files will cause the restart of the site.

The cloud Office system has more than 10,000 resident users, and even a little more than half a minute can get a lot of complaints. Based on this, we combed a set of smooth release scheme.

Implementation plan

1. A health check interface is agreed with the Nginx proxy server

2. Use the HTTP status code returned by the interface to determine whether NGX is diverting user requests (this is a standard practice in our technical department).

Nginx will not forward traffic to an instance if it returns a status code of 5XX on its interface.

Release process

The goal is to smooth the release at release time, so QA and developers follow these steps at release time:

[/publish/ngxconfig]

2. Remove an instance (for example, instance A) from the shelf (assume that the system cluster has instances A, B, and C)

Check whether the health check interface is successfully removed: this is the health check interface that we agreed with Nginx. When the health check interface is normally online, the statU is 200. When the health check interface is offline, the statU is 401.

Online status:

Offline status:

4. Observe the monitoring site until the Req and Connnectiuon traffic of the instance disappears

5. Release the version under this instance

6, open Fidller, host to the instance to be published, and determine whether the release is successful (DLL, configuration file, IIS site will briefly restart)

7, QA students go to check the grayscale A instance server, to ensure that it is running normally, so the cycle, until all servers are released.

Further optimization of ABTesting

background

Once the smooth release was done, it really helped me a lot. I didn’t have to make announcements every time I made a release. The unimportant or non-functional content was released.

However, after a long time, the number of customers increased, and then encountered a problem, that is, every big business change, large release is directly released to production, which may have risks. Users may not fully accept what the designer has designed, but once the new version is available,

Received a lot of ridicule, are users, if can be in a small range of people in the gray scale trial, complete the smooth overuse and feedback, after optimization and then on production will be better.

Therefore, we need to think and design a set of unified technical solutions. In the future, no matter cloud office or other business systems, they can first conduct experience and function verification in a small scope that can be specified through gray release.

Based on the above smoothing, we put some thought into Nginx reverse proxy server and let Nginx help us to do ABTesting solution. Here are some of the options we tried:

1. Nginx reverse proxy: Incoming IP policy

process

steps

1. Enter the cloud office system and the Nginx reverse generation server

2. Nginx reads the AB list of incoming IP addresses

3. Forwarding traffic according to IP AB list (List A travels A specific instance, list B travels the original cluster instance of cloud office)

server {
    listen 80;
    server_name officecloud.com;
    access_log officecloud.com/logs main;
    ip_list 192.168.254.4,192.168.254.170

    set $group default;
    if ($remote_addr in iplist) {
        set $group ACluster;
    }

    location / { 
        proxy_pass http://$group;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        index index.html index.htm;
    }
}

The advantages and disadvantages

1. Simple configuration. The grayscale upgrade of the original resource platform is designed and upgraded according to the IP list

2, many external computers are not fixed IP, this is suitable for the company Intranet implementation, such as the configuration of the company Intranet IP.

Nginx reverse proxy: $.Cookies policy

process

steps

1. Enter the cloud office system and the Nginx reverse generation server

Nginx reads the version of the Http request Cokie (or other key).

3. Traffic forwarding is carried out according to the Key version (for example, Version1.1 goes to a specific cluster, Version1.0 goes to a general cluster instance)

server { listen 80; server_name officecloud.com; access_log officecloud.com/logs main; Ip_list 192.168.254.4, 192.168.254.170 set $group default; If ($http_cookie ~* "version=V1.0"){set default; } if ($http_cookie ~* "version=V1.1"){set $group ACluster; } location / { proxy_pass http://$group; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; index index.html index.htm; }}

The advantages and disadvantages

$COOKIE_version = $COOKIE_version = $COOKIE_version

2, relatively stable, for users who need to open the list, add a specific version in the Cookie head, the application as long as a little development

3. The cookie may not be generated when you first visit a static page

Note: This is the best Nginx proxy solution considered by the team. Similarly, both user-agent and Header can make this type of judgment, but the Header needs to invade the underlying HttpRequest to add the service, so it is not recommended.

3. AB cluster + service agent mode

process

steps

1. Login to the cloud office system in two ways: login page: ~/login; default page: ~/default? usertoken=#usertoken#

2. During login and when the usertoken is passed in, the router accesses the routing proxy module, verifies user information, and divides users to two AB clusters based on different personnel and departments (the personnel and department belong to the AB list)

3. Skip to specific instance cluster domain names based on forwarding (You can configure AB cluster to have different domain names for easier differentiation)

The advantages and disadvantages

1. Separation from Nginx, no need to rely on the company’s common platform and technical department implementation

2. You need to apply for cluster AB, which has different domain names.

3. If the front and rear ends are separated, ensure that both static sites and service sites apply for AB clusters

4. All entrances need to be unified as agents, with a certain amount of development

application

At present, two systems have been implemented according to this scheme

Reference data: https://github.com/CNSRE/ABTe…

ABTestingGateway is a dynamic routing system of sina open source. ABTestingGateway is a grayscale publishing system that can dynamically set the traffic flow policy. It works on layer 7.

Based on the development of NGINx and NGX-Lua, the dynamic scheduling function can be realized by using Redis as the diversion policy database.

Recent hot articles recommended:

1.1,000+ Java interview questions and answers (the latest version of 2021)

2. Finally got the IntelliJ IDEA activation code through the open source project, how sweet!

3. Ali Mock tool officially open source, kill all the Mock tools on the market!

4.Spring Cloud 2020.0.0 is officially released, a new and disruptive version!

5. “Java Development Manual (Songshan version)” the latest release, speed download!

Feel good, don’t forget to click “like” + forward oh!