In the process of project iteration, it is inevitable to put the project online. Going online corresponds to deployment or redeployment, deployment corresponds to modification, and modification means risk.

There are many technologies available for deployment, some simple, some complex, some requiring downtime, and some requiring no downtime. This article will provide a brief summary of the deployment scenarios commonly used today.

Blue/Green Deployment

  1. define

A blue-green deployment is one where you run the old version, deploy the new version, and then test it. After confirming OK, the traffic is cut to the new version, and then the old version is also upgraded to the new version.

  1. The characteristics of

Blue-green deployments require no downtime and are less risky.

  1. The deployment process

  • Deploying a Version 1 application (initial state)

All traffic from external requests is directed to this version.

  • Deploy the VERSION 2 application

Version 2 code is different from version 1 (new features, Bug fixes, etc.).

  • Switch traffic from version 1 to version 2.

  • If version 2 tests work, remove resources (such as instances) that are being used in version 1, and version 2 is officially used.

  1. summary

As you can see from the process, our application is always online during the deployment process. In addition, during the launch of the new version, no content of the old version is modified. During the deployment, the status of the old version is not affected, so there is little risk. And as long as the old version of the resource is not deleted, in theory, we can roll back to the old version at any time.

  1. Notes for blue and green releases

When you switch to blue, you need to properly handle unfinished business and new business. If your database back end can’t handle it, it can be a problem.

  • There may be situations where microservice architecture applications and legacy architecture applications need to be handled simultaneously, and if the two are not properly coordinated in a blue-green deployment, there is still the potential for service downtime.

  • Synchronous migration/rollback of database and application deployment needs to be considered in advance.

  • Blue-green deployments require infrastructure support.

  • With blue-green deployments on non-isolated infrastructures (VM, Docker, etc.), the blue and green environments risk being destroyed.

  1. Strengths and Weaknesses

  • advantage

The upgrade switchover and rollback speed is very fast.

  • insufficient

The switch is full, and if there is a problem with the V2 version, it has a direct impact on the user experience. It takes twice as many machine resources.

  1. Applicable occasions

  • Scenarios with a certain tolerance for the user experience.

  • Machine resources are either surplus or can be allocated on demand (AWS cloud, or home-built container cloud).

Gray released

  1. define

Grayscale publishing is a smooth transition between black and white. AB Test is A grayscale release method, in which some users continue to use A and some users start to use B. If users have no objection to B, the scope will be gradually expanded and all users will be migrated to B. Grayscale publishing can ensure the stability of the whole system, and problems can be found and adjusted at the beginning of the grayscale to ensure its impact.

  • Grayscale release structure diagram

  1. A/B Testing

A/B testing is A way to test the performance of an application’s functionality, such as usability, popularity, visibility, and so on. A/B testing is usually done on the front end of an application, but of course needs to be supported on the back end.

The difference between A/B test and blue-green deployment is that A/B test aims to obtain representative experimental conclusions through scientific experimental design, sampling sample representation, flow segmentation and small flow testing, and ensure that the conclusions are credible in the extension to all traffic. The purpose of a blue-green deployment is to safely and stably release new versions of the application and roll them back if necessary.

  1. Published by Canary

What we usually call Canary deployment is also a gray scale release method. In the case of the original version available, a new version of the application is deployed as the canary server to test the performance and performance of the new version, so as to ensure the stability of the overall system, to find and adjust problems as soon as possible.

Canaries in mines: In the 17th century, English mine workers discovered that canaries are sensitive to gas. Canaries will stop singing at the slightest trace of gas in the air; When the gas levels exceed a certain limit, the canaries are already poisoned to death, though dull humans do not notice. With relatively rudimentary mining equipment, workers took a canary with them each time they went down the mine as a gas indicator to evacuate in case of danger.

Grayscale publishing/Canary publishing consists of the following steps:

  • Prepare artifacts for each phase of deployment, including build artifacts, test scripts, configuration files, and deployment manifest files.

  • Remove the Canary server from the load balancing list.

  • Upgrade the Canary application (drain the traffic and deploy it).

  • Automate test your application.

  • Add the Canary server back to the load balancing list (connectivity and health check).

  • If canary is successfully tested online, upgrade the remaining servers (or roll back).

In addition, grayscale publishing can also set the route weight, dynamically adjust different weight to carry out the verification of new and old versions.

  1. Strengths and Weaknesses

  • advantage

The impact on user experience is small, and problems in the grayscale publishing process only affect a small number of users.

  • insufficient

Publishing is not automated enough to cause service outages during publishing.

Rolling Update Deployment

Further optimization and improvement on the basis of Canary publishing is a highly automated publishing method with relatively smooth user experience. It is the mainstream publishing method adopted by mature technical organizations at present.

  1. define

Rolling publishing: Typically taking one or more servers out of service, performing updates, and putting them back into use. This cycle continues until all instances in the cluster are updated to the new version.

  1. The characteristics of

This deployment is more resource-efficient than a blue-green deployment — it does not require running two clusters with twice the number of instances. We can deploy in parts, such as taking out 20% of the cluster at a time to upgrade.

  1. The deployment process

  • Rolling distribution usually starts with 1, or a small scale, such as 2% server, mainly used for traffic verification, similar to Canary test.

  • Rolling publishing requires sophisticated publishing tools and smart LBS that support smooth version replacement and traffic pull in and pull out.

  • During each release, remove the traffic of the old version V1 from the LB, clear the traffic of the old version, send the new version V2, and transfer the LB traffic to the new version. This will keep the user experience as intact as possible.

  • A rolling publication typically consists of several release batches, and the number of each batch is typically configurable (which can be defined using a publication template). For example, 1 set (canary) for the first batch, 10% for the second batch, 50% for the third batch, 100% for the fourth batch. There are observation intervals between batches to ensure that there are no problems with the next batch through manual verification or monitoring feedback, so the rolling release process is generally slow (canary time is generally longer than subsequent batches, e.g. canary 10 minutes and subsequent interval 2 minutes).

  • Rollback is the reverse release process. The traffic of the new version is removed from the LB, the new version is deleted, the old version is sent, and the LB traffic is transferred to the old version. Like the release process, the fallback process is generally slow.

  1. Strengths and Weaknesses

  • advantage

The impact on user experience is small and smooth.

  • insufficient

Release and rollback times are slow. Publishing tools are complex, and LB requires smooth traffic removal and pull-in capabilities.

Other distribution methods

These are traditional publishing methods that can cover most application publishing scenarios. There are special releases for key new features or for specific scenarios.

Function switch Release

Using Feature Flag/Toggle/Switch in code to control release logic generally does not require complex release tools and intelligent LB coordination, which is a relatively low-cost and simple release method. This approach also supports the modern DevOps concept of flexible customization and self-completion. The principle of the function switch is shown in the figure below:

  1. The deployment process

  • Functional switch publishing requires a configuration center or service support such as switch center, such as Ctrip’s Apollo configuration center or open source FF4J, which support switch publishing. The industry also has dedicated capabilities to switch on and off SaaS services, such as LaunchDarkly. Through the configuration center, operations and maintenance or r&d personnel can dynamically configure the values of functional switches during runtime. Of course, feature switch publishing is only one usage scenario for configuration center, which also supports many other dynamic configuration scenarios.

  • Function switch services generally provide a client SDK for developers to integrate. At runtime, the client SDK synchronizes the latest switch values, and the technology implementations can be push, pull, or a combination of push and pull.

  • The new feature (V2 new feature) and the old feature (V1 old feature) live in the same set of code, the new feature is hidden behind the switch, if the switch is not turned on, the old code logic, if the switch is turned on, the new code logic. Technically, it can be understood as a simple if/else logic.

  • After the application goes online, the switch is not turned on for the first time, and then the o&M or R&D personnel turn on the new function through the switch center. After the flow verification, the new function has no problem, and the release is completed. If there is a problem, you can always switch back to the old functional logic by switching the center.

  1. Strengths and Weaknesses

  • advantage

The upgrade switchover and rollback speed is very fast. Compared to complex publishing tools, implementation is relatively simple and cost is relatively low. The r&d can flexibly customize the release logic and support DevOps self-publishing.

  • insufficient

The switch is full, and if there is a problem with the V2 version, it has a direct impact on the user experience. If the code is invaded, the code logic will become complex, and the old version logic needs to be cleaned regularly, resulting in high maintenance costs.

Shadow test

For some core business related to the upgrade of legacy systems, in order to ensure that nothing goes wrong, there is a big trick called shadow testing, using more complex traffic replication, playback and comparison technology. Here is a sample architecture diagram of a shadow test:

  1. The deployment process

Shadow testing is generally applicable to equivalent refactoring and migration of legacy systems, such as.net to Java, or SQLServer database to MySQL database, and external dependence should not be too much, otherwise it will require a lot of mock development, high cost of test deployment, and more complex and unstable comparison testing.

  • The goal is to migrate and upgrade the legacy service to the new experimental service.

  • Before the test begins, one Legacy service and experimental service need to be deployed in the test environment, and two copies of the production database need to be copied to the test environment. Production request logs need to be collected, typically through a Kafka queue, and then consumed in Kafka using tools like Goreplay, copied and played back, and distributed to legacy and experimental services. After receiving the response, the comparison is performed. If all the response comparison is successful, it can be considered that legacy service and experimental service are equivalent in function logic. If any response comparison fails, it is considered that the two are not equivalent in functional logic. Experimental needs to be repaired and shadow test again until all comparison is successful. Depending on the complexity and criticality of the system, the comparison test time can be as short as a few weeks to as long as half a year.

  • Shadow testing can have no impact on production flow because the bypass is performed in an independent test environment.

  1. Strengths and Weaknesses

  • advantage

It has no impact on production user experience. You can use production real traffic for testing (copy comparison).

  • insufficient

The construction complexity is very high, the technical threshold is high, the database export replication is difficult. Do not have too many external dependencies, otherwise test deployment costs are high and comparison testing is more complex and unstable.

Refer to the article

https://www.google.comhttp://t.cn/RHV3riWhttp://t.cn/RnD147P

Today’s idea

A beautiful skin is all the same, an interesting soul is one in a million.

– anonymous

More highlights:

  • Blockchain primer

  • Docker core technology and implementation principle

  • Let’s Encrypt free wildcard SSL Certificate application tutorial

  • Nine suggestions for technical realization for technical personnel

  • FRP is recommended as a good Intranet penetration tool

If you think it’s great, what are you waiting for? Click on the tip as soon as possible, iOS rich people can also yo!