Gray release (also called canary) is one of the commonly used Internet product release, as the name implies, is in black and white are a product of a smooth transition between product releases, product distributors according to certain rules, let users continue to use the original product function, the other part of the user through the new features. Further improvements may be made to the product during the transition, and after the grayscale release is complete, all users will be able to use the new product features. [1]

When the service is released, it is difficult to ensure that there is no problem, problems are inevitable, how to ensure the minimum impact when there is a problem, gray release can let us know in advance whether there is a problem with the release, first use a small amount of traffic to see the effect.

Grayscale publishing system architecture

1 Configure the center, which services perform gray scale publishing, and how to divide the traffic in gray scale publishing

2 After receiving the request from the client, the upstream service forwards the request to the new or old service based on the configuration center

The registry is responsible for the registration of new and old services

4 Downstream service A business service system that processes user requests

Difficulty in design: protocol design in grayscale publishing system

  • Data protocol
    • Fixed-length header
    • Longer body
    • Design grayscale publishing field
      • uid
      • token
      • ip
      • tag
  • Grayscale publishing strategy
    • Single strategy
      • uid/token/ip
    • Context-based grayscale publishing strategy
      • Module AC grayscale at the same time
      • tag

Grayscale Release scenario

Only a downstream service is updated

Upstream if Nginx:

  • Nginx extensions such as Openresty and Tengine can be used to embed Lua scripts for forwarding
  • Server configuration Agent dynamically updates nginx configurations, such as consul + consul-template.

Upstream RPC service:

  • The client SDK of the CONFIGURATION management platform can be integrated to update the configuration of the server in real time, implement the grayscale publishing policy, and determine whether to enable the new version service.

Downstream services:

  • New and old services are typically registered in a service registry, and services typically have their own version numbers.

Grayscale multiple modules at the same time

The gateway layer and the data access layer publish the gray scale simultaneously.

According to the protocol fields mentioned above, the incoming traffic is labeled, and all requests that go to the new gateway are tagged, and then forwarded to different data access layers at the business logic layer according to the tag.

  • All requests that go through the new Gateway are labeled
  • Labeled requests at the business layer are forwarded to the new data access layer

Database gray upgrade

Such as SqlServer migration to MySQL, or database field changes.

  • First, the data is fully replicated and double-written
  • Double write for a while
  • Get rid of the old version of DB and just write the new database

Client grayscale publishing

For Android grayscale distribution, don’t put the package on the app market at once. If you put the package on the app market, you can not control it. You can try to update a group of users through your background interface, and then gradually increase the proportion of users. Such as gray released after completion, and then submitted to the application market

The IOS App Store has a grayscale distribution mechanism, but it has strict rules. Publishing can be terminated early, but users who have been updated cannot be demoted. The default is 7 days for all grayscale upgrades.

The last

Learning record

reference

  • [1] Gray publishing – Baidu Encyclopedia
  • AB testing and grayscale publishing platform architecture design and practice
  • Nepxion-Spring Cloud grayscale publishing artifact