2019 Service Mesh Double Eleven Exam revealed

Ant Financial has been paying attention to ServiceMesh for a long time. It launched the ServiceMesher community in 2018 and now has over 4000 active developers. On the technical side, the Service Mesh scenario has gone through its exploratory phase and entered full deepwater exploration this year.

Singles’ Day 2019 was a big moment for us, we had a massive landing. As a technical person to face world-class traffic challenges, it is very nervous and exciting. What happens when Service Mesh meets Singles’ Day? What is the responsibility of Service Mesh in the continuous evolution of Ant Financial’s LDC architecture? Let’s take a look at ant Financial’s Service Mesh 11 real combat.

Basic concepts of Service Mesh

Istio clearly describes the two core concepts of Service Mesh: data plane and control plane. The data plane is responsible for network proxy, and does layer interception and forwarding on the link of service request. It can do service routing, link encryption, and service authentication in the link. The control plane is responsible for service discovery, service route management, and request measurement (which is controversial on the control plane).

We will not repeat the benefits brought by Service Mesh. Let’s take a look at ant Financial’s data and control products:

  1. ** Data plane: SOFAMosn. ** Ant Financial uses the high-performance network agent developed by Golang as the data surface of the Service Mesh to carry the massive core application traffic of Ant Financial’s Double Eleven.

  2. ** Control surface: SOFAMesh. **Istio, streamlined to Pilot and Citadel during landing, integrates directly into the data plane to avoid the overhead of an extra hop.

An overview of the arrival of Singles’ Day

This year, Ant Financial’s core applications have been fully connected to SOFAMosn, and hundreds of thousands of Mesh containers have been produced. The data load scale of SOFAMosn is tens of millions of QPS at the peak of Double 11, and the average processing time of SOFAMosn forwarding is 0.2ms.

In such a large-scale access scenario, we are faced with extremely complex scenarios, which require the full cooperation of various parties and guarantee the performance stability of the data surface to meet the requirements of large-scale promotion. The whole process is extremely challenging.

Meanwhile, the landing of Service Mesh is also a typical case of cross-team cooperation, integrating the sincere cooperation of the core, RPC, messaging, wireless gateway, control surface, security, operation and maintenance, and testing teams. Next, we will analyze the landing of Service Mesh on November 11 according to the above modules. More analysis on this number.

This paper is the first and core article of Ant Financial Service Mesh Implementation Practice Series. The author is Tian Yang (name: Liyuan), a technical expert of Ant Financial, specializing in the development of high-performance network servers, a core member of Tengine open source project, and a core member of Ant Financial SOFAMosn open source project.

Basic capacity building

Large view of SOFAMosn capabilities

SOFAMosn is divided into the following modules, including the basic capabilities of network agents and cloud native capabilities such as XDS.

Business support

As a high-performance secure network agent, SOFAMosn supports services such as RPC, MSG, and GATEWAY.

IO model

SOFAMosn supports two IO models, one is Golang classic model, Goroutine-per-Connection; One is the RawEpoll model, also known as Reactor model, I/O multiplexing + non-blocking I/O model.

For the landing scenes in Ant Financial, the number of connections is not the bottleneck, but thousands or tens of thousands. We chose Golang’s classic model. The RawEpoll model is more suitable for the access layer and gateway with a large number of long links.

Coroutines model

  • A TCP connection corresponds to a Read coroutine, which performs packet acceptance and protocol parsing.
  • A request corresponds to a worker coroutine, which performs business processing, proxy and Write logic.

In the conventional model, a TCP connection will have two Read/Write coroutines. We cancel the separate Write coroutine and replace it with the workerPool coroutine, which reduces the scheduling delay and memory footprint.

Ability to expand

Protocol extensions

SOFAMosn provides the plugin mechanism of the protocol by using the same codec engine and codec core interface, including support for:

  • SOFARPC;
  • HTTP1. X/HTTP2.0;
  • Dubbo;

NetworkFilter extension

SOFAMosn implements the network Filter extension mechanism by providing the network Filter registration mechanism and unified Packet Read/Write Filter interface. Currently, it supports:

  • TCP proxy;
  • Fault injection.

StreamFilter extension

SOFAMosn provides a stream filter registration mechanism and a unified Stream Send/Receive Filter interface to implement the stream filter extension mechanism, including:

  • Traffic mirroring;
  • RBAC authentication;

TLS secure link

As a fintech company, capital security is one of the most important links, and link encryption is one of the most basic capabilities. We have conducted a lot of investigations and tests on TLS secure links.

Through tests, the TLS of native Go has undergone extensive assembly optimization and is 80% of the performance of Nginx(OpenSSL). The Boring version of Go(which calls BoringSSL using CGO) is not superior due to cGO’s performance issues. Therefore, we finally selected TLS for native Go. We believe that the Go Runtime team will have more optimization in the future, and we will also have some optimization plans.

  • Go doesn’t have much optimization on RSA, go-Boring (CGO) is twice as powerful as Go;
  • P256 has assembly optimization on GO, ECDSA is better than Go-Boring;
  • On AES-GCM symmetric encryption, GO is 20 times more powerful than Go-Boring;
  • HASH algorithms such as SHA and MD also have corresponding assembly optimization.

In order to meet security compliance for financial scenarios, we also developed support for domestic passwords, which is not available in Go Runtime. Although the current performance is still a little behind the international standard AES-GCM, about 50%, but we have some subsequent optimization plans, please look forward to it.

Smooth upgrade capability

In order to make the release of SOFAMosn not aware of the application, we investigated and developed a smooth upgrade scheme, similar to the binary hot upgrade capability of Nginx, but the biggest difference is that the connection of the old SOFAMosn process will not break, but migrate to the new process. The socket FD of the bottom layer and application data of the upper layer are included to ensure that services are not damaged during binary publishing. In addition to supporting SOFARPC, Dubbo, messaging and other protocols, we also support TLS encryption link migration.

Container to upgrade

We will inject a new SOFAMosn, and then it will check whether there is an old SOFAMosn through the UnixSocket of the shared volume. If there is an old SOFAMosn, it will migrate the connection with the old SOFAMosn. Then the old SOFAMosn exits. This section is more detailed, involving the interaction between SOFAMosn itself and the Operator.

The connection of the SOFAMosn is migrated

The core of connection migration is mainly the migration of kernel Socket and application data, continuous connection, no perception of the user.

Metric migration of the SOFAMosn

We used shared memory to share the metric data of the old and new processes to ensure that the metric data was correct during the migration.

Memory overcommitment mechanism

  • Based on the sync. The Pool;
  • Slice reuse uses slab fine granularity to improve reuse rate.
  • Common structure reuse;

Sync. Pool is not a silver bullet and has its own problems, but as the Runtime continues to optimize sync.Pool, For example, GO1.13 uses the lock-free structure to reduce lock contention and add victim cache, which will be improved in the future.

XDS (UDPA)

Support cloud native unified data surface API, full dynamic configuration update.

preparation

Performance manometry and optimization

In the preparation process before going online, we conducted a lot of pressure measurement and optimization for the core cashier desk application in the grayscale environment, laying a solid foundation for the subsequent landing.

From the offline environment to the grayscale environment, we encounter many large-scale scenarios that are not available offline, such as tens of thousands of back-end nodes in a single instance and thousands of routing rules, which not only occupy memory, but also have a great impact on the efficiency of routing matching. For example, massive and high-frequency service publication registration also poses great challenges to performance and stability.

The entire pressure measurement optimization process lasted five months, from the initial increase of 20% overall CPU and 0.8ms RT per hop to the final increase of 6% overall CPU and 0.2ms RT per hop, the peak memory footprint optimization was 1/10 of the previous.

Add cpus as a whole Each jump RT Peak memory footprint
Before optimization 20% 0.8 ms 2365M
The optimized 6% 0.2 ms 253M
  • Partial optimization measures

In the 618 Rush, we launched some core link applications and saw CPU consumption increase by up to 1.7%. Some applications saw CPU consumption decrease by around 8% due to logic migration from Java to Go. The average latency increases by 0.17ms per hop, and the whole link of the two combined deployment systems increases by 5 to 6ms, resulting in a loss of about 7%.

When SOFAMosn is launched in the later stand-alone room, the overall performance of SOFAMosn is better under the full-link pressure test. For example, the RT of transaction payment with SOFAMosn is 7.5% lower than that without SOFAMosn.

A lot of the core optimizations and business logic optimizations like Route Cache that SOFAMosn does sink down faster bring architectural dividends.

Go Version selection

Updates to a version require a series of tests, and the new version may not always be the best fit for your scenario. At the beginning of our project, we used Go 1.9.2. After one year of iteration, we began to investigate the latest version of Go at that time, 1.12.6. We tested and verified many good optimizations of the new version, and also modified the default strategy of memory reclamation to better meet the needs of our project.

  • GC optimization to reduce long tail requests

The new version of self-preempt breaks up the long-running GC marking process in exchange for smoother GC performance and reduced latency impact on business. Go-review.googlesource.com/c/go/+/6857… Go-review.googlesource.com/c/go/+/6857…

Go 1.9.2

Go 1.12.6

  • Memory reclamation policy

In Go1.12, the memory reclamation policy is modified from the default MADV_DONTNEED to MADV_FREE. Although this is a performance optimization, in actual use, the test does not improve performance greatly, but occupies more memory, which greatly interferes with monitoring and problem judgment. We used GODEBUG=madvdontneed=1 to restore the previous policy, and there was also a discussion in the issue, this value may be changed in the future version.

runtime: use MADV_FREE on Linux if available

Go1.12 uses the default MADV_FREE policy. Inuse is 43M, but Idle has 600M and cannot be released.

Fix the Go Runtime Bug

During the early gray level verification, a serious memory leak occurred on the SOFAMosn line, 1G memory was leaked in a day. The final check was that the Writev implementation of Go Runtime was defective, resulting in the memory address of Slice being referenced by the bottom layer and GC could not release it.

We submitted a Bugfix to Go official, which has been integrated into the latest version of Go 1.13. internal/poll: avoid memory leak in Writev

After the order

SOFAMosn has experienced the big exam of Double 11 in Ant Financial, and we have more technology evolution and support scenes to follow. Welcome those who are interested to join us.

SOFAMosn:github.com/sofastack/s…

For more information about the implementation of Ant Financial Service Mesh on Nov 11, please stay tuned

Financial Class Distributed Architecture (Antfin_SOFA)