This time we share from the current situation and problems of the ten billion traffic transaction system API Gateway, expounds the relationship between micro-service architecture and API Gateway, straighten out the flow Gateway and business Gateway, and bring the most comprehensive KNOWLEDGE and experience of API Gateway. Contents:

Part I: API Gateway Overview

  • Distributed service Architecture, microservice Architecture and API gateway
  • API gateway definition and functions, concerns
  • Classification and technical analysis of API gateways

The second part: open source gateway analysis and research

  • Common open source gateway introduction
  • Comparative analysis of four open source gateway (OpenResty/Kong/Zuul2 / SpringCloudGateway, etc.)
  • Technical summary of open source gateway

Part THREE: API gateway design of ten billion traffic transaction system

  • The current situation and problems of ten billion traffic API gateway
  • Business gateway design and best practices
  • Development prospects for API gateways

Part I: API Gateway Overview

Any problem in computer science can be solved by adding an indirect intermediate layer. — David Wheeler

Distributed service Architecture, microservice Architecture and API gateway

What is an API Gateway?

In fact, gateways have deep roots in Service Oriented Architecture (SOA) and MicroServices Architecture (MSA).

More than a decade ago, banks and other financial institutions completed the national business system centralized, decentralized systems become centralized, but also brought various problems: how to deal with the rapid development of business, how to integrate and manage too many docking systems. In order to solve these problems, the industry has implemented the mid-layer gateway between channels and business systems, that is, the integrated front-end system, which ADAPTS to various channels and services, and processes various protocol access, routing and message conversion, synchronous and asynchronous invocation, etc.

People based on the concept of SOA, on the basis of comprehensive pre-configuration, further increase Service metadata management, registration, mediation, choreography, governance and other functions, gradually formed the Enterprise Service Bus (ESB).

Service-oriented Architecture (SOA) is an architectural guideline for building an enterprise IT ecosystem. SOA focuses on services, the most basic unit of business function defined by platform neutral interface contracts. By servitization of business systems, different modules can be decoupled, and various heterogeneous systems can easily realize service invocation, message exchange and resource sharing.

Unlike the isolated business systems of the past, SOA emphasizes that the entire enterprise IT ecosystem is a large whole. All business services in the entire IT ecosystem constitute the core IT resources of the enterprise. The business of each system is disassembled into modules and services of different granularity and level. Services can be assembled to a larger granularity, and services from different sources can be choreographed into the same processing process to achieve very complex integration scenarios and richer business functions.

SOA from a higher level of the whole enterprise IT to uniform design and ecological management, application software is divided into service units with different functions, and through the standard software interface to connect these services, by the SOA architecture to realize the enterprise application can be more flexible to quickly respond to the enterprise business change, realize the integration and reuse of old and new software assets, Reduce total cost of ownership of software. Of course, ESB based centralized SOA solutions also have problems, especially in the explosive development of Internet technology.

Distributed service Architecture, microservice Architecture and API gateway

In recent years, with the rapid development of Internet technology, in order to solve the problems of centralized management SOA solutions represented by ESB, the emergence of distributed service technology represented by Apache Dubbo (open source in 2011) and Spring Cloud gives another choice for SOA implementation: Decentralized Distributed Services Architecture (DSA). Distributed service architecture technologies no longer rely on specific service-centric container technologies (such as esBs), but separate service addressing and invocation completely, eliminating the need for a container as a service proxy.

On this basis, with the development of REST, Docker containerization, domain modeling, automated test operation and maintenance and other fields, microservice architecture (MSA) was gradually formed. In the microservices architecture, the granularity of services is further subdivided, and individual business services can be independently designed, developed, tested, deployed, and managed. In this case, each independent deployment unit can be maintained by different development and test teams and designed using different programming languages and technical platforms. However, each unit must use a service protocol independent of language and platform as the communication mode, as shown in Figure 7-3.

In microservices architecture, the system structure becomes very complex due to the segmentation of systems and services, and RESTAPI is the best choice due to its simplicity, efficiency, cross-platform, easy development, easy testing, and easy integration. At this point, a system similar to the synthesis front is created, which is the API Gateway. The API gateway serves as an API aggregation point and unified access point scattered in various business system microservices. External requests can access all internal REST API services through this access point.

Similar to SOA/ESB, all business service capabilities exposed inside the enterprise can be reflected through API services managed on the API gateway, so the API gateway also aggregates all IT service capabilities directly provided by the enterprise.

API Gateway technology trends

We can see from baidu index trend that SpringCloud and SOA are very hot, MSA, gRPC, Gateway also have very high attention, and the search trend of these technologies are positively correlated. On the other hand, we can see from Github’s search that there are many Gateway types of projects.

Github.com/search?o=de…

As you can see, almost half of the 100 projects in the first 10 pages use the Go language to implement the Gateway. The language classification is: Go>NodeJS/JavaScript>Java>Lua>C/C++>PHP>Python/Ruby/Perl

Definition, functions, and concerns of an API gateway

Definition of API gateway

The gateway’s role is to serve as an API architecture that protects, enhances, and controls access to API services. The role of a Gateway in an API architecture is to protect, Enrich and control access to API services.) — github.com/strongloop/…

An API gateway is a system that sits in front of an application or service that provides REST API interface services to manage authorization, access control, and traffic restrictions, etc., so that REST API interface services are protected by the API gateway and transparent to all callers. As a result, business systems hiding behind API gateways can focus on creating and managing services without having to deal with the strategic infrastructure.

In this way, the gateway system can proxy the business service apis of the business system. In this case, the gateway receives service invocation requests from other external systems and needs to access the actual service at the back end. Security-related system protection measures can be implemented while the request is accepted. When accessing back-end business services, you can make judgment based on relevant request information, route to specific business services, or call multiple services and then aggregate new data back to the caller. The gateway system can also do some process and pretreatment of the requested data, and also can do some filtering and pretreatment of the data returned to the caller, that is, according to the needs of the request header/response header, request message/response message to do some modification processing. Without this extra processing, the simplest and most straightforward proxy service API feature is what we call pass-through.

At the same time, due to the language independence of REST API, we can see that based on API gateway, our back-end service can be any heterogeneous system, no matter Java, Dotnet, Python, PHP, ROR, NodeJS, etc., as long as it supports REST API, it can be managed by API gateway.

API gateway functions

In general, AN API gateway has four functions:

  • Request access: As the access point of all API service requests, manage all access requests;
  • Business aggregation: As an aggregation point for all back-end business services, all business services can be invoked here;
  • Mediation policy: implement security, authentication, routing, filtering, flow control, caching and other policies, and carry out some necessary mediation processing;
  • Unified management: Provides a configuration management tool to centrally manage the call lifecycle and mediation policies of all API services.

API gateway concerns

From the above analysis, it can be seen that THE API gateway is not a typical business system, but an intermediate layer to provide more additional capabilities to THE API service in order to make the business system more focused on the business service itself.

Thus, when designing and implementing an API gateway, two goals need to be considered:

  1. Simple development and maintenance, saving labor cost and maintenance cost. This requires us to use a very mature, simple and maintainable technology architecture.
  2. High performance, save equipment cost, improve system throughput capacity. This requires specific design and trade-offs for API gateways.

None of this is a problem when the amount of concurrency is small. Then, once the system’s API is heavily visited, these can become critical issues.

The three most important concerns of a massively concurrent Gateway are:

  1. Maintain large inbound request access capabilities (short and long connections), such as netty based implementations.
  2. Maximum multiplexing outbound HTTP connection capability, such as an HttpClient4-based asynchronizedHttpclient implementation.
  3. Flexibly implement security, authentication, filtering, aggregation, traffic limiting, and monitoring policies.

Classification and technical analysis of API gateways

Classification of API gateways

If we think more deeply about the goals and concerns mentioned above, it becomes important to note that all the issues and features to be considered fall into two categories.

  • One type is global, completely irrelevant to back-end business systems and services, such as security policies, global flow control policies, and traffic distribution policies.
  • One type is targeted at a specific back-end business system, or a part of the service that has certain relevance to the business, and is generally directly deployed in front of the business service.

Figure 6.

Thus, with the development of the complex business system of the Internet, these two kinds of function sets have gradually formed two common gateway systems: traffic gateway and service gateway.

Traffic gateway and WAF

We define global policy gateways that are completely independent of specific back-end business systems and services, namely traffic gateways. In this way, the traffic gateway focuses on the stability and security of global traffic, such as preventing all kinds of SQL injection, blacklist and whitelist control, Loadbalance of access requests to the service system, etc., and generally has the following universal functions:

  • Global flow control
  • Log statistics
  • Preventing SQL injection
  • Preventing Web Attacks
  • Mask tool scanning
  • Whitelist control
  • And so on.

Through this list of features, we can see that the traffic gateway functions very similar to the Web application Firewall (WAF). WAF is a Web application firewall based on ngX_Lua module of Nginx/OpenResty. WAF general code is simple, with a focus on ease of use, high performance, and lightweight. The short answer is to add security-related functionality to Nginx’s proxy capabilities. To describe its principle in a word, it is to parse HTTP requests (protocol parsing module), detect rules (rule module), perform different defense actions (action module), and record the defense process (log module). A general WAF has the following functions:

  • Prevent SQL injection, local include, partial overflow, Fuzzing testing, XSS/SSRF and other Web attacks
  • Protect against stress testing tools like Apache Bench
  • Block common scanning hacker tools, scanners
  • Block picture attachment class directory execution permission, prevent Webshell upload
  • Supports IP whitelist and blacklist functions, and directly denies access from blacklisted IP addresses
  • Supports URL whitelist to define urls that do not need filtering
  • Supports user-Agent filtering, CC attack defense, and limits the number of times a URL can be accessed at a specified time
  • Cookie filtering and URL and URL parameter filtering are supported
  • Supports logging. All rejected operations are recorded in logs

Several WAF open source implementations

The above WAF content mainly refers to the following two projects:

  • github.com/unixhot/waf
  • Github.com/loveshell/n…

For an open source example of a traffic gateway, see the well-known open source project Kong (based on OpenResty).

Business gateway

We define a policy gateway for a specific back-end business system or a certain correlation between services and services, that is, a service gateway. For example, the flow control strategy for a certain system, a certain service or a certain user classification, the cache strategy for a certain type of service, the permission verification method for a specific system, the request filtering for certain user condition judgment, and the data aggregation encapsulation for specific several related apis.

The service gateway is usually deployed after the traffic gateway and before the service system, and is closer to the system than the traffic gateway. When we say API gateway most of the time, we mean business gateway in a narrow sense. And if the system is small, we will combine the two and use one gateway to handle all the work. The specific design and implementation of the service gateway will be described in detail in the following chapter.

The second part: open source gateway analysis and research

Common open source gateway introduction

At present, the common open source gateways can be roughly classified into the following categories according to language:

  • Nginx+ LuA: Open Resty, Kong, Orange, Abtesting Gateway, etc
  • Java: Zuul/Zuul2, Spring Cloud Gateway, Kaazing KWG, Gravitee, Dromara Soul, etc
  • Go: Janus, Fagongzi, grPC-gateway
  • Dotnet: Ocelot
  • NodeJS: Express Gateway, Micro Gateway

According to the number of use, maturity, etc., there are four mainstream ones:

  • OpenResty
  • Kong
  • Zuul/Zuul2
  • Spring Cloud Gateway

Nginx+Lua

OpenResty*

Project address: openresty.org/

OpenResty® is a high-performance Web platform based on Nginx and Lua that integrates a large number of excellent Lua libraries, third-party modules, and most of the dependencies. It is used to easily build dynamic Web applications, Web services and dynamic gateways that can handle ultra-high concurrency and scalability.

OpenResty® effectively turns Nginx into a powerful general-purpose Web application platform by bringing together a variety of well-designed Nginx modules, primarily developed by the OpenResty team. In this way, Web developers and system engineers can use the Lua scripting language to mobilize various C and Lua modules supported by Nginx, and quickly construct high-performance Web applications that are capable of concurrent single-machine connections over 10K or 1000K.

The goal of OpenResty® is to have your Web services run directly inside Nginx services, taking full advantage of Nginx’s non-blocking I/O model, Provide consistent, high-performance responses not only to HTTP client requests, but also to remote backends such as MySQL, PostgreSQL, Memcached, and Redis.

This is from the Chinese version of the OpenResty website: openresty.org/cn

To put it simply, OpenResty is based on Nginx, integrating Lua with Lua’s various libraries and available third-party modules, so that we can leverage the dynamic extension capabilities of Lua on top of the efficient HTTP processing that Nginx already has. Therefore, we can make a variety of Lua scripts that meet the gateway policies we need and implement our gateway system based on them.

Kong*

Project address: konghq.com/ and github.com/kong/kong

Kong is based on OpenResty and is a cloud native, fast, scalable, distributed Microservice Abstraction Layer, also called API Gateway, Also called API Middleware in Service Mesh.

Launched in 2015, Kong’s core values lie in high performance and scalability. According to the global 5000 strong organization statistics, Kong is the most widely used API gateway still being maintained in production environments.

Kong claims to be The World’s Most Popular Open Source Microservice API Gateway.

Core advantages:

  • Scalability: It is easy to scale horizontally by adding nodes, which means that large system loads can be supported with very low latency.
  • Modularity: Kong’s capabilities can be extended by adding new plug-ins that can be installed and configured via the RESTful Admin API.
  • Run on any infrastructure: Kong can run anywhere, such as in a cloud or hybrid environment, in a single or global data center.

ABTestingGateway

Project address: github.com/CNSRE/ABTes…

ABTestingGateway is a gateway that can dynamically set the shunting policy. It focuses on the related fields of grayscale publishing. It is developed based on Nginx and NGX-Lua and uses Redis as the shunting policy database to realize the dynamic scheduling function.

ABTestingGateway is a part of Sina Weibo’s internal dynamic routing system dyGateway, which has been opened source. In the previous gray scale systems based on NGINx implementation, shunting logic is often implemented through the rewrite stage of if and rewrite instructions. The advantages are high performance, but the disadvantages are limited function, error-prone, and fixed forwarding rules, only static shunting. ABTestingGateway uses NGX-lua. By enabling lua-shared-dict and lua-resty-lock as the system cache and cache lock, the system achieves performance close to native Nginx forwarding.

Features:

  • Supports multiple shunt modes, including iPRange, UIDRange, UID mantis, and specified UID shunt
  • Multi-level traffic diversion is supported. Dynamic traffic diversion policies take effect immediately without restart
  • Expansibility, provides a development framework, developers can flexibly add new diversion methods, to achieve secondary development
  • High performance, pressure test data close to native Nginx forwarding
  • The grayscale system configuration is written in the nginx configuration file for easy administrator configuration
  • Suitable for various scenarios: grayscale publishing, AB testing, and load balancing

It is known that Oceanus is also based on Nginx and NGX_LUa extension, which mainly provides service registration and discovery, dynamic load balancing, visual management, customized routing, security anti-racking, session ID reuse, fuse degradation, one-key traffic intercession and performance statistics.

JAVA

Zuul/Zuul2*

Project address: github.com/Netflix/zuu…

Zuul is Netflix’s open source API gateway system, designed primarily for dynamic routing, monitoring, resilience, and security.

The internals of Zuul can be seen simply as a collection of many different functional filters (PS: For comparison, an ESB can also be seen simply as a collection of channels and filters). These filters can be written in Groovy or other JVM-based scripts (as well as Java), placed in a specified location, and then loaded dynamically by Zuul Server polling to detect changes and take effect in real time.

Zuul is currently available in two large versions, 1.x and 2.x, which are quite different.

Zuul1.x is based on synchronous IO and is part of the Spring Cloud family bucket, which can be easily configured and used with Spring Boot/Spring Cloud.

In Zuul1.x, the types and processing flow of filters are shown in the figure below. The most important filters are Pre, Routing, and POST, which respectively apply to request processing before calling service API, direct response, and response processing after calling service API.

Zuul2.x’s biggest improvement is the implementation of asynchronous IO access requests based on Netty Server, and the implementation of back-end business service API requests based on Netty Client. This results in higher performance and lower latency. In addition, the filter types were changed to explicitly name the original three core filters as Inbound filter, Endpoint filter, and Outbound filter.

Zuul2.x

  • Service Discovery
  • Load Balancing
  • Connection Pooling
  • Status Categories
  • Retries
  • Request Passport
  • Request Attempts
  • Origin Concurrency Protection
  • HTTP/2
  • Mutual TLS
  • Proxy Protocol
  • GZip
  • WebSockets

Spring Cloud Gateway*

Project address: github.com/spring-clou…

Spring Cloud Gateway is based on Java8, Spring 5.0, Spring Boot 2.0, and Project Reactor. It was developed earlier than Zuul2 and is currently part of the Spring Cloud family.

Spring Cloud Gateway can be seen as an upgrade and replacement for Zuul1.x, which uses Netty for asynchronous IO earlier than Zuul2, enabling a simpler, more efficient API Gateway that works closely with Spring Cloud.

Spring Cloud Gateway clearly distinguishes Router and Filter, and a big feature is that it has a lot of built-in functions out of the box, which can be used by SpringBoot configuration or hand-coded chain calls.

For example, there are 10 Router types built in, so you can configure them to route according to Header, or Path, or Host, or Query.

For example, the general Filter and global Filter are distinguished, and 20 kinds of Filter and 9 kinds of global Filter are built in, which can also be used directly. Of course, custom Filter is also very convenient.

Core features:

  • Able to match routes on any request attribute.
  • Predicates and filters are specific to routes.
  • Hystrix Circuit Breaker integration.
  • Spring Cloud DiscoveryClient integration
  • Easy to write Predicates and Filters
  • Request Rate Limiting
  • Path Rewriting

gravitee gateway

Project address: Gravitee. IO/with github.com/gravitee-io…

Kaazing WebSocket Gateway

Project address: github.com/kaazing/gat… With kaazing.com/products/we…

Kaazing WebSocket Gateway is a dedicated Gateway for and processing websockets, which claims to provide world-class enterprise-class WebSocket service capabilities.

The features are as follows:

  • Standard WebSocket support, support full duplex two-way data delivery
  • Linear scaling, stateless architecture means that more machines can be deployed to extend service capabilities
  • Authentication, authentication, single sign-on support, cross-domain access control
  • SSL/TLS encryption support
  • Websocket Keepalive and TCP half-on/half-off detection
  • High availability through load balancing and clustering
  • Docker support
  • JMS/it support
  • IP white list
  • Automatic reconnection and reliable message acceptance assurance
  • Fanout processing policy
  • Real-time cache and so on

Dromara soul

Project address: github.com/Dromara/sou…

Soul is an asynchronous, high-performance, cross-language, responsive API gateway that provides unified HTTP access.

  • Support for various languages, seamless integration with Dubbo and SpringCloud;
  • Rich plug-ins support authentication, current limiting, fusing, firewall, etc.
  • The gateway can be dynamically configured with multiple rules and policies.
  • Plug-in hot swappable, easy to expand;
  • Supports cluster deployment and A/B Test.

Go

fagongzi

Project address: github.com/fagongzi/ga…

Fagongzi Gateway is a full-featured Go API gateway with a Rails Web UI management interface.

Features:

  • Flow control
  • fusing
  • Load balancing
  • Service discovery
  • Plug-in mechanism
  • Routing (shunt, replication traffic)
  • API polymerization
  • API parameter verification
  • API Access Control (whitelist)
  • API default return value
  • API custom return value
  • API result Cache
  • JWT Authorization
  • API Metric imports Prometheus
  • API failed retry
  • Back-end server health check
  • Open Management API(GRPC, Restful)
  • Supports the Websocket protocol

Janus

Project address: github.com/hellofresh/…

Janus is a lightweight API Gateway and management platform that allows you to control who, when, and how these REST apis are accessed, and it also logs all the details and errors of the access interactions. One benefit of implementing an API gateway using Go is that it generally requires only a single binary to run, with No complex dependency hell.

Features:

  • Hot loading does not need to restart the gateway process
  • Graceful closing of HTTP connections
  • Support for OpenTracing, which enables distributed tracing
  • Support HTTP / 2
  • Circuit breakers can be implemented for each API
  • Retry mechanism
  • Flow control, which can be for each user or key
  • CORS filtering can be specific to the API
  • Multiple out-of-the-box authentication protocols are supported, such as JWT, OAuth2.0, and Basic Auth
  • Docker image support

Dotnet

Ocelot

Project address: github.com/ThreeMammal…

  • routing
  • Request the aggregation
  • Service discovery (based on Consul or Eureka)
  • The service Fabric
  • WebSockets
  • Authentication and authentication
  • Flow control
  • The cache
  • Retry policies and QoS
  • Load balancing
  • Logging and Tracing
  • Request header, Query string conversion
  • Custom intermediate processing
  • Configure and manage REST apis

NodeJS

Express Gateway

Project address: github.com/ExpressGate… With www.express-gateway.io/

Express Gateway is a REST API Gateway based on NodeJS development, Express and Express middleware implementation.

Features:

  • Dynamic centralized configuration
  • API consumer and credential management
  • Plug-in mechanism
  • Distributed data storage
  • Command line tool CLI

microgateway

Project address: github.com/strongloop/… With developer.ibm.com/apiconnect

StrongLoop is a subsidiary of IBM. The Microgateway gateway is built on Node.js/Express and Nginx as IBM API Connect and is part of THE IBM Cloud ecosystem.

Microgateway is a developer-focused, extensible gateway framework that enhances our access to microservices and apis.

Core features:

  • Security and control, based on the Swagger(OpenAPI) specification
  • Built-in gateway policies, API Key validation, flow control, OAuth2.0, JavaScript scripting support
  • Implement Gateway policies (security, routing, integration, etc.) with Swagger Extensions (API Assembly)
  • You can easily customize gateway policies

In addition, Microgateway has several features:

  • By integrating Swagger, verification capability based on Swagger API definition is realized,
  • Using datastore to maintain the API data model that needs to be processed,
  • Using a streaming engine to handle multiple policies gives API designers more control over the API lifecycle

The core architecture is shown in the figure below:

Comparative analysis of four open source gateway (OpenResty/Kong/Zuul2 / SpringCloudGateway, etc.)

OpenResty/Kong/Zuul2 / SpringCloudGateway important features

The gateway Current limiting authentication monitoring Ease of use maintainability maturity
Spring Cloud Gateway It can be extended by IP, user, cluster traffic limiting, and provides corresponding interfaces Common authentication, Auth2.0 Gateway Metrics Filter Simple and easy to use Spring series is extensible and easy to configure and maintain The Spring community is mature, but gateway resources are small
Zuul2 You can use the configuration file to configure traffic limiting for a cluster or a single server. You can also use filter to implement traffic limiting The filter is implemented The filter is implemented Fewer references Poor maintainability Open source is not long, little information
OpenResty Lua development is required Lua development is required Need to develop It’s easy to use, but lua requires a lot of development Poor maintainability and a large number of Lua scripts to maintain in the future It’s very mature and there’s a lot of information
Kong Traffic limiting is performed by user by second, minute, hour, day, month, and year. Can be developed on the basis of the original code Common authentication, Key Auth authentication, HMAC, Auth2.0 Datadog can be reported to record the number of requests, the amount of request data, the amount of reply data, the interval between receiving and sending, the number of status codes, and the running time within Kong Easy to use, API forwarding through the administrator interface configuration, development needs lua scripts Poor maintainability and need to maintain a large number of Lua libraries in the future Relatively mature, user issues summary, community, plug-in open source

Take the current limiting function as an example:

  • Spring Cloud Gateway currently provides the Ratelimiter implementation based on Redis. The algorithm used is the token bucket algorithm, which is configured through the YML file.
  • Zuul2 can be configured with cluster traffic limiting and single-server traffic limiting through the configuration file, and can be extended using filter.
  • The OpenResty can use resty.limite. count, resty.limite. conn, and resty.limite. req to limit traffic.
  • Kong has a base stream limiting component that can be developed in Lua based on the base component source code.

Zuul/Zuul2 / Spring Cloud Gateway can refer to some of the function point analysis Spring Cloud Gateway author Spencer Gibb’s article: spencergibb.netlify.com/preso/detro…

OpenResty/Kong/Zuul2 / SpringCloudGateway performance test

Use 3 4 core16g memory machine, respectively, as the API service providers, Gateway, the press, using WRK as a performance testing tool, to simple OpenResty/Kong/Zuul2 / SpringCloudGateway tabloid article of performance testing.

In the figure above, Y-axis coordinates are QPS, X-axis is the data of a gateway, and each line is the data of different gateways in a scene. The test results are as follows:

  • SCG~Zuul2 << OpenResty ~< Kong << Direct (Direct connection);
  • Spring Cloud Gateway and Zuul2 have similar performance, about 40% of direct connection;
  • OpenResty and Kong are similar, about 60-70% of direct connection;
  • For large numbers of concurrent users, such as 200 or 1000 concurrent users, Zuul2 has a high probability of returning errors.

Technical summary of open source gateway

Open source gateway test analysis

From the scene to talk about performance, are playing rogue. Performance is like temperature, different occasions under the standard is not the same. It’s 18 degrees Celsius, it’s cold for the old, it’s fine for the kids, it’s hot for the penguins, and the vegetables in the fridge might go bad.

Under the same benchmark conditions, different parameters and software, relatively horizontal comparison, is valuable. For example, the same machine (e.g. 16GB /4Core), the same server (using Spring Boot, configure path API/Hello to return a HelloWorld), the same pressure measurement method and tools (e.g. WRK, 10 threads, 20 concurrent connections), We tested the limit QPS obtained by Direct access to the server (Qps-Direct, 29K), and configured a Spring Cloud Gateway to do the limit QPS for gateway access (QPs-SCG, 11K), the limit QPS (QPs-Zuul2, 13K), the limit QPS (QPs-Kong, 21K), and the limit QPS (QPs-KONG, 19K) obtained by OpenResty are configured in the same way, and the comparison is meaningful.

The performance of Kong is very good, and it is very suitable for traffic gateway. Besides, the abstraction of Service, route, upstream, consumer and plugins is also worth learning from the self-developed gateway.

For complex systems, it is not recommended to use Kong for business gateways, or more specifically, to deeply customize Kong or OpenResty on systems in the Java technology stack, mainly for engineering reasons. Give an example: if we have a lot of different business lines, authentication means is multifarious, it is related with business how much a bit. If you implement authentication in the gateway, you need to maintain a large number of Lua scripts, and introducing a new complex technology stack is not a cheap thing to do.

Spring Cloud Gateway/Zuul2 is convenient for Java technology stacks and can rely on some common Jars for business systems. Lua is inconvenient, not only because of the language, but also because of the reuse infrastructure. In addition, for the gateway system, the performance is not worse than an order of magnitude, the problem is not big, add 2 machines can be done.

Based on the current test results, if the services are at 2ms level and the performance of direct connection is 100, Kong can achieve 60, OpenResty 50, Zuul2 and Spring Cloud Gateway 35. If the service latency is larger, these gaps will be gradually narrowed.

So far Zuul2 has a lot of craters:

  1. It’s immature, it’s undocumented, it’s new, it doesn’t have a lot of real-world applications
  2. The error rate was high with high concurrency, and nearly 50% of our test scenarios had errors at 1000 concurrency

Therefore, it is recommended to use the Spring Cloud Gateway as the basic skeleton to simply use or slightly customize the business Gateway system.

Demo and test of various gateways

Most of the mock service and gateway demo code used in the above tests can be found here: github.com/kimmking/sp…

The demo also simulated a NodeJS Gateway and added keep-alive and pool. The demo’s performance test results were about 1/9 of that of direct connection, that is, about 1/4 of that of Spring Cloud Gateway or Zuul2.

Part THREE: API gateway design of ten billion traffic transaction system

The current situation and problems of API gateway of ten billion traffic transaction system

Ten billion flow system facing the business status quo

At present, we are faced with the status quo of daily tens of thousands of concurrent online long connections (not counting short connections), the total number of long connections per day is 30 million +, the number of API calls per day is more than 10 billion, and the number of transaction orders per day is 150 million.

In this case, an important goal of API gateway design is: how to provide accurate, professional and personalized services for all kinds of customers with THE help of API gateway, and ensure that customers can obtain real-time data and business capabilities of the business system.

The relationship between the gateway system and other systems

In our business, the relationship between API gateway system and other systems is roughly as follows:

Typical application scenarios of the gateway system

Our API gateway system provides services for Web and mobile APP clients, as well as API call services for a large number of API clients. It also supports REST API and WebSocket protocol.

As the pre-system of real-time trading system, it must provide customers with the latest market and trading information accurately and timely. Once data delays or errors occur, they will cause irreparable losses to customers.

In addition, for different customers and channels, the gateway system needs to provide different security, verification, flow control, cache strategies, and can at any time aggregate data from different perspectives for pre-processing, to ensure the stability and reliability of the system and real-time accuracy of data.

FIG. 16 and 17

Trading system API features

As a global trading system, the API features are summarized as follows:

  • Access is very concentrated: the very core set of apis accounts for more than half of the visits
  • Very frequent visits: Very high QPS, very high daily visits
  • Fixed data format: The data format processed by trading systems is very fixed
  • Small amount of packet data: Generally, no more than 10 KB of data is transmitted in each request
  • Worldwide distribution of users: Customers are distributed in various countries around the world
  • Divided into internal calls and external calls: except for the API directly called by the API client, other apis are called by other internal systems
  • 7×24 hours always-on service: The system needs to provide highly available, always-on service capabilities to satisfy customers in different time zones for transactions and automated policy transactions
  • External users have certain technical capabilities: external API customers generally integrate our API and realize their own trading system

Issues facing the transaction system API gateway

Problem 1: Increasing traffic

How to control the flow reasonably, how to deal with the sudden flow, how to ensure the stability of the system to the greatest extent, are important issues. Special gateway as a direct customer – facing system, any problems will be magnified a hundredfold. A lot of strange problems that no one has ever encountered can appear at any time.

Problem 2: Gateway systems are getting more complex

After years of development, the existing business gateway has a large number of business embedded in it, and there are many different business gateways without any relationship with each other, and there is no precipitation of infrastructure.

At the same time, there is too much technical debt, and the system hardcodes to realize the global gateway policy and many business rules, resulting in high maintenance costs.

Problem 3: API gateway management is difficult

The design of API monitoring indicators and data collection under massive concurrency is also a big problem. Technical support for 7×24 hours also leads to high maintenance costs.

Problem 4: Push or pull choices

Short or long connections, REST API or WebSocket?

There are many business channels (Web, App and API of different product lines form more than a dozen different channels), which makes it difficult to control user behavior.

Business gateway design and best practices

API gateway 1.0

Our API gateway version 1.0 was developed many years ago and was customized directly with OpenResty. Global security testing, routing and forwarding policies for traffic, and limiting traffic for different levels were implemented directly with Lua scripts.

As a result, after the rapid development of the business, there are many Lua scripts with the same or different functions in the system. Every time the system goes online or is maintained, several or dozens of Lua scripts need to be found to adjust the policy, which is very inconvenient, and the granularity of policy control is not fine enough.

API gateway 2.0

After the separation of traffic gateway and service gateway, the separation of traffic gateway and service gateway began in 2017. The traffic gateway continued to use OpenResty customization, with only a small number of global configuration functions and Lua scripts that were not frequently changed. The business gateway uses the Java system implemented by vert. x and is deployed between the traffic gateway and the back-end business service system, taking advantage of vert. x’s reactive programming capabilities and asynchronous non-blocking I/O capabilities, as well as the scale-out capabilities of distributed deployment, thus initially solving problems 1 and 2.

Vert.x is a framework based on event-driven and asynchronous non-blocking IO that runs on the JVM, as shown below. In vert. x, Verticle is the most basic development and deployment unit, and different vert. x can transfer data through the Event Bus, thus facilitating the implementation of high concurrency network applications. About the Vert. The analysis of the x can refer to ali lodge any blog:www.sczyh30.com/tags/Vert-x…

Vert.x also supports the Websocket protocol, so it’s easy to implement a fully asynchronous gateway system that supports REST apis and WebSockets.

Caching is an essential part of a high-performance API gateway system. Whether it is distributing hot and cold data to reduce stress on business systems, or acting as an intermediary data source to provide efficient reusable business data for service aggregation, it plays a huge role.

An excellent and efficient cache system must be designed and implemented specifically according to the characteristics of the business data it carries.

Daily monitoring of API gateways

We use a variety of tools to monitor and manage the API, including full link access tracking, connection count statistical analysis, and wave access statistics for important countries and cities around the world. The Gateway technology team keeps an eye on data trends day by day. Each business system r&d team arranges special personnel to pay attention to the API performance of its own system every day, and promotes the solution of performance problems and continuous optimization. This is the initial solution to problem 3.

Pictures 22, 23, 24, 25

External customers are recommended to use Websocket

Because external customers need to call API services themselves through the API gateway to integrate business service capabilities into their own systems. The technical ability and system processing ability of each customer are greatly different, and the use behavior is also different. For the constantly developing and changing transaction business data, if the API call frequency is too low, the real-time data will be affected; if the call frequency is too high, the system resources of both parties may be wasted. At the same time, using the message push feature of Websocket, we can control the frequency of customers receiving messages and the number of connections of a single user in the gateway system, and adjust the policy dynamically according to the situation of the business system at any time.

All things considered, Websocket is a more real-time, reliable and manageable approach than REST apis. Problem 4 is basically solved by gradually assisting and encouraging customers to use Websocket protocol.

API gateway performance optimization

API gateway system as a unified API access point, in order to provide users with the best user experience, must do long-term performance optimization work.

Not only can the API gateway optimize itself, but also the API service capability of each service system can be discovered according to the monitoring situation. Based on this, the API performance of each service system can be continuously optimized.

In a specific example, a gateway system found that the connection was often violently jitter (as shown in the figure below), which seriously affected the system stability and wasted system resources. After elimination, it was found that:

  1. Crawler IP constantly crawls our transaction data, and none of these IP addresses has any actual transaction on the platform. The new connection of the highest single crawler IP is nearly 1 million times per day, averaging more than 10 times per second.
  2. Some API customers have bugs in their programs, and the processing speed is limited. They constantly disconnect and reconnect and try to process API data again, which seriously affects the customer’s user experience. In view of the above analysis, we adopted several solutions:
  3. For the crawler IP identified every day, join the blacklist and directly restrict its access to our API gateway in the traffic gateway;
  4. For API customers with bugs, assist them to locate problems and fix bugs, and enhance customers’ confidence in using them;
  5. For customers with limited processing speed and technical ability, based on the customized Websocket service, the algorithm of sliding time window is used to optimize the distributed messages in batches when the business data changes greatly.
  6. For unlogged and unidentified API calls, the traffic gateway implements a global flow control policy to increase the cache time and limit the number of calls to ensure system stability.
  7. The service gateway further refines and controls the gateway policy in real time according to the importance level of API services and the classification of customers to ensure the use of core services and customers to the greatest extent.

Before optimization:

After the optimization:

Development prospects for API gateways

  1. The existing API Gateway is a self-developed Gateway 2.0 system based on vert. x.
  2. It is currently planned to implement Gateway 3.0, a new generation of microservices architecture, based on Spring Cloud and Spring Cloud Gateway by the end of the year.
  3. The microservices architecture Gateway is planned to integrate traffic and business gateways and add many out-of-the-box functional components as Apollo Gateway 1.0 open source.

We look forward to your participation.