1. The background

1.1 What is an API Gateway

API gateway can be regarded as the entrance of the system and the outside world, we can handle some non-business logic in the gateway, such as permission verification, monitoring, caching, request routing and so on.

1.2 Why is an API Gateway needed

  • The RPC protocol is converted to HTTP.

In internal development, we use RPC protocol (THRIFT or Dubbo) to develop and expose to internal services. When external services need to use this interface, they usually need to convert RPC protocol to HTTP protocol.

  • Request routing

In our system, because the same interface is used by both the old and the new systems, we need to route the request to the corresponding interface according to the request context.

  • Unified authentication

Authentication operations that do not involve service logic can be processed at the gateway layer instead of the service logic.

  • Unified monitoring

Since the gateway is an entry point for external services, we can monitor the data we want from here, such as incoming and outgoing parameters, link time.

  • Flow control, fuse downgrading

For traffic control, fusible downgrading non-business logic can be uniformly placed in the gateway layer.

Many businesses implement their own gateway layer to access their own services, but this is not enough for the entire company.

1.3 Unified API Gateway

A unified API gateway not only has all the features of an API gateway, but also has the following benefits:

  • Unified technical component upgrade

If there is a technical component that needs to be upgraded in the company, it needs to be communicated to every line of business, often for months. For example, if there is a major security risk for the security authentication of the entrance and it needs to be upgraded, it must not be upgraded if the speed is still so slow. Then with the unified gateway, the upgrade is very fast.

  • Unified Service Access

It is also difficult to access certain services. For example, the company has developed relatively stable service components and is promoting them in the company. This period must be very long.

  • Saving resource

Different businesses and different departments should have their own gateway layer to do this if we follow the above practice. You can imagine that if a company has 100 of these businesses, each business is equipped with 4 machines, then 400 machines will be needed. And every business development RD needs to develop this gateway layer, to maintain at any time, increase manpower. If there is a unified gateway layer, then maybe only 50 machines can do the 100 business gateway layer, and business RD does not need to pay attention to the development, online steps.

2. Design the unified gateway

2.1 Asynchronous Request

For our own implementation of the gateway layer, because only we use, the requirements for throughput is not high, so we generally call synchronous request.

For our unified gateway layer, how to access more services with fewer machines requires our asynchrony to increase throughput. There are generally two strategies for asynchrony:

  • Tomcat/Jetty+NIO+servlet3

This strategy is widely used. Jingdong, Youzan and Zuul all choose this strategy, which is more suitable for HTTP. Asynchrony can be enabled in Servlet3.

  • Netty+NIO

Netty is born for high concurrency. At present, VipSHOP gateway uses this strategy. In the technical article of Vipshop, Netty has a throughput of 30W + per second under the same situation, while Tomcat has a throughput of 13W + per second, which shows a certain gap.

Servlets can be used in more HTTP request scenarios where gateways are used, because there are more mature protocols for handling HTTP. Netty can be used if throughput is more important.

2.1.1 Full link Asynchronous

For the incoming request, we have used asynchrony. In order to achieve full-link asynchrony, we need to process the outgoing request asynchronously. For the outgoing request, we can make use of our RPC asynchrony support for asynchronous request, so the following figure can be basically achieved:

Start servlet asynchracy in the Web container, then enter the business thread pool of the gateway for business processing, then make the asynchronous call of RPC and register the business that needs to be called back, and finally perform callback processing in the callback thread pool.

2.2 Chain treatment

Among the design patterns is the chain of responsibility pattern, which prevents the sender of the request from being coupled with the receiver, making it possible for multiple objects to receive the request, connecting the objects into a chain, and passing the request along the chain until an object processes it. This pattern decouples the sender of the request from the handler of the request. We have implementations of this pattern in various frameworks, such as filter in servlet, Interceptor in SpringMVC.

This pattern is also applied in Netflix Zuul, as shown below:

This mode can be used for reference in the design of gateway:

  • PreFilters: preFilters that handle common services such as unified authentication, unified traffic limiting, fuse degradation, cache processing, and provide business side scaling.

  • RoutingFilters: Used to handle generic calls, such as protocol conversions and request routing.

  • PostFilters: postFilters used for result processing, logging, timing, etc.

  • ErrorFilters: errorFilters used to handle calls for exceptions.

This design can also be applied to great gateways.

2.3 Service Isolation

In the case of full-link asynchro, the impact between different services is small, but if some synchronous calls are made in the provided custom FiIlter, the frequent timeouts will affect other services. So we need to use isolation to reduce the interaction between businesses.

2.3.1 Semaphore isolation

Semaphore isolation only limits the total number of concurrent calls to the service or the main thread. This isolation can still affect the main thread if the remote call times out, which can affect other business. Therefore, if you simply want to limit the total amount of concurrent calls to a service or call a service that does not involve remote calls, you can use lightweight semaphores. Semaphore isolation is selected because there is no custom filter for the preferred gateway.

2.3.2 Thread pool isolation

The simplest way is to isolate different services through different thread pools. Even if the business interface has a problem, it will not affect other services because the thread pool has been isolated. In the gateway implementation of JINGdong, thread pool isolation is adopted. Important business, such as goods or orders, are processed separately through the thread pool. However, due to the unified gateway platform, if there are many lines of business, everyone thinks that their business is important and needs to be isolated by a separate thread pool. If Java language is used for development, threads in Java are heavy and resources are limited. If too many thread pools need to be isolated, it is not suitable. Threads are a lighter resource when developing gateways in other languages such as Golang, so thread pool isolation is preferable.

2.3.3 Cluster Isolation

What if some services require isolation but the unified Gateway does not have thread pool isolation? You can use cluster isolation, if you have some really important business you can apply for a separate cluster or multiple clusters for that business and isolate it from one machine to another.

2.4 Request Traffic Limiting

Traffic control can be implemented with many open source implementations, such as Alibaba’s recently open source Sentinel and the more mature Hystrix.

Traffic limiting can be classified into cluster traffic limiting and single-node traffic limiting:

  • In the case of using unified storage to store current traffic, Redis is generally used, which generally has some performance losses.
  • Single-machine flow limiting: We can directly use Guava’s token bucket to limit each machine, because there is no remote call performance consumption is small.

2.5 Fuse downgrade

This piece can also refer to the open source implementation of Sentinel and Hystrix, which is not the main point here.

2.6 Generalization Call

Generalized calls refer to transformations of some communication protocol, such as HTTP to Thrift. Some open source gateways such as Zuul are not implemented because the internal service communication protocols vary from company to company. For example, Vipshop supports HTTP1,HTTP2, and binary protocols and then converts them into internal protocols. Taobao supports HTTPS,HTTP1, and HTTP2, which can be converted into HTTP,HSF,Dubbo and other protocols.

2.6.1 Generalization Call

How do you implement a generalization call? Because protocols are difficult to be automatically converted, interfaces corresponding to each protocol need to provide a mapping. In simple terms, two protocols can be converted to a common language, so that they can be converted to each other.

In general, there are three ways to specify a common language:

  • Json: The JSON data format is simple, fast and lightweight. There is an HTTP to Dubbo project in the Dubbo ecosystem that uses JsonRpc to convert HTTP to JsonRpc to Dubbo.

For example, a www.baidu.com/id = 1 GET can be mapped to json:

The code block

Param: {"id" : 1}}Copy the code

  • XML: XML data is heavy and difficult to parse, which I won’t discuss here.

  • Custom description language: Generally speaking, this cost is relatively high and you need to define your own language to describe and parse, but its expansibility and customization are the highest. Example: Spring customizes its own SPEL expression language

For generalization calls, if you want to design JSON, you can basically meet the requirements. If you need to customize too much, you can define your own language.

2.7 Management Platform

The technical keys to implementing a gateway are described above. There is a business key to gateways that needs to be addressed here. With gateways in place, a management platform is required to configure the key technologies we have described above, including but not limited to the following:

  • Current limiting
  • fusing
  • The cache
  • The log
  • A custom filter
  • Generalization call

3. Summary

A final reasonable standard gateway should be implemented as follows:

— jingdong Vipshop Have a great Ali. Zuul
Implement key servlet3.0 netty servlet3.0 servlet3.0 servlet3.0
Asynchronous condition Servlets are asynchronous. It is not clear if RPC is asynchronous Full link asynchronous Full link asynchronous Full link asynchronous Zuul1 is blocked synchronously and Zuul2 is non-blocked asynchronously
Current limiting — — Smooth traffic limiting. Codis at first, then switch to the token bucket limiting per machine. 1. Basic flow control: API-based QPS for flow limiting. 2. Operation flow control: support APP traffic packet, flow control of APP+API+USER 33. Flow control: Weight flow control for APP access API. Ali Open Source :Sentinel A JAR package is provided: Spring-Cloud-Zuul-ratelimit. 1. Limit the flow of the requested target URL (for example, how many times a URL can be invoked per minute). 2. Limit traffic from the client’s access IP address (for example, the number of requests per minute for an IP address) 3. Traffic restriction for certain users or user groups (for example, non-VIP users can only call an API 100 times per minute) 4. Multi-dimensional mixed current limiting. At this point, it is necessary to implement some flow limiting rules orchestration mechanism. And, or, not equal relationship. Support for four kinds of storage ways ConcurrentHashMap, Consul, Redis, database.
Fusing the drop — — Hystrix — Only service level fuses are supported, not URL level fuses.
isolation Thread pool isolation — Semaphore isolation — Thread pool isolation, semaphore isolation
The cache redis — Level 2 cache, local cache +Codis HDCC local cache, remote cache, database You need to develop it yourself
Generalization call — HTTP, HTTPS, http1 http2, binary dubbo,http,nova hsf,dubbo,http,https,http2,http1 Only support HTTP

4. Reference

  • Jingdong: www.yunweipai.com/archives/23…

  • Zan gateway :tech.youzan.com/api-gateway…

  • Vipshop: mp.weixin.qq.com/s/gREMe-G7n…

  • Zuul:www.scienjus.com/api-gateway…

Snailclimb: How to design an API Gateway? Source: making