In the microservice architecture of payment system, the construction of basic services is the most important. This paper focuses on how to use Apache Thrift + Google Protocol Buffer to build basic services.

1. RPC vs Restful

In microservices, what protocol is used to build the service system is always a hot topic. The debate centers on two candidate technologies: RPC or Restful.

1. Binary RPC, represented by Apache Thrift, supports multiple languages (but not all languages) and has a four-layer communication protocol with high performance and bandwidth saving. Compared with Restful protocol, using Thrifpt RPC, the bandwidth usage is only 20% of that of Thrifpt RPC under the same hardware conditions, but the performance is improved by an order of magnitude. But the biggest problem with this protocol is that it cannot penetrate the firewall.

2. The advantage of Restful protocols supported by Spring Cloud is that they can penetrate firewalls, are easy to use, and are language independent. Basically, systems that can be implemented using various development languages can accept Restful requests. But there are disadvantages in performance and bandwidth usage.

Therefore, the implementation of microservices in the industry is basically to determine an organizational boundary, within which RPC is used; Outside the boundary, use Restful. This boundary can be business, department, or even company wide.

Selection of RPC technology

In the selection of RPC technology, the principle is to choose the familiar or internal framework of the company. If it’s a new business, there aren’t a lot of frameworks to choose from right now, but they’re confusing enough.

Apache Thrift

Many of them were used abroad, originated from Facebook, and later donated to Apache Foundation. Apache Thrift is Apache’s top-level project. Users include Facebook, Evernote, Uber, Pinterest and other big Internet companies. In the open source community, Apache Hadoop /hbase also uses Thrift as an internal communication protocol. This is the most mature framework, the advantages of stability, high performance. The disadvantage is that it only provides RPC services, and other functions, including traffic limiting, fusing, and service governance, need to be implemented by ourselves or by using third-party software.

Dubbo

Domestic use of more, from Ali Company. It is a bit weaker than Apache Thrift in performance, but it integrates a large number of microservice governance functions and is quite easy to use. The problem with Dubbo is that it hasn’t been maintained for a long time. The website shows that the last update was eight months ago.

Google Protobuf

Like Apache Thrift, Google Protobuf includes both data definitions and service definitions. The problem is that Google Protobuf has always had only an implementation of the data model, not an official implementation of RPC services. It wasn’t until 2015 that gRPC was launched as an official implementation of RPC services. But there is a lack of serious users.

The above is for qualitative comparison only. Quantitative comparison, there are a lot of information on the Internet, you can consult. In addition, there are some good RPC frameworks, such as Zeroc ICE, that are beyond the scope of this comparison.

Thrift provides a variety of high-performance transport protocols, but is not as powerful as Protobuf in terms of data definition.

1. The Protobuf compression rate and serialization/deserialization performance of data in the same format are slightly higher.

2. Protobuf supports custom annotations to data and can access these annotations through apis, making Protobuf very flexible in data manipulation. For example, you can use option to define the mapping between attributes defined by Protobuf and database columns to achieve data access.

3. Data structure upgrades are a common requirement, and Protobuf does a great job of supporting data backwards compatibility. As long as the implementation is handled properly, users of older versions will not be affected when the interface is upgraded.

The disadvantage of Protobuf is the poor implementation performance (gRPC) of RPC services. For this reason, The RPC implementation of Apache Thrift + Protobuf has become the choice of many companies.

Apache Thrift + Protobuf

As mentioned above, combining the advantages of Protobuf in flexible data definition, high performance serialization/deserialization, compatibility, and Thrift’s mature implementation in transport is the choice of many Internet companies.

Service definition:

 service HelloService{
	binary hello(1: binary hello_request);
 }Copy the code

Definition of agreement:

message HelloRequest{ optional string user_name = 1; // Access this interface user optional string password = 2; // Access this interface password optional string hello_word = 3; // Other parameters; } message HelloResponse{ optional string hello_word = 1; // The user accessing this interface}Copy the code

For a pure thrift implementation, this approach may seem cumbersome, but it offers a number of benefits in terms of extensibility, maintainability, and service governance.

Service registration and discovery

Spring Cloud provides service registration and discovery functions. If you need to implement it yourself, you can consider using Apache Zookeeper as a registry and use Apache Curator to manage Zookeeper links. It implements the following functions:

• Listen for changes to registry keys and reload the registry as soon as there are updates.

• Manage links to ZooKeeper and retry if problems occur.

The retry policy for Curator is configurable and provides the following policies:

BoundedExponentialBackoffRetry
ExponentialBackoffRetry
RetryForever
RetryNTimes
RetryOneTime
RetryUntilElapsedCopy the code

An exponential delay strategy is commonly used, such as a retry interval of 1s, 2s, 4s, 8s… Index increases to avoid killing the server.

For service registries, the registry structure needs to be designed in detail. In general, the registry structure is organized as follows:

Equipment room area - Department - Service Type - Service Name - Server ADDRESSCopy the code

Due to a certain delay in the registration and discovery of ZooKeeper, it is necessary to pay attention to the implementation. Zookeeper can only be registered after the service is successfully started. Before the zooKeeper service goes offline or restarts, disconnect the connection from ZooKeeper and stop the service.

Connection pool

RPC service access is similar to a database in that establishing a link is a time-consuming process and connection pooling is standard for service invocation. At present, there is no mature open source Apache Thrift link pool. Most Internet companies will develop their own link pool. Your own implementation can make improvements based on JDBC link Pools, such as referring to the Apache Commons DBCP link pool and using Apache Pools to manage links. In interface design, the connection pool needs to manage RPC’s Transport:

Public interface TransportPool {/** * obtains a transport * @return* @throws TException */ public TTransport getTransport() throws TException; }Copy the code

The main difficulty with connection pooling implementation is how to select connections from multiple servers to serve the current invocation. For example, there are 10 machines providing services at present, and the fourth server was allocated last time. Which server should be allocated this time? In implementation, the QOS of each machine and the current burden need to be collected to allocate an optimal connection.

API gateway

As the business of the company grows, the number of RPC services increases, which also presents a challenge for service invocation. If you have an application that calls more than one service, for that application, you need to maintain links to multiple servers. Any service restart will affect the connection pool and client access. For this reason, API gateways are widely used in microservices. An API gateway can be thought of as an access point to a collection of services. From the perspective of object-oriented design, it is similar to the facade pattern in that it encapsulates the services provided.

The gateway function

The API gateway itself does not provide a concrete implementation of the service; it dispatches the service to the concrete implementation on request. Its main functions:

1. API routing: When a request is received, it is forwarded to the worker machine of the specific implementation. Avoid creating a large number of connections using a party.

2. Protocol conversion: The original API may use HTTP or other protocols to achieve, unified encapsulation as RPC protocol. Notice that the conversion here is a batch conversion. That is, the set of apis that were originally implemented using HTTP are now being converted to RPC, and gateways are being introduced for unified processing. For the transformation of a single service, develop a separate Adapter service to perform it.

3. Encapsulate common functions: Encapsulate functions related to microservices governance on the gateway to simplify the development of microservices, including fuses, traffic limiting, authentication, monitoring, load balancing, caching, and so on.

4. Triage: Triage of access can be easily implemented by controlling the distribution strategy of the API gateway, which is particularly useful for grayscale and AB testing.

decoupling

In the implementation of RPC API gateway, the difficulty lies in how to achieve service independence. We know that using Nginx to implement HTTP routing gateways can be implemented service independent. However, RPC gateway is difficult to realize service – independent because of its non-standard implementation. The unified use of Thrift + Protobuf to develop RPC services simplifies the development of API gateways, avoids the adjustment of gateways for each service coming online, and decouples gateways from specific services:

• Each worker machine that implements the service registers the service with ZooKeeper.

• The API gateway receives changes from ZooKeeper, updates the local routing table, and records the mapping between service and worker (connection pool).

• When the request is submitted to the gateway, the gateway can extract the service name from the RPC request, then find the corresponding worker machine (connection pool) according to the name, call the service on the worker, and return the result to the caller after receiving the result.

Permissions and others

An important feature of Protobuf is that the serialization of data has nothing to do with the name, only the attribute type and number. In this way, the inheritance of the class is realized indirectly. We can parse the deserialization stream of Girl and Boy using the Person class as shown below:

message Person {
  optional string user_name = 1; 
  optional string password = 2; }message Girl {
  optional string user_name = 1; 
  optional string password = 2; 
  optional string favorite_toys = 3; }message Boy {
  optional string user_name = 1; 
  optional string password = 2; 
  optional int32  favorite_club_count = 3; 
  optional string favorite_sports = 4; }Copy the code

As long as the input parameters of the service are properly choreographed, common attributes are represented by fixed numbers, and the input parameters can be resolved using common base classes. For example, if the first and second elements of all input must be user_name and password, then we can use Person to parse the input, allowing for uniform authentication of the service and QPS control based on the results of the authentication.

Seven, fusing and current limiting

Netflix Hystrix offers a good implementation of fuses and limiting, see the project description on GitHub. Here is a brief description of fusing and current limiting principle.

Circuit Breaker Patten is generally used. When an error occurs in a service and the number of errors per second reaches the threshold, the service does not respond to the request but directly returns an error indicating that the server is busy to the caller. After a delay, try to open 50% of the access, if the error is still high, continue the circuit breaker; Otherwise, return to normal.

Traffic limiting limits the access to services based on the access party, IP address, or domain name. Once the number exceeds a specified limit, the access is prohibited. In addition to using Hystrix, consider using Guava RateLimiter if you want to implement it yourself

8. Service evolution

As the number of visits to the service increases, the implementation of the service evolves to improve performance. The main methods are read and write separation, caching and so on.

Reading and writing separation

For entity services, read and write separation is the first step to improve performance. Read/write separation is generally implemented in the following ways:

1. Use the master-slave replication mode on homogeneous databases: Common databases, such as MySQL, HBase, and Mongodb, all provide the master-slave replication function. Data is written to the master library, and operations such as reading and retrieval are performed from the slave library to achieve read and write separation. This method is simple to implement, without additional development of data synchronization program. Generally speaking, the performance on the read will be poor for the database with transaction requirements for writing. Sharding requests can be made by adding slave libraries, but this can also increase costs.

2. Separate read and write on heterogeneous databases. Take advantage of different databases to synchronize data from the master to the slave via messaging mechanisms or other means. For example, MySQL is used as the master library to write data. When data is written, messages are sent to the message server. After the synchronization program receives the messages, the data is updated to the read library. You can use Redis, Mongodb and other in-memory databases as read libraries to support reading by ID. Use Elastic as a slave library to support search.

3. Micro-service technology is a topic that programmers cannot do without. Here, I recommend a learning platform for communication: Architecture Exchange Group 650385180, which will share some videos recorded by senior architects: Spring, MyBatis, Netty source code analysis, high concurrency, high performance, distributed, microservice architecture principles, JVM performance optimization has become an architect’s essential knowledge system. You can also get free learning resources. The following course chart is also available in the group. I believe that for those who have already worked and met technical bottlenecks, there will be content you need here.

The cache using

Using slave libraries can also result in very high slave library costs if the data volume is large. For most of the data, such as the order base, it usually takes only a period of time, say three months. Longer periods of data access are very low. In this case, there is no need to load all the data into the expensive read library, which in this case is cache mode. In cache mode, data update strategy is a big problem.

• For data with low real-time requirement, passive update strategy can be considered. That is, when data is loaded into the cache, the expiration time is set. General in-memory databases, including Redis, Couchbase, and so on, support this feature. When the data expires, it becomes invalid. When it is accessed again, the system triggers the process of reading and writing data from the primary database.

• For data with high real-time requirements, the policy of active update should be adopted, that is, the cached data should be updated immediately after Message is received.

Of course, as the service evolves, there will be an impact on the implementation of the original service. Considering that one of the implementation principles of microservices is that a service only manages one repository, the original service is split into multiple services. In order to maintain customer stability, the original service is re-implemented as a service gateway, providing services as proxies for sub-services.

That’s all about RPC and microservices. Here’s an introduction to the thrift + Protobuf implementation specification.

Annex 1. Basic service design specification

Basic service is the lowest module in the service stack of microservices. Basic service directly deals with data storage and provides basic operations of adding, deleting, modifying and checking data.

1.1 Design specification is attached

The file specification

The RPC interface file name is xxx_rpc_service.thrift. Protobuf Parameter file name is xxx_service.proto.

Both files use UTF-8 encoding.

Naming conventions

The service name is XXXXService. XXXX is an entity and must be a noun. The following are reasonable interface names.

OrderService
AccountServiceCopy the code

Attachment 1.2 Method design

Because basic services mainly solve data read and write problems, external interfaces can be standardized into basic interfaces such as add, delete, modify, search, and statistics by referring to database operations. Interfaces are named after operations + entities, such as createOrder. The input and output parameters of the interface are named according to the interface name +Request and the interface name Response. This approach makes the interface easy to use and manage.

file: xxx_rpc_service.thrift

**/ namespace Java com.phoenix. Service /** * provides basic operations on XXX entities. **/ service XXXRpcService {/** * create entity * Input parameters: * 1. CreateXXXRequest: create a request. * Output parameters * createXXXResponse: The creation is successful, returns the ID list of the created entity; * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: Mandatory parameter is not provided. **/ binary createXXX(1: binary create_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: errors. notFoundException) /** * update entity * input parameters: * 1. UpdateXXXRequest: update request, support updating multiple entities at the same time; UpdateXXXResponse: The update is successful and returns a list of ids of the entities that are updated. * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: This entity was not found on the server side. **/ binary updateXXX(1: binary update_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: errors. notFoundException) /** * delete entity * Input parameters: * 1. RemoveXXXRequest: delete request by ID. * Output parameters * removeXXXResponse: If the deletion succeeds, the ID list of the deleted entity is returned. * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: This entity was not found on the server side. **/ binary removeXXX(1: binary remove_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: errors. notFoundException) /** * Obtain entity by ID * input parameters: * 1.getxxxRequest: Get request, by ID, support to get multiple entities at once; * Output parameter * getXXXResponse: returns the corresponding entity list; * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: This entity was not found on the server side. **/ binary getXXX(1: binary get_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: errors. notFoundException) /** * Query entity * Input parameters: * 1. QueryXXXRequest: query condition; QueryXXXResponse: Returns a list of corresponding entities; * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: This entity was not found on the server side. **/ binary queryXXX(1: binary query_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: errors. notFoundException) /** * Count the number of eligible entities * Input parameters: * 1. CountXXXRequest: query condition; CountXXXResponse: Returns the number of entities; * Exception * 1. UserException: The input parameter is incorrect; * 2. "systemExeption" : * 3. NotFoundException: This entity was not found on the server side. **/ binary countXXX(1: binary count_xxx_request) throws (1: Errors.UserException userException, 2: Errors.systemException, 3: Errors.notFoundException) }Copy the code

Attached is 1.3 parameter design

The input and output parameters of each method are represented by protobuf.

file: xxx_service.protobuf

/** ** here is copyright declaration **/ option java_package ="com.phoenix.service";
import "entity.proto";
import "taglib.proto"; /** */ message CreateXXXRequest {optional string user_name = 1; // Access this interface user optional string password = 2; // The password to access this interface will be repeated XXXX XXX = 21; // Physical content; } /** * CreateXXXResponse {repeated int64 ID = 11; // List of successfully created entity ids}Copy the code

Attachment 1.4 Abnormal design

RPC interfaces also do not need too complicated exceptions, and generally define three types of exceptions.

file errors.thrift

/** * Errors caused by the caller, such as invalid parameters, lack of required parameters, no permissions, etc. * This exception is usually retried. ** */ exception UserException {1: Required ErrorCode error_code; 2: optional string message; } /** * Caused by a server error, such as database failure. This also includes QPS exceeding the limit, at which point rateLimit returns the assigned QPS limit; ** */ exception systemException {1: Required ErrorCode error_code; 2: optional string message; 3: i32 rateLimit; } /** * The object cannot be found based on the given ID or other criteria. * **/ exception systemException { 1: optional string identifier; }Copy the code

Appendix II service SDK

Of course, RPC services should not be provided directly to the business side, but to a packaged client. In general, clients need to encapsulate common functionality, including service discovery, RPC connection pooling, retry mechanisms, and QPS control, in addition to providing proxies to access servers. Here is the design of the service SDK. Using Protobuf as input and output parameters directly, the code developed is cumbersome:

GetXXXRequest.Builder request = GetXXXRequest.newBuilder();
request.setUsername("username");
request.setPassword("password");
request.addId("123");

GetXXXResponse response = xxxService.getXXX(request.build());
if(response.xxx.size()==1)
XXX xxx = response.xxx.get(0);Copy the code

As above, there is a lot of repetitive code that is not intuitive or convenient to use. Therefore, the client SDK needs to be used to make a layer of encapsulation for the business side to call:

Public XXX getXXX(String ID){getxxxRequest.Builder Request = getxxxRequest.newBuilder (); public XXX getXXX(String ID){getxxxRequest.builder request = getxxxRequest.newBuilder (); request.setUsername("username");
request.setPassword("password");
request.addId("123");

GetXXXResponse response = xxxService.getXXX(request.build());
if(response.xxx.size()==1)
return response.xxx.get(0);
returnnull; }}Copy the code

Providing a corresponding client-side SDK for all server-side interfaces is also one of microservices architecture best practices. Once this encapsulation is complete, the caller can use the interface as if it were a normal interface without knowing the implementation details.