Traditional applications use local transactions and distributed transactions to ensure data consistency. However, in the microservice architecture, data is private to the service and needs to be accessed through the API provided by the service. Therefore, distributed transactions are no longer suitable for microservice architecture. How does a microservice architecture ensure data consistency? This article will talk about this topic. Traditional distributed transaction is not the best choice for data consistency in the service Micro service architecture should satisfy data consistency principle in the end Micro service architecture to realize eventual consistency of three models Reconciliation is the last final line of defense Traditional distributed transaction Let’s look at the first part of the traditional use of local transactions and distributed transaction to ensure consistency.
Traditional stand-alone applications typically use a relational database, with the benefit that the application can use ACID. To ensure consistency, we just need to start a transaction, change (insert, delete, update) many rows, and commit the transaction (roll back the transaction if there is an exception). Further, with data access technologies and frameworks in development platforms such as Spring, we need to do less and just focus on the data itself changing. As the scale of an organization increases and the volume of services increases, a single application and database cannot support the huge volume of services and data. In this case, the application and database need to be separated. As a result, one application needs to access two or more databases at the same time. Initially we used distributed transactions to ensure consistency, also known as the two-phase commit protocol (2PC).
Local and distributed transactions are now mature enough for a rich introduction that we won’t discuss them here. Let’s talk about why distributed transactions are not appropriate for microservices architecture.
First, data access becomes more complex with microservice architectures because the data is microservice private and the only way to access it is through the API. This packaged data access approach makes microservices loosely coupled and independent of each other, making performance scaling easier. Second, different microservices often use different databases. Applications generate different types of data, and a relational database is not always the best choice. For example, an application that generates and queries strings uses Elasticsearch’s character search engine; An application that generates social image data could use a graph database, such as Neo4j. Microservices-based applications generally use a combination of SQL and NoSQL. But most of this non-relational data does not support 2PC. You can see that distributed transactions are no longer an option in microservices architecture. The ultimate consistency principle is based on CAP theory and requires a choice between availability and consistency. If you choose to provide consistency, you pay the price of blocking other concurrent access until consistency is satisfied. This can last for an indefinite period of time, especially if the system is already exhibiting high latency or if a network failure causes a loss of connection. Based on current success, availability is generally the better choice, but maintaining data consistency between services and databases is a very fundamental requirement, and the ultimate consistency should be chosen in the microservices architecture. Final consistency means that all data copies in the system reach a consistent state after a certain period of time. Of course, the final consistency is selected to ensure that the final period of time is within the acceptable range of users. So how do we achieve final consistency? By its very nature, consistency ensures that all services contained within a business logic will either succeed or fail. So how do we choose our direction? Guaranteed success or guaranteed failure?
We say the business model dictates our choice. There are three modes to achieve final consistency: reliable event mode, business compensation mode, and TCC mode. Reliable Event Pattern The reliable event pattern is an event-driven architecture where microservices publish an event to the message broker when something important happens, such as updating a business entity. The message broker pushes events to the microservices that subscribe to these events, and when the microservices that subscribe to these events receive this event, they can complete their business and may cause more events to be published. 1. If the order service creates an order to be paid, issue a “Create Order” event.
2. Payment service consumption “create order” event, release a “Payment completion” event after the payment is completed.
3. The “Payment completed” event of order service consumption, and the order status is updated to waiting for delivery.
Thus the completed business process is implemented. But it’s not a perfect process.
This process can lead to inconsistencies where a microserver fails to publish an event after updating a business entity; Although the microservice published the event successfully, the message broker failed to push the event correctly to the subscribed microservice; The microservice that receives the event re-consumes the event. The reliable event pattern is to ensure reliable event delivery and avoid repeated consumption. Reliable event delivery is defined as atomic business operations and release events for each service. The message broker ensures that the event is delivered at least once. Avoiding repeated consumption requires the service to be idempotent, such as the payment service cannot be paid more than once because of the repeated receipt of events. Because the current popular message queues all implement the event persistence and at least once delivery mode, “message broker to ensure that the event is delivered at least once” has been satisfied, today does not expand. The following content is mainly discussed from two aspects of reliable event delivery and implementation idempotency. Let’s look at reliable event delivery first. First let’s look at an implementation snippet, taken from a production system.
Based on the above code and comments, at first glance, three things can happen: the operation on the database succeeds, and the delivery of the event to the message broker succeeds. Failed to operate the database and will not post events to the message broker. The operation on the database succeeded, but the delivery of the event to the message broker failed, throwing an exception, and the operation just performed to update the database will be rolled back. From the above analysis, there seems to be no problem. However, careful analysis is not difficult to find the defects, in the above process there is a hidden time window.
When microservice A sends the event, the message broker may have processed it successfully, but when it returns the response, the network is abnormal, causing the append operation to throw an exception. The end result is that the event is posted and the database is rolled back.
If microservice A goes down between post completion and the database commit operation, the database operation will also be rolled back due to an abnormal connection shutdown. The end result is that the event is delivered and the database is rolled back. This implementation often runs for a long time without problems, but when it does, it can be confusing and hard to see where the problems are. Here are two implementations of reliable event delivery. 1. Local event table The local event table approach stores events and business data in the same database, uses an additional “event recovery” service to recover events, and local transactions ensure atomicity in updating business and publishing events. To allow for some delay in event recovery, the service can publish an event to the message broker as soon as it completes the local transaction.
Microservers record business data and events in the same local transaction. The microservice releases an event in real time and immediately notifies the associated service. If the event is successfully released, the recorded event is deleted immediately. The event recovery service periodically recovers unpublished events from the event table and re-publishes them. The recorded events are deleted only after the re-publishing succeeds. The operation of article ⅱ is mainly to increase the real-time performance of published events, while the third ensures that the events will be published. Local event table business system and event system are closely coupled, and additional event database operations will bring additional pressure to the database, which may become a bottleneck. 2. External event table The external event table method persists events to the external event system. The event system needs to provide real-time event service to accept microservices to publish events, and the event system also needs to provide event recovery service to confirm and recover events.
The business service sends the event through the real-time event service request to the event system before the transaction is committed, and the event system only records the event without actually sending it. After the submission, the business service sends the confirmation to the event system through the real-time event service. After the confirmation of the event, the event system actually releases the event to the message broker. During business rollback, a business service cancels events to the event system through real-time events. What if the business service stops serving before sending an acknowledgement or cancellation? Event The event recovery service of the system periodically checks the status of unconfirmed events to the service service, and determines whether to publish or cancel the events according to the status returned by the service service. In this way, the business system and the event system can be decoupled independently, and both can scale independently. But this approach requires an additional send operation and an additional query interface from the publisher. After introducing reliable event delivery, some events are idempotent and some are not. An event that is idempotent in itself is idempotent if it describes a fixed value at a point in time (e.g., an account balance of 100) rather than a conversion instruction (e.g., an increase in the balance of 10). We need to be aware that the number and order of possible events is unpredictable, and we need to ensure that the order of idempotent events is executed, otherwise the outcome is often not what we want. If we receive two events successively, (1) the account balance is updated to 100, and (2) the account balance is updated to 120. 1. Event received by micro service (1)
2. Micro Service receives event (2) 3. Micro Service receives event (1) again obviously the result is wrong, so we need to ensure that event (2) cannot be processed once event (1) is executed, otherwise the account balance is still not the result we want. A simple way to ensure the order of events is to add a timestamp to the event. The microservice records the last time stamp of each type of event, and if it receives an event with a timestamp older than ours, it discards the event. If events are issued on different servers and time synchronization between servers is a problem, it is safer to replace the timestamp with a global increasing sequence number. For non-idempotent operations, the main idea is to store the execution result for each event. When receiving an event, we need to query whether the event has been executed according to the ID of the event. If the event has been executed, the last execution result is directly returned; otherwise, the execution event is scheduled.
With this in mind we need to consider the overhead of repeatedly executing an event and storing the results of a query. If the cost of processing an event repeatedly is low or very few events are expected to be received repeatedly, you can choose to process an event repeatedly and have a unique constraint exception thrown by the database when the event data is persisted. If the cost of processing an event repeatedly is much higher than that of an additional query, use a filtering service to filter out the repeated events. The filtering service uses the event store to store the events and results that have already been processed. When receiving an event, the filtering service queries the event store to determine whether the event has been processed. If the event has been processed, the filtering service directly returns the stored result. Otherwise, the business service is scheduled to perform processing and the result of processing is stored in the event store. Normally the method above can run well, if our service is the RPC class service we need to be more careful, possible problems lies in the fact that (1) filtering services will only after business processing complete events results stored in the storage, but in front of the business processing is completed may have received a recurring events, Since it is RPC service, it cannot rely on the uniqueness constraint of database. (2) The processing result of the business service may appear location state, which generally occurs when the request is normally submitted but no response is received. For problem (1), you can record the event processing process according to steps. For example, the event processing process can be received, send request, receive reply, and Complete. The advantage is that the filtering service can detect duplicate events in time and further process them differently according to the event state. For problem (2), the actual processing status of the event can be determined by an additional query request. It should be noted that the extra query will bring a longer delay, and some RPC services may not provide a query interface at all. In this case, only temporary inconsistencies can be received, and reconciliation and manual access are used to ensure consistency. Compensation mode For the convenience of description, two concepts are defined here: Service exception: a situation where the service logic is wrong, for example, the account balance is insufficient or the commodity inventory is insufficient. Technical exception: non-service logical exception, such as network connection exception or network timeout. Compensation mode uses an additional coordination service to coordinate each microservice that needs to ensure consistency. The coordination service invokes each microservice in sequence. If a microservice invocation is abnormal (including business and technical exceptions), all the previously successfully invoked microservices will be cancelled. Compensation mode is recommended only when business exceptions cannot be avoided, and if possible the business mode should be optimized to avoid requiring compensation transactions. For example, business anomalies with insufficient account balance can be avoided by freezing the amount in advance, and merchants can be required to prepare additional inventory for insufficient commodity inventory. We illustrate the compensation model through an example. A travel company provides the business of booking trips, which can book air tickets, train tickets, hotels and so on in advance through the company’s website. Suppose a customer’s schedule is: shanghai-Beijing flight x at 9 o ‘clock on June 19th. 3 nights in xx Hotel. Beijing – Shanghai 17 o ‘clock train on June 22. After the customer submits the itinerary, the itinerary booking business of the travel company calls the flight booking service, hotel booking service and train booking service sequentially. The whole booking is not complete until the final train booking service is successful.
If the train ticket booking service is not successfully invoked, all flights and hotels booked before will have to be cancelled. The cancellation of hotels and flights booked before is the compensation process. To reduce development complexity and improve efficiency, coordinated service implementation is a common compensation framework. The compensation framework provides the ability to orchestrate services and automate compensation. To implement the compensation process, we need to do two things: first, identify the failed steps and states, and thus the scope for compensation. In the example above we need to know not only that step 3 (booking the train) failed, but also why it failed. If you return without a ticket because you booked a train service, the compensation process only needs to cancel the first two steps; However, if the failure is due to a network timeout, the compensation process needs to include a third step in addition to the first two steps. The second is to be able to provide the business data used by the compensation operation. For example, the compensation operation requirements of a payment microservice include the transaction id, account number and amount at the time of payment. In theory, the actual completion of the compensation operation can be based on a unique business flow ID, but providing more elements is beneficial to the robustness of the microservice. The microserver can do business checks when receiving the compensation operation, such as checking whether the accounts are equal, whether the amount is consistent, and so on.
The way to achieve the above two points is to record a complete business flow. The state of the business flow can be used to determine the steps that need to be compensated, and the business flow provides the required business data for the compensation operation.
When a reservation request from a customer arrives, the coordination service (compensation framework) generates a globally unique business serial number for the request and records the complete status while invoking the individual work services. Record the business flow that calls bookFlight, call the bookFlight service, and update the business flow state. Record the business flow that calls bookHotel, call the bookHotel service, and update the business flow state. Record the business flow calling bookTrain, call the bookTrain service, update the business flow state. When an exception occurs when invoking a service, such as the step 3 (train reservation) exception.
The coordination service (compensation framework) also logs the status of step 3, along with an additional event indicating an exception to the business. Then it is time to perform the compensation process. The scope of the compensation can be known from the state of the business flow, and the business data needed for the compensation process can be obtained from the recorded business flow. It is impossible for a generic compensation framework to know in advance what business elements microservices need to record. Then you need a way to ensure the extensibility of the business flow, and here are two methods: large tables and associated tables.
A large table is designed with a large number of spare fields in addition to the required fields. The framework can provide auxiliary tools to help map business data to spare fields. The association table is divided into frame table and service table. The technical table stores the technical data required for compensating operations. The service table stores the service data. Large tables are easy to implement for the framework layer, but there are some difficulties, such as how many fields are appropriate to reserve and how much length should be reserved for each field. Another difficulty is that if you query data from the data level, it is difficult to see the business meaning of the standby field, and the maintenance process is not friendly. The associated table is more flexible in business elements and can support different business types to record different business elements. However, the framework is more difficult to implement, and each query requires complex associated actions, which can affect performance. With the complete flow record above, the coordination service can complete the compensation process in the event of an exception based on the state of the working service. However, due to network reasons, the compensation operation is not guaranteed to be 100% successful. At this time, we need to do more.
As a service invocation process, the compensation process also has the case of unsuccessful invocation. In this case, the mechanism of retry is needed to ensure the success rate of compensation. This, of course, requires that the compensation operation itself be idempotent. The realization of idempotence was discussed earlier. Retry immediately if it is only a failure will put unnecessary stress on the working service, and we will choose a different retry strategy depending on the cause of the service execution failure.
1) If the cause of the failure is not temporary and the business error is caused by business factors (such as business factor check failure), such error can be automatically recovered without retransmission, then the retry should be terminated immediately. 2) If the error is caused by some rare exceptions, such as data loss or errors during network transmission, you should try again immediately, because similar errors are rarely repeated. 3) If the error is caused by a busy system (such as HTTP 500 or another return code) or a timeout, wait for some time and try again. The retry operation usually sets the maximum number of retries. If the number of retries reaches the upper limit, no retries are performed. This time should be through a means to inform the relevant personnel to deal with. If the retry policy still fails, you can increase the waiting time gradually until the waiting time reaches the upper limit. If a large number of operations need to be retried at any one time, the compensation framework needs to control the flow of requests to prevent undue stress on the working service. In addition, there are a few additional notes about the compensation mode. Microservices implement compensation operations that do not simply fall back to the state at which the business occurred, because there may be other concurrent requests that change the state at the same time. The compensation is usually done by inverse operation. The compensation process does not need to be performed in the exact opposite order of business occurrence, but can be performed first, depending on the degree of reuse of the work service, or even concurrently. The compensation process of some services is dependent. If the compensation operation of the dependent service is not successful, the compensation process will be terminated in time. If the work services included in a business do not all provide compensation operations, then we should choreograph the services that provide compensation operations first so that we have a chance to compensate if the work services that follow fail. Compensation interfaces for work services should be designed based on business elements that coordinate service requests, not response elements for work services. Because there are cases where timeouts need to be compensated, the compensation framework cannot provide the business elements needed to compensate. TCC mode (try-confirm-Cancel) A complete TCC service consists of one primary service and several secondary services. The primary service initiates and completes the entire service activity. In TCC mode, the secondary service provides three interfaces: Try, Confirm, and Cancel.
1. Try to complete all service checks. Reserve required service resources. 2. Release the service resources reserved in the Try phase. The Cancel operation is idempotent. The TCC service is divided into two phases.
Phase 1: The master business service invokes the try operation of all slave businesses separately and registers all slave business services in the activity manager. The second phase begins when all the try operations from the business service are called successfully or one of the try operations from the business service fails. Phase 2: The activity manager performs confirm or Cancel based on the results of phase 1. If all the first phase try operations are successful, the activity manager invokes all confirm operations from the business activity. Otherwise, all cancel operations from the business service are invoked. It should be noted that the confirm or Cancel operation in the second stage itself is also a process to meet the final consistency. The invocation of confirm or Cancel may fail due to some reason (such as network), so activity management is required to support retry. This also requires the confirm and Cancel operations to be idempotent. One of the more obvious defects in the compensation mode is that there is no isolation. Inconsistencies are visible to other services from the first work service step until all work services are completed (or the compensation process is completed). In addition, the guarantee of final consistency is fully dependent on the robustness of the coordination service. If the coordination service is abnormal, consistency cannot be achieved. In the TCC mode, all business operations are isolated (guaranteed by the business level) until an explicit confirm action. In addition, the work service can realize autonomous micro-service by specifying the timeout period of the try operation and actively cancel the reserved service resources. TCC mode and compensation mode need coordination service and work service, coordination service can also be implemented as a general service framework. Unlike the compensation pattern, the TCC service framework does not need to record detailed business flows, and the business elements that perform confirm and Cancel operations are provided by business services.
Until step 4 confirms the order, the order is in a pending state and will not take effect until an explicit confirm is made.
If one of the three services fails the try operation, it can submit cancel to the TCC service framework, or do nothing and have the work service handle the timeout itself.
The TCC pattern also does not guarantee 100% consistency. If the TCC service framework fails to commit confirmations to a working service (such as a network failure) after the business service submits confirmations to the TCC service framework, then inconsistencies, commonly known as heuristic Exceptions, occur. It should be noted that in order to ensure the success rate of business, the business service must support retry when it submits confirm to the TCC service framework and TCC service framework submits confirm/ Cancel to the work service. Therefore, the realization of Confirm/Cancel must be idempotent. If the business service fails to submit confirm/ Cancel to the TCC service framework, there is no inconsistency because the service will eventually time out and cancel. Heuristic exceptions are inevitable, but they can be minimized by setting appropriate timeouts, retries, and monitoring measures. If a heuristic exception appears, it can be remedied by artificial means. Reconciliation is the last line of defense. If some business is due to instantaneous network failure or call timeout, the above three modes can generally be solved. However, in today’s cloud computing environment, many services depend on the availability of external systems, and periodic reconciliation is required in some important business scenarios to ensure true consistency. For example, there will be reconciliation between the payment system and the bank at the end of every day.