With the rapid development of business and the increasing complexity of business, almost every company’s system will move from monomer to distributed, especially to micro-service architecture. This article summarizes the most classic solution of distributed transaction and shares it with you.

The basic theory

Before explaining the specific scheme, let’s take a look at the basic theories involved in distributed transactions.

Let’s take transfer as an example. A needs to transfer 100 yuan to B, so the balance of A needs to be -100 yuan, and the balance of B needs to be +100 yuan. The whole transfer should ensure that A-100 and B+100 are both successful or fail at the same time. Take a look at how this problem is solved in various scenarios.

The transaction

The ability to operate on multiple statements as a whole is called a database transaction. A database transaction ensures that all operations within the scope of that transaction can either succeed or fail altogether.

Transactions have four attributes: atomicity, consistency, isolation, and persistence. These four properties are often referred to as ACID properties.

  • Atomicity: All the operations in a transaction are either completed or not completed at all, and do not end somewhere in between. When an error occurs during the execution of a transaction, it is restored to the state it was in before the transaction started, as if the transaction had never been executed.
  • Consistency: The integrity of the database has not been compromised before the start of the transaction and after the end of the transaction. Integrity includes foreign key constraints, application-defined constraints, and so on that are not broken.
  • Isolation: The ability of a database to allow multiple concurrent transactions to read, write and modify its data at the same time. Isolation prevents data inconsistencies due to cross-execution when multiple transactions execute concurrently.
  • “Durability” : After the transaction is over, changes to the data are permanent, not lost in the event of a system failure.

Distributed transaction

The inter-bank transfer business of A bank is A typical distributed transaction scenario. Suppose A needs to transfer between banks to B, then the data of two banks are involved, and the ACID of transfer cannot be guaranteed through A local transaction of A database, but can only be solved through distributed transactions.

Distributed transactions mean that the initiator of the transaction, the resource and the resource manager and the transaction coordinator are located on different nodes of the distributed system. In the above transfer business, User A-100 operations and User B+100 operations are not on the same node. Essentially, distributed transactions are designed to ensure that data operations are performed correctly in distributed scenarios.

In the distributed environment, in order to meet the requirements of availability, performance and degraded service, and reduce the requirements of consistency and isolation, distributed transactions follow the BASE theory on the one hand (the related theories of BASE involve a lot of content, those who are interested can refer to the BASE theory) :

Eventual Consistency Similar to Eventual Consistency, distributed transactions follow the ACID specification in part:

Atomicity: Strict adherence to consistency: Strict adherence to consistency after completion of a transaction; Consistency in transactions can ease isolation appropriately: parallel transactions cannot be influenced; Transaction intermediate result visibility allows for secure relaxation of persistence: strict compliance

Distributed transaction solutions

Two-phase commit /XA

XA is a distributed transaction specification proposed by the X/Open organization. The XA specification mainly defines the interface between the (global) transaction manager (TM) and the (local) resource manager (RM). Local databases such as MySQL play the role of RM in XA

XA is divided into two stages:

The first stage (Prepare) : This is when all participants RM are ready to execute the transaction and lock up the required resources. When a participant is ready, it reports to TM that it is ready. The second phase (COMMIT/ROLLBACK) : After the transaction manager (TM) confirms that all participants (RM) are ready, a COMMIT command is sent to all participants. At present, the mainstream databases basically support XA transactions, including MySQL, Oracle, SQL Server, Postgre

XA transactions consist of one or more resource managers (RM), a transaction manager (TM), and an ApplicationProgram.

Taking the transfer above as an example, the timing diagram for a successful XA transaction is as follows:

If any of the participants prepare fails, then TM notifies all participants who have completed the prepare to roll back.

XA transactions are characterized by:

  • Simple, easy to understand and easy to develop
  • The resources are locked for a long time and the concurrency is low

If the reader wants to explore XA further, refer to DTM for Go and Seata for Java

SAGA

SAGA is a scheme mentioned by SAGA in this database paper. The core idea is to split a long transaction into several local short transactions, coordinated by the Saga transaction coordinator, complete normally if it ends normally, and invoke the compensation operation once, in reverse order, if a step fails.

Taking the transfer above as an example, the timing diagram for a successfully completed Saga transaction looks like this:



Features of Saga transactions:

  • High degree of concurrency, not like XA transactions as long locked resources
  • Normal operations and compensation operations need to be defined, and the amount of development is greater than XA
  • The consistency is weak. As for the transfer, it may happen that User A has deducted the money, and the transfer finally fails

There are many Saga contents in this paper, including two recovery strategies, including concurrent execution of branch transactions. Our discussion here only includes the simplest Saga

Saga is applicable to many scenarios, such as long transactions and business scenarios that are not sensitive to intermediate results

If the reader wants to explore Saga further, refer to DTM in Go and Seata in Java

TCC

About the concept of TCC (try-confirm-cancel), It was first proposed by Pat Helland in a paper titled Life Beyond Distributed Transactions: An Apostate’s Opinion published in 2007.

TCC is divided into three phases

  • Try phase: Attempt execution, complete all business checks (conformance), reserve necessary business resources (quasi-isolation)
  • Confirm stage: Confirm the actual execution of the business, do not make any business check, only use the business resources reserved in the Try stage, Confirm operation requires an idempotent design, the Confirm failed to need to retry.
  • Cancel phase: Execution is cancelled, freeing the business resource reserved in the Try phase. The exception handling scheme in the Cancel stage and the Confirm stage is basically the same, which requires idempotent design.

Take the above transfer as an example, the amount is usually frozen in Try but not deducted, the amount is deducted in Confirm and the amount is unfrozen in Cancel. The timing diagram of a successfully completed TCC transaction is as follows:

TCC features are as follows:

  • High degree of concurrency, no long-term resource locking.
  • The Try/Confirm/Cancel interface is required due to the large amount of development.
  • The consistency is good, and the transfer failure after the Saga deduction will not occur
  • TCC applies to Order-type business, which has constraints on the intermediate state

If the reader wants to explore TCC further, refer to DTM for Go and Seata for Java

Local message table

The idea of a local message table was originally written by Dan Pritchett, an architect at eBay, to the ACM in 2008. The core of the design is to ensure the execution of tasks that require distributed processing asynchronously by means of messages.

The general process is as follows:



Writing the local message and the business operation in one transaction guarantees the atomicity of the business and the sending message, and either they all succeed or they all fail.

Fault Tolerant Mechanism:

  • When a balance deduction transaction fails, the transaction is rolled back directly with no further steps
  • Round-order production message failure, increase balance transaction failure will be retried

Local message table features:

  • Long transactions only need to be split into multiple tasks, easy to use
  • Producers need to create additional message tables
  • Polling is required for each local message table
  • If the logic of the consumer does not succeed through retry, then more mechanisms are needed to roll the operations back and forth

Suitable for businesses that can be executed asynchronously and where subsequent operations do not need to be rolled back

Transaction message

In the local message table scheme mentioned above, the producer needs to create additional message tables and poll the local message table, so the business burden is heavy. Alibaba’s open source RocketMQ after 4.3 officially supports transactional messaging, which essentially puts a local message table on RocketMQ, addressing the atomicity of message sending and local transaction execution on the production side.

Transactional message sending and submission:

  • Send a message (half message)
  • The server stores the message and responds to the written result of the message
  • Execute the local transaction based on the sent result (if the write fails, the Half message is not visible to the business at this time and the local logic is not executed)
  • Perform a COMMIT or ROLLBACK based on the state of the local transaction (the COMMIT operation publishes the message, which is visible to the consumer)

The flow chart of normal transmission is as follows:

Compensation process:

For transaction messages without COMMIT/ROLBACK (pending state messages), a “lookback” is initiated from the server side by a Producer who receives the lookback message and returns the status of the corresponding local transaction. The scheme of transaction messages for COMMIT or ROLBACK is very similar to the mechanism of the local message table. The main difference is that the local table operation is replaced with an unlookup interface

The transaction message features the following:

  • Long transactions only need to be split into multiple tasks, and provide a anti-lookup interface, easy to use
  • If the logic of the consumer does not succeed through retry, then more mechanisms are needed to roll the operations back and forth

Suitable for businesses that can be executed asynchronously and where subsequent operations do not need to be rolled back

If you want to explore transaction messages further, you can refer to RocketMQ, and DTM also provides a simple implementation to help you learn about transaction messages

Best effort notice

The initiating notifier tries its best to inform the receiving party of the result of business processing through a certain mechanism. Specifically including:

There is a notification mechanism for repeated messages. Since the receiving party may not have received the notification, there should be some mechanism to repeat the notification of the message. Message proofing mechanism. If no notification is made to the recipient despite the best efforts, or the recipient needs to consume the message again after consuming the message, the recipient can take the initiative to query the message information from the notifying party to meet the demand. Both the local message table and the transactional messages described earlier are reliable messages; how are they different from the best effort notifications described here?

Reliable message consistency. The initiator notifier needs to ensure that the message is sent and sent to the receiving notifier. The key to the reliability of the message is guaranteed by the initiator notifier.

The notifying party tries its best to notify the business processing result to the receiving party, but the message may not be received. In this case, the receiving notifying party needs to take the initiative to call the interface of the initiating notifying party to query the business processing result. The key to the reliability of the notification is the receiving notifying party.

On the solution, try best to inform the need for:

  • Provide an interface that enables the receiver to query the results of the business process through the interface
  • Message queue ACK mechanism, message queue in accordance with the interval of 1min, 5min, 10min, 30min, 1h, 2h, 5h, 10h, gradually increase the notification interval, until reaching the notification required time window upper limit. No further notice

Maximum effort notification is applicable to the type of business notification. For example, the result of WeChat transaction is notified to various merchants through maximum effort notification, which has both callback notification and transaction query interface

AT transaction mode

This is a transaction mode in Seata, Alibaba’s open source project, which is also known as FMT in Ant Financial. The advantage of this transaction mode is that it is similar to XA mode, in that the business does not need to write all kinds of compensation operations, and the rollback is completed automatically by the framework. The disadvantage is similar to AT, in that there is a long time of locking, which does not meet the high concurrency scenarios. If you are interested, you can refer to SEATA-AT

Network exceptions in distributed transactions

In each link of distributed transactions, network and business failures may occur, which require the business side of distributed transactions to achieve three characteristics of anti-air rollback, idempotency and anti-suspension. The following TCC transactions illustrate these abnormal situations:

Empty rollback:

The two-stage Cancel method is called without calling the TCC resource Try method, and the Cancel method needs to recognize that this is an empty rollback and directly return success.

The reason for this is that when a branch transaction is recorded as a failure due to service outage or network exception, the Try phase is actually not executed at this time. When the failure is recovered, the Cancel method of the second phase will be called for the rollback of the distributed transaction, thus forming an empty rollback.

Idempotent:

Since any request may have network exceptions and duplicate requests, all distributed transaction branches need to ensure idempotency

Suspension:

Suspension means that for a distributed transaction, the second phase of the Cancel interface is executed before the Try interface.

When the branch transaction try is called by RPC, the branch transaction is registered first and then the RPC call is executed. If the network of the RPC call is congested at this time, after the RPC timeout, TM will inform RM to roll back the distributed transaction. It may be that after the rollback is completed, the RPC request will reach the participant for actual execution.

Let’s look at the timing diagram of a network exception to better understand the above problems

Cancel is executed before the Try when the business processing request 4 is required. Cancel is repeated when the null rollback business processing request 6 is required. When the idempotent business processing request 8 is required, the Try is executed after Cancel and the suspension is required

In the face of the complex network exceptions mentioned above, all the proposed solutions are that the business side queries whether the associated operation has been completed through the unique key, and returns success directly if it has been completed. The related judgment logic is complex, error-prone, and the business burden is heavy.

Here we present a sub-transaction barrier technique, which is our first. In our project DTM, the method ThroughBarrierCall is provided. The prototype of the method is as follows:

func ThroughBarrierCall(db *sql.DB, transInfo *TransInfo, busiCall BusiFunc)

Business developers, writing their own related logic in BusiCall, call this function. ThroughBarrierCall ensures that busiCall will not be called under scenarios such as empty rollback and suspension. When the business is called repeatedly, there is idempotent control to ensure that it is committed only once.

Subtransaction barriers manage TCC, Saga, XA, transaction messages, etc., and can be extended to other domains as well

The principle of transaction barrier technology is, in the local database, set up branch sub_trans_barrier transaction state table, the only key to global transaction id – child transaction id – the transaction branch name (try | confirm | cancel)

  • Open the transaction
  • If it is a Try branch, then insert ignore inserts gid-branchid-try and, if successful, invokes the barrier logic
  • If it is a Confirm branch, insert ignore inserts gid-branchid-confirm and, if successful, invokes the barrier logic
  • If it is a branch of Cancel, insert ignore inserts gid-branchid-try, then gid-branchid-cancel. If the try is not inserted and the Cancel is inserted successfully, then the inside barrier logic is called
  • The barrier logic returns success, commits the transaction, and returns success
  • The barrier logic returns an error, rolls back the transaction, and returns an error

In this mechanism, the problems related to network anomalies are solved

  • Null compensation control — If the Try does not execute and executes CANCEL directly, then Cancel inserts gid-branchid-try successfully, avoiding the logic inside the barrier and ensuring null compensation control
  • Idempotency control – Any branch cannot repeatedly insert a unique key, which guarantees no duplicate execution
  • Anti-hang control — If the Try is executed after Cancel, then if the inserted gid-branchid-try fails, it will not be executed, ensuring anti-hang control

A similar mechanism exists for Saga transactions.

Invented transaction barrier technology, for the DTM, its significance lies in the design simple and easy to implement algorithms, provides a simple and easy to use interface, at first, its significance lies in the design simple and easy to implement algorithms, provides a simple and easy to use interface, with the help of the two, developers completely liberated from network exception handling.

The technology currently needs to be paired with the DTM transaction manager, and the SDK is currently available to Go developers. SDKs for other languages are in the works. For other distributed transaction frameworks, as long as appropriate distributed transaction information is provided, the technology can be implemented quickly according to the above principles.

conclusion

This paper introduces some basic theories of distributed transaction, and explains some common distributed transaction schemes. In the last part of the paper, the causes, classification and elegant solutions of transaction anomalies are given.

Distributed transaction itself is a technical problem, and the specific scheme to be used in a business still needs to be chosen by oneself according to its own business characteristics. Each scheme needs to be considered in the actual implementation process with a large number of points and complexity, so distributed transaction should be used with caution.