What is idempotency

Idempotence is a mathematical and computer concept in which an operation of one element is idempotence, acting on any element twice will have the same effect as acting on it once. In computer programming, an idempotent operation is characterized by any number of executions having the same effect as a single execution.

An idempotent function or method is a function that can be executed repeatedly with the same parameters and obtain the same result. These functions do not affect system state, nor do they have to worry about system changes caused by repeated execution.

What is interface idempotency

In HTTP/1.1, idempotency is defined. It describes that one or more requests to a resource should have the same effect on the resource itself (except for network timeouts), that is, the first request has side effects on the resource, but subsequent requests do not have side effects on the resource.

The side effect here is that you don’t spoil the results or have unexpected results. That is, any number of executions will have the same impact on the resource itself as one execution.

Why is idempotence necessary

When the interface is called, it normally returns a message and does not commit again. However, problems may occur when the following situations occur, such as:

  • Front-end repeated submission of forms: When filling in some forms, the user completes the submission. In many cases, due to network fluctuations, the user does not respond to the successful submission in time. As a result, the user thinks that the submission is not successful, and then clicks the submit button all the time.
  • Malicious user brushing: For example, when the user voting function is implemented, if a user repeatedly votes for a user, the interface will receive the voting information repeatedly submitted by the user, and the voting result is seriously inconsistent with the facts.
  • Interface timeout repeated submission: In most cases, the HTTP client tool enables the timeout retry mechanism by default. When a third party invokes an interface, a retry mechanism is added to prevent request failures caused by network timeout. As a result, one request is submitted multiple times.
  • Repeated consumption of messages: When using MQ messaging middleware, repeated consumption occurs if the messaging middleware fails to submit consumption information in a timely manner.

The biggest advantage of using idempotent is that the interface guarantees any idempotent operation, avoiding unknown problems with the system caused by retries and so on.

The influence of idempotence on the system

Idempotency is to simplify the logical processing of the client side and can place repeated submission and other operations, but it increases the logical complexity and cost of the server side, mainly:

  • The function of parallel execution is changed to serial execution, which reduces the execution efficiency.
  • Complicate business functions by adding additional control idempotent business logic;

Therefore, it is necessary to consider whether to introduce idempotency when using the interface. According to the actual service scenario, except for special service requirements, interface idempotency is generally not needed.

Idempotency of Restful APIS

There are idempotent lines and methods that cannot guarantee idempotent lines in several popular Restful HTTP interface methods, as follows:

  • The square root is idempotent
  • X is not idempotent
    • It may or may not be idempotent, depending on the actual business logic

How do you achieve idempotency

Scheme 1: Unique primary key of the database

Scheme described

The realization of unique primary key of database mainly uses the unique constraint of primary key in database. Generally speaking, unique primary key is more suitable for idempotency of “insert”, which can ensure that only one record with the unique primary key can exist in a table.

When using a database unique primary key to achieve idempotency, it is important to note that the primary key is generally not augmented in the database, but is used as the primary key by a distributed ID to ensure global uniqueness of ids in a distributed environment.

Applicable operation:

  • The insert
  • Delete operation

Restrictions on use:

  • Need to generate globally unique primary key ID;

Main process:

Main process:

  • ① The client performs the creation request and invokes the server interface.
  • ② The server executes the business logic, generates a distributed ID, acts as the primary key of the data to be inserted, and then executes the data insertion operation and the corresponding SQL statement.
  • ③ The server inserts the data into the database. If the data is inserted successfully, the interface is not invoked repeatedly. If a duplicate primary key exception is thrown, the record already exists in the database and an error message is returned to the client.

Solution 2: Optimistic database lock

Scheme Description:

Optimistic database locking scheme is generally only applicable to the process of “update operation”. We can add an extra field in the corresponding data table in advance to act as the version identifier of the current data. Each update to this data in the database table will use the version id as a condition, and the value will be the value of the version ID in the last data to be updated.

Applicable operation:

  • The update operation

Restrictions on use:

  • Additional fields need to be added in the corresponding service table of the database;

Description Example:

For example, the following data table exists:

To prevent repeated updates each time an update is performed, we usually add a version field to record the current version of the record, so that the value is added during the update, so that the update operation can determine that the information under the corresponding version has been updated.If version=5 is updated, you must specify the version number to be updated each time.

UPDATE my_table SET price=price+50,version=version+1 WHERE id=1 AND version=5
Copy the code

Select * from SQL WHERE id=1 AND version=5; select * from SQL WHERE id=1 AND version=5; This preserves the idempotent nature of the update, and multiple updates have no effect on the results.

Solution 3: Anti-duplicate Token

Scheme Description:

In the case of continuous clicks by the client or timeout retries by the caller, such as order submission, this operation can be implemented by Token mechanism to prevent repeated submission.

In simple terms, the caller first requests a global ID (Token) from the backend when invoking the interface. The backend requests the global ID together with the request (the Token is best placed in Headers). The backend needs to treat the Token as a Key. The user information is used as the Value in Redis to verify the Key Value content. If the Key exists and the Value matches, the deletion command is executed, and then the following business logic is normally executed. If there is no corresponding Key or Value mismatch, a repeat error message is returned to ensure idempotent operations.

Applicable operation:

  • The insert
  • The update operation
  • Delete operation

Restrictions on use:

  • Need to generate globally unique Token string;
  • Need to use the third-party component Redis for data validation;

Main process:

  • ① The server provides an interface to obtain a Token. The Token can be a serial number, distributed ID or UUID string.
  • ② The client invokes the interface to obtain the Token. In this case, the server generates a Token string.
  • ③ Then the string is stored in the Redis database and the Token is used as the Redis key (note the expiration time).
  • ④ Return the Token to the client. After the client gets it, it should be saved in the hidden field of the form.
  • ⑤ When the client submits the form, it stores the Token into the Headers, and then carries the Headers with the service request.
  • ⑥ After receiving the request, the server obtains the Token from the Headers and checks whether the key exists in Redis based on the Token.
  • ⑦ The server determines whether the key exists in Redis. If the key exists, it will delete it and then execute the business logic normally. If not, throw an exception and return a duplicate error message.

Note that performing Redis lookups and deletions in concurrent cases requires atomicity, otherwise idempotency may not be guaranteed in concurrent cases. It can be implemented using distributed locks or Lua expressions to unregister queries and deletes.

Scheme 4. Downstream transmission of unique serial numbers

Scheme Description:

The so-called request sequence number, in fact, is each request to the server with a unique serial number in a short period of time, the serial number can be an ordered ID, or an order number, generally generated by the downstream, when calling the upstream server interface to attach the serial number and ID for authentication.

When the upstream server receives the request information, it combines the serial number with the downstream authentication ID to form a Key for operating Redis, and then queries Redis to see if there is a Key value pair corresponding to the Key. According to the result:

  • If yes, it indicates that the downstream request of the sequence number has been processed. In this case, you can directly respond to the error message of repeated request.
  • If not, the Key is used as the Key of Redis, the downstream Key information is used as the stored value (for example, some business logic information transmitted by downstream merchants), the Key value pair is stored in Redis, and then the corresponding business logic can be executed normally.

Applicable operation:

  • The insert
  • The update operation
  • Delete operation

Restrictions on use:

  • Require a third party to deliver a unique serial number;
  • Need to use the third-party component Redis for data validation;

Main process:

Main steps:

  • ① The downstream service generates the distributed ID as the serial number, and then performs the request to invoke the upstream interface, along with the “unique serial number” and the requested “authentication credential ID”.
  • ② The upstream service carries out security effect test to detect whether there are “serial number” and “credential ID” in the parameters transmitted downstream.
  • ③ The upstream service checks whether there is a Key consisting of the serial number and authentication ID in Redis. If there is a Key, the upstream service throws a repeated exception message and responds to the downstream error message. If it does not exist, the combination of the serial number and authentication ID is used as the Key, and the downstream Key information is used as the Value, which is then stored in Redis, and the incoming service logic is normally executed.

When inserting data into Redis in the previous step, be sure to set the expiration time. This ensures that if the interface is repeatedly called within this time range, it will be able to make a judgment call. If you do not set the expiration time, it is likely that unlimited data will be stored in Redis, causing Redis to not work properly.

Example of implementing interface idempotent

Here we use the duplicate proof Token scheme, which guarantees idempotency in different request actions. The implementation logic can be seen in the “duplicate proof Token” scheme above, and the code to implement this logic is written below.

Maven introduces dependencies

Maven tools are used to manage dependencies, and SpringBoot, Redis, and Lombok dependencies are introduced in pom.xml.

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion > 4.0.0 < / modelVersion > < the parent > < groupId > org. Springframework. Boot < / groupId > The < artifactId > spring - the boot - starter - parent < / artifactId > < version > 2.3.4. RELEASE < / version > < / parent > < the groupId > mydlq. Club < / groupId > < artifactId > springboot - idempotent - token < / artifactId > < version > 0.0.1 < / version > <name>springboot-idempotent-token</name> <description>Idempotent Demo</description> <properties> < Java version > 1.8 < / Java version > < / properties > < dependencies > <! --springboot web--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <! --springboot data redis--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.apache.commons</groupId>  <artifactId>commons-pool2</artifactId> </dependency> <! --lombok--> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>Copy the code

2. Set parameters for connecting to Redis

In the application configuration file, set the parameters for connecting to Redis as follows:

Spring: redis: SSL: false host: 127.0.0.1 Port: 6379 database: 0 timeout: 1000 Password: lettuce: pool: max-active: 100 max-wait: -1 min-idle: 0 max-idle: 20Copy the code

3. Create and verify the Token utility class

Create a Service class for manipulating tokens, which contains the Token creation and verification methods:

  • Token creation method: Use the UUID tool to create a Token string, set idempotent_token: + Token string as the Key, use user information as the Value, and save the information in Redis.
  • Token authentication method: Receives Token string parameters, adds Key prefix to form Key, passes in value value, executes Lua expression (Lua expression ensures atomicity of command execution) to search for corresponding keys and delete them. After the command is executed, verify the returned result. If the result is not null or zero, the verification succeeds. Otherwise, the command fails.
import java.util.Arrays; import java.util.UUID; import java.util.concurrent.TimeUnit; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.data.redis.core.StringRedisTemplate; import org.springframework.data.redis.core.script.DefaultRedisScript; import org.springframework.data.redis.core.script.RedisScript; import org.springframework.stereotype.Service; @Slf4j @Service public class TokenUtilService { @Autowired private StringRedisTemplate redisTemplate; /** * private static final String IDEMPOTENT_TOKEN_PREFIX = "idempotent_token:"; /** * Create Token to store in Redis, Return the Token * * @param Value Value used for authentication * @return Generated Token String */ public String generateToken(String Value) {// Instantiate an ID utility object String token = uuID.randomuuid ().toString(); String Key = IDEMPOTENT_TOKEN_PREFIX + token; / / store Token to Redis, and set the expiration time is 5 minute redisTemplate opsForValue (). The set (key, value, 5, TimeUnit. MINUTES); // Return Token return Token; } /** ** Verify Token correctness ** @param Token Token string * @param value value Auxiliary verification information stored in Redis * @return verification result */ public Boolean ValidToken (String token, String value) {// Set Lua script, where KEYS[1] is key, KEYS[2] is value String script = "if redis. Call ('get', KEYS[1]) == KEYS[2] then return redis. Call ('del', KEYS[2]) KEYS[1]) else return 0 end"; RedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class); String Key = IDEMPOTENT_TOKEN_PREFIX + token; Long result = redistemplate. execute(redisScript, arrays.aslist (key, value)); If (result! = 0) // if (result! = 0) = null && result ! = 0, l) {log. The info (" authentication token = {}, key = {}, value = {} "success, token, key, value); return true; } the info (" authentication token = {}, key = {}, value = {} failure ", token, key, value); return false; }}Copy the code

4. Create the Controller class for the test

Create a Controller class for testing, which contains the interface to obtain Token and idempotency of the test interface, as follows:

import lombok.extern.slf4j.Slf4j; import mydlq.club.example.service.TokenUtilService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.*; @Slf4j @RestController public class TokenController { @Autowired private TokenUtilService tokenService; /** * @return Token String */ @getMapping ("/ Token ") public String getToken() { The content stored here is only an example. It serves as an auxiliary verification to make its verification logic more secure. For example, the user information is stored here for the purpose of: // -1), use "token" to verify whether the corresponding Key exists in Redis // -2), use "user information "to verify whether the Redis Value matches. String userInfo = "mydlq"; / / access Token string, and returns the return tokenService. GenerateToken (the userInfo); } /** * Interface idempotent test interface ** @param Token Idempotent token String * @RETURN Execution result */ @postMapping ("/test") public String Test (@requestheader (value = "token") String token) {// Obtain user information (here using analog data) String userInfo = "mydlq"; Boolean result = tokenservice.validToken (Token, userInfo); Return result? "Normal call" : "repeated call "; }}Copy the code

5. Create the SpringBoot boot class

Create a boot class to start the SpringBoot application.

import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); }}Copy the code

6. Write test classes to test

Write a test class to test whether the same interface can be accessed multiple times.

import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.http.MediaType; import org.springframework.test.context.junit4.SpringRunner; import org.springframework.test.web.servlet.MockMvc; import org.springframework.test.web.servlet.request.MockMvcRequestBuilders; import org.springframework.test.web.servlet.setup.MockMvcBuilders; import org.springframework.web.context.WebApplicationContext; @Slf4j @SpringBootTest @RunWith(SpringRunner.class) public class IdempotenceTest { @Autowired private WebApplicationContext webApplicationContext; @test public void interfaceIdempotenceTest() throws Exception {// Initialize MockMvc MockMvc MockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); / / call the access Token interface String Token = mockMvc. Perform (MockMvcRequestBuilders. Get ("/Token "). The accept (MediaType. TEXT_HTML)) .andReturn() .getResponse().getContentAsString(); Log.info (" Token string: {}", Token); For (int I = 1; i <= 5; I ++) {log.info(" call test interface {} time ", I); / / call the validation interfaces and print the result String result = mockMvc. Perform (MockMvcRequestBuilders. Post ("/test "). The header (" token ", token) .accept(MediaType.TEXT_HTML)) .andReturn().getResponse().getContentAsString(); log.info(result); If (I == 0) {assert.assertequals (result, "normal call "); } else {assert. assertEquals(result, "repeated call "); }}}}Copy the code

The following information is displayed:

[main] IdempotenceTest: Obtained Token string: 980EA707-CE2E-456E-A059-0A03332110b4 [main] IdempotenceTest: Obtained Token string: 980EA707-CE2E-456E-A059-0A03332110b4 The first call to test interface [main] IdempotenceTest: normal call to test interface [main] IdempotenceTest: Second call to test interface [main] IdempotenceTest: Call [main] IdempotenceTest repeatedly: call the test interface for the third time [main] IdempotenceTest: Call test interface [main] IdempotenceTest: repeat call [main] IdempotenceTest: repeat call 5th call test interface [main] IdempotenceTest: repeat callCopy the code

The final summary

Idempotency is a very common and important requirement in development, especially in payment, order and other money-related services, ensuring interface idempotency is especially important. In actual development, we need to flexibly choose the realization mode of idempotency for different business scenarios:

  • For orders with unique primary keys, the “unique primary key scheme” can be used.
  • For update scenario operations, such as updating order status, it is simpler to implement an “optimistic locking scheme”.
  • For upstream and downstream, where downstream requests upstream, the upstream service can use the “downstream pass unique sequence number scheme” more reasonable.
  • Similar to the scenario of repeated submission, repeated ordering, and no unique ID number, the “Anti-duplicate Token solution” can be implemented faster through the combination of Token and Redis.

The above is just some suggestions. Once again, to realize idempotence, it is necessary to understand its own business needs first and realize it according to the business logic, so as to deal with every node details and improve the overall business process design, so as to better ensure the normal operation of the system. To conclude this post with a brief summary: