Writing in the front

With the increase of business volume, the company has spent several months recently to fully access Redis in the project. In the development process, it found that there was a lack of specific practical information on the market, especially in the Node.js environment. The information that could be found was either too simple or not true, and most of them were elementary. Therefore, I decided to share the company’s achievements during this period. I will use several articles to introduce several scenarios of Redis in detail, hoping that everyone can learn and make progress together. Here’s the first, seckill scene.

Business analysis

In actual business, seckill includes many scenarios, which can be divided into three stages before, during and after seckill. From the perspective of development, the specific analysis is as follows:

  1. Before the second kill: mainly to do a good job of caching, in order to deal with frequent user visits, because the data is fixed, you can make the elements of the product details page static, and then useCDNOr the browser caches it.

  2. Seckill: mainly inventory check, inventory deduction and order processing, the characteristics of this step are

    • In a short period of time, a large number of users are buying at the same time, the flow of the system suddenly surges, and the pressure of the server increases instantly (high instantaneous concurrent access).
    • The number of requests is greater than the inventory of goods, such as 10,000 users snapping up the goods, but the inventory is only 100
    • Limit purchases to a certain period of time
    • Limit purchases to a single user and avoid brushing
    • Snap is dealing with the database, the core function is to place orders, inventory can not be deducted into a negative number
    • The operation of the database reads more than writes, and the read operation is relatively simple
  3. After the second kill: mainly some users to view the purchase order, processing refund and processing logistics and other operations, at this time the user request has declined, the operation is relatively simple, the server pressure is not big.

Based on the above analysis, this paper focuses on the development of seckill, and those interested in other parts can search for information and try it.

The development environment

Database: Redis 3.2.9 + MySQL 5.7.18 Server: Node.js v10.15.0 Test Tool: Jmeter-5.4.1

In actual combat

Database preparation



As is shown in the figure,MysqlYou need to create three tables in the

  • Product table, used to record product information, fields are Id, name, thumbnail, price, status, etc
  • The seckill activity table is used to record the detailed information of the seckill activity. The fields are ID, ID of the products participating in the seckill, inventory, the start time of the seckill, the end time of the seckill, and whether the seckill activity is valid, etc
  • The order table is used to record the data after placing an order. The fields are ID, order number, product ID, purchasing user ID, order status, order type and seckill activity ID, etc

The following is to create the SQL statement for your reference

CREATE TABLE `seckill_goods` (
    `id` INTEGER NOT NULL auto_increment,
    `fk_good_id` INTEGER,
    `amount` INTEGER,
    `start_time` DATETIME,
    `end_time` DATETIME,
    `is_valid` TINYINT ( 1 ),
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;
CREATE TABLE `orders` (
    `id` INTEGER NOT NULL auto_increment,
    `order_no` VARCHAR ( 255 ),
    `good_id` INTEGER,
    `user_id` INTEGER,
    `status` ENUM ( '-1', '0', '1', '2' ),
    `order_type` ENUM ( '1', '2' ),
    `scekill_id` INTEGER,
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;
CREATE TABLE `goods` (
    `id` INTEGER NOT NULL auto_increment,
    `name` VARCHAR ( 255 ),
    `thumbnail` VARCHAR ( 255 ),
    `price` INTEGER,
    `status` TINYINT ( 1 ),
    `stock` INTEGER,
    `stock_left` INTEGER,
    `description` VARCHAR ( 255 ),
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;

Please note that the product table is not the key point in this business. The following logic takes the product with id=1 as an example. Create a record with 200 inventory in the seckill activity table and use it as the seckill test data.

INSERT INTO `redis_app`.`seckill_goods` ( `id`, `fk_good_id`, `amount`, `start_time`, `end_time`, `is_valid`, `comment`, `created_at`, `updated_at` ) VALUES ( 1, 1, 200, '2020-06-20 00:00:00', '2023-06-20 00:00:00', 1, '... ', '2020-06-20 00:00:00', 'the 2021-06-22 10:18:16');

Seckill interface development

First, let’s talk about the specific development environment in Node.js:

  • webFramework usingKoa2
  • mysqlOperation usage based onpromisetheNode.js ORMtoolSequelize
  • redisoperationioredislibrary
  • encapsulationctx.throwExceptionMethod is used to handle errors, encapsulationctx.sendThe method is used to return the correct result. See the complete code at the end of the article for details

Secondly, analyze the logic to be processed by the interface. The steps and order are as follows:

  1. Basic parameter check
  2. Determine if the product is on sale
  3. Determine if the seckill activity is valid
  4. Determined whether seckill activity started and ended
  5. Determine if seckill items are sold out
  6. Gets the login user information
  7. Determine whether the logged-in user has grabbed it
  8. Buckle inventory
  9. Place the order

Finally, according to the analysis, the above steps are preliminarily realized with code, as follows:

Const moment = require('moment'); // Add database model file const SeckillModel = require('.. /.. /dbs/mysql/models/seckill_goods'); const ordersModel = require('.. /.. /dbs/mysql/models/orders'); // Introduce a utility function or utility class const UserModule = require('.. /modules/user'); const { random_String } = require('.. /.. /utils/tools/funcs'); Class Seckill {/** * Seckill interface ** @method post * @param good_id product id * @param accessToken user Token * @param path */ async doSeckill(ctx, next) { const body = ctx.request.body; const accessToken = ctx.query.accessToken; const path = body.path; // Check if (! accessToken || ! Path) {return ctx.ThrowException (20001, 'Parameter error! '); }; Const seckill = await seckillModel.findOne({where: {fk_good_id: ctx.params.good_id,}}); if (! Seckill) {return ctx.throwexception (30002, 'There is no rush on this product! '); }; // If (!! Seckill.is_valid) {return ctx.throwException(30003, 'Activity terminated! '); }; If (moment().isbefore (moment(seckill.start_time)) {return ctx.throwexception (30004, ' '); } if(moment(.isAfter(moment(seckill.end_time)) {return ctx.throwexception (30005, ' '); } if(seckill.amount < 1) {return ctx.throwexception (30006, 'Sold out! '); }; Const UserInfo = await UserModule.getUserInfo(AccesToken); const UserInfo = await UserModule.getUserInfo(AccesToken); if (! UserInfo) {return ctx.throwException(10002, 'User does not exist! '); }; Const OrderInfo = await OrdersModel.finDone ({where: {user_id: {user_id:}) const OrderInfo = await OrdersModel.finDone ({where: {user_id:}); userInfo.id, seckill_id: seckill.id, }, }); If (orderInfo) {return ctx.ThrowException (30007, ' '); }; // Callback const count = await seckill. Decrement ('amount'); If (count.amount <= 0) {return ctx.throwexception (30006, ' '); }; // Order const OrderData = {Order_no: date.now () + Random_String (4); // Order number is the current timestamp plus 4 random numbers. Ctx.params.good_id, user_id: userinfo.id, status: '1', // -1 Canceled, 0 Not paid, 1 Paid, 2 Refunded order_type: ", "// 1", "// 1", "// 1", "// 1", "// 2", "// 2"; const order = ordersModel.create(orderData); if (! Order) {return ctx.throwexception (30008, ' '); }; Ctx.send ({path, data: 'Succeed! '}); } } module.exports = new Seckill();

At this point, seckill interface with the traditional relational database implementation is completed, the code is not complicated, the comments are very detailed, we can understand it without special explanation, then can it work normally, the answer is obviously negative through JMeter simulation of the following test:

  • Simulate 5000 concurrent 2000 users to seckill, will findmysqlQuote ustimeoutWrong, at the same timeseckill_goodstableamountThe field becomes negative,ordersThere are also more than 200 records generated in the table (the exact number varies depending on the environment), which means that it is oversold and does not comply with the seckill rule
  • Simulate 10000 and seckill a single user.ordersIf there is more than one record in the table (the data may vary from environment to environment), this indicates that a user has bought more than one record for the activity, which does not comply with the seckill rule

A look at the code reveals the problem:

  • Step 2, determine whether the product has been snapped up

    Direct query in MySQL, because it is in the seckill scenario, the concurrency will be very high, a large number of requests to the database, obviously MySQL is unable to handle, after all, MySQL can only support thousands of levels of concurrent requests per second

  • Step 7: Determine whether the logged-in user has grabbed it

    With a user under the high concurrency last order has not yet been generated successfully, again to determine whether to still make no, in this case the code doesn’t do any restrictions to deduction and order operation, thus gives rise to a single user to buy more products, with a user according to the requirements of this activity can only buy a discrepancy

  • Step 8: Withhold inventory operation

    Suppose there are 1000 requests at the same time, and the inventory of these 1000 requests is all 200 when they judge whether the product is out of stock in Step 5. Therefore, all these requests will execute Step 8 to deduct the inventory, then the inventory will definitely become negative, that is, the phenomenon of oversold will be generated

The solution

After analysis, three problems need to be solved:

  1. The seckill data needs to support high concurrent access
  2. A user can only purchase once for this activity, that is, the purchase limit problem
  3. Reduction of inventory can not be deducted into a negative number, the order number can not exceed the set inventory number, that is, the problem of oversold

As an in-memory database, Redis can support high concurrency due to its high speed of processing requests. For oversold situations, where the inventory deduction is negative, Redis can provide Lua scripts to guarantee atomicity and distributed locking to solve the problem of inconsistent data under high concurrency. Redis’ distributed lock solves the problem of only one purchase per user. Therefore, you can try to use Redis to solve the above problems.

  • In order to support a large number of highly concurrent inventory check requests, it is necessary to useRedisSave seckill activity data (i.eseckill_goodsTable data) so that requests can be made directly fromRedisIf there is any inventory left after the query is completed, it can be accessed directly fromRedisMedium minus inventory

  • The inventory deduction operation is carried out in Redis, but because Redis deduction operation is divided into two steps: read and write, that is to say, the data must be read for judgment before the subtraction operation is performed, so if the two operations are not well controlled, the data will be corrected and the oversold phenomenon will still appear. To ensure that concurrent access is correct, atomic operations are needed to solve the problem. Redis provides a solution that uses Lua scripts to include multiple operations to achieve atomicity. Here is an explanation of the atomicity in the official Redis documentation

    Atomicity of scripts


    Redis uses the same Lua interpreter to run all the commands. Also Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed. This semantic is similar to the one of MULTI / EXEC. From the point of view of all the other clients the effects of a script are either still not visible or already completed.

  • useRedisImplement distributed locking to lock the inventory and write order operations to ensure that a user can only buy once.

Access to Redis

First, instead of using the SECKILL_GOODS table, the new seckill activity logic changes to insert data into Redis with a hash type and a key of SECKILL_GOOD_ + product ID. Now suppose you add a new record with a key of SECKILL_GOOD_1 and a value of

{ amount: 200, start_time: '2020-06-20 00:00:00', end_time: '2023-06-20 00:00:00', is_valid: 1, comment: '... '}

Second, create a Lua script to ensure the atomicity of the deduction operation. The script reads as follows

if (redis.call('hexists', KEYS[1], KEYS[2]) == 1) then
  local stock = tonumber(redis.call('hget', KEYS[1], KEYS[2]));
  if (stock > 0) then
    redis.call('hincrby',  KEYS[1], KEYS[2], -1);
    return stock
  end;
  return 0
end;

Finally, complete the code. The complete code is as follows:

// const moment = require('moment'); const Op = require('sequelize').Op; const { v4: uuidv4 } = require('uuid'); // Add database model file const SeckillModel = require('.. /.. /dbs/mysql/models/seckill_goods'); const ordersModel = require('.. /.. /dbs/mysql/models/orders'); // const Redis = require('.. /.. /dbs/redis'); // Introduce a utility function or utility class const UserModule = require('.. /modules/user'); const { randomString, checkObjNull } = require('.. /.. /utils/tools/funcs'); Const {SECKILL_GOOD, LOCK_KEY} = require('.. /.. /utils/constants/redis-prefixs'); Lua script const {stock, lock, unlock} = require('.. /.. /utils/scripts'); class Seckill { async doSeckill(ctx, next) { const body = ctx.request.body; const goodId = ctx.params.good_id; const accessToken = ctx.query.accessToken; const path = body.path; // Check if (! accessToken || ! Path) {return ctx.ThrowException (20001, 'Parameter error! '); }; ${SECKILL_GOOD}${goodId} '; const key = '${SECKILL_GOOD}'; const seckill = await redis.hgetall(key); if (! CheckObjNull (seckill)) {return ctx.ThrowException (30002, 'There is no rush on this item! '); }; // If (!! Seckill.is_valid) {return ctx.throwException(30003, 'Activity terminated! '); }; If (moment().isbefore (moment(seckill.start_time)) {return ctx.throwexception (30004, ' '); } if(moment(.isAfter(moment(seckill.end_time)) {return ctx.throwexception (30005, ' '); } if(seckill.amount < 1) {return ctx.throwexception (30006, 'Sold out! '); }; Const UserInfo = await UserModule.getUserInfo(AccesToken); const UserInfo = await UserModule.getUserInfo(AccesToken); if (! UserInfo) {return ctx.throwException(10002, 'User does not exist! '); }; / / whether the logged in user has grabbed the const orderInfo = await ordersModel. FindOne ({where: {user_id: the userInfo. Id, good_id: goodId, status: { [Op.between]: ['0', '1'] }, }, }); If (orderInfo) {return ctx.ThrowException (30007, ' '); }; ${userinfo.id}:${goodId} '; ${userinfo.id}:${goodId} '; // The key of the lock is const uuid = uuidv4(); const expireTime = moment(seckill.end_time).diff(moment(), 'minutes'); Const tryLock = await redis.eval(lock, 2, [lockKey, 'releaseTime', uuid, expireTime]); Try {if (tryLock === 1) {const count = await redis.eval(stock, 2, [key, 'amount', '', ']); If (count <= 0) {return ctx.ThrowException (30006, ' '); }; // Order const OrderData = {Order_no: date.now () + RandomString (4); // Order number is the current time stamp and 4 random numbers are used as order number. GoodId, user_id: userinfo. id, status: '1', // -1 cancelled, 0 not paid, 1 paid, 2 refunded order_type: '2', / / 1 normal order for 2 seconds to kill order / / seckill_id: seckill. Id, / / SEC kill activity id, do not maintain in the redis seconds kill activity id the comment: "', / / note}; const order = ordersModel.create(orderData); if (! Order) {return ctx.throwexception (30008, ' '); }; } } catch (e) { await redis.eval(unlock, 1, [lockKey, uuid]); Return ctx.throwException(30006, 'This item is out of stock! '); } ctx.send({path, data: 'Snap! '}); } } module.exports = new Seckill();

Here are four main changes to the code:

  1. Step 2, determine whether the product is in the buying process and go insteadRedisIn the query
  2. Step 7: Determining whether the logged-in user has grabbed it, because there is no panic buying activity to be maintainedid, so use user insteadid, productidAnd statestatusjudge
  3. Step 8: Remove inventory and use it insteadluaThe script toRedisDeduct the inventory
  4. Locks storage and write to database operations

The operation of the order still takes place in the MySQL database, because most of the requests are intercepted in Step 5, and MySQL is fully capable of handling the rest of the requests.

Through the JMeter test again, it is found that the order table is normal and the deduction of inventory is normal, indicating that the problem of overselling and purchase restriction have been solved.

Other problems

  1. Other techniques used in the seckill scenario are based on Redis’ support for high concurrency, key-value pair databases and support for atomic operations. In this case, Redis is used as a seckill solution. In more complex seckill scenarios, in addition to using Redis, there are several other techniques that can be used if necessary:

    • Current limiting, funnel algorithm, token bucket algorithm, etc
    • Cache, to cache hot data in memory, as far as possible to ease the pressure of database access
    • Peak-clipping, the use of message queue and caching technology to make the instantaneous high traffic into a period of smooth traffic, such as the customer buying success, immediately return the response, and then asynchronously process the following steps through the message queue, SMS, log writing, update the database with low consistency, and so on
    • Asynchronous, suppose that the merchant creates a seckill activity for fans only. If the merchant has fewer fans (suppose less than 1000), the seckill activity will be directly pushed to all the fans. If the user has more followers, the program will immediately push the seckill activity to the top 1000 users, and the rest users will use message queue to delay the push. (The number 1000 depends on the specific situation. For example, if 99% of merchants have less than 2000 followers, and only 1% of users have more than 2000 followers, then this number should be set to 2000.)
    • Shunt, single server can not be on the cluster, through the load balance to handle the request together, disperse the pressure

    The application of these technologies will make the whole seckill system more perfect, but the core technology is still Redis. It can be said that the seckill system implemented by Redis is enough to cope with most scenarios.

  2. The Redis robustness case uses the stand-alone version of Redis, and a single node is rarely used in a production environment because

    • High availability cannot be achieved
    • Even if aAOFLog andRDBSnapshot solution to ensure that data is not lost, but can only be placedmasterOnce the machine fails, the service will not be able to run, and even if the corresponding measures are taken, the data will inevitably be lost.

    Therefore, Redis master-slave mechanism and clustering mechanism are necessary in a production environment.

  3. Redis distributed locking issues

    • Single point distributed lock, the distributed lock mentioned in the case, is actually more accurately called single point distributed lock, for demonstration purposes, but, single point distributed lockRedisDistributed locks should definitely not be used in production environments for reasons similar to point 2
    • Distributed locking, based on a master-slave mechanism (multiple machines), is also insufficient becauseredisThis is done asynchronously when master-slave replication is performed, as inclientAAfter acquiring the lock, the masterredisCopy data fromredisThe process crashed, causing the lock not to be copied to the slaveredis, and then fromredisElect an upgrade to the mainredisAnd create a new LordredisThere is noclientASet the lock at this timeclientBAttempting to acquire the lock, and being able to acquire the lock successfully, invalidates the mutex.

    In response to the above problems, Redis officially designed Redlock, Node.js in the corresponding resource library is Node-Redlock, can be installed with NPM, at least three independent servers or clusters can be used, providing a very high fault tolerance rate, in the production environment should be priority to adopt the deployment of this scheme.

conclusion

The characteristics of seckill scenario can be summarized as instantaneous concurrent access, more reads and less writes, limited time and limited amount. In the development, overselling phenomenon and purchase limit such as scalpers’ competition for tickets should be avoided. Based on the above characteristics and problems, the development principle is as follows: Writing data to memory instead of hard disk, asynchronous processing instead of synchronous processing, atomic execution of inventory holding operations, and locking of single-user purchases all fit the bill, so Redis was chosen to solve the problem.

The seckill scenario is a relatively complex scenario in e-commerce business, and this article only introduces the most core logic. The actual business may be more complex, but it only needs to be expanded and optimized on the basis of this core. The solution of seckill scenario is not only suitable for seckill, similar to grabbing red envelopes, grabbing coupons and grabbing tickets, etc. The idea is the same. The idea of the solution can also be applied to separate purchase restriction, half price of the second piece and inventory control, and many other scenarios, we should be flexible to use.

The project address

https://github.com/threerocks/redis-seckill

The resources

https://time.geekbang.org/column/article/307421

https://redis.io/topics/distlock