Abstract:Millions of people are placing orders at the same time, so why are your chances of getting one? How can e-commerce enterprises master the sales data of hundreds of millions during the promotion period?

This article shares from the Huawei cloud community “618 technology special series (2) millions of people place orders at the same time seckill, why is it more and more easy to grab?” The original author: Torchbearers of Technology.

Add the desired goods to the shopping cart, click the settlement, payment, product page sales +1, inventory -1, if only dozens of people order, the system easily cope with, but when the number of millions, millions, is another situation.

E-commerce enterprises are faced with the pressure of traffic access during the big promotion period: the surge of visitors to the website makes the single server run overloaded, which leads to the website access lag or failure, which seriously affects the user experience. Elastic load balancing service can easily solve this problem.

Elastic Load Balancing: Moving to Where Pressure Is High

Elastic Load Balance (ELB) is a traffic distribution control service that distributes the access traffic to multiple servers at the back end according to the forwarding strategy. ELB of Huawei Cloud can expand the external service capability of the application system through traffic distribution, and improve the availability of the application system by eliminating the single point of failure.

In the face of different e-commerce business needs, ELB can deal with it flexibly. For example, for businesses with large traffic access, the corresponding forwarding rules can be set through ELB to evenly distribute the traffic to multiple back-end cloud servers. For businesses with tidal effect, back-end cloud servers can be added and removed from the ELB at any time, such as adding servers before the first wave of high tide, removing them in the middle, and adding them again at the end.

This can not only ensure the promotion period, the product page will not be blocked because of too many visits, but also to control the cost of spending. But how operators know when to add/remove servers, and when to set forward rules for ELB, requires a basic prejudgment of the overall pace of promotions and their own product data.

Therefore, during the promotion period, in addition to the pressure of access traffic, what makes people more headache is the ability of real-time analysis and management of data.

Data warehouse: there is no goods, sold how many single, system door clear

Generally speaking, e-commerce data is mainly divided into two parts:

  • The first type is transaction-oriented orders, goods, activities, shipments and other data;
  • The second category is the real-time logs of the platform operation and the behavioral data generated by the user’s activities on the platform.

If companies can’t cope with the business demands of massive online transactions and “real-time analytics,” they won’t be able to take the initiative in initiatives like “618.”

For example, the original system of a sports apparel e-commerce enterprise adopts traditional solutions, and the transaction and BI are independent. The trading platform is built by distributed middleware + stand-alone database. Because this plan does not have strong data consistency ability, the data in the system at the same time may be incomplete and inaccurate. For example, 100 orders have been sold one minute ago, but the inventory has not changed, which brings great difficulties for the sales of orders. In order to ensure the final consistency of the data, the data of the trading system needs to be synchronized to the BI system after several hours through the ETL tool, so real-time analysis cannot be done, and the sales and operation supervisors cannot grasp the business situation in real time.

To solve this problem, they later adopted Huawei Cloud Mixed-Load Data Warehouse (DWS). DWS adopts the design concept of “one database and two uses”. A data warehouse cluster can not only support ultra-high concurrency and low delay business transaction requests, but also support complex massive data analysis and BI applications, reducing development, operation and maintenance costs.Compared with the original system, the timeliness of BI system is greatly improved, and the data analysis performance is improved by 3 times.

While achieving real-time data consistency, DWS also ensures the accuracy of individual data and “zero” time delay in operating reports.

DWS can support the writing capacity of ten thousand level TPS on a small scale, and it can reach one million to ten million level TPS after horizontal expansion, supporting the e-commerce enterprise to pass “Double 11” and “12-12” smoothly. Moreover, because it has the distributed transaction ACID characteristic, it has the strong consistency guarantee of data. At any time, it can guarantee the accuracy and integrity of the trading system data and ensure the single data is correct. In addition, the high availability architecture design of DWS provides automatic data backup function, and the reliability reaches 11 nines after the decimal point, so as to guarantee the business data not to be lost.

Data Visualization: It’s up to you to see if sales break records

When we integrate the data of different system modules into the data storehouse for further cleaning, integration and rule processing to realize real-time, complete and accurate data, we also need to push the data visualization platform to the product and operation personnel for visual analysis.

Huawei Mall has previously moved its e-commerce big data applications from the TIDB+Spark cluster to the data warehouse platform based on Huawei Cloud with DWS database as the core.

In terms of real-time data integration, Huawei Cloud provides CS stream computing services based on two stream computing engines, Flink and Spark. As the Flink community grows and FlinkSQL makes it much easier for application developers to implement stream computing in the form of SQL.

Huawei Cloud encapsulates the common message-oriented middleware into the Source operator corresponding to data input. The logical layer is represented by SQL, and various data storage platforms are encapsulated as the sink operator. Developing and publishing a short input/output task using the native Flink API would have taken at least 2 hours to complete, but now the one-stop streaming platform can complete it in 10 minutes. And after checking the grammar, most problems can be avoided in this link. After publishing, you can check the input and output data information through the visual interface of data for checking and monitoring.

The BI data visual module selects Huawei cloud DLV service, which is mainly used in large-screen scenes. In a big promotion activity of Huawei Mall, a command screen was temporarily added to view the data. Two developers worked for two hours overtime, one as the front desk and the other to deal with the data logic. Version 0.1 was soon completed and released, and then the effect was continuously iterated and optimized.

Figure: Kanban board of business data of an enterprise

In addition to the core data warehouse equipment, DayU data development suite platform can greatly improve the release efficiency. DayU takes the data scheduling platform as the core as an extension, and integrates data monitoring, metadata database management and data service release. These services have been gradually used in the big data platform of Huawei Mall.

For a data API release, the developer starts a Job on the DLF and drops in three tasks: CDM (to integrate TIDB tasks into DWS), 2 DWS SQL (one to do DWR layer rule processing, one to do data DWD result rendering), and then API publish the result table data through DayU’s DLG service, which can be called by other domains.

E-commerce enterprises can solve the problems of infrastructure and focus more on the development of business logic without considering service load balancing, disaster recovery and so on. It also reduces the burden of operation and maintenance and the labor cost. For consumers, the shopping experience is better during the big promotion period, and they can even place an order during the peak traffic period to grab the desired goods.

The last

In the battle for consumers, advances in technology are crucial. From the generation of consumer demand to the in-depth purchase behavior, from the supply chain to the analysis and management of platform transaction data, from the low delay of live broadcast to the security of business data, the iteration of cloud computing, AI, big data and other technologies has changed the online shopping model step by step, and also defined a new business model.

In the past, AR/VR technology has provided new possibilities in content presentation and consumer interaction. With the maturity of 5G technology, and after high investment and trial and error in the early stage, VR shopping simulating real scenes may make shopping while staying at home become a normal shopping mode.

On the other end of the consumer side — the intelligent logistics side, IoT, edge computing, machine vision, unmanned driving, these technologies have been changing the traditional logistics warehousing and delivery system, from automated three-dimensional warehousing, automatic transportation, automatic sorting to robot operation, orderly and unified automatic operation to improve the operational efficiency.

618 is a shopping carnival festival for consumers. It is also a big test for e-commerce enterprises, testing their technical strength behind the record performance. From another perspective, 618 not only boosts domestic demand, but also tightens the rope behind technological innovation and industrial upgrading.

Why can’t you stop buying and buying when your budget is three times over unconsciously? If you want to know why our wallets are always empty whenever we have a big promotion? Behind this is the lack of self-control, or the e-commerce platform is too good at reading people’s minds, we might as well from the technical dimension, to find out.

Click on the attention, the first time to understand Huawei cloud fresh technology ~