This article is based on a presentation by Meituan senior technical expert Bin Song at the ArchSummit Architect Summit.

background

Meituan Takeout has been developing for five years, and instant logistics exploration has also experienced more than three years. The business has been incubated from zero to take shape. During the whole process, we have accumulated some experience in the construction of distributed and high-concurrency systems. The main gains include two things:

  1. Real-time logistics services have very low tolerance for failures and high latency. As the service complexity increases, the system is required to be distributed, scalable, and disaster tolerant. The real-time logistics system gradually implements the architecture upgrade of the distributed system, and finally solves the risk of system downtime.

  2. On cost, efficiency and experience of the three elements of the core, real-time logistics system of AI technology, from ETA, scheduling, capacity planning, pricing, capacity intervention, subsidies, accounting, business operations, voice interaction, LBS mining, indicators monitoring, etc., the business break through combining structure upgrade, to promote the scale, experience, the effect of cost reduction.

This paper mainly introduces the technical obstacles and challenges encountered in the layer by layer evolution of Meituan real-time logistics distributed system architecture:

  • Large scale calculation of orders, riders and supply and demand matching process.
  • Meet holiday or bad weather, order gathering effect, flow peak is more than ten times the usual.
  • Logistics contract fulfillment is the key link between online and offline, with very low fault tolerance, no downtime, no order loss, and high availability requirements.
  • Data real-time, accuracy requirements are high, very sensitive to delay, anomaly.

Meituan real-time logistics architecture

Meituan instant logistics platform mainly focuses on three things: first, it provides SLA for users to fulfill the contract, including calculating ETA of delivery time and pricing of delivery fee; The second is to match the most suitable rider in the context of multi-objective (cost, efficiency, experience) optimization; The third is to provide auxiliary decision-making in the complete performance process of the rider, including intelligent voice, route recommendation, store reminder, etc.

Behind a series of services, meituan’s powerful technical system supports the distribution business architecture system, which is based on the platform, algorithm, system and service. The huge logistics system can not be separated from the support of distributed system architecture, and this architecture should ensure high availability and high concurrency.

Distributed architecture is a kind of architecture system as opposed to centralized architecture. Distributed architecture applies CAP theory (Consistency, Availability, Partition Tolerance). In a distributed architecture, a service is deployed among multiple peer nodes, which communicate with each other through the network. These nodes form a service cluster to provide highly available and consistent services.

In the early days, Meituan was divided into several vertical service architectures based on business areas. With the development of business, the hierarchical service architecture is made from the perspective of availability. Later, the business development became more and more complex, and gradually evolved into a microservice architecture after considering multiple aspects such as operation and maintenance and quality. There are two main principles to follow: don’t get into the design of microservices architecture too early, and good architectures are evolved, not designed in advance.

Distributed System practice

The figure above shows a typical distributed system structure under The Meituan technology system: It relies on the common components and services of Meituan to complete partition expansion, disaster recovery, and monitoring capabilities. Front-end traffic is distributed and load balanced through HLB. Within the partition, services communicate with services through OCTO, providing services registration, automatic discovery, load balancing, fault tolerance, gray publishing, and so on. You can also communicate via message queues, such as Kafka and RabbitMQ. Use Zebra at the storage tier to access distributed databases for read and write operations. Use CAT (Meituan open source distributed monitoring system) to collect, report and monitor distributed services and system logs. Distributed caching uses a combination of Squirrel+Cellar. Distributed task scheduling is done by Crane.

In practice, there are several problems to be solved. The typical one is the scalability of the cluster. The scalability of the stateful cluster is relatively poor. At the same time, there will be problems of node hot spots, including uneven resources, uneven CPU usage, and so on.

First of all, the technical team of the distribution background transforms the stateful nodes into stateless ones by upgrading the architecture, and enables small business nodes to share the computing pressure through the parallel computing capability, so as to achieve rapid expansion.

The second problem is to solve the consistency problem. For scenarios where both DB and cache are written, the business write cache cannot guarantee data consistency. Meituan mainly uses Databus to solve the problem. Upstream from Databus, business Binlog changes can be monitored and piped to ES and other DBS, or other KV systems, taking advantage of the high availability features of Databus to ensure that data is eventually synchronized to other systems.

The third is that we have been spending energy to solve the problem, is to ensure the cluster high availability, mainly from three aspects to start, prior to more is to do the full link pressure assessment, estimate the peak capacity; Periodic cluster health check Random failure drills (service, machine, component). Do abnormal alarm in the event (performance, business indicators, availability); Fast fault location (single machine failure, cluster failure, IDC failure, component exception, and service exception); Collect system changes before and after faults. Focus on system rollback after the event; Capacity expansion, current limiting, fusing, downgrading; Nuclear weapons are the last straw.

Rapid deployment & Dr Of a single IDC

After a single IDC failure, the entry service can recognize the failure and automatically switch traffic. Rapid expansion of a single IDC, data synchronization in advance, service deployment in advance, open inbound traffic after Ready; All data synchronization and traffic distribution services are required to have automatic fault detection and fault service removal. Capacity expansion and scaling by IDC.

Multi-center attempt

Meituan IDC uses zones as its unit, but the zones cannot be expanded because resources are full. The solution of Meituan is to form a virtual center with multiple IDCs as the partition unit. Services are deployed indiscriminately in the center; If the center capacity is insufficient, add a new IDC to expand the capacity.

An attempt at unitization

Compared with multicenter, unitary is a better solution for partition Dr And capacity expansion. Meituan routes traffic by region or city based on service characteristics. In data synchronization, there will be delays in different places. When a local or remote SET fails, you can quickly switch the SET to another SET to take over the traffic.

Intelligent logistics core technical ability and platform precipitation

Machine learning platform is a one-stop offline to online model training and algorithm application platform. The purpose of building this platform is to solve the contradictory problems of multiple application scenarios of the algorithm, repeated wheel building, and inconsistent online and offline data quality. If the process is not clear and coherent, there will be low efficiency of iteration, data quality and other obstacles in the online deployment of features and models.

JARVIS is an intelligent business operation and maintenance AIOps platform for stability assurance. It is mainly used to deal with many alarm sources when the system fails, there will be a large number of repeated alarms, effective information is easy to be submerged and other problems. In addition, the operation and maintenance faults of small-scale distributed clusters in the past mainly rely on people and experience to analyze and locate, resulting in low efficiency and slow processing speed. The expected results of each fault processing are unstable, and the effectiveness and timeliness cannot be guaranteed. The AIOps platform is needed to solve these problems.

Challenges ahead

After Review and Review, we found that the future challenge is very big. The micro service is no longer “micro”, and the service will expand when the business complexity increases. Secondly, any slight delay in the network structure of service cluster may lead to network amplification effect. In addition, AIOps needs to focus on how to quickly locate and handle faults in complex service topologies. Finally, after the unitization, meituan’s business deployment capability is challenged greatly from the operation and maintenance of cluster to the operation and maintenance of unit.

Author’s brief introduction

Bin Song, senior technical expert of Meituan, has been involved in distributed system architecture and high concurrency system stability guarantee for a long time. At present, I am the backstage technical leader of instant logistics team. Joined Meituan in 2013, and participated in the c-terminal of Meituan takeout and the establishment of instant logistics system from scratch. Now I lead the team to be responsible for the r&d, operation and maintenance of business systems such as scheduling, clearing and settlement, LBS, pricing, algorithm data platform, stability guarantee platform and other technical platforms. Recently, I focused on THE direction of AIOps to explore how to better guarantee system stability under the architecture of high concurrency and distributed system.

Recruitment information

Meituan distribution technology team sincerely invites senior technical experts and architects in LBS, scheduling and performance platform, settlement platform, AIOps, machine learning platform and algorithm engineering. To build the largest single instant distribution network and platform in the industry, jointly face the challenges of complex business and high concurrent traffic, and meet the era of comprehensive intelligence of distribution business. Interested students are welcome to submit their resumes to [email protected], [email protected].