Welcome to Tencent cloud community, get more Tencent mass technology practice dry goods oh ~

Author: Wang Jingbo

Introduction:

“Live-streaming barrage is one of the core functions of the live-streaming system. How to quickly make a missile barrage system with good scalability? How to deal with rapid business development? I’m sure many engineers/architects have their own ideas.

Live bullet screen refers to the users, gifts, comments, likes and other messages in the live broadcast room, which is an important means of interaction in the live broadcast room. From November 2015 to now, the System has undergone three stages of evolution and can support millions of users online at the same time

This paper explains the balanced evolution process of the live broadcast barrage system according to the development stage of the project. The three stages are fast online, high availability guarantee system construction and long connection evolution.

Rapid on-line

A message model

At the initial stage of design, the core requirement of meipai live live barrage system is to go online quickly and support millions of users online at the same time.

Based on these two points, our strategy is to use HTTP polling in the first and middle phases and replace it with long connections in the second phase.

So while the business team is developing the HTTP solution, the basic r&d team is also working hard to develop the long-link system.

Live broadcast messages, compared to IM scenarios, have the following characteristics:

  • Messages are required to be timely, and outdated messages are not important to users.
  • Loose group chat, users at any time into the group, at any time out of the group.
  • After the user joins the group, the message during the offline period (answering the call) does not need to be resend.

For users, there are three typical operations in the live broadcast room:

  • Enter the live broadcast room and pull up the list of users who are watching the live broadcast.
  • Receive continuous barrage messages from the broadcast room.
  • Text yourself.

We treat gifts, comments, user data as messages.

After consideration, SortedSet of Redis is selected to store messages, and the message model is as follows:

  • The user sends a message through Zadd, where score is the relative time of the message.
  • Receive messages from the live broadcast room through the ZrangeByScore operation, polling every two seconds.
  • Enter the live broadcast room, get the user’s list, through the Zrange operation to complete.

So the general process is:

  • The process of writing a message is: front-end machine -> Kafka -> processor -> Redis.
  • The process for reading messages is: front-end -> Redis.

There is, however, a hidden concurrency problem: users can lose messages.

[image upload failed…(image-fe088C-1510643718048)]

As shown in the figure above, a user starts pulling from comment number 6, and two users are Posting comments at the same time, comment number 10 and 11.

If comment number 11 is written first, the user happens to pull numbers 6,7,8,9,11. The next time the user pulls the message, it starts pulling from number 12. The result: the user doesn’t see message number 10.

To solve this problem, we added two mechanisms:

  • On the front end, the same message type in the same broadcast room is written to the same partition in Kafka.
  • On a processor, the same type of message in the same broadcast room, guaranteed serial writing to Redis by synchronized.

Once the message model and concurrency problems were solved, the development was relatively smooth, and the system was soon online, achieving the preset goals.

After on-line exposure problem is solved

After the launch, with the gradual increase of message volume, the system has exposed three serious problems, we solved one by one.

Problem 1: Messages are written in Redis. If there is a large amount of messages in a broadcast room, the messages will accumulate in Kafka, resulting in a large message delay.

Solutions:

  • Message writing process: front-end machine -> Kafka -> Processor -> Redis.
  • Front-end: if the delay is small, only one Kafka partion is written; If the latency is large, the live message type is written to multiple partions in Kafka.
  • Processor: if the delay is small, lock serial write Redis; If the delay is large, the lock is cancelled. Therefore, there are four combinations, four stalls, respectively: a partion, lock serial write Redis, maximum concurrency: 1. Multiple partitions, lock serial write Redis, maximum concurrency: number of Kafka partion. A partion that writes to Redis in parallel without locking. Maximum concurrency: the number of thread pools on the processor. Multiple partions write to Redis in parallel without locking. Maximum concurrency: number of Kafka partitions Number of processor thread pools.
  • Judgment of delay degree: when the front-end machine writes the message, it prints the unified time stamp of the message. After the processor gets the message, the delay time = current time-timestamp.
  • Slot selection: automatic slot selection, granularity: a certain message type in a broadcast room.

Problem 2: When users poll for the latest news, Redis ZrangByScore is required. Redis Slave has a high performance bottleneck.

Solutions:

  • Local cache, the front end machine every 1 second or so pull a broadcast room message, the user to the front end machine polling data, from the local cache to read data.
  • The number of messages returned is automatically adjusted according to the size of the live broadcast room. Small live broadcast rooms return messages with a larger time span, while large live broadcast rooms have stricter restrictions on the time span and number of messages.

One of the biggest differences between the local cache and the local cache is the cost.

If all the broadcast messages are cached, assuming that there are 1000 broadcast rooms at the same time, each broadcast room has 5 message types, local cache pulls data every 1 second, and 40 front-end machines, then the access QPS to Redis is 1000 * 5 * 40 = 200,000.

The cost is too high, so we only automatically enable local cache in the large studio, not in the small studio.

Question 3: Bullet screen data also supports playback. After the live broadcast, the data is stored in Redis. During playback, the data will compete with the live data for Redis CPU resources.

Solutions:

  • After the live broadcast, data is backed up to MySQL.
  • Added a set of replay Redis.
  • Add a local cache for playback on the front-end server.

** Read data in the following order: local cache -> Redis -> mysql. Both LocalCache and playback Redis can store only part of the data of a certain type of broadcast message, effectively controlling the capacity; Redis uses the sortedSet data structure so that the data structure is consistent throughout the system.

High availability assurance

Deploy two equipment rooms in the same city

The dual machine room is divided into master machine room and slave machine room. The write is in the master machine room, and the read is shared by two slave machine rooms. So as to effectively ensure that single room failure, can be quickly recovered.

Rich means of downgrading

[image upload failed…(image-4fBF87-1510643718048)]

Service monitoring for all links

[image upload failed…(image-8e9bFF-1510643718048)]

After the completion of the construction of high availability guarantee, there were four live broadcasts of TFBOYS in the United States. The peak online audience of these four live broadcasts reached nearly one million, with 28.6 million views, 29.8 million comments and 2.623 billion likes. During the live broadcasts, the system operated steadily and successfully withstood the pressure.

Replace short connection polling scheme with long connection

Long connection overall architecture diagram

[image upload failed…(image-409dFC-1510643718048)]

Details:

  • Before using the long connection, the client invokes the routing service to obtain the IP address of the connection layer. Feature of the routing layer: a. Can be gray scale in percentage; B. You can set the uid, deviceId, and whitelist. Blacklist: Long connections are not allowed. Whitelist: Long connections are allowed even if they are closed or not in the grayscale range. These two features ensure the smooth operation of long and short connection switching.
  • ** Client features: **a. Supports both long and short connections, which can be determined based on the routing service configuration. B. Automatic degradation. If the long connection fails for three consecutive times, the long connection is automatically degraded to the short connection. C. Automatically reports long-link performance data.
  • The connection layer is only responsible for maintaining long connections with clients, without any push business logic. This greatly reduces the number of reboots, thus maintaining a stable user connection.
  • The push layer stores the subscription relationship between users and live broadcast and is responsible for specific push. The whole connection layer and push layer have nothing to do with direct broadcast business, and do not need to be aware of business changes.
  • The long connection service module is used to verify the user entering the live broadcast room.
  • The communication between the servers uses the TARDIS framework developed by the basic r&d team for service invocation, which is based on gRPC and uses ETCD for service discovery.

Long connection message model

We adopted the subscription push model, and the following figure is the basic introduction:

[image upload failed…(image-6d6F47-1510643718048)]

For example, user 1 subscribes to A live broadcast, and A live broadcast has new messages:

  • After the push layer queries the subscription relationship, it knows that user 1 has subscribed to A live broadcast and that user 1 is on the node of connection layer 1, so it will inform the connection layer of new messages.
  • After receiving the notification message, connection layer 1 waits for a short period of time (millisecond level), pulls the message from user 1 again, and pushes the message to user 1.

If it is a large live broadcast room (with many subscribers), the notification/pull model of the push layer and the connection layer will be automatically downgraded to the broadcast model. As shown below:

[image upload failed…(image-d2f9c9-1510643718048)]

We went through three iterations of the client side and realized the replacement of long connection to short connection at both ends (Android and iOS). Because of the support of gray scale and black and white list, the replacement is very smooth and the user is not aware of it.

Summary and Prospect

This paper reviews the development process of the system, achieves the predetermined goal of using polling in the early and middle stages, using long connection in the late stages, and practices the principle of balanced evolution.

From the perspective of development, the future plans to do things include: for the machine room in Beijing, some areas in the south will have a long connection time. How do we bring long connections closer to the user. Further evolution of the message model.

Author: Wang Jingbo, graduated from Xi ‘an Jiaotong University. He once worked for netease and Sina Weibo, where he was responsible for the construction of open platform business and technical system. He joined Meitu in September 2015 and worked in the Architecture Platform Department. Currently, he is responsible for the research and development of some core businesses and infrastructure, including bullet screen service, Feed service, task scheduling and quality control system. More than ten years of back-end research and development experience, has rich back-end research and development experience, for the construction of high availability, high concurrency system has more practical experience. Talk to him at [email protected].

Source: reprinted from the high availability architecture wechat public number, 51CTO technology stack wechat public number

reading

One-stop to meet the e-commerce festival cloud computing needs of the secret information entropy in the game live cover screening application “Tencent cloud game developers technology salon” On November 24 shenzhen station registration open chat game acceleration

Has been authorized by the author tencent cloud community released, reproduced please indicate the article source The original link: cloud.tencent.com/community/a…

Massive technical practical experience, all in Tencent cloud community!