The original link: www.jianshu.com/p/7e97af441…

Due to business requirements, it is necessary to realize real-time access to the updated data function of the server. Based on this demand, research and implementation of the technical solution are carried out, and finally the MQTT +ProtocolBuffer scheme is adopted for real-time data push based on long connection. See this paper for the specific implementation scheme; This paper includes three parts: 1. Technology selection 2. Technical scheme practice 3

Technology selection

In the process of investigation, it was found that the demand function could be realized by push. However, after various considerations, it was found that the self-built long connection for real-time data push was the most suitable solution to meet the demand. In the research process, the scheme of push manufacturer and the scheme of this paper are compared, and the results are as follows:

Push the SDK Integrated APNs Keep long connection
The aurora push is is
A push is is
Baidu to push is no
Tencent messenger pigeons is no
In this paper, solutions no is

From the table, the scheme in this paper is just a simplified version of aurora push and Individual push, which can be realized by aurora push and individual push, or data push by push; But in terms of specific requirements, it can better reflect the feasibility and necessity of this scheme; Specific analysis is as follows:

  1. Push scheme has great uncertainty and high message loss rate
  2. Aurora push and Individual push SDK adopt the long connection mode, but the long connection protocol is unknown, and the data transmission format and link existence state are not controlled
  3. Custom scheme, because of the adoptionMQTT 和 Protocol BufferThe two protocols are compatible with weak network conditions, have relatively low requirements for device performance, and have good data format optimization, and the link status is completely controllable, which can monitor whether the link is available in real time, which is very convenient for analyzing and solving problems.

Technical scheme practice

Solution process

The general process of the whole technical solution is shown as follows:

In the whole process, the real-time transmission of messages is realized by pushing background and long connection SDK. The SDK based on this scheme is responsible for the validity and optimization function of maintaining the long connection in the background.

Refer to the following figure for the interaction parties and interaction processes involved in the detailed process:

Solution architecture

In the whole real-time data transmission scheme, it involves the establishment and maintenance of long connection, encapsulation of communication data according to protocol, encapsulation and optimization of service data and other functions. The whole scheme is sorted out here, and its structure is as follows:

In the overall structure, it is divided into business layer, application layer, transmission layer and network layer. The business layer, application layer and transmission layer constitute the SDK of real-time data push service. 1. Business layer: mainly deals with business data, message storage, message loss, repeated processing, and data format transformation, etc.; 2. Application layer: this layer is the core layer of the entire SDK, which encapsulates MQTT protocol, reconnection of disconnection and secure connection, etc. 3. Transmission layer: the main core of this layer is to call system API to realize long connection, data packet, timeout retransmission and other processing;

The following is a detailed arrangement and description of the whole SDK architecture level and the modules contained in each level. The diagram below:

Business layer This layer mainly solves data format transformation, message loss and duplication, message storage, data statistics and other functions. Here are some of these features:

  1. Data format conversion:This module is responsible for theProtocolBufferType of data converted toJSONdata
  2. Message repetition and loss processing: This module will judge whether the received message is repeated according to the rules, or whether the message has been lost before the message is received. After processing, if the message is repeated, it will not be transmitted to the business side. If there is a message loss, it will tell the business side so that the business side can deal with the relevant situation.
  3. Message storage: According to a certain cache policy, the message is stored for a short time in order to solve the problem of message weight judgment and processing loss.
  4. Data statistics: this module will make statistics on the message, distinguish the repetition, loss and other cases, and report the data to the server, convenient to verify the feasibility and problems of the whole program;

Application layer The name of this layer mainly comes from MQTT, because MQTT belongs to the protocol of the business layer, and the operations of this layer are mostly the implementation and optimization of the protocol. It mainly includes the realization of MQTT, heartbeat mechanism, disconnection reconnection mechanism, link state management, etc.

  1. MQTTThe implementation of the: This module mainly uses tripartite libraryMQTTClient, the tripartite library pairMQTTThe protocol is fully encapsulated and the current phase of the program is onMQTTThe implementation of the protocol temporarily relies on the library;
  2. Heartbeat mechanism: Due to the early stage of the project and the scheme to be verified, this module temporarily uses a relatively simple fixed heartbeat mechanism and sets the heartbeat interval to 60S. As for the heartbeat interval, the NAT timeout time of each carrier is uncertain, so the heartbeat interval is set to be small. This interval may cause an increase in mobile phone traffic and power consumption. This module is a key optimization module for the follow-up.
  3. Disconnection reconnection mechanism: Due to network switchover, mobile network base station change, and network disconnection and other problems, the disconnection reconnection mechanism is specially provided. The current scheme of this mechanism is as follows: Reconnection is carried out at the interval of 2 to the power of n, and n is the number of reconnection times. When n is equal to 6, no retry is given.
  4. Link state management: this module is responsible for the application of the foreground, background in different states, link processing management;

Transmission layer This layer is mainly for calling system API to splice and assemble Socket data, initiate connection, close connection and other operations;

The specific modules of the scheme have been introduced above. The general process of the scheme is summarized as follows:

This process shows the general process of SDK from establishing links, receiving data, processing TCP packets, analyzing MQTT communication data, and processing ProtocolBuffer data format. At this point, the entire existing programme ends;

Future optimization direction

Considering the unstable situation of mobile phone power, flow and network connection, the above scheme is obviously not perfect. For the above scheme, the following plans are made:

Intelligent Heartbeat Strategy

Because heartbeat is a necessary means to ensure the survival of the connection, the existence of heartbeat is beyond doubt, but the existence of heartbeat will also increase the consumption of electricity and flow. In this paper, the author puts forward the following targeted optimization ideas for the heartbeat strategy of the scheme, especially puts forward the following scheme — intelligent heartbeat strategy; The general process is as follows:

Optimization of disconnection reconnection mechanism

Caused by abnormal possible server, client group dropped, the client will immediately initiate reconnection, under the condition of the users is small, the client collective reconnection to the server can withstand, but when a user level reaches a certain amount, a large number of client reconnection, are likely to produce heavy storm, the abnormal, led directly to the server again Therefore, a more appropriate strategy should be considered for reconnection mechanism to avoid reconnection storm.

Network Connection optimization

Direct IP connection, link multiplexing, packet size control, etc.