The background,

Meituan is the world’s largest Internet + lifestyle service platform, providing online and offline e-commerce services to 320 million active users and more than 5 million high-quality merchants. Adhering to the mission of “help everyone eat better, live better”, our business covers more than 200 categories and 2,800 city and county network, in catering, takeout, hotel travel, beauty, family, leisure and entertainment and other fields with a leading market position.

With the rapid development of various businesses, the mobile R&D team of Dianping has grown from a “small workshop” to a “regular army” of 1,000 people who can fight together. Our mobile project architecture has also changed dramatically to adapt to business development, which puts forward higher requirements for mobile continuous integration, and the whole mobile R&D team has also ushered in new opportunities and challenges.

Ii. Problems and challenges

Currently, there are more than 600 mobile client component libraries, and several mobile projects have code volumes of millions of lines, with hundreds of release integration requirements every day. It is an important mission of our department to ensure the smooth development and integration of nearly 1,000 mobile developers. However, the road ahead is never smooth. On the road to our goal, we still face many problems and challenges, mainly including the following aspects:

Project dependency complexity

The figure above shows only a small part of the dependencies between the components of our mobile project. Imagine that the dependencies between the more than 600 components are as dazzling as the road traffic network of a city. This complex dependency between components will inevitably lead to two serious problems. First, if a business needs to modify the code, it is very likely to affect other businesses, which will make many r & D students fearful at work and more timid in doing projects. Second, it is also a headache to manage the tedious dependencies between these components. Now the average number of dependencies of each component is more than 70, and the largest number even reaches more than 270. It is extremely difficult to maintain these dependencies manually.

Trivial RESEARCH and development process

To fulfill a complete functional requirement, mobile r&d needs to go through component release, component integration, packaging and testing in addition to code development. If the tests find bugs that need to be fixed, then go through component release, component integration, packaging, testing again until the tests pass the delivery. The r&d students need to manually submit MR, manually upgrade components, manually trigger packaging and manually monitor the status of the process in real time during the whole process, so the R&D will be frequently interrupted to track the connection of the processing process, which is bound to seriously affect the development concentration and reduce the r&d productivity.

Slow build

At present, the construction time of Dianping’s iOS project has increased from 20 minutes two years ago to more than 60 minutes now, and the Android project has also increased from 5 minutes to 11 minutes. The increasing construction time of mobile project has seriously affected the efficiency of mobile terminal development integration. And with the rapid expansion of the business, the project code continues to grow. In order to adapt to the rapid development of business, it has become urgent to find effective ways to speed up the construction of mobile projects.

App Quality Assurance

There are many performance quality indicators to evaluate App, such as CPU usage, memory usage, traffic consumption, response time, online Crash rate, package body and so on. Online Crash has a direct impact on user experience. If users flash back when using the App, they are likely to give a “one-star” bad rating. Package size is an important factor affecting new users to download your App. If the package size is too large, users may lose interest in your App. Therefore, reducing the online Crash rate of App and controlling the size of App package are important goals that every mobile r&d should pursue.

Complex project dependency, trivial RESEARCH and development process, slow construction speed, and App quality assurance are all problems that every mobile project will encounter in the process of team and business development. This paper will explain step by step how we solve these problems in actual practice according to the practical experience accumulated in dianping mobile App over the years.

MCI architecture

MCI (Mobile Continuous Integration) is a set of effective architecture system concluded by dianping Mobile terminal team over the years. It can actually solve the problems of complex dependence, trivial RESEARCH and development process and slow construction speed in mobile projects. Meanwhile, mobile projects connected with MCI architecture system can truly and effectively improve the quality of App.

The complete MCI architecture is shown in the figure below:

MCI architecture system includes mobile CI platform, process automation construction, static inspection system, log monitoring & analysis, information management configuration. In addition, MCI also adopts binary integration measures to improve the construction speed of MCI.

Build a mobile CI platform

We build mobile CI platform to ensure that mobile R&D can complete business R&D integration without affecting the extremely complex project dependence; Secondly, we designed reasonable CI strategies to help mobile developers get out of the daunting “quagmire” of dependency management.

Process automation construction

On the basis of building mobile CI platform, we automate MCI process to solve trivial problems in r&d process, so as to liberate mobile R&D productivity.

Increase build speed

In the case that CI platform ensures the correctness of integration, we rely on flattening and optimization of integration methods to improve the construction speed of MCI and further improve the efficiency of research and development.

Static inspection system

We have established a complete set of self-developed static inspection system. According to the characteristics of mobile projects, MCI online carries out all-round static inspection to promote the quality of App.

Log monitoring & Analysis

We log the complete process of MCI system to facilitate the traceability and troubleshooting of problems, and further optimize the PROCESS of MCI and monitor the health status of mobile App projects through data analysis.

Information Management Configuration

Finally, in order to facilitate the management of mobile projects connected to MCI, we built a unified project information management and configuration platform.

Next, we will take a detailed look at how the MCI architecture is built step by step to solve the various problems we face.

4. Build a mobile CI platform

4.1 Build a mobile CI platform

We have studied the current popular CI system in the industry, such as: Travis CI, CircleCI, Jenkins and Gitlab CI conducted a survey and finally chose to build a mobile continuous integration platform based on Gitlab CI, considering code security, scalability and page operability according to the characteristics of mobile projects. And of course we use Jenkins to do some supporting work. The CI core architecture of MCI system is shown in the figure below:

Noun explanation:

  • Gitlab CI: Gitlab CI is short for Gitlab Continuous Integration.
  • Runner: Runner is the interface that Gitlab CI provides to register CI servers.
  • Pipeline: Can be understood as an assembly line, containing different tasks at different stages of CI.
  • Trigger: Triggers, such as Push code or the submission of Merge Request, Trigger the corresponding Trigger to proceed to the next process.

The advantages of this architecture are extensibility, customization, and concurrency. First of all, CI server can be expanded arbitrarily. In addition to dedicated server, ordinary personal PC can also be used as CI server (the disadvantage is that the performance is worse than the server, and the task execution time is longer). Secondly, the Pipeline of each integration task is customizable. Integration projects hosted in MCI can customize the Pipeline to match their own needs. Finally, the task execution of each integration project is concurrent, so component code integration can occur without interference between lines of business.

4.2 CI process design

A complete component integration process includes two stages: component library release and integration to target App project. As shown below:

In the first stage, after the daily function development is completed, the R & D will send PR to the designated branch, and after the code Review, component library compilation and static check is correct, the version will be automatically released into the component pool. All components that go into the component pool can be reused across different App projects.

In the second stage, the r&d will integrate the components into the designated App project as required. The correctness of component A itself has been verified in the component library release of the first stage, and the second stage is to check whether the change of component A has any impact on other components that originally depend on it in the target App. Therefore, the first step is to analyze which components are relied on by component A in the target App. The target App project compiles and conducts static analysis on the integrated component library according to their respective access standards. After checking without error, the integrated component library is finally merged into the release branch.

Components are properly integrated into the target project through a two-stage CI process of component release and integration. The components with problems will be blocked out of the project, so it will not affect the normal development and release integration of other businesses, and the r&d process of each business is independent and controllable.

4.3 Design a reasonable CI strategy

Whether a component’s release and integration passes CI checks depends on the component’s current dependencies and whether the component itself is compatible with the target project. In order to reduce the complexity of component dependency management, we design a reasonable release integration strategy to help mobile r&d get out of the complicated version dependency management dilemma.

Component Integration Strategy

Each component has its own dependencies. Different components may depend on the same component. During the integration of components into the target project, the following problems may occur:

  • Version integration conflict: a component that has a conflict with the version number of an existing dependency in the target project during integration.
  • App test package instability: The version of a component dependency changes causing App test packages with different dependencies to be printed at different times.

Frequent version integration conflicts will lead to low integration efficiency of business collaborative development, and the instability of App test package will bring great trouble to r&d tracking. The root of the problem is that the target project uses the dependencies of each component for integration. Therefore, by displaying the specified component version number in the integration project and forbidding dynamic dependence, the stability and reliability of App test package are ensured, and the integration conflict of component versions is also solved.

Component release policy

Component publishing to a component pool also involves managing dependencies. The simplest way is to specify the version numbers of all dependencies. This has the advantage of being straightforward, but the developer needs to have a good understanding of the functionality of the dependencies in different versions. As stated in the component integration policy, the version of each component in an integration project is display-specified and uniquely determined, and the version number of a specified dependency in a component does not play a role in an integration project. So we automatically rely on the latest version in the component pool for component release. The benefits of this design are:

  • Avoid version dependencies in mobile development.
  • Provides a powerful driving mechanism for iteration of changes to the underlying components.

When the interface and design of the base component library change greatly, it can strongly promote the business layer components to adapt accordingly, ensuring the high agility in the highly decoupled project architecture. However, this ability should not be abused, and should be properly arranged according to the business iteration cycle, and advance notice mobilization work.

Five, process automation construction

The main reason for the trivia of the R&D process is that r&d requires human participation in every step of the CONTINUOUS integration process. Once we take mobile r&d out of the continuous integration process, it will naturally increase r&d productivity. We optimized the MCI process through project integration release process automation and optimized test package distribution.

Project integration process hosting

Component release, component integration and App packaging in the R&D process are all standardized processes in continuous integration. We use process management tools to automatically connect these steps. The r&d students only need to pay attention to code development and Bug fixing.

The implementation scheme of the process hosting tool is as follows:

  • Automatic process execution: Tasks are automatically executed sequentially through managed queues, and Webhook is used to monitor process status.
  • Critical node notification: after the critical node process is successfully executed, the notification will be sent so that the R&D can know the process status clearly.
  • Process exception notification: Once the continuous integration process is abnormal, such as project compilation failure, static check failure, etc., the r&d team will be notified immediately to deal with it in time.

Packaged release process hosting

Both iOS and Android need to do a series of procedures before releasing App packages to the market, for example, iOS needs to export IPA packages for backup, save symbol tables to resolve online Crash, and upload IPA packages to iTC (iTunes Connect). In addition to package backup and saving Mapping files to parse online Crash, Android also needs to release App packages to different channels, making the whole package release process more complicated and tedious.

Before there was no MCI process hosting, every App release day, the r and D students would guard in front of the packaging machine as if facing a great enemy. They would fight their way through many difficulties until all App packages were “shipped” to the designated place, and they were very tired. Just like the project integration process hosting, we made the whole packaging and release process into the whole process hosting, the unattended automatic packaging and release way liberated the r & D students, the r & D students no longer have to spend every day, go out early and come back late, kneel on the keyboard (covering their face).

Package distribution process construction

Does this scenario sound familiar to QA and R&D? Bugs are a bridge between QA and development, but the lack of unified package management and distribution makes it difficult to quickly locate and trace faulty packages. In order to reduce ineffective communication between QA and R&D and optimize the package distribution process, we urgently need a platform for unified management and distribution of App packages within the company, so MCI App came into being.

MCI App provides the following functions:

  • Download and install different types and versions of apps.
  • View basic App package information (packager, packaging time, package version, code commit point, etc.).
  • View information about all component libraries integrated with the current version of the App package.
  • Check App package usage.
  • Query App release schedule.
  • Share the link to download the App package.

In the future, MCI App will also support the query of project integration status, App release reminders and problem feedback, and integrate the whole process of mobile research and development.

6. Speed up construction

The two most time-consuming steps in mobile project construction are component dependency calculation and project compilation.

** Component dependency calculation **

Component dependency calculation is to calculate all related dependencies and their versions according to the integration components specified in the project. When there are many integration components in the project, recursive calculation of dependencies and their versions is a very time-consuming operation, especially to deal with related dependency conflicts.

** Project compilation **

Project compilation time is proportional to the amount of code of project engineering. The group’s business is developing rapidly, and the amount of code is also expanding rapidly.

In order to improve the construction speed of the project, we completely eliminate the component dependence calculation time by relying on the flattening method, and reduce the project compilation time by optimizing the project integration method.

It depends on flattening

The core idea of dependency flattening is to display and specify the dependency and dependency version number in advance, so that the time-consuming of component dependency calculation can be completely eliminated by fixing the dependency and dependency version, which greatly improves the speed of project construction. At the same time, reliance on flattening has the following additional benefits:

  • Reduce the burden of r&d dependency maintenance.
  • App projects are more stable and don’t have problems with automatic upgrades of dependencies.

Optimized integration mode

Generally, component code is integrated into the target project by source code. The biggest disadvantage of this integration method is the slow compilation speed. For an App with millions of lines of code, the project compilation time will be more than 40 minutes or even longer if the source code integration method is adopted.

Using source code integration

Using binary integration

In fact, component code can also be integrated into the target project in binary mode:

Compared with source integration, binary component packages are pre-compiled and only need to be linked in the integration process without compilation. Therefore, binary integration can greatly improve the speed of project compilation.

Binary integration optimization

To further improve the efficiency of binary integration, we also do a few small things:

** (1) Multithreaded download **

Although binary integration can reduce project compilation time, binary packages must be downloaded remotely to the CI server. We have changed the default single-thread download policy to use multiple threads to download binary packages to improve download efficiency.

** (2) Binary packet cache **

R&d triggers different integration tasks on MCI, and the binary packages of components used by these integration tasks are mostly the same except for the upgraded components. Therefore, we cache the binary packages of components on CI server for sharing among different tasks to further improve the construction speed of the project.

Binary integration results

We adopted binary integration in MCI and after a series of optimizations, the compilation time of iOS projects was reduced by 60% and that of Android projects was reduced by nearly 50%, greatly improving the efficiency of project construction.

Seven, static inspection system

In addition to completing daily requirements development, improving code quality is a required part of every r&d process. Many online problems could have been avoided if every mobile developer had strictly followed mobile programming specifications and best practices in their daily development. In fact, it is difficult to implement these mobile programming specifications and best practices effectively in the long run, so we need to implement these mobile programming specifications and best practices into static inspection enforcement to effectively nip the problem in the cradle.

Statically check infrastructure

The simplest method of static checking is text matching, which is logically simple but has limitations. For example, the static checking code written is difficult to maintain, and the text matching ability is limited to some complex logic processing. There are many static analysis tools for Objective-C and Java, such as OCLint, FindBugs, CheckStyle, etc., but these tools have a high threshold of customization. In order to reduce the cost of static inspection access, we independently developed a static analysis framework –Hades that meets the needs of MCI.

Hades features:

  • Complete code semantic understanding
  • Ability of global analysis
  • Support incremental analysis
  • Low access cost

The core idea of Hades is the semantic expression of structured data in AST (Abstract Syntax Tree) generated by source code, on which we can build a series of static analysis tools and services. As a static analysis framework, Hades is not limited to the production of Lint tools, but we also want to use this structured semantic representation to gain a deeper understanding of code. Therefore, we can build semantic model database of project code with the help of document database (e.g. CouchDB, MongoDB, etc.), so that we can build view through JS map-reduce to quickly retrieve what we need to find. The technical implementation of Hades will be explained in detail in a subsequent technology Blog, so stay tuned.

MCI static check status

At present, MCI has launched more than 20 static checks covering basic code specifications, non-empty features, multithreading best practices, resource legitimacy, startup process control, dynamic behavior control and other static checks, which effectively promote the improvement of App code quality.

Log monitoring & Analysis

MCI is an important platform for continuous integration of Dianping mobile terminal. Stability and efficiency is the first goal to be achieved. Log monitoring is an important means to promote MCI towards stability and efficiency. We have logged the whole MCI process to facilitate problem tracing and troubleshooting. The following are some online monitoring items.

Process time monitoring analysis

By monitoring and analyzing the execution time of each step in the MCI process, we can make targeted optimizations to improve integration speed.

Abnormal process monitoring analysis

We monitor abnormal flows and notify the flow initiator. In addition, we analyze the causes of jobs that fail frequently. Some CI environment or network problems can be solved automatically by MCI, while other exceptions caused by code errors will guide the mobile r&d to troubleshoot and resolve the problems.

Inclusion monitoring analysis

We monitor the total size, executable files and pictures of the package in all aspects. The change trend of the package is clear at a glance, and we can feel the abnormal change of the package in the first time.

In addition, we also monitor the success rate of MCI integration, binary coverage and other aspects, so as to know the whole PROCESS of MCI clearly and make MCI run stably and efficiently.

9. Information management configuration

At present, THE MCI platform has been connected to several mobile projects of the company. In order to access MCI projects for unified and convenient information management, we built the MCI information management platform — Mocha. The Mocha platform includes project information management, configuration of static check items, and integrated query of component release.

Project information Management

Mocha platform is responsible for registering and accessing the basic information of MCI projects, including project address and project leader, and managing the rights of members of each project.

Configure static check items

MCI supports different static check items for different projects. On the Mocha platform, you can customize the static check items required by the project and check the configuration of the static check whitelist.

Component release integration query

Mocha platform supports component history integration record query, easy to troubleshoot and trace problems.

As a visual configuration system for mobile integration projects, the Mocha platform is an important complement to MCI. It makes it easy and fast for mobile projects to access MCI, and more configuration items will be added to the Mocha platform in the future.

X. Summary and outlook

Starting from the business complexity of Dianping mobile project, this paper introduces in detail the idea and best practice scheme of building a stable and efficient mobile continuous integration system, solves the problems caused by complex project dependency, and improves the construction speed through dependency flattening and binary integration. On this basis, the self-developed static check infrastructure Hades reduces the threshold of static check access to help improve the quality of App; Finally, the full process hosting capability provided by MCI can significantly increase mobile r&d productivity.

At present MCI for iOS, Android native code project integration has provided quite complete support. In addition, MCI also supports the continuous integration of Picasso Project, a high performance cross-platform dynamic framework developed by The Platform, which focuses on dynamic UI construction across iOS, Android, Web and mini program. Of course, there are similarities and differences between continuous integration for mobile native projects and continuous integration for dynamic projects. In the future, MCI will further explore in the field of mobile engineering and escort the booming development of mobile terminal business.

Author’s brief introduction

Zhicong, aN iOS technical expert of Dianping, focuses on the development of mobile toolchain, and has a deep understanding and rich practical experience in mobile continuous integration and static analysis platform construction.

Xing Yi, Dianping Android technology expert, focuses on mobile continuous integration, static analysis, static App infrastructure construction.

Team to introduce

Dianping Mobile R&D Center, Base Shanghai, provides Meituan with basic mobile terminal infrastructure services, including network communication, mobile monitoring, push touch, dynamic engine, mobile R&D tools, etc. At the same time, the team also carries out business research and development work such as traffic distribution, UGC, content ecology, personal center, etc., waiting for heroes focusing on mobile terminal research and development for years. Welcome to send resume: dawei.xing#dianping.com.