01 background

Due to the high performance requirements of map rendering, navigation and other core functions in navigation applications, a large number of functions in amap client are implemented by C++. With the rapid development of business, the map engine library alone has more than 40 modules, and the engineering configuration is extremely complex. The original construction and continuous integration technology can no longer meet the increasing demand changes.

In addition to the complexity of millions of lines of code, the construction and continuous integration of the C++ engine library project (the engine library for short) in the amap client also face the following challenges:

Support multi-team collaboration: multi-team means multiple operating systems and multiple ides. Reducing the difficulty of engineering configuration under different operating systems and different ides is one of the key problems to be solved. Support multi-service line customization: The engine library provides support for mobile phone, vehicle, open platform and other business lines, but each business line has different demands, so it needs to be able to build according to functions; Support vehicle environment: Among many business lines, AMAP Has a very special business line, namely AMAP AUTO. The car machine directly faces the major car factories and many equipment vendors, the environment is customized, and the construction of a variety of tool chains. If a set of build profiles are customized for each VEHICLE environment, the maintenance cost will be very high. Therefore, how to meet the diverse build requirements of vehicles with a set of build profiles becomes an urgent problem to be solved. In addition, due to historical reasons, the source code and dependency libraries in the engine repository are mixed and stored in a Git repository, which causes two problems:

With the increase of build times, Git repositories are getting bigger and bigger, and the check out of code and dependent libraries is getting slower and slower, which greatly affects local development and packaging efficiency. Lack of unified management, confusion of dependencies, often build failure due to dependency problems, or although the build is successful, but errors occur at runtime; These challenges and historical legacy issues have seriously hindered the improvement of r&d efficiency. To this end, we conducted in-depth research and analysis on the existing build and continuous integration tools, combined with its own business characteristics, and finally developed Amap C++ native build tool Abtor and continuous integration tool Amap CI.

02 Local Build

Analysis of existing tools

C++ is a language near the bottom. Different hardware, operating systems, compilers, and cross-compilation make C++ very difficult to build. In response to these problems, the C++ community has produced a number of excellent build tools, such as Make and CMake.

Make, or GNU Make, was released in 1988 as a tool for executing makefiles. The basic syntax of makefiles includes targets, dependencies, commands, and so on. When certain files change, only objects that directly or indirectly depend on those files need to be rebuilt, which greatly improves compilation speed.

The combination of Make and Makefile can be regarded as project management tools, but they are too basic and have high barriers and restrictions in terms of cross-platform use. In addition, large project construction can encounter the problem of Makefile inflation.

CMake was created in 2000 as a cross-platform compilation, testing, and packaging tool. It converts the configuration file into a Makefile and runs the Make command to compile the source code into an executable program or library. CMake belongs to the Make family. Configuration files are more readable than makefiles and support cross-platform construction with high build performance.

However, CMake also has two obvious shortcomings. First, the complexity of configuration files is much higher than that of other modern languages, and there is a certain learning cost for beginners of CMake syntax. Second, it is not friendly to use with different ides.

As you can see, Make and CMake still have a low level of abstraction, which puts too much pressure on the builder. In order to reduce the cost of building, a number of new C++ building tools have emerged in the C++ community, including Google’s Bazel and Ninja, as well as SCons. The features and disadvantages of these tools are as follows:

Through the above research and analysis of existing C++ building tools, it can be concluded that each tool has both advantages and disadvantages. In addition, considering the challenges and historical problems faced by the amap engine library project, we found that none of the above tools could perfectly meet the business needs, and the transformation cost was very high. Therefore, we decided to build our own C++ local construction tool based on CMake, namely Abtor, which is now used in the engine library project.

Abtor

First, we need to explain the question: what is Abtor?

Abtor is a C++ cross-platform build tool. Abtor uses Python to write build scripts, generate CMake configuration files, and generate build files through built-in CMake components, and finally produce executable programs or libraries. It abstracts the build description, making complex compilers and connectors transparent to developers; It provides powerful built-in functionality that makes it easier for developers to write build scripts.

Second, we need to address the question, what is the build process for Abtor?

As shown in the figure above, the entire process of an Abtor build is:

Write Abtor build scripts; Parse the Abtor build script; Detect dependencies, identify conflicts, and download required dependencies from Ali OSS; Generate cmakelists.txt and make a Makefile using the built-in CMake; Compile, link, and generate target files corresponding to the platform; Publish the target file to Alicoss; In addition, the function of controlling access to the release library is also added to ensure the security of the release library.

Finally, we need to explore the question, what does Abtor solve?

In the opening background, we mentioned some of the challenges and problems that hinder r&d effectiveness, which Abtor needs to solve, so Abtor has the following characteristics:

More extensive cross-platform: support MacOS, iOS, Android, Linux, Windows, QNX and other platforms; Effective multi-team collaboration: well combined with IDE, and support a set of configuration to generate different projects, so as to achieve the consistency of project configuration; High customization: support flexible customization of toolchain and construction parameters, and provide strong support for complex construction of vehicle and machine through built-in toolchain configuration; Source code and dependency separation: support source code dependence and library dependence, source code through Git management, build inventory placed in Ali Cloud, source code and product completely separated; Good build performance: Build large projects quickly to improve development efficiency; As can be seen from the above features, Abtor effectively solves the pain points faced by existing build tools in Autonavi’s business. However, Rome was not built in a day, and Abtor is also constantly improving. Here are three problems encountered in the development process of Abtor. The project configuration is consistent

In the daily development process, the project debugging is particularly important. The developers of the C++ engine library project in the amap client involve several departments and groups. The technology stacks these groups excel at, the platforms they use, and the development tools they are used to. If a separate engineering configuration is established for each platform, the workload and subsequent maintenance costs can be imagined.

For these reasons, Abtor has built-in IDE functionality, which allows developers to create project configurations from a single set of configurations combined with the Abtor command with one click, and achieve consistency of project configurations across different platforms. Engineering configuration alignment brings the following benefits to library development:

Simple command, reduce learning cost, developers only need to memorize abtorw project [IDE name];

The configuration file will not expand rapidly due to the increase of IDES. Developers can change the build command, such as Abtorw Project Xcode or Abtorw Project VS2015, to generate the corresponding project project;

It is conducive to the cooperation between departments and the rapid integration of new people. Developers can choose IDE for development according to their preferences, which greatly improves the development efficiency.

Currently, Abtor supports Xcode, Android Studio, Visual Studio, Qt Creator, CLion, and more.

Construction of complex vehicle-machine environment

As a very important business line of AmAP, the construction environment faced by vehicles is complex and changeable, and manufacturers often customize the tool chain by themselves. If all projects need to modify configuration files every time a device is added, the cost is still very high. To solve this problem, Abtor offers two approaches:

Built-in toolchain configuration: Completely transparent to the developer, he doesn’t need to change any configuration to build artifacts for the appropriate platform;

Support for custom configuration plug-ins: developers write configuration plug-ins according to rules, and Abtor will detect the plug-in during construction, and build according to the set toolchain and construction parameters;

In addition, we have docker-treated all vehicle-vehicle environment, and managed the online and offline of vehicle-vehicle Docker environment in a unified manner through the Docker control center. By using the above built-in tool chain configuration function of Abtor to build vehicle mechanism construction parameters, we can realize the environment switching and other operations without the developer’s perception. It effectively solves the construction problem of complex vehicle-machine environment.

The main steps of docker-based vehicle mechanism construction are as follows:

Toolchain installation: generally provided by the manufacturer, we will install the toolchain into the basic Docker image; Docker publishing: Publishing the image to the Docker repository; Abtor adaptation: one-time adaptation tool chain, and built-in configuration, developers can use the configuration through the Abtor version upgrade; Service configuration update: Managed by Jenkins, supports batch update of Abtor version without affecting current compilation requirements; Service monitoring: Managed by Jenkins, the Docker service in abnormal state will be automatically restarted; Docker-based car mechanism construction diagram is as follows:

Dependency management

Dependencies are a common problem for all build tools, and diamond dependencies are particularly common. As shown in the figure below, assuming that A relies on B and C, and that B and C depend on different versions of D, respectively, with only minor differences between D, this will compile, but may eventually cause unexpected problems at runtime.

Without a mechanism to detect them, diamond dependencies can be difficult to detect, and the consequences can be very serious, such as leading to widespread crashes on the line. So the analysis and solution of dependency problem is very important.

Currently, Java has mature dependency management solutions such as Maven on the market, but C++ does not. Abtor specially establishes dependency management mechanism to ensure compilation correctness.

What does Abtor do with dependency management? Here is an idea for your reference:

Establish Abtor server, used for library publishing, and dealing with dependencies; After each library is built in the cloud, the version information that the library depends on will be stored in the cloud database. Abtor resolves the version information of all dependent libraries before building locally/in the cloud; Recursively searching the dependency information corresponding to these sub-libraries can list the information of all dependent libraries. Check if the same library name exists in the list of dependent libraries with different versions: if not, continue with the build; If there is the same library name, it indicates that there is a conflict between the dependent libraries. In this case, the construction is interrupted and the information of the conflicting libraries is displayed. The developer can continue the construction after the conflict is resolved. According to the above ideas, we ensure the consistency of library dependencies and avoid the problem of diamond dependencies. In addition, if a library is dependent on and updated by other libraries, the libraries that depend on it should also be built to ensure consistency of dependencies. This trigger update to dependency builds is implemented on the Amap CI, which is described in more detail in section 3.

The engineering practice

After covering some of the basics of Abtor, we’ll look at how it is used in daily development.

The following figure shows the directory structure of the Abtor project. There are two types of files that developers need to care about, one is the source file directory (SRC) and the other is the Abtor core configuration file (abtor.proj).

├─ │ ├─ ├─ abtor_├.propertiesConfiguration file, which can specify Abtor version information│ └ ─ ─ abtor - wrapper. PyDownload the Abtor version and call the Abtor entry function├ ─ ─ abtor proj# Abtor core configuration file├ ─ ─ abtorw# Initial execution script for Linux/Mac├ ─ ─ abtorw. Bat# Initial execution script for Windows└ ─ ─ the SRC └ ─ ─ main. CThe source file to compile
Copy the code

The organization of source directories is not much different from that of the Make family of build tools. Here’s a look at the Abtor core configuration file:

#! /usr/bin/python
# -*- coding: UTF-8 -*-

The following is Python syntax

# specify the compiled source code
header_dirs_list = [abtor_path("include")]    # dependent header file directory
binary_src_list = [abtor_path("src/main.c")]  # source

cflags = " -std=c99 -W -Wall "
cxxflags = " -W -Wall "

# specify compilation binary
abtor_ccxx_binary(
  name = 'demo',
  c_flags = cflags,
  cxx_flags = cxxflags,
  deps = ["Add: 1.0.0.0"].Specify the dependent library information
  include_dirs = header_dirs_list;
  srcs = binary_src_list
)
Copy the code

As you can see from the figure above, the Abtor core configuration file has the following characteristics:

  • Written in Python, easy to use;
  • Abstract the build description like abtor_CCxx_binary, reduce the threshold of use; Provide built-in functionality such as abTOR_path to improve development efficiency;
  • Through the above source file directory organization and Abtor

By writing the core configuration file, we have completed the Abtor configuration of the project, and then we can build, publish, or directly build the project through the built-in commands of Abtor. We believe that developers can build and publish with Abtor without any problems even if they are not very good at building principles.

03 Continuous Integration

Problems faced

As shown in the figure below, the entire development workflow can be divided into several phases: code -> Build -> Integrate -> test -> deliver -> deploy. After using Abtor to solve a number of challenges and problems with native builds, we moved on to the whole continuous integration phase.

Continuous integration refers to the delivery of individual pieces of software to the software as a whole, with frequent integration in order to find errors more quickly. It comes from Extreme Programming (XP), one of the original 12 practices of XP. For the engine base, the continuous integration solution should have the ability to build object files of different platforms and architectures in batches at a time, as well as the ability of operation and maintenance management and message management.

Jenkins was initially used in the Autonavi engine library for continuous integration. Because the engine library development adopts the method of pulling branches from Git repository for version management, Jenkins Job needs to be manually established and corresponding scripts need to be modified in each version iteration. In addition, an additional Jenkins Job dependent on library relationship needs to be built for linkage compilation.

Assuming 100 projects, 101 Jenkins jobs need to be manually created for each release iteration. Similar operations were repeated with each iteration, requiring a lot of coordination, and as more iterations became available, these Jenkins jobs became unmaintainable. This is a very serious problem encountered by Jenkins continuous integration scheme in the development of Autonavi engine base.

For these reasons, it is imperative to have a continuous integration system where developers don’t have to maintain Jenkins, don’t have to deploy a build environment, and can build object files for all platforms with a single trigger event without knowing the build details. So we decided to build our own continuous integration platform, Amap CI.

Amap CI

The Amap CI platform uses Gitlab’s Git Webhook for continuous integration. Gitlab receives the developer’s Tag push event, calls back to the background service of the CI platform, and then the background service distributes tasks according to the running status of the build machine. When there are many build tasks, the CI platform will wait until there are build resources before redistributing the tasks.

Amap CI platform is composed of task management, Jenkins management, construction management, notification management, webpage front-end display, etc. The overall architecture diagram is as follows:

Through the Amap CI platform, we achieved the following objectives:

Scalable: All build machines are registered to access, making it very easy to expand the build machine, reducing the pressure caused by peak builds;

Visualization: Abtor Server is transparent to developers. CI platform interacts with Abtor Server to provide visual functions such as conflict check, dependency view and library download for developers.

Intelligent: Standard Jenkins Job build templates are built into THE CI platform. Developers are not aware of these templates and do not need to make any changes. They only need to submit a tag information through Git to achieve the whole platform build, so as to achieve a one-click tag build;

Automation: The service analyzes the Gitlab Hook Tag push information and pulls the code, then parses the corresponding configuration file and all the platform information to build. Based on this information, the CI platform allocates the build machine and executes the Abtor command to build and publish. All this is done automatically;

Immediacy: Once a build is started, a spike message is sent with a link to the build in addition to the summary. Developers can click on the link to track progress. A message is sent when a build succeeds or fails, so developers can move on to the next step or deal with build errors;

Extensible: CI platform provides extensible docking mode, convenient for autonavi or Ali’s other platforms, such as Titan platform, CT platform, Aone platform, so as to realize the development loop of coding, construction, testing and release;

Of these purposes, the most important for the Amap CI platform is automation. Let’s focus on whole-tree linkage compilation in automation.

Whole tree linkage compilation

In Part 2, we mentioned that if a library is relied on and updated by other libraries, the libraries that depend on it should also be built to ensure consistency of dependencies, which is one of the key points of build automation. Amap CI uses the whole tree linkage compilation scheme to solve this problem.

The developer builds a corresponding version build tree on the CI platform, which lists the build sequence of each library, as shown in the figure below. CI platform will build based on this build tree, and the dependent libraries will be built first, and then automatically trigger the construction of their superior libraries after completion, and so on, and finally form a multi-fork tree. In this multi-fork tree, the corresponding libraries are concurrently built from the leaf nodes in hierarchical order, which is called whole-tree linkage compilation.

According to the above ideas, we guarantee the dependency consistency in continuous integration. Developers only need to care about the library they are responsible for, and then tag it to trigger the generation of all libraries that depend on that library, thus avoiding the problem of inconsistent dependencies.

The engineering practice

After covering some of the fundamentals of Amap CI, we’ll show you how you should use Amap CI in your daily development.

When a new project is integrated into Amap CI platform, it first needs to add the WEB hook url of CI platform to the configuration of Gitlab, and then write the configuration file ci_config. json. So far, a new project has been integrated, which is very simple. Let’s focus on ci_config.json.

Ci_config. json is the core configuration file. Its structure is as follows:

CI_CONFIG. Json DEMO: (json) {" mail ":" [email protected] ", # email notification "arch" : "Android, iOS, Mac, Ubuntu64, Windows". # build platform "build_vars":"-v -v ", # build parameter "modules":{# build module "amap":{# build module "amap":{# build platform "name":"feature1", # build platform "build_vars":"-v -v ", # build parameter "modules":{ Feature1 "macro":" -dFeature_demo1 =True" FEATURE_DEMO1}, {"name":"feature2", # feature2" macro":" -dfeature_demo2 =True" FEATURE_DEMO2}]}, "auto":{FEATURE_DEMO2}}, "auto":{FEATURE_DEMO2}, "auto":{ Feature1 "macro":" -dFeature_demo1 =True" FEATURE_DEMO1}, {"name":"feature3", # feature3" macro":" -dfeature_demo3 =True" # FEATURE_DEMO3}}Copy the code

As you can see from the figure above, the configuration file describes information about mail notifications, build platforms, build parameters, and provides good support for multi-line of business customization. Amap CI build time read the file, parse configuration macro in different projects, and through the parameter passed to Abtor, on the other hand, developers use these macros in your code for code isolation, build will choose corresponding source code to compile, according to these macros to support multiple lines of business of different needs, to achieve maximum reuse of code level.

At present, there are hundreds of projects with Amap CI access and hundreds of thousands of times of compilation. At the same time, the construction performance and success rate have been greatly improved compared with before, and new projects are still being added to the construction platform. It can be said that Amap CI platform is a solid guarantee for the rapid iterative development of Amap client C++ project.

04 Future Outlook

It has been more than three years since mid-2016, when we surveyed existing build tools. Three years is long enough for us to turn our vision into reality, long enough for us to refine Abtor, long enough for us to develop Amap CI. Three years is a very short time, and it is just the beginning of a system development life cycle.

About the future, our plan is to develop in the direction of development of closed-loop, namely through coding, construction, integration, testing, delivery and deployment and so on each link in the link, solve the problem of closed-loop business development, realize the entire development process automation, further freeing the developer from the cumbersome process, make the staff have the energy to do more valuable things.