AIOps community is initiated by Cloud Intelligence. Aiming at operation and maintenance business scenarios, AIOps community provides the overall service system of algorithms, computing power and data sets as well as solution exchange community for intelligent operation and maintenance business scenarios. The community is committed to spreading AIOps technology, aiming to solve the technical problems of intelligent operation and maintenance industry together with customers, users, researchers and developers from various industries, promote the implementation of AIOps technology in enterprises, and build a healthy and win-win AIOps developer ecosystem.

Introduction to the

This paper takes the DODB product of cloud intelligence digital operation and maintenance data platform as an example. The CLOUD Intelligence RESEARCH and development team analyzes and compares the number of Java code lines, number of Java files, ratio of code comments, size of released package, component dependence and other key data between the historical version and the new version of the product. Find out the existing problems in code, project management and developers in the process of historical version development, and finally summarize the effective methods to improve the efficiency of research and development such as reforming code and formulating norms.

I. Data comparison

As you can see from the chart below, in the new release, the percentage of code comments decreased by nearly 150%, the number of project branches decreased by nearly 100%, and all other statistics were significantly different.

Lines of code statistics

• In the past

• now

Code comment ratio

• In the past

• now

Sonar scan

• In the past

• now

Data summary and comparison

indicators	In the past	now	Comparing the results
Number of Project Modules	12	8	A 33% reduction in
Total lines of code	174252	136484	A 22% reduction in
Number of Java code lines	90310	50719	A 44% reduction in
Number of Java files	982	583	A 41% reduction in
Code comment ratio	3%	7%	133% increase
Sonar – the number of bugs	92	0	A 100% reduction in
Number of external HTTP interfaces	160	100	A 37.5% reduction in
Number of external DubbO interfaces	100	25	A 75% reduction in
Release package size	226M	170M	A 25% reduction in
Number of project branches	300 (reference)	20	A 93% reduction in
Write performance (single 10K)	88W/min	187W/min	113% increase
The component dependence	Ignite, vertx	Delete ignite, vertx	5.3.1 Replace Ignite with Redis and completely remove VerTX
Micro service	1	3	5.3.1 Support services run independently as storage modules or non-storage modules

Ii. Review of historical issues

Based on the comparative analysis of the above data, this chapter makes a retrospective analysis of the problems existing in the historical version development, such as code, project management and developers. For details, please refer to the following:

Code side issues

• The number of project modules is large, and some modules that are no longer used still exist;
• There is no uniform package name distinction for each module module, which leads to the existence of a large number of classes with the same fully qualified name, and the inconsistent loading sequence leads to various problems.
• There are a large number of code blocks that are no longer used in the project, which increases the learning cost and troubleshooting difficulty of newcomers; For example, only 25 dubbo interfaces are used externally but 100 are provided externally.
• There are many duplicate codes in the project, and similar services are copied and pasted seriously, resulting in high maintenance costs.
• Repeat wheel more, such as HttpclientUtil, etc.;
• The annotation rate is not enough, a lot of code without annotation, no test cases;
• The transformation of Vertx framework into SpringBoot container is not complete, there are still many Vertx logic and SpringBoot coexist;
• Adding new modules requires a lot of SQL and manual writing of “basic” code for the Controller, Service, and Dao layers;
• Projects rely on ignite group clusters, resulting in stateful services, most personnel are not familiar with Ignite, resulting in more difficult problem solving;

Project management problems

• Without strong specifications, more members can not support efficient collaborative development;
• Branch management is not standardized, resulting in too many branches, less active branches, hundreds of branches difficult to maintain;
• It is delivered in the form of branches, but the branches are still submitted after delivery, which makes it difficult to track the release version;
• There is no code management standard, and multiple function omissions and multifunctions lead to branch rollback;
• Without code review system, code quality cannot be guaranteed, and conflicts are often resolved by force.

Developer side issues

• Most members are not familiar with the code, or only part of the code;
• Many members are afraid to change the code, much less delete it; In the name of not affecting other functions, copy and paste a lot of duplicate code and logic, resulting in maintenance difficulties;
• Early code design is more reasonable, with the increase of personnel, a lot of code heavy copy light design, just to complete the function, not considering maintenance expansion, no design at all;
• Some staff are not skilled in Git, resulting in forced coverage of remote branches from time to time, without permission control;

Examples of partial problems

• Fully qualified classes with exactly the same name

• Repeat the wheel

3. Solution to the problem

In order to solve the code, project management and other problems existing in the development process of the historical version of the product, the cloud intelligence R&D team has formulated the development, interface and other related specifications; At the same time, the strict implementation of the specification is also guaranteed by other operations such as restricting the permission of the code repository.

Formulate relevant specifications

No rules, no fangyuan, feasible norms are the premise of guaranteeing team combat effectiveness. The specification should be formulated in order to improve the level of the team without restricting the enthusiasm of members.

• The development of specification
• DODB interface specification
• Develop a specification implementation plan
• DODB version Tag naming specification
• Development design template
• Mention test template
• Meeting specifications

Ensure strict implementation of standards

To do good work must first sharpen its tools, the specification provides the corresponding tools, good use can achieve twice the result with half the effort; At the same time, there is a relevant submission process to ensure the implementation of the specification, not let the specification become a mere formality.

• Set gitLab permissions to ensure mandatory code review

No one is allowed to submit code to the protected branch. Merge process is required

Merge List interface

Merge Details Interface

• The remote branch is forcibly deleted

• Track production deliveries with tags, requiring test deliveries to be tagged

Disallow tag deletion

Tag Tag list

Tag Details

How to transform the existing code

Transformation principle

• Does not affect the current development and delivery schedule, must not affect each other, similar to the air tanker, both refueling, aircraft can not stop;
• Ensure compatibility and upgrade, and the original data and services are not affected.
• The external interface (such as Dubbo) cannot be modified in principle to ensure the normal function of the caller. In special cases, it can be communicated and cooperated (such as the overload of dubbo method must be modified).
• Unneeded code will be deleted, later need to find from the historical version, prohibit batch comment out code;
• Code design should follow the principle of high cohesion and low coupling to ensure reusability and portability.
• The goal is consistent, the transformation step by step, to ensure full testing.

Partial transformation logic

• Delete the code that is no longer used. The history can be viewed in gitlog, such as BDP-plugin-SDK, bDP-plugin-Zabbix and other modules.
• Focus on solving the problem of repeated wheel building. Replace some general logic, such as HttpClientUtils and DB operations with existing tool classes.
• Writing database code tedious, the introduction of Mybatis -plus operation mysql

<! <dependency> <groupId>com.baomidou</groupId> <artifactId>mybatis </artifactId> <version>${mybatis-plus.version}</version> </dependency>Copy the code

• External DUBBo interfaces BDP-RPC and BDP-rpc-model are merged into BDP-rpc-core to provide external duBBo interfaces. The unused interfaces are deleted by compiling each caller project
• Completely remove verTX and Ignite dependencies

• The functions and naming of each module have been standardized, and the original code has been modified and adjusted uniformly

The name of the module	function	The package name prefix
bdp-api	Restful API and business logic implementation, can only be relied on by BDP-standalone	com.cloudwise.bdp.api
bdp-base	Business object and interface definitions	com.cloudwise.bdp.base
bdp-commons	Common Util	com.cloudwise.bdp.commons
bdp-pipeline	Pipeline data processing implementation	com.cloudwise.bdp.pipeline
bdp-rpc-core	RPC interface declarations and RPC entity class definitions cannot rely on other modules.Modify the external RPC interface with caution	com.cloudwise.bdp
bdp-standalone	The main project, DODB main entry class is DodbServiceApplication	com.cloudwise.bdp.standalone
bdp-store-ck	ClickHouse access implementation	com.cloudwise.bdp.store.ck
bdp-store-common	Data layer storage port declaration	com.cloudwise.bdp.store.common

• Call Gitlab-API to clean up the backup centrally for long-inactive branches

/** * github-api * https://github.com/help/api/api_resources.md */ private static final String GITLAB_URL = "https://github.com/api/v4/projects/2393/repository"; private static final String PRIVATE_PARAM = "*****************"; /** * delete the merged branch. */ @test public void cleanBranches() {Map<String, Object> paramMap = maps.newhashMap (); */ @test public void cleanBranches() {Map<String, Object> paramMap = maps.newhashmap (); paramMap.put("private_token", PRIVATE_PARAM); paramMap.put("per_page", 10086); String body = HttpUtil.get(GITLAB_URL + "/branches", paramMap); List<Branche> branches = JSON.parseArray(body, Branche.class); Sort (Comparator.comparingLong(o -> o.goetCommit ().getcommitted_date ().getTime()))); Log.error (" Number of branches :{}", branches. Size ()); branches.forEach(item -> log.info("{}", item)); // Clean up long-inactive branches..... }Copy the code

Release packages to slim down

To see how the distribution package is built, see assembly.xml configuration

• What’s in the release pack

• Do not duplicate the configuration files into the JAR package

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <includes> <include>com/cloudwise/bdp/**</include> </includes> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>${start-class}</mainClass> <classpathPrefix>.. /lib/</classpathPrefix> </manifest> </archive> </configuration> </plugin>Copy the code

• Focus on the lib package, you can check from the following aspects

Focus on whether packages that are too large are necessary for the project

Whether there are duplicate dependencies

* For example, netty-all is a collection of netty-, so do not rely on it repeatedly. If the version is inconsistent, it will cause problems.

Batik-all is a collection of batik-*, excluding all subpackages

After the investigation, trace the dependent source through relevant plug-ins and eliminate it accurately

<plugin>

   <groupId>org.apache.maven.plugins</groupId>

   <artifactId>maven-dependency-plugin</artifactId>

   <version>${maven.dependency.version}</version>

</plugin>
Copy the code

1. Developers should be clear about what relying on packages does, and should not rely on a bunch of useless packages to increase the size of the release package unnecessarily, which will also bring dangers;

2. During code review, the modification of dependency files (Pom.xml file of Maven project) should be paid close attention, and arbitrary introduction and modification of dependencies are prohibited;

3. Each product line can adjust the construction mode and greatly reduce the size of the integration package on the basis of unifying the dependent versions.

Reform summary

• Optimization is not a step in place, can be carried out gradually, the goal can be clear;
• Complexity is the root of all problems. If you can solve a problem with one line of code, you don’t need two.
• All functions are realized by code, writing good code is very important, can not take writing good code as a small thing, just to complete the required function of the stack of code is very simple, but write clear and easy to understand and can complete the required function of the code is not simple;
• You’ll never write “perfect” code, use tools and process specifications to ensure this, and test it well;

Learn more

Cloud Intelligence Is an open-source, lightweight, converged, and intelligent Operation and maintenance (O&M) integrated Operation Management Platform (OMP). It provides functions such as Management, deployment, monitoring, inspection, self-healing, backup, and recovery, providing users with convenient O&M capabilities and service Management. In addition to improving the efficiency of operation and maintenance personnel, it improves business continuity and security.

Click on the link below to like OMP and send it to Star to learn more

Making address:Github.com/CloudWise-O…

Gitee address:Gitee.com/CloudWise/O…