It has been seven years since the initial development of Qiniuyun. Currently, qiniuyun has more than 300 r&d students and six business lines: cloud storage, CDN, live cloud, container cloud, big data and artificial intelligence laboratory. Besides qiniuyun’s excellent product r&d team, there is also an important department responsible for quality management, engineering empowerment, and process improvement — Qiniuyun Engineering Efficiency Department.

This issue of cattle said, by the seven niuyun engineering efficiency department responsible person Li Qian and we discuss the problem of engineering efficiency. Share her journey and the evolution of the whole team from 0 to 1.

#01 Development history of engineering Efficiency Department

I think organizational change should be based on a more favorable perspective for business and development, not necessarily a testing team, quality assurance department, engineering department, etc. What’s more important is what you can offer in the context of the company, and whether that will make the business better.

At the same time, the biggest feeling in a startup is to embrace change. In this process, you will find that a lot of things are not ready for you, you have to constantly adapt, constantly find the maximum value, improve yourself. Now LET me give you a brief introduction to the development of engineering Efficiency Department:

  • In 2014, the Engineering Efficiency Department stepped in and was mainly responsible for following up the front and back end tests. The first step we need to do is to build a complete test environment;
  • In March 2015, a test group was established to provide daily quality assurance for the three business lines (storage, data processing and CDN testing). From the beginning we tend to solve problems in an automated way, so we hire test developers. That’s when we introduced the basic CI/CD delivery capabilities;
  • In February 2016, the quality assurance Department was established to serve the whole research and development, and began to write test services, improve the ability of CI/CD, and automate pipeline. In addition, we also manage the company’s internal platform and tool chain, and write a set of quality and efficiency analysis platform;
  • In February 2017, the platform service group was established, and the efficiency platform was developed, and the container was introduced to be used as the most basic technology in the efficiency platform. At the same time, the department was officially renamed As Engineering Efficiency Department (V1.0);
  • From 2017 to 2018, the internal organizational structure of engineering Efficiency Department (V2.0) was optimized to adapt to the rapid growth of Qiniu business and empower the technical system.

#02 Team structure of engineering Efficiency Department

As a company in cloud computing for basic services, we want to help our customers provide technology capabilities faster and better. Our internal vision is to shorten the distance between quality code and production line/customer. This is also a very important distance in engineering.

Our system is divided into three main parts: quality management, engineering empowerment, and process improvement. ** although there are dotted lines and solid lines, it is an organic whole. We not only need to do quality, but more importantly, how to do the best quality under the condition of efficiency.

  • Quality management: dedicated QA students in-depth research and development business, identify quality risks, establish quality feedback loop. At the same time around the target to deepen the degree of automation, adapt to more interactive scenarios, big customers. The main personnel here are test development engineers;
  • Process improvement: PMO process management ensures that our entire process is as simple as possible and suitable for the development of the company. When appropriate, we will use it to spread best practices.
  • Engineering empowerment: it’s the backbone of CI/CD, and it’s our axis, and it allows us to track all the work we do on a good platform. Improve and optimize the quality and efficiency measurement system, so that we have a sense of purpose, know what Angle we should solve the problem.

In a word, engineering efficiency is mainly concerned with the quality control of r&d and delivery links in the delivery chain, as well as the overall delivery ability of efficiency.

#03 Inheritance and upgrading of quality awareness

So how to achieve full quality awareness, code delivery and service upgrade, there are three main points:

  1. Engineers have the ultimate pursuit of quality. Code and bugs are written by humans, so we hold engineers accountable for their code — Eat Your own dog Food.
  2. Build – in quality setup. Internal quality determines external quality. All processes before delivery should be advanced as far as possible and internal quality should be improved. For example: single test, static scan analysis, integration testing, etc., every failure should be fixed. Each step should provide a verification mechanism, so that the code has a way to verify, responsible for the overall quality;
  3. DevOps engineering culture is introduced: if we do something more than twice, we need to think about whether it’s actually more efficient. So I summed up a sentence: Everything is Code.

#04 Technology Evolution

In this regard, I would like to talk to you about the basic technical practice path of the quality team, which every QA student should do, and every development student should understand.

First, I’d like to explain why there is a step-by-step effect like the one above. A lot of our products go from zero to one to ten, so we have a lot of work to do in the beginning, rather than doing the same thing all the time. Every tech person has to evolve, shifting their focus from A to B to C.

  • In the first phase, we improve the quality of the code itself, the quality built in, and we provide feedback to the developers to let them know what they are doing wrong.
  • The second stage, for the product business, output API-level test services;
  • The third stage, from the code analysis to improve the test coverage, based on the test coverage, to assist the precision test;
  • In the fourth phase, mock or callback tests are performed for inter-service dependencies.
  • In the fifth stage, error injection is considered to simulate the response performance in the case of failure.
  • In the sixth stage, we will introduce more dimensions of detection methods, not only E2E, but also business-based call chain measurement, service health status, etc., to evaluate each iteration of the product.
  • In the seventh stage, the output, summary and abstraction of each stage will be servitized and productized to provide quality assurance in the service mode.

#05 Platform Evolution

Platform evolution is important, and we need to consider whether it is possible to move more things onto services. At the same time, platformization is the stage after servitization — how to integrate services into the whole process, and be completely managed, providing certain engineering capabilities. Our goal is to quantify product quality and efficiency, provide quality and effectiveness analysis capabilities, and identify weaknesses.

The main features listed at the service level or module level include single-test coverage data, static scanning and code inspection, integration test coverage, defect and incident analysis, release tracking, service detection, competitive benchmark detection, and toolbox.

These are the background of our quality and performance platform. These are some of the indicators, including winners and runners-up, which serve as a kind of incentive and evaluation. In my opinion, there is no harm without contrast, and there is no progress without contrast. We will quantify some key indicators and introduce incentives accordingly. But rather than reward the champion, we reward each team according to their growth rate.

Engineering efficiency platform is Devops or CI/CD platform in narrow sense, and software engineering information platform in broad sense.

  • Compilation: Jira HOOK is used to trigger tasks periodically, and GitHub declares tasks to be discussed at the same time to trigger system compilation. Push the build result into a container and Archive the result into a storage for backup.
  • Deployment: Group services, provide certain assembly capabilities, make multiple services into service groups, and support real-time log retrieval and upload the generated logs to Pandora big data platform.
  • Testing: Because the test service is similar to the production service, the deployment is similar. We will generate a test report and Log to the quality platform. The important thing about testing is that interacting with the quality performance platform generates a lot of data, and every data is recorded;
  • Distribution: We do distribution for object storage. Containerized deployment uses the container platform for publishing; Physical deployments are distributed in a unified location.

Let me share with you the results of seven cows in practice. Directions will be taken from two aspects of the diagram above: quality and efficiency.

Quality: single test coverage of core services includes uploading, downloading, deleting, live streaming and streaming, etc. The coverage of these core services is more than 60%, compliance is about 80%, pipeline pass rate is more than 80%, integration test coverage is 35%;

Efficiency: 2-10 minutes of pipeline construction, defect resolution rate of 82%+, release frequency of 40+ per week, core service MTTR within 2 hours.

Efficiency: 2-10 minutes of pipeline construction, defect resolution rate of 82%+, release frequency of 40+ per week, core service MTTR within 2 hours.

Finally, a little insight: Do the right thing and don’t put limits on yourself. Especially in a startup, you’ll find a lot of things to do, and maybe you think there are some things you should do and some things you shouldn’t do, but I don’t think there are any should’s or shouldn’t’s, as long as it’s right, you have to push to get it down.

Follow the public account “Seven Niuyun” for more information