takeaway
Today, the era of pure Internet industry has come to an end, and the growth dividend and domestic demographic dividend accompanying the birth of mobile Internet have basically come to an end. At a time when all industries were immersed in the surge of mobile Internet users brought about by the rapid growth of the market, enterprises could have been ignored even if their development methods were crude and their costs were high. At the same time, under the general trend of sino-US trade war and national deleveraging, traditional industries are suffering from domestic and foreign troubles. At this time, the efficiency and cost of r&d management of enterprises are gradually emerging. They must cultivate internal skills and enter the era of intensive cultivation.
Zhang Hailong, CEO of CODING, believes that in this context, all enterprises need to transform. Efficient development management like DevOps and a better version control process; At the same time, many emerging Internet companies are rapidly transforming in this respect. If enterprises still adhere to the rules and use the previous waterfall management mode, the gap will become bigger and bigger in the future.
As an accepted and understood set of cultural values and practices, DevOps has been proven to help organizations of all sizes improve their R&D management processes, accelerate iteration and release, and ensure r&d quality and safety, while providing quick and effective feedback on product development. From the 27,000 DevOps survey responses Puppet has received over the past six years, there is ample evidence that DevOps practices have led to upgrades in R&D management that lead to greater efficiency, resulting in improved productivity, profits, and market share (1).
Today’s case is Gary Gruver, HP LaserJet Printer Firmware Development Director describes the DevOps transformation of the entire firmware Development team in his book A Practical Approach To Large-Scale Agile Development. This case was shared not only because the entire r&d team achieved a 2x or 3x efficiency increase after the successful transformation, but also because the r&d team was positioned to provide firmware development for the entire HP LaserJet product line, not the typical App or website development team.
This case shows us that DevOps is not just a tool for Internet giants like Google, Amazon, and Netflix, but a tool that can improve the efficiency of all enterprises that need r&d, testing, and operations to be closely connected to better achieve their corporate goals. If an elephant like HP LaserJet’s firmware development team can turn around, you can transform your business.
Enjoy!
Problem detection (firmware development becomes a bottleneck of the business)
Gary’s r&d team is primarily responsible for writing firmware code that runs on HP LaserJet’s full line of products (printers, copiers, scanners, etc.). At the time, the consumer printer market was a completely red Sea market, full of giants, with new features or products coming out almost every month.
Although Gary’s firmware development team has over 600 engineers worldwide and maintains over 10,000,000 lines of code, the team has faced significant challenges in trying to implement these new features.
At the time, the entire firmware development team was only able to complete two iterations per year, spending most of their time porting the code to new printers. Gary estimates that at most 5% of r&d time is spent developing or maintaining new and innovative features.
As a result, Gary’s firmware r&d team became the bottleneck of LaserJet Business Division. According to their r&d management mode at that time, it was almost impossible to add new functions to LaserJet’s firmware to adapt to market competition.
“Our marketing folks come to us every once in a while with creative ideas and emphasize how important these features are to our customers. But all we can tell them is, pick two features on your list and they’ll be available in six to 12 months.”
Over time, the Marketing Department basically gave up raising new functional requirements to the firmware team, which is undoubtedly a serious blow to HP in the increasingly competitive printer market. What’s more, r&d costs increased by nearly 2.5 times that year, while 80 to 90 percent of the resources were actually invested in firmware porting and repairing new products.
Lead times become a big problem
In the book, Gary mentions some statistics about the delivery cycle that should be common in software development projects we are familiar with:
- Only 5% of r&d time is spent developing new features
- 15-20% of the development time is spent merging code into the trunk
- There were eight groups, each with a team leader responsible for submitting code every day
- It takes a week after a group leader submits the code to determine whether it has been successfully merged into the trunk
- Then you have to wait a day to get to the “integration phase,” and then you have to spend another day running compatibility tests
- Finally, six weeks of manual integration testing is required
This means that every engineer has to wait more than 8 weeks to find out how the code works!
Backward feedback mechanisms
“If our feedback mechanism could be more efficient and timely, r&d efficiency could be improved a lot. The development process was often stuck with code issues submitted eight weeks earlier, and it took ages to figure out what the problem was.”
Under such feedback mechanism, it is not conducive for engineers to learn and summarize how to prevent the same problems from happening again in the future. The only thing that will happen is that the engineer will be blamed by management for a mistake that was made 8 weeks ago and spend a lot of time fixing the problem and fixing the Bug.
This is a very important problem, and if engineers don’t get timely and efficient feedback, they will have to spend a lot of time fixing code that was submitted long ago, which will have a significant impact on development efficiency.
Introduce continuous integration and architectural change
Architectural change
Under the previous development management model, each different LaserJet product had to create a separate branch of code that used the #ifdef’s precompile command in C to enable or disable specific features (such as paper size) on different product types.
This mode will cause a lot of problems, for example, with the continuous increase of product line, more and more independent code branches need to be maintained in real time, and each maintenance will produce relatively independent firmware version, which requires independent testing of 叒 Yi, and a lot of resources will be wasted in these repeated efforts.
So the first goal was to use Trunk to create a platform for all engineers to work in a unified code repository, eliminating the individual branches of code through Trunk architecture, Change printer functionality from #ifdef’s precompile instruction in C to XML configuration files at run time.
Today, HP LaserJet’s Trunk architecture supports more than two dozen printer products.
“Creating a Trunk to unify the code, rather than managing the code across products, will bring tremendous efficiency gains to the enterprise.” — Gary commented on the impact of the architecture change. “Next, we need to automate testing. Without automated testing, continuous integration will produce a lot of ineffective results quickly.”
Automatic test
To develop automated tests, the entire r&d team built a series of automated components, including code review and code integration tests, that continuously tested the backbone code in the background.
Firmware testing for printers is a challenging task, and the most reliable way to do it is to actually print on a physical printer, but the problem is that the entire firmware development team needs over 150,000 hours of testing per day under the new architecture.
“There aren’t enough trees on the planet to allow us to do a real-world test of this magnitude.”
Because the closer the test environment is to the real test environment, the test cost will increase accordingly, so the main goal of the research and development at that time was to reduce the cost through as many online tests as possible in the form of simulators. To ensure that test colleagues got feedback as quickly as possible, Gary’s team spent six weeks setting up the infrastructure of the test environment to support testing: four clusters of servers, and two more in India, each consisting of 500 servers, each running four virtual machines. A total of 240,000 virtual printers continuously run the new firmware and report results.
Through the establishment of an automated test environment, the entire firmware r&d team has realized an efficient feedback mechanism, and now engineers can get the submission results quickly after submitting the code, and the whole regression test only takes 24 hours.
Code reviews
After the successful introduction of continuous integration and automated testing, another problem arose — the entire r&d team needed to adapt to new habits. Every now and then, when an engineer submits code that crashes the Trunk or the test environment, everyone has to stop working and start looking at what’s wrong and fixing bugs. There was even a crash that lasted for five days, causing great frustration for the team as no one could commit new code. It turned out that while upgrading the test environment, I had accidentally raised the bar for one of the tests, resulting in negative feedback for all the tests.
Gary’s solution was to introduce a code review process to ensure that only correct, high-quality code was reviewed and incorporated into the trunk, ensuring that the number of people affected by code quality was minimized and the r&d efficiency was maximized.
DevOps transformation results
Gary’s team transformation results are as follows:
- 75,000 to 100,000 lines of code are changed every day
- 100-150 code submissions per day
“In the past, we couldn’t change 100,000 lines of code a day. DevOps with continuous integration and continuous delivery makes it much more flexible.”
More importantly, Gary’s firmware development team made the following progress:
- New feature development time increased from 5% to 40%! (An eight-fold increase)
In this day and age, software is becoming more and more important to every company; Better tool coordination, automation, and continuous learning are also becoming increasingly important through DevOps practices. Even if a large, decentralized, relatively conservative team like the firmware development team at HP LaserJet can make a DevOps transition, it is reasonable to assume that all organizations, regardless of their mission, can accomplish their goals more effectively by practicing DevOps.
Reference
- Puppet Labs 2017 State of DevOps Report
- A Practical Approach To Large-Scale Agile Development
- http://www.slideshare.net/gbgruver/spark-2013-presentation-of-making-the-enterprise-agile