【 Abstract 】 Test strategy document is usually a long, text-based form, the writing cost is higher, and the written few people go to see, the existence of real death. This article introduces the visual way, the test strategy is expressed in a diagram, and done on a page, such a strategy diagram is very clear, the key information is clear, and provides a larger space for discussion, to prevent rigidity, can really play the role of the strategy.

“What is the testing strategy?”

“The test strategy includes #&~+-= ~ *-+$…”

“Is there anything special about your project strategy?”

“For our project, the content of the testing strategy is a little too much, where to start?”

Does that scene sound familiar? Are you aware of the testing strategy you are currently using?

I. Common testing strategies

1.1 Content and form of test strategy

As we all know, a testing strategy consists of the following two aspects:

  • To test (What)? Test what refers to what quality requirements are, what aspects need to pay attention to quality, such as the functional scope of the application, performance, safety, ease of use and other non-functional requirements.
  • How to test (How)? How to measure means to help the system achieve quality requirements, not only manual and automated testing methods, but also all the processes, environments, infrastructure and personnel that serve for quality assurance.

In order to describe clearly what to test and how to test it, the test policy is usually a long document, consisting of several chapters; Mainly with text description, with only a small amount of illustrations.

Images from web: https://wenku.baidu.com/view/…

1.2 Pain points of the test strategy

Lengths of text are mostly inconvenient:

  • It is difficult to write: It is difficult to write long test strategy documents, especially for those who are not good at writing and have a background in science and engineering.
  • Not easy to read: It is not easy to quickly find out the key information in a long test strategy document. The details that may be accidentally missed are the most important parts because of the length of the document. There is often a lot of information that is not important.
  • Maintaining and updating pains: Policy documents can never be static, making updating and maintaining these long documents a nightmare. It often starts out fine, but as time goes on, updates and maintenance get more and more cumbersome.
  • Lose the value of the policy: Because it is not easy to read, maintain and update, in fact, many people on the team may not be very clear about the content of the policy document. Such a policy document is dead and cannot really serve as a guide to the policy.
  • Anti-Agile: Agile development focuses on shortening feedback cycles and delivering high quality software products quickly. Spending too much effort on writing and maintaining a long document that doesn’t serve a strategic purpose is obviously not agile and very painful.

The importance of testing strategy is self-evident, but could there be a better way to express it in a way that makes it less painful? Jamie McIndoe, in “Testing Stuff – A One-Page Test Strategy”, first proposed the idea of visualizing A Test Strategy graph on A single Page.

As we all know, graphical expression is intuitive, clear, easy to identify key information, and easy to remember. It would be great if the testing strategy could be graphically illustrated on one page.

Let’s take a look at what a graphical testing strategy looks like.

Second, graphical testing strategy

2.1 One page

As the name implies, graphicalization is the use of diagrams to describe the content of the testing strategy, but does not translate each chapter directly from the original text description into a diagram. We considered drawing the key components of the test strategy on a page.

Of course, the one-page test strategy only presents the key information in graphical form, not the whole test strategy. Behind the one-page paper is the full communication of the team and the consensus on all aspects of the strategy, which requires the team to do a lot of work together. This highly simplified presentation is designed to give the team more room for discussion and a page that is easier to modify so that it is more adaptable to change and truly meets the needs.

Based on Jamie McIndoe’s idea of visual test strategy, my proposed test strategy map contains the following information:

  • Guiding Principle: Team is responsible for quality;
  • How to test: test to the left, lean test, test to the right, covering the test process, test type, test method, etc.
  • What to test: including function, performance and safety, etc.

The following will introduce the contents of these parts by taking the Blue Whale Project as an example. For the Blue Whale project, it was an offshore delivery project that took place over 10 years, using an agile development model, releasing every four to five weeks, following agile development practices.

For example, the testing strategy for the Blue Whale project looks like this:

2.2 Detailed introduction of each part

Now, let’s look at the implications of the various components of this testing strategy.

2.2.1 Guiding principles

Blue Whale project adopts the agile development mode, quality is not a matter of a single role, the team is responsible for quality is the guiding principle of project quality assurance, all team members need to have a consistent understanding of this, everyone should have the awareness of paying attention to quality. For more on teams taking responsibility for quality, see my blog post on how good teams take responsibility for quality.

2.2.2 Test Left Shift and Quality Built-in

The two most critical aspects of agile testing are Test early and Test often, which is the idea of testing left and quality built in.

The test shift to the left requires that the rationality of the demand itself should be verified at the beginning of the demand analysis stage. It is not only necessary to build the right product, but more importantly, to build the right product, which requires a good source of demand. Therefore, we can see that the process in the strategy begins with requirements analysis.

Quality built in is to have quality-related activities at every stage of the software development life cycle, integrate quality into every step of the development, obtain quick feedback through CI/CD and do a good job in the prevention of software defects, so as to reduce the massive repair costs caused by late exposure of defects.

The development life cycle of the Blue Whale project is mainly reflected in the following seven links. Each link has its corresponding test activities, and each activity has the participation of different roles.

2.2.3 Lean testing

Lean testing can be understood as targeting business value and delivering high quality software at the lowest possible cost. That is to say, testing should be done at the point where value can be demonstrated, and effective coverage should be achieved to reduce waste. There are two frameworks in the strategy chart of Blue Whale project to help us test more effectively, namely, test quadrant and test layering.

1) Test quadrant

In Lisa Crispin and Janet Gregory’s book, Agile Software Testing: A Practical Guide for Testers and Agile Teams, we see an introduction to the Agile Test Quadrant. Since the role of this quadrant framework is not limited to agile environments, I call it the test quadrant here.

The test quadrant matrix has four parts, called four quadrants. The bottom side is technology-oriented testing, the top side is business-oriented testing; The tests to support the team are on the left, and the tests to evaluate the product are on the right.

  • Support team testing

Testing in support of the team is used to tell the team what code to write, to clarify requirements, and to aid in design. The first quadrant is technology-oriented support team testing, mainly TDD, which helps build the internal quality of the product, namely code quality assurance, such as unit testing and API testing. The second quadrant is business-oriented support team testing, which determines the desired behavior of the system from a higher level in a way that can be understood by business experts to ensure the external quality of the product.

Testing in these two quadrants provides quick feedback and ensures quick resolution of problems, both guiding feature development and providing a safety net against refactoring and the introduction of new code that leads to unwanted behavior.

  • Testing to evaluate the product

Programmers can write code that makes the business-oriented tests on the left pass, but it may not produce what the customer really wants, so you need quadrants 3 and 4 to evaluate the product.

The third quadrant is business-oriented evaluation of the product tests, which help confirm whether the product is actually built by mimicking the way real users use the application. The fourth quadrant is technology-oriented product testing, which uses tools and corresponding technologies to evaluate non-functional features such as performance, robustness, and safety, and considers these tests at every step of the development cycle.

The information generated from the tests in these two quadrants should be fed back to the left of the quadrant matrix and used to create new tests to drive the next step of development, forming a virtuous enhancement loop.

  • Use of the test quadrant

The order of the quadrants is independent of the order in which tests are executed. Agile development often starts with customer testing (business-oriented testing). Factors related to the timing of test execution are usually:

  • Risk of product release;
  • Customer’s requirements for product objectives;
  • Whether to develop on legacy systems or to build new systems from scratch;
  • Test resources available, etc.

The test quadrant provides a framework for thinking about what tests are needed to ensure quality, and can be combined to conduct the appropriate tests, depending on the project. The test quadrant of Blue Whale project shown in the strategy diagram reflects different test types from those introduced in Lisa’s book. It is the test that needs to be executed at the moment, which is determined according to the project’s current cooperation mode with customers, business requirements, quality requirements, etc. For example, the safety test is divided into business part and technical part.

2) Test layering

As for the idea of test layering, what you might be familiar with is the test pyramid, which is basically for automated tests, divided into layers based on what the tests can cover. Pyramid refers to the proportion of tests, which is reflected in the fact that there are more unit tests at the bottom level and less tests at the top level, showing a pyramid structure.

The lower the test is, the closer it is to the code, the lower the writing cost, the faster the execution speed, and the more accurate the problem positioning, but it is far from the business and cannot reflect the business value well. The higher the level of testing is, the closer it is to the business and the better it can reflect the business value, but it has the disadvantages of unstable, slow execution speed and high implementation cost. Therefore, it is necessary to weigh the pros and cons, and determine the proportion of each layer of testing based on the actual goals of the project.

Whether the exact ratio is a pyramid structure, a honeycomb structure or something else is not fixed and will not be fixed. It may be influenced by value objectives, pain points, quality requirements, technical architecture, skill level, etc.

The strategy diagram for Blue Whale shows the current test hierarchy, similar to a honeycomb structure, because more of Blue Whale’s automated testing is the API testing portion that ensures connectivity between services based on the characteristics of the microservice architecture, while the proportion of unit testing and end-to-end testing is reduced. For more details on testing strategies for the Blue Whale project, please refer to my blog post Microservice Testing Thinking & Practicing.

2.2.4 Test right shift

Due to the increasingly complex ecological environment in which software systems live, the evolution of technical architecture, the increase of business complexity and data volume, and the development of infrastructure bring more uncertainties, the quality assurance of software systems has been unable to be done in the test environment, we need to shift our attention to the right to the production environment. This is the idea of testing moving to the right, which is QA in Production.

Production environment is different from the characteristics of the test environment, production environment of QA is not a test environment directly by QA, but need to take advantage of its characteristics through technical methods to collect the production environment of all the available data, including log, user behavior, user feedback and so on, use these data to analyze and optimize the business and development process of the development and testing work, Forming a development process and production environment information analysis of a virtuous cycle system.

The Blue Whale project has done a lot of work on this, and for more details, see my blog post on QA in production environments and how QA and OPS work together to combat vulnerable software systems.

2.2.5 measure what

The reason why I put this at the end of the introduction is that the previous part of “how to test” has already covered the content to be tested, the Blue Whale project test content mainly includes: function, performance and safety.

These three aspects of testing are similar, from requirements analysis to production environment to consider relevant testing, to achieve quality built in, safety built in and continuous performance testing. Quality build-in in terms of function is described in the section “Test Left Shift and Quality Build-in” above. For safety and performance strategies, please refer to the following diagram. Due to the limited space, this article will not go into details.

Third, test the correct way to open the policy

One-page testing strategy has obvious advantages. It is more concise and clear than traditional policy documents, and the key information is clear at a mile. Let’s take a look at what else needs to be noted once the test strategy is graphical.

3.1 Goal Driven

Although there are many test policy templates that can be found by searching the Internet, testing policies should not be one-size-fits-all and should not be rigidly applied to a common template. The testing strategy is influenced by many factors, such as: business value, quality requirements, current pain points, technical architecture, technical capabilities, focus of work, performance requirements, project status, and so on.

The development of testing strategy requires clear objectives, comprehensive consideration of various factors, weighing the pros and cons, and finding the most suitable strategy for the current state of your project. In other words, the testing strategy should be goal driven.

3.2 the evolution type

In different stages of the project, there will be different quality objectives, and with the evolution of the architecture and the development of the business, the quality assurance work of the software system also needs to be adjusted accordingly. Therefore, the test strategy should also be evolutionary and be adjusted on demand.

Graphic testing strategies are highly concise, have more room for discussion and play, and have obvious advantages in preventing rigidity and maintaining evolution.


The testing strategy is important, and the content is important. It needs to be value-driven, constantly measured, and adapted and evolved according to project specific circumstances. Policy documents do not stick to the form, using graphical method, intuitive, clear expression, is a very good form of organization, can effectively overcome the pain of conventional text-based documents, recommend everyone to use.


  • One – page test strategy diagram
  • Jamie McIndoe Testing of Stuff – A One – Page Test Strategy: https://making.stuff.co.nz/te…
  • Say good team responsible for quality: https://www.bylinzi.com/2019/…
  • QA in Production:https://martinfowler.com/arti…
  • The blue whale project of test strategy thinking and practice of service test: https://www.bylinzi.com/2018/…
  • Production environment QA:https://www.bylinzi.com/2016/…
  • QA work together with Ops dozen rebel vulnerable software system: https://www.bylinzi.com/2018/…

Source: By the Woods

Author: Lin Bingyu

Disclaimer: The article was forwarded on IDCF public account (devopshub) with the authorization of the author. Quality content to share with the technical partners of the Sifou platform, if the original author has other considerations, please contact Xiaobian to delete, thanks.

Every Thursday in July at 8 p.m., “Dong Ge has words” R & D efficiency tool special, public account message “efficiency” can be obtained address

  • July 8, Leansoft – Wenyang Zhou “Azure DevOps Toolchain”
  • July 15, Aliyun Intelligent Senior Product Expert – Chen Xun “Practice of Efficiency Improvement in Complex Research and Development Collaboration Mode”
  • On July 22, Yang Yang, solution architect at GitLab, shared “Exploring the Automated Testing of Infrastructure as Code”
  • On July 29, Bytedance Product Manager — Xianbin Hu shared “Automated Testing: How to Be Offensive and Defensive”.
  • Wang Zhi, head of Acoustics AgoracicD System, shared “Creating a Closed Loop of Software Delivery Quality Assurance from 0 to 1” on August 5.