Shengcong Cheng – Product Manager of CODING

Automated testing is the foundation of continuous testing

In DevOps’ high-frequency delivery scenario, teams tend to fall into the “choose between speed and quality” trap: short lead times in order to embrace changing requirements, and then frequent changes lead to more problems, resulting in delays in development and rollout, resulting in compressed testing times and inadequate testing. Faced with this situation, how can teams improve the efficiency of test execution? One of the first things that comes to mind is automated testing — replacing repetitive manual testing with automated testing, which can be performed faster and save testing time. In addition, due to the relatively fixed time of each execution of automation, and the high consistency of the preset test behavior of the program, the stability and repeatability of the test reach a high standard, which can well achieve the goal of “rapid recurrence of software defects”.

If the greatest value of automated testing for regression testing scenarios in the traditional waterfall model, where testing time is relatively abundant, is in terms of cost savings, the greater value of automated testing in the agile and DevOps era is in frequent validation and providing quick feedback. It can be said that the basis of continuous testing practice is automated testing, only the degree of automation is high enough to meet the needs of continuous delivery of high frequency version.

Automated test strategy

Automated testing has great value, but that doesn’t mean we should invest in all types of automated testing without limit. Automated testing is to verify that the given logic meets expectations, and the maintenance cost of automated code is huge when requirements change frequently. Therefore, we need appropriate strategies to guide the implementation of automation — the pyramid model.

From The Test Automation Pyramid: Your Essential Guide for Test Automation

The Pyramid of automated testing changed in his first book in 2009 In Software Development using Scrum, the top three layers were UI, Service, and Unit. Later, with the implementation of agile testing practices, the industry gradually formed a recognition of UI Tests, API Tests, and Unit Tests from top to bottom, plus manual exploratory Tests at the top, further enriching the test pyramid (both automated and manual). This triangle, narrow at the top and wide at the bottom, provides a visual guide for our automation investment at each level: unit testing at the bottom is the most, interface testing is in the middle, and UI testing is the least.

As shown in the pyramid model, the lower level of unit/interface testing has some advantages over the upper level of UI testing: it is easier to write and locate defects in the code because it is closer to production code; Because the test object has smaller granularity and fewer dependencies, it is more efficient to execute; The cost of maintenance is lower because the test object is more stable, etc. Of course, the advantage of closer testing is that it is easier to see the value of the test because it is more reflective of the business needs. In the DevOps era, interface integration testing for the middle tier should be a priority because it can maintain relatively low maintenance costs and reflect the value of business logic based on the balance of speed and quality, especially when all aspects of automation are still in their infancy.

The test pyramid, which grew out of agile practices, is used as a reference to continually adjust our test automation efforts, and the team’s test cases and execution will gradually be well balanced.

The value of precision testing

Industry surveys in recent years, however, have shown that the adoption of DevOps has led to increased investment in automated testing, a direct result of which is more and more automated testing code. But with the rapidly increasing number of automated codes, will automation achieve the desired results?

In reality, organizations are not getting the expected value from increased coverage of automated tests because the execution of automated code is not as “free” as we might think, often for two reasons:

  1. Teams generally treat automated code execution as a part of CI and only use it as a regression scenario, but the long time required for full regression limits the frequency of execution.
  2. The technical barriers to building the pipeline and matching the tools on top of it, coupled with the amount of work required by these operations, make it difficult for everyone to automate tests all the time.

This results in regression tests taking longer and longer to execute as automation coverage increases, reducing the frequency of automated tests and ultimately questioning the value of automated code. In addition to enhance automation coverage is, in fact, we also need to change the “use case for each test execution to cover the more is better” idea: we should not make test set for “no” too much redundancy, but needs to be based on business risk optimization test coverage, in order to realize more test in a limited range input-output ratio, precision testing.

CODING makes test execution more free

In order to make test execution more “free”, CODING has built the ability to automate test execution in the cloud, aiming to solve the “last mile” problem of automated testing to achieve:

  1. The freedom to choose a set of examples for each execution;
  2. Everyone has the freedom to perform tests.

Next, let’s look at how to perform tests “freely” in CODING test management:

  1. Firstly, automatic code registration is carried out in the CODING automation use case library to confirm that the automated code already exists in the code hosting, the existing automation code base is registered, and the relevant language/framework is set up.

  1. The test function list of automatic code base is analyzed, and the matching relation between function cases and automatic functions in use case management is established, and the automation coverage is obtained. By matching the automated code with the functional use cases, it helps us create an intuitive sense of the value that the automated code generates.

  1. Create test plans for automated execution, circle examples.

    Select suitable for automated test subset is need professional knowledge test, and the implementation of fixed scope full regression testing takes too long, or repeated mechanical smoke test execution does not reflect the test of the new requirements in time, this is the ascension of automated test coverage after still failed to reach the expected value. CODING provides an on-demand way to circle test subsets to create test plans, accurately execute relevant automated code subsets, and quickly feedback results, thereby eliminating concerns about the length of the automation runtime and maximizing the value of the automated code that the team has worked hard to produce.

  1. The test plan is executed, and the matching automation cases are executed in the background and the execution results of the corresponding functional cases are updated. After the automation is complete, untested or failed use cases can be manually verified and use case task status updated.

  1. Click generate test report, and the system will automatically analyze and evaluate the quality according to the collected data. Test traceability makes test reporting more convincing and helps teams keep risk to a low level.