Ham Vocke, a consultant at Thoughtworks In Germany, has written a full introduction to the concept of a test pyramid and an example of how to use it on the Martin Fowler website. This article covers many software testing and quality engineering best practices to help readers establish the right software testing philosophy. 译 文 : The Practical Test Pyramid[1]

The “test pyramid” is a metaphor for how to group tests according to their different granularity and how many tests should be included in each group. Although the concept of a test pyramid has been around for some time, teams still need to work hard to make it work in practice. This article reviews the original concept of the test pyramid and shows how to put it into practice, including what types of tests should be covered at different levels of the pyramid, and gives examples of how these tests can be implemented.

Software needs to be tested before it is put into production environment. With the maturity of software development methods, software testing methods are becoming more and more mature. Instead of requiring a large number of manual testers, the development team automates much of the testing effort. Automated testing allows teams to know in seconds or minutes whether a software problem has occurred, rather than waiting days or weeks.

Automated testing reduces feedback loops that go hand in hand with agile development practices, continuous delivery, and DevOps culture, and having effective software testing methods allows teams to test quickly and with confidence.

This article explores what a comprehensive test portfolio should look like to provide fast response, high reliability, and maintainability, whether we’re building microservices architectures, mobile applications, or iot ecosystems. We’ll also delve into the details of building efficient and readable automated tests.

The importance of automation

Software has become such an important part of the world we live in that it has evolved far beyond its early singular goal of making business more efficient. Today, many companies are trying to figure out how to become the best digital companies. As users, each of us interacts with more and more software every day. The wheels of innovation are turning faster and faster.

If we want to keep up with the pace of innovation, we have to find ways to deliver software faster without sacrificing quality. Continuous delivery is one such practice that helps us ensure that software can be released to production at any time in an automated manner. In the continuous delivery practice, we automate test software by building a build pipeline ** and deploying it to test and production environments.

It is increasingly impossible to build, test, and deploy an ever-increasing amount of software manually unless we are willing to spend all our time doing manual repetitive work instead of focusing on delivering software that works. Automating everything — from build to test, deployment and infrastructure — is the only way we can move forward.

Figure 1: Use the build pipeline to automatically and reliably put software into production

Traditional software testing is purely manual, deploying the application into a test environment and then performing some black-box testing, such as clicking on the user interface to see if there is a problem. These tests are typically specified by test scripts to ensure that testers perform consistent checks.

Obviously, testing all changes manually is time-consuming, repetitive, and tedious. Repetition leads to boredom, and boredom leads to mistakes that make you want a different job every weekend.

Fortunately, there is a remedy for repetitive tasks: automation.

As software developers, automating repetitive testing can have a major impact on our lives. By automating these tests, you no longer have to blindly follow point-and-click protocols to check that the software is still working, making it easier to change the code. If you’ve ever tried large-scale refactoring without a proper test suite, I bet you know how scary the experience can be. How do we know if anything has been broken during refactoring? Of course, we can execute all the manual test cases, and that’s one way to do it. But honestly, would anyone really like that? What if we could make a massive change and know if something was broken over a cup of coffee? That sounds more interesting, if you ask me.

Test the Pyramids

If you want to get serious about automated testing of software, there’s one key concept you should know: the test pyramid. Mike Cohn in his book “Succeeding with Agile” changed his life. This is a good metaphor for thinking about different levels of testing and how much testing to do at each level.

Figure 2: Test pyramid

Mike Cohn’s original test pyramid consists of three layers, from bottom to top:

  1. Unit testing
  2. Service test
  3. UI test

Unfortunately, if you look closely, the concept of a pyramid test falls short. Some people think that Mike Cohn’s test pyramid has a less than ideal name or concept, and I agree. From a modern point of view, testing pyramids seems simplistic and therefore potentially misleading.

Still, because of its simplicity, the test pyramid is essentially a good rule of thumb when it comes to building your own test suite, and it’s best to remember the two points Cohn made initially in the test pyramid:

  1. Write tests in different granularity
  2. The higher the hierarchy, the fewer tests should be done

Stick to a pyramid shape to build a healthy, fast, and maintainable test suite, starting with lots of small, fast unit tests, followed by some coarse-grained tests and fewer advanced tests that test your application end-to-end. Be careful not to end up with the ice cream cone test [2], which would be a nightmare to maintain and run.

Don’t get too hung up on the names of each layer in the Cohn test pyramid. In fact, they are very misleading, and service testing is a difficult term to understand (as Cohn himself has mentioned, many developers ignore this layer entirely [3]). Obviously, with single-page application frameworks like React, Angular, ember.js, UI testing doesn’t have to be at the top of the pyramid, because you can unit test your UI in these frameworks.

Given the shortcomings of the initial naming, it is perfectly acceptable to name the test hierarchy by another name as long as it is consistent in the code base and team discussions.

Libraries and tools that we use

  • JUnit[4]: Test actuator
  • Mockito[5]: Simulate dependencies
  • Wiremock[6]: Piling external services
  • Pact[7]: Write CDC tests
  • Selenium[8]: Write UI-driven end-to-end testing
  • Restful [9]: Write restful API-driven end-to-end testing

Sample application

I wrote a simple microservice with a test suite [10] that contains tests for different levels of the test pyramid.

The sample application represents the characteristics of a typical microservice, providing a REST interface, interacting with a database, and getting information from third-party REST services. The program is implemented based on Spring Boot[11], which should be understandable even if you have never used Spring Boot.

Be sure to check out the code [10] on Github, where readme contains instructions for executing the application and its automated tests on a computer.

function

The functionality of the application is simple, providing a REST interface with three endpoints:

GET /hello # always returns "Hello World". GET /hello/{lastName} # Find someone with the supplied lastname and return "hello {Firstname} {lastName}" if found. GET /weather # Return to current weather conditions in Hamburg, Germany.Copy the code

Profile architecture

Externally, the system has the following architecture:

Figure 3: External architecture of a microservice system

Our microservice provides a REST interface that can be invoked through HTTP. For some endpoints, the service gets the information from the database. In other cases, the service will invoke an external weather API[12] via HTTP to retrieve and display the current weather conditions.

Internal architecture

Internally, Spring Service is a typical Spring architecture:

Figure 4: Internal architecture of microservices

  • ControllerThe class provides a REST endpoint and handles HTTP requests/responses
  • RepositoryClass provides the database interface and is responsible for writing and reading data to the persistent store
  • ClientThe class communicates with other apis, and in our case, it gets the JSON response from darkSky.net’s weather API over HTTPS
  • DomainClasses reflect our domain model[13], including domain logic (which, to be honest, is pretty trivial in our example).

Experienced Spring developers may notice that a common layer is missing: inspired by domain-driven Design [14], many developers build a Service layer consisting of service classes. In this application, I decided not to include a service layer, either because the application was simple enough that the service layer would be an unnecessary layer of indirection, or because I thought people were doing too much on the service layer. I often come across codebase where the entire business logic is reflected in the service classes and the domain model becomes only the data layer, not the behavior layer (anaemic domain model [15]). For critical applications, this wastes a lot of potential resources to keep the code well-structured and testable, and underutilizes the power of object orientation.

Our repository is straightforward and provides simple CRUD functionality. To keep the code simple, we used Spring Data[16]. Spring Data provides a simple, universal implementation of the CRUD repository that you can use directly without having to build your own wheels. It also provides an in-memory database for testing, without the need for a real PostgreSQL database in production.

Take a look at the code base and familiarize yourself with the internal architecture, which is very useful for testing your application later!

Unit testing

The foundation of a test suite consists of unit tests that ensure that a unit of the code base is working as expected. Unit tests have the narrowest scope of any test in the test suite, and will vastly outnumber any other type of test.

Figure 5: Unit tests typically replace external dependencies with emulators (Test doubles)

What are unit tests?

If you ask three different people what “unit” means in the context of a unit test, you might get four different, slightly different answers. To some extent, this is a self-defining question, and it doesn’t matter if there is no standard answer.

If functional languages are used, the unit is likely to be a single function. The unit test calls a function with different parameters and ensures that it returns the expected value. In an object-oriented language, a unit can be a single method or a complete class.

Dependence and Solitary

Some argue that all dependencies of the unit under test (for example, other classes called by the class under test) should be replaced with mocks or stubs to achieve perfect isolation and avoid side effects and complex test setups. Others argue that only dependencies that are slow or have large side effects (for example, classes that access databases or make network calls) should be mocked or stubbed.

Sometimes [17], these two types of tests are referred to as solitary unit tests (solitary unit tests, which are stub tests for all dependencies) and dependent unit tests (Solitary Unit tests, Tests that allow interaction with real dependencies) (Jay Fields’ Using Unit Tests Effectively [18] coined these terms). If you have time, you can read more about the pros and cons of these two different schools of thought [19].

It doesn’t really matter whether you do stand-alone unit tests or rely on them. What matters is that you write automated tests. Personally, I use both methods. If using real dependencies is inconvenient, use mocks and stubs. If you want to involve the true dependent parties and make the testing more confident, keep only the outermost layer of the service.

Mocking and Stubbing

Mocks and stubs are two different types of mocks (Test doubles [2]). Many people use the terms Mock and Stub interchangeably, but I think it’s best to be precise and remember their specific properties. Test simulators can help you replace objects that will be used in production during testing.

In short, this means replacing the real thing (such as a class, module, or function) with a mock version. The mock version looks and behaves like the real version (responding to the same method call), but its response is a forged response defined at the beginning of the unit test.

Not only unit tests use emulation, but more sophisticated emulators can be used to simulate entire parts of a system in a controlled manner. In unit testing, however, you’re likely to encounter a lot of mocks and stubs (depending on whether you use dependent or stand-alone unit tests), and many modern languages and libraries make it increasingly convenient to set up mocks and stubs.

Regardless of the technology you choose, there’s a good chance that a standard library for your language or a popular third-party library provides an elegant way to set up mocks. Even if you write your own mock from scratch, you simply write a mock implementation with the same signature as the real class/module/function and set up the mock object in your test.

Unit tests will execute very quickly, and on a normal machine you can expect to run thousands of unit tests in a matter of minutes. Test a small portion of the code base independently, avoiding accessing databases, file systems, or triggering HTTP queries (by using mocks and stubs on those parts) to keep the test speed.

Once you get the hang of writing unit tests, you’ll become more proficient. Simulate external dependencies, set input data, invoke test objects, and check that the return value meets expectations. Study test-driven Development [21] and let unit tests guide Development. When applied correctly, TDD can help us follow good processes and make good and maintainable designs, while automatically generating comprehensive, fully automated test suites. But this is no silver bullet. Give it a real chance and see if it’s right for you.

Testing what?

The advantage of unit testing is that you can write unit tests for all production code classes, regardless of their functionality or which layer they belong to in the internal structure. We can unit test a controller just like we can unit test a repository, a domain class, or a file reader. Just stick to the rule of thumb of one test class per production class and you’re off to a good start.

Unit test classes should at least test the public interface of the class; private methods cannot be tested because they cannot be called from the test class. Protected or package-private interfaces can be accessed from the test class (assuming that the package structure of the test class is the same as that of the production class), but testing these methods may be too much.

When writing unit tests, there is a fine line: They should ensure that all important code paths are tested (both the main path and the edge case) and that they should not be too tightly coupled to the implementation.

Why is that?

Tests that are too close to the production code quickly become annoying. As soon as the production code is refactored (a quick review: Refactoring means changing the internal structure of the code without changing the externally visible behavior), the unit tests fail.

This removes one of the great benefits of unit testing: being a safety net for code changes. You quickly get tired of stupid tests that fail every time you refactor, resulting in more work than help, and you ask whose idea were these stupid tests?

So what to do? Do not reflect the internal structure of your code in unit tests; test for observable behavior. Think about it:

If you input the values x and y, will the result be Z?

Instead of:

If x and y are input, does the method call class A first, then B, and then return the result of class A plus the result of class B?

Private methods are often seen as implementation details, which is why you shouldn’t even feel the urge to test them.

I often hear opponents of unit testing (TDD) argue that writing unit tests is a pointless exercise because all methods must be tested to achieve high test coverage. They often cite scenarios where overeager team leaders force them to write unit tests for getters and setters and all the other trivial code to achieve 100% test coverage.

This is completely wrong.

Yes, you should test the public interface. More importantly, however, don’t test trivial code. Don’t worry, Kent Beck said it’s okay [23]. You won’t gain anything from testing simple getters or setters or other trivial implementations (for example, without any conditional logic). Time saved for another meeting, hooray!

But I really need to test the private method

If you find yourself desperate to test private methods, take a step back and ask yourself why.

I’m pretty sure this is more of a design issue than a test scope issue. You will most likely feel the need to test a private method because it is complex and requires a lot of clunky setup to test it through the class’s public interface.

Whenever I find myself in this situation, I usually conclude that the class I’m testing is already too complex. It does too much and violates the principle of single liability – principle S of the five Principles of SOLID[22].

For me, the solution that usually works is to split the original class into two classes. It usually takes only a minute or two of thought to come up with a good way to divide a large category into two subcategories of single responsibility. I move the private methods (the ones I desperately want to test) into the new class and have the old class call the new method. Great, that private method that was hard to test is now public and easy to test. In addition, I have improved the code structure by adhering to the single responsibility principle.

The test structure

A good structure for all tests (not limited to unit tests) is:

  1. Setting test Data
  2. Call the method under test
  3. The assertion returns the expected result

A good mnemonic for remembering this structure is Arrange, Act, Assert[24]. Another method comes from BDD, the “Given, When, then[25]” triplet, where given reflects the Settings, when is the method call, and then is the assertion part.

This pattern can also be applied to other more advanced tests. In each case, they ensure that the tests remain easy to read and consistent. In addition, tests written with this structure tend to be shorter and more expressive.

Implementing unit tests

Now that we know what to test and how to construct unit tests, we can finally see an example in action.

Take a simplified version of the ExampleController class as an example:

@RestController
public class ExampleController {

    private final PersonRepository personRepo;

    @Autowired
    public ExampleController(final PersonRepository personRepo) {
        this.personRepo = personRepo;
    }

    @GetMapping("/hello/{lastName}")
    public String hello(@PathVariable final String lastName) {
        Optional<Person> foundPerson = personRepo.findByLastName(lastName);

        return foundPerson
                .map(person -> String.format("Hello %s %s!",
                        person.getFirstName(),
                        person.getLastName()))
                .orElse(String.format("Who is this '%s' you're talking about?", lastName)); }}Copy the code

We wrote unit tests based on JUnit (the Java Testing Framework De facto standard) and used Mockito to replace the real PersonRepository class with the stub of the test, thereby defining stub methods in the test to return forged responses. Stub makes testing simpler and more predictable, making it easy to set up test data.

According to arrange, Act, and Assert, we write two unit tests: a normal case test and a test that cannot search for people. The first normal test case creates a new Person object and tells the mock repository to return it when called with “Pan” as the lastName parameter. The test then calls the method under test and finally asserts that the response is equal to the expected response.

The second test works in a similar way, but tests scenarios where a person cannot be found based on a given parameter.

Professional test assistants It’s great that we can write unit tests for an entire code base, regardless of the level of application architecture. This example shows a simple unit test of the controller. Unfortunately, this approach has a disadvantage for Spring controllers: Spring MVC controllers make heavy use of annotations to declare paths to listen to, HTTP operations to handle, parameters to resolve from URL paths or query parameters, and so on. Simply calling the controller in a unit test does not test all of these critical pieces. Fortunately, the Spring folks provide a great testing assistant that you can use to write better controller tests. This tool, MockMVC[60], provides a nice DSL that can be used to trigger mock requests to the controller and check that everything is ok. An example [61] is included in the sample code base. Many frameworks provide testing assistants to help you better test specific parts of your code base. Review the documentation for the frameworks you choose and see if they provide any useful assistance for automated testing.

Integration testing

All important applications will be integrated with other parts (databases, file systems, network calls to other applications). When writing unit tests, these parts are often left out for better isolation and faster testing. However, you still need to test how your application interacts with other parts. This is where integration tests are needed [26], which test the integration of the application with all parts outside the application.

For automated testing, this means not only running the application, but also the components that are integrated with it. If you test integration with the database, you need to run the database when you run the tests. For tests that read files from disk, you need to save the files to disk and load them in the integration test.

As mentioned earlier, the term “unit testing” is vague, especially when it comes to “integration testing.” For some, integration testing means testing the entire stack of applications connected to other applications in the system. I prefer narrower integration testing, testing integration points one at a time and replacing individual services and databases with test simulators. Combining contract tests with running contract tests against test simulators, and real implementations, can build integration tests that are faster, more independent, and often easier to reason about.

Narrow integration tests lie at the boundaries of services, and conceptually they are always about triggering an operation that results in integration with external parts (file systems, databases, standalone services). Database integration tests look like this:

Figure 6: Database integration tests integrate code with a real database

  1. Start the database
  2. Connect the application to the database
  3. A function call that triggers writing data to the database in your code
  4. Reads data from the database and checks that the expected data has been written to the database

Another example is testing the integration of our service with another standalone service through the REST API, as follows:

Figure 7: This integration test checks that the application communicates correctly with the standalone service

  1. Start the application
  2. Start an instance of a standalone service (or a test simulator with the same interface)
  3. Trigger a function call in your code that reads data from the API of a separate service
  4. Check that the application can parse the response correctly

Integration testing (similar to unit testing) can be quite white-box. Some frameworks allow you to start an application while still emulating the rest of the application to check that the correct interaction is taking place.

It is more common than you might think to write integration tests for all blocks of code that serialize or deserialize data. Consider the following situation:

  • Invoke the REST API of the service
  • Read and write to the database
  • Call another application’s API
  • Read and write queues
  • Write to the file system

Writing integration tests around these boundaries ensures that reading/writing data to these external dependencies works.

When writing narrow integration tests *, you should focus on running external dependencies locally: starting a local MySQL database, testing a local ext4 file system. If you want to integrate with a separate service, you can run an instance of that service locally or build and run a mock version that mimics the behavior of the real service.

If you cannot run third-party services locally, you should choose to run a dedicated test instance and point to that test instance when running integration tests to avoid integration with the actual production system in automated tests. Handling thousands of test requests on a production system is a sure way to drive people crazy because you mess up their logs (at best) and even break their services (at worst). Integration with services over the network is typical of broad Integration tests, which make them slow and often difficult to write.

Integration tests are at a higher level on the test pyramid than unit tests. Slow parts such as integrating file systems and databases tend to be much slower than running unit tests using stubs for those parts. They can also be harder to write than small and isolated unit tests, since external dependencies must be handled as part of the test. However, these tests have the advantage of assuring us that the application works correctly with all the external dependencies that need to communicate, whereas unit tests cannot solve this problem.

Database integration

PersonRepository is the only repository class in the codebase that relies on Spring Data and has no actual implementation, just extending the CrudRepository interface and providing a method header. The rest is Spring’s magic.

public interface PersonRepository extends CrudRepository<Person.String> {
    Optional<Person> findByLastName(String lastName);
}
Copy the code

Through the CrudRepository interface, Spring Boot provides a full-featured CRUD library that includes methods such as findOne, findAll, Save, Update, and Delete. The custom method (findByLastName()) extends the basic functionality and provides a way to get a Person by last name. Spring Data analyzes the return type of the method and its method name, and checks the method name against naming conventions to determine what it should do.

Although Spring Data does the heavy lifting for the database, I still wrote a database integration test. You might say this is testing the framework, which is something we should avoid doing because we’re not testing our code. However, I believe there needs to be at least one integration test here. First, we need to test whether our custom method findByLastName actually behaves as expected. Second, prove that our repository uses Spring connections correctly and can connect to the database.

To make it easier to run the tests on our own machine (without installing the PostgreSQL database), the tests were connected to the in-memory database H2.

I defined H2 as a test dependency in build.gradle. The application. Properties in the test directory does not define any Spring. datasource properties, which tells Spring Data to use an in-memory database. When it finds H2 on the CLASspath, it uses it when it runs the test.

When a real application is run using an int profile (for example, by setting SPRING_PROFILES_ACTIVE=int as an environment variable), it connects to the PostgreSQL database defined in application-int.properties.

I know there are a lot of horrible Spring details to know and understand. To do this, you have to look at a lot of documentation [27]. The generated code looks simple, but it can be difficult to understand without knowing the details of Spring.

In addition, using an in-memory database is risky. After all, integration testing is for a different type of database than in production. Go ahead and decide for yourself if you like Spring’s magic and simple code, or its more explicit, verbose implementation.

Enough explained, here’s a simple integration test that saves Person to the database and finds it by its last name:

@RunWith(SpringRunner.class)
@DataJpaTest
public class PersonRepositoryIntegrationTest {
    @Autowired
    private PersonRepository subject;

    @After
    public void tearDown(a) throws Exception {
        subject.deleteAll();
    }

    @Test
    public void shouldSaveAndFetchPerson(a) throws Exception {
        Person peter = new Person("Peter"."Pan");
        subject.save(peter);

        Optional<Person> maybePeter = subject.findByLastName("Pan"); assertThat(maybePeter, is(Optional.of(peter))); }}Copy the code

As you can see, integration tests follow the same Arrange, Act, and Assert structures as unit tests, which is a common concept!

Integration with standalone services

Our microservice communicates with DarkSky.net, a weather REST API. We certainly want to ensure that the service sends the request and parses the response correctly.

We want to avoid using real Darksky services when running automated tests. Quota limits on free plans are only part of the problem; the real reason is decoupling. Our tests should be independent of DarkSky.net and should run even if our machines are unable to access the Darksky server or the Darksky server is down for maintenance.

We ran integration tests by running our own mock Darksky service to avoid using the real Darksky service. This sounds like a daunting task, and it’s easy to do thanks to tools like Wiremock. Look at this:

@RunWith(SpringRunner.class)
@SpringBootTest
public class WeatherClientIntegrationTest {

    @Autowired
    private WeatherClient subject;

    @Rule
    public WireMockRule wireMockRule = new WireMockRule(8089);

    @Test
    public void shouldCallWeatherService(a) throws Exception {
        wireMockRule.stubFor(get(urlPathEqualTo("/ - test - API - key / 53.5511, 9.9937,"))
                .willReturn(aResponse()
                        .withBody(FileLoader.read("classpath:weatherApiResponse.json"))
                        .withHeader(CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                        .withStatus(200)));

        Optional<WeatherResponse> weatherResponse = subject.fetchWeather();

        Optional<WeatherResponse> expectedResponse = Optional.of(new WeatherResponse("Rain")); assertThat(weatherResponse, is(expectedResponse)); }}Copy the code

We instantiate a WireMockRule on the fixed port (8089) to use Wiremock. We can set up the Wiremock service through the DSL, define the endpoint that should listen, and set the response that should be returned.

You then call the method you want to test, which calls the third-party service and checks that the results are properly parsed.

It is important to understand how the test knows that it should call the mock Wiremock server and not the real Darksky API. The secret lies in the application.properties file in SRC /test/ Resources, which is the properties file that Spring loads when you run the test. In this file, we overwrite the configuration with values suitable for testing purposes, such as API keys and urls, so that the test calls the mock Wiremock server instead of the real one:

weather.url = http://localhost:8089
Copy the code

Note that the port defined here must be the same as the one we defined when we instantiated WireMockRule in our test. In our tests, replacing the URL of the real weather API with the URL of the simulated weather API can be done by injecting the constructor of the WeatherClient class:

@Autowired
public WeatherClient(final RestTemplate restTemplate,
                     @Value("${weather.url}") final String weatherServiceUrl,
                     @Value("${weather.api_key}") final String weatherServiceApiKey) {
    this.restTemplate = restTemplate;
    this.weatherServiceUrl = weatherServiceUrl;
    this.weatherServiceApiKey = weatherServiceApiKey;
}
Copy the code

This tells the WeatherClient to read the weatherUrl value from the weather.url parameter we defined in the application properties.

With tools like Wiremock, it is very easy to write narrow integration tests for individual services. Unfortunately, there is a drawback to this approach: how do we ensure that the simulated service we set up behaves like the real service? With the current implementation, the standalone service can change the API and the tests can still pass. For now, we’re just testing whether WeatherClient can parse the response sent by the mock service. It is a start, but very fragile. Using end-to-end testing and running tests against test instances of real services, rather than using mock services, solves this problem but makes us dependent on the availability of the test services. Fortunately, there is a better solution to this dilemma: Running contract tests against mock services and real services ensures that the mock services used in integration tests are trusted replicas. Let’s see how this is done.

The test of contract

Modern software development organizations divide the development of the same system among different teams as a way of extending the development effort. Teams build independent, loosely coupled, non-conflicting services and integrate them into one large, unified system. The recent discussion of microservices has focused on this point.

Splitting a system into many small services usually means that these services need to communicate with each other through specific interfaces (hopefully well defined, but sometimes due to unexpected growth).

Interfaces between different applications can take different forms and technologies. Common ones are:

  • HTTPS based REST and JSON
  • Use RPC such as gRPC
  • Build event-driven architectures through queues

For each interface, there are two parties involved: the producer and the consumer. Producers provide data to consumers, who process the data they get from producers. In REST, producers build a REST API that contains all the required endpoints, and consumers call this REST API to retrieve data or trigger changes in other services. In an asynchronous, event-driven architecture, producers (often called publisher) publish data to queues, and consumers (often called subscriber) subscribe to these queues and read and process the data.

Figure 8: The specification that each interface has a producer (or publisher) and a consumer (or subscriber) interface can be considered a contract.

If the consumption and production of services are dispersed among different teams, interfaces between these services must be explicitly specified (so-called contract contracts). Traditionally, companies have dealt with this problem in the following way:

  • Write a long, detailed interface specification (contract)
  • The service is provided according to the defined contract implementation
  • Throw the interface specification to the consumer team
  • Wait for them to implement the part that uses the interface
  • Run some large-scale manual system tests to see if everything works
  • Let’s hope both teams stick to interface definitions forever and don’t screw it up

More modern software development teams have replaced Steps 5 and 6 with something more automated: automated contract testing [28] to ensure that consumer and producer implementations follow defined contracts. This is a good suite of regression tests to ensure that contract deviations are caught early.

In a more agile organization, a more efficient and less wasteful path should be taken. Building applications in the same organization, we should be able to talk directly to developers of other services, rather than just throw in documents stuffed with details. After all, they are colleagues, not third-party vendors who can only communicate with them through customer support or other legal means stipulated in the contract.

** Consumer-driven Contract tests (CDC Tests, consumer-driven Contract Tests)** Let consumers drive the implementation of contracts [29]. With CDC, consumers of interfaces can write tests that examine the data they need to retrieve from the interface. The consumer team then publishes these tests so that the producer team can easily pick them up and execute them. The service provider team can now develop the API by running CDC tests, and once all the tests pass, it knows that it has achieved what the consuming team needs.

Figure 9: Contract testing ensures that the provider of the interface and all consumers comply with the defined interface contract. With CDC tests, consumers of the interface publish requirements in the form of automated tests that providers continually acquire and execute

This approach allows producer teams to implement only what is really needed (keep it simple, YAGNI(You ain’t gonna need it), etc.). The team providing the interface should constantly acquire and run these CDC tests (in the build pipeline) so that any disruptive changes can be found immediately. If the interface is corrupted, the CDC tests fail, preventing the execution of the change in question. As long as the test stays green, the team can make any changes without worrying about other teams. Consumer-driven contract approaches typically follow the following process:

  • The consumer team writes automated tests based on all consumer expectations
  • Publish tests to producer teams
  • The producer team constantly runs CDC tests and keeps them green
  • Once the CDC tests are interrupted, both sides communicate with each other

CDC testing is a big step toward building an autonomous team if your organization adopts a microservices architecture. CDC testing is an automated way to facilitate team communication, ensuring that interfaces between teams are working at all times. The failed CDC test was a good indicator that we should communicate directly with the affected teams about the upcoming API changes and figure out how we should proceed.

A simple CDC test implementation can be as simple as making a request against an API and asserting what needs to be included in the response, and these tests can then be packaged as executable files (.gem,.jar,.sh) and uploaded to a place where other teams can access them (for example, an artifact repository like Artifactory[30]).

Over the past few years, the CDC approach has become more popular, and several tools have been built to simplify writing and publishing.

Pact[31] is probably the most prominent of these tools, providing a sophisticated way to write tests for consumers and producers, stand-alone service stubs out of the box, and support for exchanging CDC tests with other teams. Pact has been ported to many platforms and can be used with JVM languages, Ruby,.NET, JavaScript, and more.

Pact is a smart choice if you want to try CDC but don’t know how to get started. Reading the documentation [32] can be overwhelming at first, and being patient will help us gain a clear understanding of the CDC, which will help us more easily advocate the use of CDC when working with other teams.

Consumer-driven contract testing can be a real game changer, building autonomous teams that act quickly and confidently. I suggest you research the concept and give it a try. A reliable set of CDC tests is invaluable for rapid testing without breaking other services.

Consumer testing (our team)

Since our microservices use the weather API, it is our responsibility to write consumer tests that define the contract (API) between our microservices and the weather service.

First, we include a library for writing PACT consumer tests in build.gradle:

TestCompile (' au.com.dius:pact-jvm-consumer-junit_2.11:3.5.5)Copy the code

With this library, we can implement consumer tests and use Pact’s mock service:

@RunWith(SpringRunner.class)
@SpringBootTest
public class WeatherClientConsumerTest {

    @Autowired
    private WeatherClient weatherClient;

    @Rule
    public PactProviderRuleMk2 weatherProvider =
            new PactProviderRuleMk2("weather_provider"."localhost".8089.this);

    @Pact(consumer="test_consumer")
    public RequestResponsePact createPact(PactDslWithProvider builder) throws IOException {
        return builder
                .given("weather forecast data")
                .uponReceiving("a request for a weather request for Hamburg")
                    .path("/ - test - API - key / 53.5511, 9.9937,")
                    .method("GET")
                .willRespondWith()
                    .status(200)
                    .body(FileLoader.read("classpath:weatherApiResponse.json"),
                            ContentType.APPLICATION_JSON)
                .toPact();
    }

    @Test
    @PactVerification("weather_provider")
    public void shouldFetchWeatherInformation(a) throws Exception {
        Optional<WeatherResponse> weatherResponse = weatherClient.fetchWeather();
        assertThat(weatherResponse.isPresent(), is(true));
        assertThat(weatherResponse.get().getSummary(), is("Rain")); }}Copy the code

Look closely, you will find WeatherClientIntegrationTest WeatherClientConsumerTest and are very similar. Instead of using Wiremock as a service mock, we used Pact this time. In fact, consumer tests work exactly like integration tests, where we replace the real third-party service with stubs, define the expected response, and check that the client can parse the response correctly. In this sense, WeatherClientConsumerTest itself is a special integrated test. The advantage of this test over wiremock-based tests is that each run generates a pact file (found in target/pact /< pace-name >.json) that describes the expectations for the contract in a special JSON format, This can be used to verify that the stub service behaves like the real service. We can give the contract file to the team providing the interface, and they can write producer tests based on the contracts defined in it, so that they can test whether their API meets all our expectations.

As you can see, this is where the consumer-driven part of the CDC comes in. Users drive the implementation of the interface by describing their expectations, and producers must ensure that all expectations are met and fulfilled. Nothing fancy, no YAGNI or anything like that.

Contract files can be submitted to the producer team in a number of ways; one simple way is to check them into version control and tell the producer team to get the latest version of the contract file. A more advanced approach is to use an artifact repository, a service like Amazon’s S3, or a protocol broker. But we can start with something simple and then make a choice based on our needs.

In a real application, integration testing and client-side consumer testing are not required at the same time. The sample code base includes both methods to show how to use either test. If you want to write CDC tests based on PACT, it is recommended that you stick with the latter. The work of writing tests is the same, and the benefit of using PACT is that you automatically get pact files with contract expectations that other teams can use to easily implement producer tests. Of course, this only makes sense if you can convince other teams to use contracts as well. If that doesn’t work, using a combination of integration tests and Wiremock is a good plan B.

Producer testing (other teams)

Producer testing must be implemented by the person providing the weather API. We are using the public API provided by DarkSky.net. In theory, darksky’s team will implement producer testing on their end to check for any violations of the contract between their application and our service.

It was clear that they didn’t care about our small test application and wouldn’t implement CDC testing for us. This is the big difference between a public-facing API and an organization that adopts microservices. Public apis can’t take every consumer into account, or they won’t grow. In an organization where applications are likely to be targeted at a small number of users, perhaps dozens at most, producer tests for these interfaces can be well written to keep the system stable.

The producer team implements a producer test by taking the contract file and running it against the services provided. The test reads the contract file, s simulates the test data, and runs the service against the expectations defined in the contract file.

Pact has written several libraries to implement producer testing, which are outlined in the GitHub repository [33] and you can choose the library that best fits the current technology stack.

For convenience, we assume that the Darksky API is also implemented in Spring Boot. In this case, the Spring Pact Provider [34] can be used, which can be easily attached to Spring’s MockMVC mechanism. Suppose the DarkSky.net team would implement the following producer tests:

@RunWith(RestPactRunner.class)
@Provider("weather_provider") // same as the "provider_name" in our clientConsumerTest
@PactFolder("target/pacts") // tells pact where to load the pact files from
public class WeatherProviderTest {
    @InjectMocks
    private ForecastController forecastController = new ForecastController();

    @Mock
    private ForecastService forecastService;

    @TestTarget
    public final MockMvcTarget target = new MockMvcTarget();

    @Before
    public void before(a) {
        initMocks(this);
        target.setControllers(forecastController);
    }

    @State("weather forecast data") // same as the "given()" in our clientConsumerTest
    public void weatherForecastData(a) {
        when(forecastService.fetchForecastFor(any(String.class), any(String.class)))
                .thenReturn(weatherForecast("Rain")); }}Copy the code

As you can see, all the producer tests need to do is load pact files (for example, use the @PactFolder annotation to load the pact file you downloaded earlier), and then define how to provide test data with predefined states (for example, using Mockito). There is no need to implement custom tests; all this comes from contract files. It is important that the producer test matches the producer name and state declared in the consumer test.

Producer testing (our team)

We’ve seen how to test the contract between our service and the weather provider. Through this interface, our services act as consumers and our weather services act as producers. Further thinking, our service also acts as a producer for others: we provide a REST API that provides several endpoints for others to use.

Since we’ve just learned that contract tests are very popular, we’ve of course written contract tests for this contract as well. Fortunately, we were using consumer-driven contracts, so all the consumer teams sent us contracts that we could use to implement producer testing of the REST API.

Let’s start by adding Spring’s Pact Provider library to the project:

TestCompile (' au.com.dius:pact-jvm-provider-spring_2.12:3.5.5)Copy the code

To implement producer tests, we follow the same pattern as described earlier. For simplicity, we simply check into the repository the contract files provided by the consumer [35], which makes this easier. In a real world scenario, a more complex mechanism might be used to distribute pact files.

@RunWith(RestPactRunner.class)
@Provider("person_provider")// same as in the "provider_name" part in our pact file
@PactFolder("target/pacts") // tells pact where to load the pact files from
public class ExampleProviderTest {

    @Mock
    private PersonRepository personRepository;

    @Mock
    private WeatherClient weatherClient;

    private ExampleController exampleController;

    @TestTarget
    public final MockMvcTarget target = new MockMvcTarget();

    @Before
    public void before(a) {
        initMocks(this);
        exampleController = new ExampleController(personRepository, weatherClient);
        target.setControllers(exampleController);
    }

    @State("person data") // same as the "given()" part in our consumer test
    public void personData(a) {
        Person peterPan = new Person("Peter"."Pan");
        when(personRepository.findByLastName("Pan")).thenReturn(Optional.of (peterPan)); }}Copy the code

The ExampleProviderTest needs to provide state from the PACT file, and that’s it. Once we run the producer tests, Pact takes the Pact file and makes an HTTP request against our service, which then responds based on the set state.

UI test

Most applications have some kind of user interface. In general, in the context of web applications, you are talking about web interfaces. People often forget that REST apis or command-line interfaces are just as much user interfaces as fancy Web user interfaces.

UI tests are used to test that an application’s user interface works correctly. User input should trigger the correct action, the data should be presented to the user, and the UI state should change as expected.

UI testing and end-to-end testing are sometimes thought of as the same thing (as in Mike Cohn’s case). To me, this conflates two orthogonal concepts.

Yes, testing an application end-to-end usually means driving testing through a user interface. However, the reverse is not true.

Testing the user interface does not have to be done in an end-to-end manner. Depending on the technology used, testing the user interface can be as simple as writing unit tests for the front-end javascript code.

In traditional Web applications, user interface testing can be done through tools like Selenium[8]. If the REST API is the user interface, then appropriate integration tests should be written against the API.

With a Web interface, you may need to test the UI in a number of areas: behavior, layout, usability, and compliance with your company’s design, but these are just a few.

Fortunately, testing the behavior of the user interface is simple. Click here, enter data there, and hope that the state of the user interface changes accordingly. Modern single-page application frameworks (React [36], vue.js[37], Angular[38], etc.) often have their own tools and helper classes that allow us to thoroughly test interactions in a low-level way (unit testing). Even if we implement our own front end using native javascript, we can use regular testing tools like Jasmine[39] or Mocha. For more traditional, server-rendered applications, Selenium based testing will be the best choice.

Testing the layout of your Web application can be difficult, and depending on your application and user needs, you may want to make sure that code changes don’t break the layout of your site.

The problem is that computers are notoriously bad at checking whether something “looks good” (perhaps some clever machine learning algorithm could change that in the future).

If you want to automatically check the design of your Web application during the build process, you can try some tools. Most of these tools are based on Selenium to open web applications in different browsers and formats, take screenshots and compare them to previous screenshots. If there is an unexpected difference between the old and new screenshots, the tool reports it.

Galen[41] is one of these tools. However, if there are special requirements, it is not difficult to develop your own solution. To achieve a similar goal, some of the teams I’ve worked with have built alineup [42] and its Java version, Jlineup [43], both of which employ the Selenium based approach described earlier.

Once you want to test usability and the “look good” factor, leave the field of automated testing and rely on exploratory testing [44], usability testing (which can even be as simple as hallway testing [45]), or showing users whether they like using the product, Whether you can use all the features without getting frustrated or annoyed.

End to end testing

Driving a deployed application through the user interface is one way to test the application end-to-end, and the WebDriver-driven UI testing described earlier is a good example of end-to-end testing.

Figure 11: End-to-end testing of a fully integrated system

End-to-end Tests (also known as Board Stack Tests[46]) give us the most confidence when we need to determine whether the software is working. Selenium[8] and the WebDriver protocol [47] allow deployed services to be automatically driven through a (headless) browser to perform operations such as clicking, entering data, and checking user interface state. We can use Selenium directly or use tools built on top of it, Nightwatch[48] being one of them.

End-to-end testing has its own problems. They are notoriously fragile, often failing for unexpected and unforeseen reasons, and these failures are often false positives. The more complex the user interface, the more volatile the tests. Browser quirks, timing issues, animations, and unexpected pop-up dialogs are just a few of the problems that take a lot of debugging time.

Another big issue with microservices architecture is who is responsible for writing these tests. Because it spans multiple services (the entire system), there is no single team responsible for writing end-to-end tests.

If you have a focused quality assurance team, they look like a good fit. However, a centralized QA team is itself a classic anti-pattern that should not be seen in the DevOps world. In the DevOps world, teams should be truly cross-functional. So there is no simple answer to who should be responsible for end-to-end testing, perhaps there is a community of practice or quality association within your organization that deals with these issues, so finding the right answer is very much up to your organization.

In addition, end-to-end testing requires a lot of maintenance and is very slow to run. Consider a scenario where there are multiple microservices and you can’t even run end-to-end tests locally because that would require all microservices to be started locally. Unless you’re lucky, running hundreds of applications on your development machine is a sure-fire way to run out of memory.

Because end-to-end testing is expensive to maintain, you should strive to minimize the number of end-to-end tests.

Think about what is the high-value interaction between the user and the application, try to define the user journey of the core value of the product, and translate the most important steps of the user journey into automated end-to-end testing.

If you’re building an e-commerce site, the most valuable customer journey is probably when the user searches for the product, puts it in the shopping basket, and checks out, and that’s it. As long as the journey continues, there won’t be much trouble. You may also find one or two more important user journeys that you can turn into end-to-end testing. Any more might cause more pain than good.

Remember: There are many lower levels in the test pyramid, where we have tested various edge cases and integration with other parts of the system, and there is no need to repeat these tests at higher levels. It was only a matter of time before high maintenance costs and a large number of false positives slowed us down and led to a loss of trust in testing.

User interface end-to-end testing

Selenium[8] and the WebDriver protocol [47] are the tools of choice for many developers for end-to-end testing. With Selenium, we can choose our favorite browser and have it automatically call the site, dot here and there, enter data, and check the content of the user interface for changes.

Selenium requires a browser that can be launched and used to run tests. Different “drivers” can be used for different browsers. Select one [49](or more) and add them to build.gradle. Regardless of which browser you choose, you need to ensure that your team and all the developers on the CI server have the correct version of the browser installed locally. Keeping in sync can be a pain, and for Java, there is a nice little tool, WebDriverManager [50], that automatically downloads and sets up the correct version of the browser you want to use. Add these two dependencies to build.gradle and you’re ready:

TestCompile (' org. Seleniumhq. Selenium, selenium - chrome - driver: 2.53.1 ') TestCompile (' IO. Making. Bonigarcia: webdrivermanager: 1.7.2 ')Copy the code

Running a full-fledged browser in the test suite can be cumbersome, especially if the Server running the pipeline fails to launch the browser and user interface (for example, because there is no X-Server available) while the practice continues to deliver. You can solve this problem by starting a virtual X-Server like XVFB [51].

The latest trend is to run WebDriver tests using a * headless * browser (that is, a browser without a user interface). Until recently PhantomJS[52] was the leader in headless browsers for automated testing, and PhantomJS has suddenly become obsolete since Chromium[53] and Firefox[54] announced headless mode in browsers. After all, it’s better and more convenient for developers to test a site using a browser that users actually use, such as Firefox and Chrome, than using a mock browser.

Headless Firefox and Chrome are relatively new and have not yet been widely adopted for WebDriver testing. We wanted to make things easier, and instead of wasting time using cutting-edge headless patterns, stick with Selenium and the classic way of using regular browsers. A simple end-to-end test to launch Chrome, navigate to our services, and check the site’s content. Such tests are as follows:

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class HelloE2ESeleniumTest {

    private WebDriver driver;

    @LocalServerPort
    private int port;

    @BeforeClass
    public static void setUpClass(a) throws Exception {
        ChromeDriverManager.getInstance().setup();
    }

    @Before
    public void setUp(a) throws Exception {
        driver = new ChromeDriver();
    }

    @After
    public void tearDown(a) {
        driver.close();
    }

    @Test
    public void helloPageHasTextHelloWorld(a) {
        driver.get(String.format("http://127.0.0.1:%s/hello", port));

        assertThat(driver.findElement(By.tagName("body")).getText(), containsString("Hello World!")); }}Copy the code

Please note that if You have Chrome installed on the system on which you are running this test, the test will only run on that system (your local machine, or CI server).

Testing is simple, start the entire application on a random port using @Springboottest, then instantiate a new Chrome WebDriver, navigate to our microservice’s /helloendpoint, and check if “Hello World!” is printed on the browser window. . Cool!

REST API end-to-end testing

It’s a good idea to avoid using a graphical user interface when testing your application, which is more reliable than a full end-to-end test, while still covering most of the application. This comes in handy when testing through an application’s Web interface is particularly difficult. Maybe there’s not even a Web UI, but just a REST API(because there’s a one-page application that communicates with the API, or just because you don’t like the pretty stuff). Either way, testing under a graphical user interface through the Subcutaneous Test[55] will give us the confidence to go further. Sample code for a REST API test is as follows:

@RestController
public class ExampleController {
    private final PersonRepository personRepository;

    // shortened for clarity

    @GetMapping("/hello/{lastName}")
    public String hello(@PathVariable final String lastName) {
        Optional<Person> foundPerson = personRepository.findByLastName(lastName);

        return foundPerson
             .map(person -> String.format("Hello %s %s!",
                     person.getFirstName(),
                     person.getLastName()))
             .orElse(String.format("Who is this '%s' you're talking about?", lastName)); }}Copy the code

The REST-Assured [9] library is very helpful when testing the REST API’s services, providing a good DSL for triggering real HTTP requests against the API and evaluating the response received.

First, add the dependencies to build.gradle.

TestCompile (' IO. Rest assured: rest assured: 3.0.3 ')Copy the code

Based on this library, you can implement end-to-end testing for REST apis:

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class HelloE2ERestTest {

    @Autowired
    private PersonRepository personRepository;

    @LocalServerPort
    private int port;

    @After
    public void tearDown(a) throws Exception {
        personRepository.deleteAll();
    }

    @Test
    public void shouldReturnGreeting(a) throws Exception {
        Person peter = new Person("Peter"."Pan");
        personRepository.save(peter);

        when()
                .get(String.format("http://localhost:%s/hello/Pan", port))
        .then()
                .statusCode(is(200))
                .body(containsString("Hello Peter Pan!")); }}Copy the code

We also launch the entire Spring application through @Springboottest. In this case, we define PersonRepository with @autowire so that the test data can be easily written to the database. Now, when we ask the REST API to say “hello” to “Mr Pan”, we get a greeting. Amazing!!! If we didn’t have a Web interface, end-to-end testing would be enough.

Acceptance Testing – Does your feature work?

The higher you move up the test pyramid, the more likely you are to enter the realm of testing whether the functionality you are building works correctly from the user’s point of view. Here you can think of the application as a black box and change the focus of testing from

When I enter values x and y, the return value should be z

become

*** Assuming (given)* the user has logged in *** and (and)* has a “bike” product *** When (when)* the user navigates to the “Bike” details page *** and (and)* clicks the “Add to Basket” button then (then) the “bike” should be in the basket

Sometimes we hear the terms functional testing [56] or acceptance testing [57]. Some people think functional testing and acceptance testing are different things, while others think they are the same thing. People will argue endlessly about wording and definitions, and this discussion usually leads to considerable distress.

Here’s the thing: We should make sure our software works correctly from a user perspective, not just a technical one. It doesn’t really matter what you call these tests, pick a term, be consistent, and write tests.

This is also the time when people talk about BDD and implement test tools as BDD. BDD or BDD-style test writing is a great way to shift thinking from implementation details to user requirements. Go and have a try.

You don’t even need to use a mature BDD tool like Cucumber[58] (although you can). Some assertion libraries, such as chai.js[59], allow assertions to be written using Should-style keywords, which can make tests read more like BDD. Even without using a library that provides this notation, well-built code will allow us to write tests that focus on user behavior. Some helper methods/functions can help us go far:

# a sample acceptance test in Python

def test_add_to_basket() :
    # given
    user = a_user_with_empty_basket()
    user.login()
    bicycle = article(name="bicycle", price=100)

    # when
    article_page.add_to_.basket(bicycle)

    # then
    assert user.basket.contains(bicycle)
Copy the code

Acceptance tests can have different levels of granularity. In most cases, the service is tested at a fairly high level and through the user interface. Technically, however, it is not necessary to write acceptance tests at the highest level of the test pyramid. If the design and scenario of the application allow us to write acceptance tests at a lower level, do so; it is better to do low-level testing than high-level testing. The concept of acceptance testing — proving that your features are correct for users — is completely orthogonal to the test pyramid.

Exploratory testing

Even the most assiduously automated testing is not perfect, sometimes edge cases are missed in automated testing, sometimes it is almost impossible to write unit tests to detect specific bugs, and some quality issues don’t even show up in automated testing (for design or usability reasons). Although we want our tests to be automated, manual testing is still a good idea.

Figure 12: Use exploratory testing to discover quality issues that the build pipeline missed

Exploratory testing [44] is a manual testing method that emphasizes the freedom and creativity of testers in finding quality problems in operating systems. Just spend some time on a regular schedule, roll up your sleeves, and try to break the application under test, with a destructive mindset, and figure out how to cause problems and bugs in the application. Keep a record of everything you find for future use. Watch out for bugs, design issues, slow response times, missing or misleading error messages, and anything else that annoys you as a user of software.

The good news is that we can happily automate most of our discoveries using automated tests. Writing automated tests for bugs found ensures that the same bugs don’t recur in the future. In addition, it helps us narrow down problems as we fix bugs.

During exploratory testing, problems that went unnoticed in the build pipeline can be discovered. Don’t get depressed. This is a good indication of the maturity of the build pipeline. With any feedback, make sure to take action: think about what can be done to avoid such problems in the future, we might miss out on a set of automated tests, maybe it’s just in this iteration hasty automated testing, and require a more thorough test in the future, there may be a new tool or method allows us to avoid this kind of problem in the future work. Make sure we’re doing something about it to help streamline and overall software delivery become more mature over time.

Confusion about testing terminology

It’s always difficult to talk about different test categories, and what I mean by unit testing may be slightly different from what you think, and in integration testing, it’s even worse. For some, integration testing is a very broad activity that involves testing many different parts of the entire system. For me, this is a fairly rigorous affair, testing the integration of one external dependency at a time. Some call it integration test, some component test, and some service test. One might even say that these three terms are completely different things. There is no right or wrong, and the software development community has not settled on a well-defined term for testing.

Don’t get too hung up on ambiguity, whether it’s called end-to-end testing, a Board Stack Test, or a functional Test. Maybe integration testing means different things to you and people in other companies, and it doesn’t matter. It would be nice if there were some clearly defined terms and broad consensus. Unfortunately, this is not realistic. Because of all the nuances in writing tests, it’s really more like an obscure scope than a set of discrete, unambiguous categories, which makes it difficult to agree on a name.

The important thing is to find the terminology that works for you and your team. Think about the different types of tests to be written, agree on a name within the team, and agree on the scope of each test type. The real concern is whether this can be done within a team (or even an organization). Simon Stewart[62] summed it up nicely in describing the approach they use at Google, which I think is a good example of not getting too hung up on names and naming conventions.

Integrate tests into the deployment pipeline

For teams that practice continuous integration or continuous delivery, there must be a deployment pipeline [63] that can run automated tests every time a software change occurs. Typically, the pipeline is broken down into phases that give us more confidence in deploying software into production. So how do you put different types of tests on the deployment pipeline? To answer this question, consider the fundamental value of continuous delivery (actually one of the core values of EXTREME programming and agile software development [64]) : rapid feedback.

A good build pipeline lets us know as soon as we screw up. No one likes to wait an hour to find out that the latest change broke some simple unit test. If the assembly line takes that long to give feedback, chances are you’ve already gone home. By placing fast-running tests early in the pipeline, feedback can be obtained in seconds, or even minutes. Conversely, longer-running tests — typically tests with a larger scope — can be placed at a later stage to avoid delays in getting feedback from fast-running tests. As you can see, the stages that define the deployment pipeline are driven not by the type of test, but by the speed and scope of the test. Remember, the decision to put some smaller, fast-running integration tests and unit tests in the same phase made a lot of sense because it gave us faster feedback without needing to draw a line in the middle of the test type.

Avoid repeated testing

Now that you know that you should write different types of tests, there is another pitfall to avoid: repeating tests at different levels of the pyramid. While it might be tempting to say there aren’t many tests, I assure you there are. Every test in a test suite is not free, there is an added burden, time to write and maintain tests, time to read and understand others’ tests, and, of course, time to run tests.

As with production code, keep it simple and avoid duplication. Two rules of thumb to keep in mind in the context of implementing a test pyramid:

  1. If a higher-level test finds an error and there is no lower-level test coverage, a lower-level test needs to be written
  2. Push tests as far down as possible

The first rule is important because low-level testing can better narrow down errors and reproduce problems in an independent manner. It can run faster and less bloated when debugging problems. They will be good regression tests for the future. The second rule is important to keep your test suite up to speed. If all conditions have been confidently tested in lower-level tests, there is no need to keep higher-level tests in the test suite, since these tests do not increase confidence in working software. On a daily basis, repeated tests can become annoying, the test suite gets slower and slower, and more tests need to be changed as the code changes its behavior.

Let’s put it another way: if higher levels of testing give us more confidence that the application is working correctly, then we should keep them. Writing unit tests for the Controller class helps test the logic of the Controller itself. But there is no way to tell whether the REST endpoint provided by the controller actually responds to the HTTP request. So, we can move the test pyramid up and add a test to check it precisely, but that’s about it. We don’t have to test all of the conditional logic and edge cases. There’s no need to test in high-level tests what low-level tests already cover. Make sure high-level tests focus on what low-level tests don’t cover.

I’m serious when it comes to eliminating tests that don’t provide any value, and I remove advanced tests that are already covered at lower levels (because they don’t provide additional value). If possible, I’ll replace higher-level tests with lower-level tests. Sometimes it’s hard, especially if you know how hard it is to come up with a test. Beware of the sunk cost fallacy and cut out redundant tests. There is no reason to waste more valuable time on tests that no longer provide value.

Write clean test code

As with writing code, writing good, clean test code requires great care. Here are some tips on how to write maintainable test code before using an automated test suite:

  1. Test code is just as important and needs the same level of care and attention as production code. “This is just test code“Is not a valid reason to be sloppy with your test code
  2. Test one condition at a time, which helps keep the test short and easy to understand
  3. “arrange, act, assert“Or”given, when, then“Is a good way to maintain a good test structure
  4. Readability is important, so don’t go DRY, and if you can improve readability, you can accept copying. Try to find a balance between DRY And DAMP(Descriptive And Meaningful Phrases)[65] code
  5. When in doubt, the “rule of three “[66] is used to decide when to refactor. Use this rule before reuse

conclusion

That’s all! I know this article explaining why and how to test software is long and hard, but the good news is that the information is timeless and type independent. Whether it’s microservices, iot devices, mobile applications, or Web applications, the lessons learned here can be applied to all of these areas.

I hope there is something useful in this article for you. You can go ahead and look at the sample code and introduce some of the concepts explained here into your own test suites. Having a solid test portfolio takes effort. Trust me, it will pay off in the long run and make your life as a developer much calmer.

References: [1] The Practical Test Pyramid: martinfowler.com/articles/pr… [2] Testing Pyramids: watirmelon blog/Testing – pyr… [3] The forgotten layer of The test automation pyramid: www.mountaingoatsoftware.com/blog/the-fo… [4] JUnit: junit.org/ [5] Mockito: site.mockito.org/ [6] Wiremock: wiremock.org/ [7] Pact: docs.pact.io/ [8] Selenium: Docs.seleniumhq.org/ [9] REST assured: github.com/rest-assure… [10] Sprint – testing: github.com/hamvocke/sp… [11] Sprint the Boot: projects. Spring. IO/spring – the Boot… [12] Darksky: darksky.net/ [13] Domain Model: en.wikipedia.org/wiki/Domain… [14] Domain – Driven Design: en.wikipedia.org/wiki/Domain… [15] Anemic Domain Model: en.wikipedia.org/wiki/Anemic… [16] Sprint Data: the projects. Spring. IO/spring – the Data… [17] Unit Test: martinfowler.com/bliki/UnitT… [18] Working Effectively with the Unit Tests: leanpub.com/wewut [19] away, aren ‘t stubs: martinfowler.com/articles/mo… [20] Test Double: martinfowler.com/bliki/TestD… [21] the Test – Driven Development: en.wikipedia.org/wiki/Test-d… [22] SOLID: en.wikipedia.org/wiki/SOLID_… [23] How deep are your unit tests: stackoverflow.com/questions/1… [24] Arrange, Act, Assert: xp123.com/articles/3a… [25] Given the When, Then: martinfowler.com/bliki/Given… [26] Integration Test: martinfowler.com/bliki/Integ… [27] Boot features embeded database support: docs. [28] Contract Test: martinfowler.com/bliki/Contr… [29] Consumer Driven Contracts: martinfowler.com/articles/co… [30] JFrog Artifactory: www.jfrog.com/artifactory… [31] : Pact github.com/realestate-… [32] Pact Document: docs. Pact. IO / [33] Pact- JVM: github.com/DiUS/pact-j… [34] Pact JVM Provider Spring: github.com/DiUS/pact-j… [35] Spring Testing Consumer: github.com/hamvocke/sp… [36] React: facebook.github.io/react/ [37] Vue.js: vuejs.org/ [38] Angular: angular.io/ [39] Jasmine: jasmine.github.io/ [40] Mocha: mochajs.org/ [41] Galen Framework: galenframework.com/ [42] lineup: Github.com/otto-de/lin… [43] jlineup: github.com/otto-de/jli… [44] Exploratory Testing: en.wikipedia.org/wiki/Explor… [45] Hallway Testing: en.wikipedia.org/wiki/Usabil… [46] Board Stack Test: martinfowler.com/bliki/Broad… [47] WebDriver: www.w3.org/TR/webdrive… [48] Nightwatch: nightwatchjs.org/ [49] Selenium Driver: www.mvnrepository.com/search?q=se… [50] webdrivermanager: github.com/bonigarcia/… [51] Xvfb: en.wikipedia.org/wiki/Xvfb [52] PhantomJS: phantomjs.org/ [53] Headless Chrome: Developers.google.com/web/updates… [54] Firefox Headless Mode: developer.mozilla.org/en-US/Firef… [55] Subcutaneous Test: martinfowler.com/bliki/Subcu… [56] Functional Testing: en.wikipedia.org/wiki/Functi… [57] Acceptance tsting in extreme programming: en.wikipedia.org/wiki/Accept… [58] Cucumber: Cucumber. IO / [59] Chai: chaijs.com/guide/style… [60] Spring MVC Test Server: docs. Spring. IO/Spring/docs… [61] ExampleControllerAPITest: github.com/hamvocke/sp… [62] the Test Sizes: testing.googleblog.com/2010/12/tes… [63] Deployment Pipeline: martinfowler.com/bliki/Deplo… [64] Values of XP: www.extremeprogramming.org/values.html [65] What does DAMP not DRY mean when talkingabout unit tests: Stackoverflow.com/questions/6… [66] Rule of Three: blog.codinghorror.com/rule-of-thr…

Hello, MY name is Yu Fan. I used to do R&D in Motorola, and now I am working in Mavenir for technical work. I have always been interested in communication, network, back-end architecture, cloud native, DevOps, CICD, block chain, AI and other technologies. The official wechat account is DeepNoMind