1. Classification of reporting framework

  • XUnit Style XML report
  • HTML report for the MVN Surefire plug-in
  • Allure2 Multilingual test report

2. Generate test reports

2.1 xUnit Style XML report

All XUnit test reports are applicable, such as Java Junit, Python PyTest, etc

  • 1) The following test classes are availableJunit4DemoTest
package junit4demo;

import org.junit.*;
import org.junit.runners.MethodSorters;
import static org.junit.Assert.assertTrue;

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class Junit4DemoTest {

    @BeforeClass
    public static void beforeAllTestCase(a){
        System.out.println("I'm @beforeClass, I'm the first step");
    }

    @AfterClass
    public static void afterAllTestCase(a){
        System.out.println("I'm @afterClass. I'm the last step.");
    }

    @Before
    public void beforeTestCase(a){
        System.out.println("I'm @Before, come to me Before use case execution.");
    }

    @After
    public void afterTestCase(a){
        System.out.println("I'm @after, come to me After the use case executes.");
    }

    @Test
    public void testDemoC(a){
        System.out.println("testDemoC");
        assertTrue(true);
    }

    @Test
    public void testDemoA(a){
        System.out.println("testDemoA");
        assertTrue(false );
    }

    @Test
    public void testDemoB(a){
        System.out.println("testDemoB");
        assertTrue(true); }}Copy the code

MVN test-dtest =Junit4DemoTest; Execution after we can target. The surefire – report. Surefile suite path TEST – junit4demo. See file Junit4DemoTest. XML, which is xUnit style XML report

  • testsuiteDisplay the name of the test class, the execution time, the number of cases executed, the number of errors, the number of skips and the number of failures:
  • propertyDefines the execution environment, Java and Java library version, etc. :
  • testcaseShows the name of the test class and the execution time of the test class
  • If a use case fails, the failure information is printed under the Failure TAB, and the results of the run before the failure are printed:
  • The full report is as follows:
    <?xml version="1.0" encoding="UTF-8"? >
    <testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" name="junit4demo.Junit4DemoTest" time="0.234" tests="3" errors="0" skipped="0" failures="1">
      <properties>
        <property name="gopherProxySet" value="false"/>
        <property name="awt.toolkit" value="sun.lwawt.macosx.LWCToolkit"/>
        <property name="file.encoding.pkg" value="sun.io"/>
        <property name="java.specification.version" value="1.8"/>
        <property name="sun.cpu.isalist" value=""/>
        <property name="sun.jnu.encoding" value="UTF-8"/>
        <property name="java.class.path" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/test-classes:/Users/qinzhen/Documents/TestDev/MyTraining/XUnit / target/classes: / Users/qinzhen/m2 / repository/org/testng/testng / 6.14.3 / testng - 6.14.3. Jar: / Users/qinzhen/m2 / repository/c Om/beust jcommander / 1.72 / jcommander - 1.72. The jar: / Users/qinzhen/m2 / repository/org/apache - extras/beanshell/BSH/b6 / BSH - 2.0 2.0 B6. Jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - engine / 5.5.2 / junit - Jupiter - engine - 5.5.2. The jar: / Users / Qinzhen/m2 / repository/org/API / 1.1.0 apiguardian apiguardian/apiguardian - - API - 1.1.0. Jar: / Users/qinzhen/m2 / repository/org Junit/platform/junit - platform - engine / 1.5.2 / junit - platform - engine - 1.5.2. Jar: / Users/qinzhen/m2 / repository/org/opentest4j / opentest4j / 1.2.0 / opentest4j - 1.2.0. Jar: / Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - Commons / 1.5.2 / jun It - platform - the Commons - 1.5.2. Jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - API / 5.5.2 / junit - Jupiter - API - 5.5.2. The jar: / Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - runner / 1.5.1 / junit - platform - runner - 1.5.1. Jar: / The Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - the launcher / 1.5.1 / junit - platform - the launcher - 1.5.1. Jar: / Users/q Inzhen/m2 / repository/org/junit/platform/junit - platform - suite - API / 1.5.1 / junit - platform - suite - API - 1.5.1. Jar: / Users/qinzhe N/m2 / repository/junit/junit / / junit 4.12-4.12. The jar: / Users/qinzhen/m2 / repository/org/hamcrest/hamcrest - core / 1.3 / hamcrest - c Ore - 1.3. The jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter / 5.5.0 - M1 / junit - Jupiter - 5.5.0 - M1. Jar: / Users/qin Zhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - params / 5.5.0 - M1 / junit - Jupiter - params - 5.5.0 - M1. Jar: / Users/qinzhen/m2 / repository/org/apache/maven/surefire/surefire - testng - utils / 2.22.0 / surefire - testng - utils - 2.22.0. Jar: / Users/qinzhen/m2 / r Epository/org/apache/maven/surefire/surefire - grouper 2.22.0 / surefire - grouper - 2.22.0. Jar."/>
        <property name="java.vm.vendor" value="Oracle Corporation"/>
        <property name="sun.arch.data.model" value="64"/>
        <property name="java.vendor.url" value="http://java.oracle.com/"/>
        <property name="user.timezone" value=""/>
        <property name="java.vm.specification.version" value="1.8"/>
        <property name="os.name" value="Mac OS X"/>
        <property name="user.country" value="CN"/>
        <property name="sun.java.launcher" value="SUN_STANDARD"/>
        <property name="sun.boot.library.path" value="/ Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib"/>
        <property name="sun.java.command" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/surefire/surefirebooter7582774321603719140.jar /Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/surefire 2019-10-15T08-12-59_318-jvmRun1 surefire5421147855410474454tmp surefire_03665361789485499624tmp"/>
        <property name="test" value="Junit4DemoTest"/>
        <property name="surefire.test.class.path" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/test-classes:/Users/qinzhen/Documents/TestDev/MyTraining/XUnit / target/classes: / Users/qinzhen/m2 / repository/org/testng/testng / 6.14.3 / testng - 6.14.3. Jar: / Users/qinzhen/m2 / repository/c Om/beust jcommander / 1.72 / jcommander - 1.72. The jar: / Users/qinzhen/m2 / repository/org/apache - extras/beanshell/BSH/b6 / BSH - 2.0 2.0 B6. Jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - engine / 5.5.2 / junit - Jupiter - engine - 5.5.2. The jar: / Users / Qinzhen/m2 / repository/org/API / 1.1.0 apiguardian apiguardian/apiguardian - - API - 1.1.0. Jar: / Users/qinzhen/m2 / repository/org Junit/platform/junit - platform - engine / 1.5.2 / junit - platform - engine - 1.5.2. Jar: / Users/qinzhen/m2 / repository/org/opentest4j / opentest4j / 1.2.0 / opentest4j - 1.2.0. Jar: / Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - Commons / 1.5.2 / jun It - platform - the Commons - 1.5.2. Jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - API / 5.5.2 / junit - Jupiter - API - 5.5.2. The jar: / Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - runner / 1.5.1 / junit - platform - runner - 1.5.1. Jar: / The Users/qinzhen/m2 / repository/org/junit/platform/junit - platform - the launcher / 1.5.1 / junit - platform - the launcher - 1.5.1. Jar: / Users/q Inzhen/m2 / repository/org/junit/platform/junit - platform - suite - API / 1.5.1 / junit - platform - suite - API - 1.5.1. Jar: / Users/qinzhe N/m2 / repository/junit/junit / / junit 4.12-4.12. The jar: / Users/qinzhen/m2 / repository/org/hamcrest/hamcrest - core / 1.3 / hamcrest - c Ore - 1.3. The jar: / Users/qinzhen/m2 / repository/org/junit/Jupiter/junit - Jupiter / 5.5.0 - M1 / junit - Jupiter - 5.5.0 - M1. Jar: / Users/qin Zhen/m2 / repository/org/junit/Jupiter/junit - Jupiter - params / 5.5.0 - M1 / junit - Jupiter - params - 5.5.0 - M1. Jar: / Users/qinzhen/m2 / repository/org/apache/maven/surefire/surefire - testng - utils / 2.22.0 / surefire - testng - utils - 2.22.0. Jar: / Users/qinzhen/m2 / r Epository/org/apache/maven/surefire/surefire - grouper 2.22.0 / surefire - grouper - 2.22.0. Jar."/>
        <property name="sun.cpu.endian" value="little"/>
        <property name="user.home" value="/Users/qinzhen"/>
        <property name="user.language" value="zh"/>
        <property name="java.specification.vendor" value="Oracle Corporation"/>
        <property name="java.home" value="/ Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre." "/>
        <property name="basedir" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit"/>
        <property name="file.separator" value="/"/>
        <property name="line.separator" value="& # 10;"/>
        <property name="java.vm.specification.vendor" value="Oracle Corporation"/>
        <property name="java.specification.name" value="Java Platform API Specification"/>
        <property name="java.awt.graphicsenv" value="sun.awt.CGraphicsEnvironment"/>
        <property name="surefire.real.class.path" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/surefire/surefirebooter7582774321603719140.jar"/>
        <property name="sun.boot.class.path" value="/ Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib/resources. The jar: / Library/Java/JavaVirtualMachine S/jdk1.8.0 _221. JDK/Contents/Home/jre/lib/rt. The jar: / Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib / sunrsasign. Jar: / Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib/jsse jar: / Library/Java/JavaVirt UalMachines/jdk1.8.0 _221. JDK/Contents/Home/jre/lib/jce jar: / Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/H Ome/jre/lib/charsets. Jar: / Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib/JFR jar: / Library/Java / JavaVirtualMachines/jdk1.8.0 _221. JDK/Contents/Home/jre/classes"/>
        <property name="sun.management.compiler" value="HotSpot 64-Bit Tiered Compilers"/>
        <property name="java.runtime.version" value="1.8.0 comes with _221 - bl1"/>
        <property name="user.name" value="qinzhen"/>
        <property name="path.separator" value=":"/>
        <property name="os.version" value="10.14.5"/>
        <property name="java.endorsed.dirs" value="/ Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib/endorsed inside"/>
        <property name="java.runtime.name" value="Java(TM) SE Runtime Environment"/>
        <property name="file.encoding" value="UTF-8"/>
        <property name="java.vm.name" value="Java HotSpot(TM) 64-Bit Server VM"/>
        <property name="localRepository" value="/Users/qinzhen/.m2/repository"/>
        <property name="java.vendor.url.bug" value="http://bugreport.sun.com/bugreport/"/>
        <property name="java.io.tmpdir" value="/var/folders/55/lg_7qtv13d93hk9v7kjkn_p80000gn/T/"/>
        <property name="java.version" value="1.8.0 comes with _221"/>
        <property name="user.dir" value="/Users/qinzhen/Documents/TestDev/MyTraining/XUnit"/>
        <property name="os.arch" value="x86_64"/>
        <property name="java.vm.specification.name" value="Java Virtual Machine Specification"/>
        <property name="java.awt.printerjob" value="sun.lwawt.macosx.CPrinterJob"/>
        <property name="sun.os.patch.level" value="unknown"/>
        <property name="java.library.path" value="/Users/qinzhen/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/E xtensions:/usr/lib/java:."/>
        <property name="java.vm.info" value="mixed mode"/>
        <property name="java.vendor" value="Oracle Corporation"/>
        <property name="java.vm.version" value="25.221 - bl1"/>
        <property name="java.ext.dirs" value="/ Users/qinzhen/Library/Java/Extensions: / Library/Java/JavaVirtualMachines jdk1.8.0 _221. JDK/Contents/Home/jre/lib/ext: / Li brary/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java"/>
        <property name="sun.io.unicode.encoding" value="UnicodeBig"/>
        <property name="java.class.version" value="52.0"/>
      </properties>
      <testcase name="testDemoA on testDemoA(junit4demo.Junit4DemoTest)" classname="junit4demo.Junit4DemoTest" time="0.007">
        <failure type="java.lang.AssertionError">java.lang.AssertionError
    	at junit4demo.Junit4DemoTest.testDemoA(Junit4DemoTest.java:40)
    </failure>
        <system-out><! [CDATA[I'm @beforeClass, I'm @beforeClass, I'm @before, use case execution comes to me Before testDemoA I'm @after, use case execution comes to me After]]></system-out>
      </testcase>
      <testcase name="testDemoB on testDemoB(junit4demo.Junit4DemoTest)" classname="junit4demo.Junit4DemoTest" time="0.001"/>
      <testcase name="testDemoC on testDemoC(junit4demo.Junit4DemoTest)" classname="junit4demo.Junit4DemoTest" time="0"/>
    </testsuite>
    Copy the code

2.2 HTML report for the MVN Surefire plug-in

The MVN site generates the basic configuration of the project website and then executes MVN Surefire-Report: Report to generate the corresponding report:

2.3 Allure2 Test Report

2.3.1 What is Allure2?

  • Take a look at allure2’s official explanation

    It’s the next generation of Allure Report, a flexible, lightweight multilingual testing reporting tool that adds steps, attachments, metrics, and more

2.3.2 Installing Allure2

  • For the installation of different systems, such as Linux, Mac and Windows, specific methods are given on the official website:
  • Here to the Mac system demonstration, directly in the terminal to execute the commandbrew install allureAfter the implementationallureThe installation is successful if you see the following display, which shows the various command parameters of Allure:

    Or useallure --versionCommand is also used to confirm the successful installation

2.3.3 Generation of allure2 simple report

  • Look again at the official website to the tutorial:
  1. Perform the test in the IDE first, intarget/surefire-reportsDirectory to generate the corresponding XML report
  2. To performallure serveThe command

  • The effect is shown in mytarget/surefire-reportsThere are two XML reports generated by test classes in the directory

    afterallure serve /Users/qinzhen/Documents/TestDev/MyTraining/XUnit/target/surefire-reportsThe following page is displayed:

  • The report analysis
    • Overview: the home pageOverviewIs a general overview of which test classes are executed, and what are the results of pass and fail, etc
    • Categories: Categorize failed results
    • Suites: Show specific implementation of all use cases

      Graphs: Displays basic data in Graphs

    • Timeline: a Timeline that can be pulled to review the test. (The official picture is quoted here. The demonstration project is not executed well and the effect is not obvious.)
    • Behaviors: Test cases are grouped according to different characteristics
    • Packages: Shows the Packages in which different use cases are located

3, Allure detailed report generation

3.1 How Allure works

  • Add the dependency and configuration of Allure to the test framework
  • Perform the test
  • Generate allure – results
  • allure generate allure-results -o allure-report

3.2 generate allure – results

Allure is available in almost every language, and its website also provides a way to generate reports for various language frameworks, as shown below:

<properties>
    <aspectj.version>1.8.10</aspectj.version>
</properties>

<dependencies>
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-junit4</artifactId>
        <version>LATEST_VERSION</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.20</version>
            <configuration>
                <testFailureIgnore>false</testFailureIgnore>
                <argLine>-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.ja r"</argLine>
                <properties>
                    <property>
                        <name>listener</name>
                        <value>io.qameta.allure.junit4.AllureJunit4</value>
                    </property>
                </properties>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.aspectj</groupId>
                    <artifactId>aspectjweaver</artifactId>
                    <version>${aspectj.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>.Copy the code

Note: the version of allure-junit4 relies on LATEST_VERSION can be viewed from the official website:

2) Execute the MVN Clean test again. After executing the MVN Clean test, you can see that there is a new folder allure- Results in the project

3.3 Annotation Features

3.3.1 Overview of annotations

The official website reads as follows:

annotations instructions
@DisplayName Set the use case presentation name
@Description describe
@Step Set the steps and complete the steps description
@Attachments Add attachments
Links: @link, @issue, @tmslink Link tests to certain resources, set bug numbers, and set use case numbers
@Severity Set the bug level to Blocker, Caitical, minor, Normal, trivial
  • @Link, the website explains

    From the instructions and examples provided on the official website, it can be seen that the Link address can be set through @link, but additional configuration is required in the first example@LinkAt present, my understanding is to show the address that needs to be linked, which has no other practical functions (there may be other functions, but the knowledge reserve is not enough, to be explored)@LinkIs to set the specific location and type of link,nameIn is the configuration{}{} can be removed from the configuration, so that name can be used as a single hyperlink name. It needs to be created in the same directory as testTest Resources RootFolder of typeresourcesAnd create a configuration fileallure.properties

    allure.results.directory=target/allure-results
    allure.link.mylink.pattern=http://xxx.com/
    allure.link.issue.pattern=https://example.org/issue/{}
    allure.link.tms.pattern=https://example.org/tms/{}
    Copy the code

    == language is relatively obscure to describe, the following practical demonstration section will be much clearer ==

  • @Severity, the official website does not directly explain several levels of Severity. When we call this interface in IED, we can see the following levels through code prompts

3.3.2 Practical operation demonstration

1) Basic annotations with @Link
  • Annotate the use case testDemoA@DisplayName@Description,@Link,@Issue,@Severity
  • Create in the test sibling directoryTest Resources RootFolder of typeresourcesAnd create a configuration fileallure.properties

    mvn clean test -> allure serve allure-resultsPost-generated test report from the report can see the changes brought about by annotations:

    Clicking on bugAddress will take you directly to the project address I have set:

2) @step Step display
  • newloginMethod and add@StepAnnotation, “login step”; In the method againtestDemoACall the method twice inlogin:

    Test results:

    As you can see from the test results, the Allure report prints out the corresponding method call execution steps

3) @attachment screenshots

There are two approaches for screenshot functionality on the website, @Attachment annotation and Allure. Addattinfringement API, so here is API for demonstration

  • Add a screenshot to the login() method:

    Test results:

    It can be seen from the test results that each time login is executed, screenshots are saved, so that listening can be set up during UI automation, screenshots are taken when use cases fail or screenshots are set up in key steps, and finally displayed in the test report for convenient analysis of test results

3.4 Dynamically generating test steps

You can dynamically add steps to use cases. In actual applications, you can insert log information for printing

3.5 Jenkins combination

Fill in the Jenkins section

4. Link address

Allure official GitHub: github.com/allure-fram… Github.com/allure-fram… Allure2 official GitHub: github.com/allure-fram… Qameta. IO /allure/#_ab…