Testing is an important factor in the reliability, maturity, and maintainability of an IaaS software. The tests are fully automated in the ZStack. This automated test system consists of three parts: integration test, system test and module based test. The integration tests are built on Junit, using emulators. With the various functions provided by the integrated test system, developers can quickly write test cases to verify a new feature or bug fix.
An overview of the
This key factor, in building a reliable, mature, and maintainable software product, is architecture; This is the design principle that we have always believed in. ZStack has put a lot of effort into designing an architecture that keeps the software stable, whether it’s adding new features, routine errors, or tailoring for specific purposes; Our previous articles, ZStack – In-process Microservices Architecture, ZStack – General plug-in System, ZStack – Workflow Engine, and ZStack – tagging system, have represented some of our attempts. However, we also fully understand the importance of testing in software development. ZStack, from day one, set the goal that every feature must be guaranteed with test cases, that testing must be fully automated, and that writing unit tests should be the only way to validate a new feature or any code changes. To achieve this goal, we divided our test system into three components: integration test, system test and module test. They are classified by their concerns and functions.
- The integrated test system is built in Junit, all using emulators; The test cases are stored in ZStack’s Java source code; Developers can easily start the test suite using regular Junit commands.
- The System Test System is a separate Python project called ZStack-Woodpecker, based on the ZStack API; Test everything in a real hardware environment.
- Module based test system* Building on the theory of module-based testing is a subproject of ZStack-Woodpecker. The test cases in this system will continue to execute the API, in a random fashion, until some predefined condition is met.
Starting with this article, we will have a series of three articles detailing our test architecture to show you how we guarantee each feature of ZStack.
Curious readers may have asked the question on their minds why we didn’t mention unit testing, a testing concept that is probably the most famous and emphasized by every dispassionate test-driven developer. We do have unit tests. If you read the subsequent section: Test Frameworks, you may be confused as to why the name used in the command is similar to: UnitTest Balabala, but in this article is called Integration Tests.
At first, we thought of our tests as unit tests, because each use case is used to validate an individual component, not the entire software; For example, a use case, TestCreateZone, tests only the Zone service, and other components like VM service and storage service will not even be loaded. However, the way we do test does differ from the traditional concept of unit testing, which is to test a small piece of code, usually as a white-box test against an internal structure, using the mock and stub methodology. The current ZStack has about 120 test cases that meet this definition, while the remaining 500 or so do not. Most test cases, even those that focus on individual services or components, are more like integration test cases in that multiple dependent services, components are loaded to perform a single test activity.
Most of our simulator-based test cases, on the other hand, actually test at the API level, which by definition of unit testing is black box testing in favor of integration testing. Based on these facts, we eventually changed our mind and we were going to do integration tests, but with a lot of the old naming, like UnitTest Balabla.
Integration testing
From our previous experience, we are acutely aware that one of the main reasons developers continue to ignore testing is that writing tests is hard, sometimes even harder than implementing a feature. When we designed the integrated test system, one of the things that went back and forth was to take the burden off the developers as much as possible and let the system do most of the boring, tedious work itself. For almost all test cases, there are two types of repetitive work. One is to prepare a minimal but working piece of software; For example, in order to test a zone, you only need the core library and zone service to be loaded. There is no need to load the other services because we don’t need them. The other is the preparation environment; For example, a test VM creation use case would require an environment with a zone, a cluster, a host, storage, networks, and all other necessary resources in place; Developers don’t want to repeat boring things like creating a zone and adding a host before they can actually start testing their stuff; Ideally, they can get a ready environment with minimal effort to focus on what they want to test. We solve all of these problems with a framework built on top of JUnit. Before everything starts, since ZStack manages all components using Spring, we create a BeanConstruct so testers can specify which components they want to load on demand:
public class TestCreateZone {
Api api;
ComponentLoader loader;
DatabaseFacade dbf;
@Before
public void setUp(a) throws Exception {
DBUtil.reDeployDB();
BeanConstructor con = new BeanConstructor();
loader = con.addXml("PortalForUnitTest.xml").addXml("ZoneManager.xml").addXml("AccountManager.xml").build();
dbf = loader.getComponent(DatabaseFacade.class);
api = new Api();
api.startServer();
}
Copy the codeIn the example above, we added three Spring configurations to BeanConstructor whose names imply that the account service, zone service, and other library components included in portalForUnitTest.xml will be loaded. In this way, testers can tailor the software to a minimum size, containing only the components needed to speed up the testing process and make things easier to debug.
Environment deployer
To help testers prepare an environment that contains all the necessary dependencies for the activity to be tested, we created a deployer that reads an XML configuration file to deploy a complete emulator environment:
public class TestCreateVm {
Deployer deployer;
Api api;
ComponentLoader loader;
CloudBus bus;
DatabaseFacade dbf;
@Before
public void setUp(a) throws Exception {
DBUtil.reDeployDB();
deployer = new Deployer("deployerXml/vm/TestCreateVm.xml");
deployer.build();
api = deployer.getApi();
loader = deployer.getComponentLoader();
bus = loader.getComponent(CloudBus.class);
dbf = loader.getComponent(DatabaseFacade.class);
}
@Test
public void test(a) throws ApiSenderException, InterruptedException {
InstanceOfferingInventory ioinv = api.listInstanceOffering(null).get(0);
ImageInventory iminv = api.listImage(null).get(0);
VmInstanceInventory inv = api.listVmInstances(null).get(0);
Assert.assertEquals(inv.getInstanceOfferingUuid(), ioinv.getUuid());
Assert.assertEquals(inv.getImageUuid(), iminv.getUuid());
Assert.assertEquals(VmInstanceState.Running.toString(), inv.getState());
Assert.assertEquals(3, inv.getVmNics().size());
VmInstanceVO vm = dbf.findByUuid(inv.getUuid(), VmInstanceVO.class);
Assert.assertNotNull(vm);
Assert.assertEquals(VmInstanceState.Running, vm.getState());
for (VmNicInventory nic : inv.getVmNics()) {
VmNicVO nvo = dbf.findByUuid(nic.getUuid(), VmNicVO.class);
Assert.assertNotNull(nvo);
}
VolumeVO root = dbf.findByUuid(inv.getRootVolumeUuid(), VolumeVO.class);
Assert.assertNotNull(root);
for (VolumeInventory v : inv.getAllVolumes()) {
if(v.getType().equals(VolumeType.Data.toString())) { VolumeVO data = dbf.findByUuid(v.getUuid(), VolumeVO.class); Assert.assertNotNull(data); }}}}Copy the codeIn this TestCreateVm cases above, the deployer reads a configuration file, stored in a deployerXml/vm/TestCreateVm. The XML, and then to deploy a complete, ready to create a new vm environment; Further, we actually let the deployer create the VM, as you don’t see any code calling api.createvMByFullConfig () in the test method; Testers really do is to verify whether the VM is according to the us in deployerXml/VM/TestCreateVm. The conditions specified in the XML created correctly. Now you can see how easy it is for testers to write only about 60 lines of code and then test the most important part of an IaaS software, creating the VM. The configuration file testCreatevm.xml in the example above looks like this:
<deployerConfig xmlns="http://zstack.org/schema/zstack">
<instanceOfferings>
<instanceOffering name="TestInstanceOffering"
description="Test" memoryCapacity="3G" cpuNum="1" cpuSpeed="3000" />
</instanceOfferings>
<backupStorages>
<simulatorBackupStorage name="TestBackupStorage"
description="Test" url="nfs://test" />
</backupStorages>
<images>
<image name="TestImage" description="Test" format="simulator">
<backupStorageRef>TestBackupStorage</backupStorageRef>
</image>
</images>
<diskOffering name="TestRootDiskOffering" description="Test"
diskSize="50G" />
<diskOffering name="TestDataDiskOffering" description="Test"
diskSize="120G" />
<vm>
<userVm name="TestVm" description="Test">
<rootDiskOfferingRef>TestRootDiskOffering</rootDiskOfferingRef>
<imageRef>TestImage</imageRef>
<instanceOfferingRef>TestInstanceOffering</instanceOfferingRef>
<l3NetworkRef>TestL3Network1</l3NetworkRef>
<l3NetworkRef>TestL3Network2</l3NetworkRef>
<l3NetworkRef>TestL3Network3</l3NetworkRef>
<defaultL3NetworkRef>TestL3Network1</defaultL3NetworkRef>
<diskOfferingRef>TestDataDiskOffering</diskOfferingRef>
</userVm>
</vm>
<zones>
<zone name="TestZone" description="Test">
<clusters>
<cluster name="TestCluster" description="Test">
<hosts>
<simulatorHost name="TestHost1" description="Test"
managementIp="10.0.0.11" memoryCapacity="8G" cpuNum="4" cpuSpeed="2600" />
<simulatorHost name="TestHost2" description="Test"
managementIp="10.0.0.12" memoryCapacity="4G" cpuNum="4" cpuSpeed="2600" />
</hosts>
<primaryStorageRef>TestPrimaryStorage</primaryStorageRef>
<l2NetworkRef>TestL2Network</l2NetworkRef>
</cluster>
</clusters>
<l2Networks>
<l2NoVlanNetwork name="TestL2Network" description="Test"
physicalInterface="eth0">
<l3Networks>
<l3BasicNetwork name="TestL3Network1" description="Test">
<ipRange name="TestIpRange1" description="Test" startIp="10.0.0.100"
endIp="10.10.1.200" gateway="10.0.0.1" netmask="255.0.0.0" />
</l3BasicNetwork>
<l3BasicNetwork name="TestL3Network2" description="Test">
<ipRange name="TestIpRange2" description="Test" startIp="10.10.2.100"
endIp="10.20.2.200" gateway="10.10.2.1" netmask="255.0.0.0" />
</l3BasicNetwork>
<l3BasicNetwork name="TestL3Network3" description="Test">
<ipRange name="TestIpRange3" description="Test" startIp="10.20.3.100"
endIp="10.30.3.200" gateway="10.20.3.1" netmask="255.0.0.0" />
</l3BasicNetwork>
</l3Networks>
</l2NoVlanNetwork>
</l2Networks>
<primaryStorages>
<simulatorPrimaryStorage name="TestPrimaryStorage"
description="Test" totalCapacity="1T" url="nfs://test" />
</primaryStorages>
<backupStorageRef>TestBackupStorage</backupStorageRef>
</zone>
</zones>
</deployerConfig>
Copy the codeThe simulator
Most integration test cases are built on emulators; Every resource that needs to communicate with a back-end device has an emulator implementation; For example, KVM emulators, virtual routing virtual machine emulators, and NFS primary storage emulators. Because today’s resource backends are python-based HTTP servers, most emulators are built with Apache Tomcat embedded with HTTP servers. A short snippet of the KVM emulator looks like this:
@RequestMapping(value=KVMConstant.KVM_MERGE_SNAPSHOT_PATH, method=RequestMethod.POST)
public @ResponseBody String mergeSnapshot(HttpServletRequest req) {
HttpEntity<String> entity = restf.httpServletRequestToHttpEntity(req);
MergeSnapshotCmd cmd = JSONObjectUtil.toObject(entity.getBody(), MergeSnapshotCmd.class);
MergeSnapshotRsp rsp = new MergeSnapshotRsp();
if(! config.mergeSnapshotSuccess) { rsp.setError("on purpose");
rsp.setSuccess(false);
} else {
snapshotKvmSimulator.merge(cmd.getSrcPath(), cmd.getDestPath(), cmd.isFullRebase());
config.mergeSnapshotCmds.add(cmd);
logger.debug(entity.getBody());
}
replyer.reply(entity, rsp);
return null;
}
@RequestMapping(value=KVMConstant.KVM_TAKE_VOLUME_SNAPSHOT_PATH, method=RequestMethod.POST)
public @ResponseBody String takeSnapshot(HttpServletRequest req) {
HttpEntity<String> entity = restf.httpServletRequestToHttpEntity(req);
TakeSnapshotCmd cmd = JSONObjectUtil.toObject(entity.getBody(), TakeSnapshotCmd.class);
TakeSnapshotResponse rsp = new TakeSnapshotResponse();
if (config.snapshotSuccess) {
config.snapshotCmds.add(cmd);
rsp = snapshotKvmSimulator.takeSnapshot(cmd);
} else {
rsp.setError("on purpose");
rsp.setSuccess(false);
}
replyer.reply(entity, rsp);
return null;
}
Copy the codeEach emulator has a configuration object, like KVMSimulatorConfig, that testers can use to control the behavior of the emulator.
The test framework
Since all test cases are in fact Junit test cases, testers can run each test case individually using the usual Junit commands, for example:
[root@localhost test]# mvn test -Dtest=TestAddImage
Copy the codeAnd all use cases in a test suite can be executed with a single command, for example:
[root@localhost test]# mvn test -Dtest=UnitTestSuite
Copy the code
Use cases can also be executed within a group, for example:
[root@localhost test]# mvn test -Dtest=UnitTestSuite -Dconfig=unitTestSuiteXml/eip.xml
Copy the codeAn XML configuration file lists the use cases in a group. For example, the IP. XML above looks like:
<UnitTestSuiteConfig xmlns="http://zstack.org/schema/zstack" timeout="120">
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip1"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip2"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip3"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip4"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip5"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip6"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip7"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip8"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip9"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip10"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip11"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip12"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip13"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip14"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip15"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip16"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip17"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip18"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip19"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip20"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip21"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip22"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip23"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip24"/>
<TestCase class="org.zstack.test.eip.TestVirtualRouterEip25"/>
<TestCase class="org.zstack.test.eip.TestQueryEip1"/>
<TestCase class="org.zstack.test.eip.TestEipPortForwardingAttachableNic"/>
</UnitTestSuiteConfig>
Copy the codeMultiple use cases can also be executed in a single command by filling in their names, for example:
[root@localhost test]# mvn test -Dtest=UnitTestSuite -Dcases=TestAddImage,TestCreateTemplateFromRootVolume,TestCreateDataVolume
Copy the codeNote: Visually ZStack has adopted another type of Integration Test, see: ZStack WiKi: Integration Test Framework for Managing Nodes based on emulators
conclusion
In this article, we introduce the first part of the ZStack automated test system, integration testing. With it, developers can write code with 100% confidence. And writing test cases is no longer a daunting and boring task; Developers can complete most use cases with less than 100 lines of code, which is very easy and efficient.