The Internet of Things requires a different testing perspective
The Internet of Things is big, probably bigger than most embedded manufacturers have ever dealt with. Furthermore, a single device is only one part of a larger whole made up of many kinds and types of devices. In my last article, I discussed how the promise of the Internet of Things is both in the services it provides and in the aggregation of many “things” that are valuable to businesses. Dealing with this scale is important in (1) the need for equipment (” things “), and (2) the value it adds to overall automation (” services “) throughout the software development life cycle to meet cost, schedule, and time-to-market constraints. Testing, in particular, must be extended to fit the expected operating environment, in addition to providing support for key non-functional requirements such as performance and security.
Service Internet needs automation
As the scale of iot implementation increases, automation in software development becomes critical. As connectivity and scale increase, security and non-functional requirements become more important. During the integration and system test phases, service-based automated testing becomes critical, along with security, stability, and performance testing.
Let’s take a look at Parasoft’s depth and breadth in test automation, as shown in Figure 1, as it applies to every stage of the SDLC. The key takeaway is how each solution complements and expands as the product grows. Unit tests complement static and runtime analysis. During the integration process, unit testing evolves to apis and service testing tools, and then to service virtualization.
Figure 1: Test automation solutions and where they are introduced into the software development lifecycle.
Most developers are familiar with some aspect of test automation; However, many people don’t think in terms of services or service-based testing and virtualization. As the pattern shifts from “things” to “services,” the benefits of service-based testing become more apparent.
Risk and cost reduction of service-based testing in the Internet of Things
As products move into the integration phase of the development process, testing becomes more complex and expensive. This is also the stage where many critical errors are discovered, including security and performance issues that cannot be detected at the unit level. A big risk is the availability of testers, LABS, and hardware resources.
The solution here is automation. Automation provides a way to improve test productivity, repeatability, and system testing scope. Automated, service-based testing approaches are critical to the success of newly developed iot devices.
Its advantages include the following:
- Improve test productivity
Automation assists in generating service-based tests and repeatable and scalable test loads. The runtime analysis tool runs with live tests to detect and track errors. Defects are much easier to find and fix than manual testing. Once the fix has been made, automation can provide simple regression testing. Support repeated testing as part of a continuous testing, agile, or iterative development process.
- Eliminate hardware resource bottlenecks
One of the biggest problems in test scheduling is the availability of dependent components. These components may be other devices on the network, or higher-level functions in the Internet of Things network. Emulating these dependencies removes testing bottlenecks while making the entire virtual lab available on every tester and developer’s desktop.
- Scale for current and future needs
As system integration progresses, the scope of testing increases to include more and more dependent components. Scalability is key in iot systems, as the devices under test perform in highly complex environments. In an appliance lifecycle environment, complexity increases, and automated testing must be extended with production.
- Practical and realistic performance and safety testing
Development teams do their best to test performance and security during the integration process, but without automation, setting up real-world scenarios in the lab is often time-consuming and expensive. The service-oriented approach provides a framework for specifying and verifying performance and testing security.
Beyond functional Testing: Safety and performance
Pure functional testing is not enough to get the device to market. Non-functional requirements such as security and performance are critical, but some of the most difficult features to test. A device with poor performance or security is simply not competitive, but meeting time-to-market constraints and properly testing performance and security is a serious challenge for iot devices. Service orientation provides a common way to specify requirements (for example, based on the performance of each service) and test security (for example, attacks that exploit exposed services and apis).
Performance and load testing
Performance and safety requirements are likely to be expressed in the form of quality-of-service statements. For example, the HVAC system might be required to maintain the building temperature at 75 degrees for two hours based on a temperature difference of 5 degrees outside, reporting the current temperature every 10 seconds. If you design a thermostat for this product, you can test functionality at a single level, and perhaps performance as well. However, if a deployed system consists of hundreds of thermostats, the performance of the device is only a fraction of the complex network of other devices.
Security testing
Security requirements are often system-level and vague. In the case of hVAC systems, the thermostat may be required not to fail under heavy network loads. Denial-of-service attacks rely on flooding a target with massive amounts of traffic, sometimes malformed packets. In these cases, penetration and fuzziness tests are used to ensure that the device can withstand the harsh network environment. The extensibility available in virtual environments exceeds the possibilities of real hardware.
Service virtualization
The next logical step in service-based test automation is virtualization. By simulating all the dependency components and the full set of test stimuli, a complete virtual environment is possible. The benefit of service virtualization is not only a real and repeatable test environment, but also the ability to replicate and deploy at will, creating a “virtual lab” — one that is just as effective as the real world, but at a fraction of the cost.
conclusion
In general, test automation is critical to meeting iot product goals, such as time-to-market and budget. A service-based approach improves test productivity and allows for highly repetitive and scalable testing of performance and security, as well as the ability to build virtual test environments. As the complexity of iot devices and their deployment environments increases, the need for scalable service-level testing becomes more pressing.