Software testing is picking bugs? Maybe you have cognitive bias

“What is software testing?” This seemingly simple problem is actually the most difficult one. Say it is simple, because this is a basic problem, do software testing work for many years of friends, naturally know what is software testing. It’s hard because “software testing” has many connotations, and it’s not easy to understand all of them. If we ask software developers what software testing is, we will probably get a wide range of answers. People have different understandings of software testing. Now the most common understanding is that software testing is about finding bugs and defects.

But some might think of software testing as:

Check whether the software products meet the design requirements;

Verify consistency of software product requirements, design and implementation;

Confirm whether the software products meet the actual needs of users;

Comprehensive evaluation of software product quality;

Provide software product quality information;

Reveal the quality risks of software products;

Lower guarantee cost greatly reduces the cost of inferior quality.

Verification and validation;

Investigation, analysis and comparison;

Keep exploring;

…

There’s too much understanding, and there’s nothing wrong with it, just different ways of looking at things. Although when answering questions, it is easy to blurt out, not carefully consider, only see one side of software testing, without systematic analysis of “what is software testing”.

Let’s discuss “what is software testing”, because what is understood is what is done. Correct understanding means correct operation; On the contrary, where there is wrong understanding, there is wrong operation. Therefore, first help readers to “software testing” to establish a correct and comprehensive understanding, build a complete “software testing” outline, not fall into the “blind men feel the elephant” predicament, have a one-sided understanding of software testing. Then, we move on to a discussion of processes, methods, techniques, and practices. That is, before the comprehensive discussion of “whole-process software testing”, we need to find a common language, that is, to reach a consensus on some basic concepts of software testing, for the subsequent communication to remove obstacles.

Software testing basic cognition – positive and negative thinking

What is software testing? It is often answered that software testing is about finding bugs in a software product. Some people say, no, software testing is to verify whether the characteristics of software products meet the needs of users. In fact, both of the above answers are true, and they explain both the pros and cons of software testing.

In the early days, people regarded “testing” more as “checking” the product, checking whether each function of the software works properly. As Bill Hetzel defined software testing in 1983, “Software testing is a series of activities to evaluate the characteristics or capabilities of a program or software system and determine whether it achieves the expected results.” From this definition, we can see at least two things.

Testing attempts to verify that the software “works,” that is, that the software functions are performed correctly.
Testing activities are based on people’s “assumptions” or “expected results.” The term “envisioned” or “expected results” here refers to requirements definition, software design results.

But at the same time, we know that there is a rule of software testing: testing is not exhaustive. The test will face a large amount of test data, test scenarios or code paths, etc. The test is just a sample experiment, which cannot prove that the software is correct, but only show that the defects discovered are indeed defects. But if the problem is not found, it does not mean that the problem does not exist, but the potential problem in the software has not been found. As Glenford J. Myers, author of The Art of Software Testing, says, testing should not focus on verifying that software works, but rather on finding as many errors as possible by using reverse thinking. He believes that from a psychological point of view, if the purpose of testing is to “verify that the software works”, it is very bad for testers to find software errors. Thus, in 1979 he gave a different definition of software testing: “Testing is the process of executing a program or system in order to find errors.” From this definition, assuming that software is always buggy (which it is) and buggy, testing is about finding bugs, not proving that the program is bug free.

Extending this definition, a successful test is one that finds software problems, otherwise the test is worthless. This is just like a patient (because it is a patient, assuming that it is really sick), to the hospital to do the corresponding examination, the result did not find the problem, it shows that the physical examination is a failure, a waste of the patient’s time and money. To reverse way of thinking to guide people to prove that the software is “not work”, will promote our thinking developers understand the erroneous zone on demand, bad habits, the boundary of the program code, the invalid data input, etc., find the weak link of system or to identify the system complex area, goal is to find all sorts of problems in the system.

Human activities are highly purposeful, and the establishment of appropriate goals plays a significant psychological role. If the purpose of the test is to prove that there are no errors in the program, the subconscious mind may unconsciously try to do so. In the process of testing, it will not deliberately select some test data to try to make the program error, and select some commonly used data, the test is easy to pass, but not easy to find problems. If the purpose of the test is to prove that there is a bug in the program, we will try to select test data that is easy to find bugs in the program, so that defects can be found earlier and faster. After all, developers strive to build software with positive thinking, so reverse thinking can improve our testing efficiency.

Reverse thinking also has a disadvantage, easy to fall into the local depth test, lack of breadth. If you test something because you think it’s flawed, and then you drill down, it’s easy to overlook some areas. There are few defects, but if there are serious defects, we cannot afford them. Therefore, positive thinking is also valuable. It will urge all function points of the software system to verify their correctness one by one, and check which function is more important. Positive thinking gives us more breadth — good test coverage.

In order to do a good test, there must be both depth and breadth; Be efficient and have the integrity of the test work itself. Therefore, we should combine positive thinking and reverse thinking organically to achieve a balance between efficiency and quality. In other words, we use reverse thinking more when we need efficiency, and positive thinking more when we need test breadth to ensure complete test quality. This balance is also reflected in different application fields, such as national defense, aerospace, banking and other key software systems, can not withstand any failure of the system. Because these failures have the full potential to lead to catastrophic events, the emphasis is on verify to ensure very high software quality. And the general business application software or service, the quality goal is set at the “user acceptable level”, in order to reduce the cost of software development, accelerate the speed of software release, is conducive to the expansion of the market, can emphasize the reverse thinking, find out most defects as soon as possible.

From narrow testing to broad testing

As mentioned earlier, Glenford J. Myers gave a simple definition of software testing: “Program testing is the process of executing a program for the purpose of finding errors”, which also reflected the very limited understanding of software testing at that time. This is also influenced by the waterfall model of software development, which regards software testing as a stage after programming. The only way to find problems is to wait for the code to be developed and then execute the program and operate the software as users do. This is called dynamic testing.

The cost of discovering and fixing defects in the requirements phase is different. If defects in requirements are discovered and fixed during the requirements phase, it is very cheap to modify the requirements document. If the defects produced in the demand stage are not discovered in the demand stage and are discovered after the design is completed, the requirements and design need to be modified, resulting in increased costs. If the defects produced in the requirements stage are not found in the requirements and design stage and are discovered after the code is written, the requirements, design and code need to be modified, and the cost will be higher. Design problems are found in the design stage, as long as the design is modified, if found in the later stage, the path of rework will become longer, and the cost of repair will naturally increase. The later defects are discovered, the higher the cost of repairing them. Figure 1-1 shows the cost of repairing defects at different stages, so it requires us to find defects as soon as possible.

Figure 1-1 Cost of repairing defects at different stages

In order to find defects as early as possible, it is necessary to extend software testing to requirements and design stage, that is, to review or verify the periodic results of software products — requirements definition documents and design technical documents. This is different from software Quality Assurance (QA). Although QA focuses on review, it focuses on review process and review management, including standardized review of requirements, design, coding and testing processes. The requirements and design review mentioned here is still the inspection or verification of software products, but the requirements documents and design documents are only periodic products of software products. According to the definition of “software = program + document + data structure”, requirement documents and design documents are also part of software, and software testing naturally includes verification of requirements and design.

Based on the above considerations, the early dynamic testing is extended to static testing, that is, from the narrow sense of software testing to the broad sense of software testing.

In the narrow sense, software testing: dynamic testing – tests performed by running programs, testing is only the stage after programming, which is also determined by the traditional waterfall model.

Generalized software testing: dynamic testing + static testing, requirements review, design review, code review (including static analysis of code) are also included in the software testing work. This makes “software testing” no longer stay in a certain stage after programming, but become a quality assurance activity throughout the whole software development cycle, which is also the original idea of “software testing throughout the whole process” in this book.

Static testing is to review software or phased results when the software system is not running, including requirements review, design review, code review, etc. The introduction of static testing, you can find the problem as soon as possible, eliminate the problem in the bud, will be produced at each stage of the defects in time to remove, greatly improve the quality of products, effectively reduce the cost of enterprises.

Quality based cognition

Software testing is not equivalent to SOFTWARE quality assurance (SQA), but it is one of the main means of software quality assurance. When we talk about software testing, the word “quality” is definitely involved. Based on the understanding of quality, software testing is to evaluate the quality of software products and improve the quality information related to software products. Even though we think from Section 1.1 that software testing is about finding bugs in a software product, what is a “defect”? To put it simply, defects are the opposite of quality, and anything that goes against quality can be considered a software defect (though in technical terms, the problem can be classified as “internal error, external failure”, etc.). So to understand software testing, you have to understand software quality.

Speaking of “quality”, we are all familiar with the concept of “bad quality will be whatever, good quality will be whatever”, but it may not be easy for us to give a formal definition of quality. We can also refer to international standards to know how to define quality. For example, IEEE Std 829-2008 defines quality as the extent to which a system, component, or process meets a particular requirement, the needs or expectations of customers/users. The higher the satisfaction, the better the quality. For example, from the perspective of a software requirements definition document, the more the requirements it describes match the actual business requirements of the customer, the more likely the software implemented in the future will meet the customer’s business requirements, which also means the higher quality of the requirements document. But in that sense, it’s still very broad and difficult to measure quality. So how do you evaluate quality? In what dimensions is quality measured? This brings us to the mass model. Based on the quality model, we can know the attributes (or dimensions) of quality, and then evaluate the attributes one by one. It is not necessary to evaluate the software quality as a whole, but to evaluate and break the quality one by one according to each dimension.

In the past, software quality was divided into internal quality, external quality and use quality, such as code standardization, complexity, coupling, etc., can be regarded as internal quality, internal quality and external quality share a quality model. International/national standards now combine internal and external quality into product quality. Product quality can be regarded as the inherent internal characteristics and external performance of the software system itself, while service quality is perceived by customers or users from the perspective of use. Because quality is relative to customers, there is no quality without customers, quality is customer satisfaction. In the past, it was believed that internal quality influenced external quality and external quality influenced usage quality, while usage quality depended on external quality and external quality depended on internal quality. Today it can be understood that the quality of products affects the quality of use, and the quality of use depends on the quality of products.

1. The quality of the product

According to the international standard IEEE 24765-2010, product quality refers to all inherent characteristics of a product clearly capable of meeting express and implicit requirements under specific conditions of use. According to ISO 25010:2011, the quality model has been increased from the original 6 features to 8 features, and “security, compatibility” has been added. As shown in Figure 1-2, the content marked in blue is newly added or modified. Security here refers to information Security, which was originally placed under “functionality”, but now most products are network products. Security is becoming more and more important, so it is necessary to measure it as a separate dimension. Today, system interconnection is very common. Secondly, there are more and more terminal devices. In addition to the traditional PC, there are many smart mobile devices, such as mobile phones, tablets, smart bracelets, smart watches, etc., which require the system to have good compatibility. These features correspond to test types, such as functional tests, performance tests (efficiency), compatibility tests, security tests, and so on.

Figure 1-2ISO 25010 2016 Product quality model

Functional suitability: software is capable of realizing functions that meet its design specifications and user needs, and correctness, completeness and suitability are emphasized.

Efficiency: under specified conditions, the time characteristics (such as response speed) shown by the software to the operation and the degree of effective use of computer resources (including memory size, CPU occupancy time, etc.) to achieve a certain function. High local resource occupancy is usually a performance bottleneck. The number of concurrent users and connections that the system can support depends on the scalability of the system.

Compatibility refers to coexistence and interoperability. Coexistence requires software to be compatible with system platform, subsystem, third-party software, etc., and to be appropriately handled for internationalization and localization. Interoperability requires efficient docking between system functions, including apis and file formats.

Ease of use (usability) : for a software, user study, operation, prepare the input and output to understand the extent of the efforts, such as easy and convenient installation, easy to use, friendly interface, and can apply to different characteristics of the user, including for people with disabilities, people with defects can provide the products use effective ways or methods (da).

Reliability: the degree/probability that the software can maintain its normal functional operation and performance level under specified time and conditions. For example, the higher the maturity, the higher the reliability; Reliability was measured by Mean Time To Failure (MTTF) or Mean Time Between Failures (MTBF).

Security: Data transmission and storage are required to ensure its security, including user identity authentication, data encryption and integrity verification. All key operations are logged and operations performed by different user roles can be reviewed. It involves confidentiality, integrity, non – resist repudiation, audit, authenticity.

Maintainability: The maintainability of software modifications when requirements, environment changes, or software errors occur. It involves modularity, reusability, easy analysis, easy modification, easy testing and so on

Portability: the ease with which software can be moved from one computer system or environment to another, or how easy it is for a system to work with external conditions. It involves adaptability, installability, and replaceability.

2. Use quality

According to THE ISO/IEC 25010 standard, software testing also pays attention to the quality of use, as shown in Figure 1-3. In the use of quality, not only contains the basic functional and non-functional properties, such as function (effective and useful), efficiency (performance), safety, etc., also requires the user to take pleasure in the process of using the software product, the product trust, products should not be to bring economic, health and environmental risks to users, and can deal with the context of the business, covering the full business areas.

Figure 1-3 Attribute description using quality

To understand the quality of use, here are three examples.

An example from my own experience. I installed an English learning software on my mobile phone, which automatically downloaded multiple language libraries used by the software (such as New Concept English, Cet-6 English, etc.). It did not judge whether my mobile phone was connected to Wi-Fi or 3G/4G when I was giving a lecture. As a result, my data greatly exceeded the plan limit and generated an extra 300 yuan of data fee. From the point of view of function, automatic download is a good feature, but there is a great economic risk, there are obvious defects in the quality of use.

【典型范例1-2】 When we play a game, we are addicted to a certain game in terms of the quality of the product itself. It’s a good product, no problem. However, from the point of view of the quality of use, it will harm the health of players and have health risks, so it is necessary to set up anti-addiction function.

When we use baidu Map, Didi Taxi and other software, we are often on the street. It’s fine if you’re standing on a sidewalk or in a safe place, but it’s a safety risk if you’re using it while crossing the street. Such software should be alerted, or they take the risk.

Risk-based perception

Since there is no way to prove that software is correct and software testing itself always has a certain degree of risk, software testing is considered as an activity to evaluate the potential risks in a software system. From the point of view of risk, software testing is to continuously evaluate the quality risk of software products, guide the software development work, and then reduce the risk of the final released software to the minimum. The risk-based knowledge of software testing is mainly reflected in two aspects:

Software testing is not just about a single defect, it is about identifying some kind of quality risk or an area of potential risk from the problems found.

Software testing is seen as a dynamic quality control process that detects the whole process of software development, detects unhealthy signs at any time, evaluates new risks in time, sets new monitoring benchmarks, and continues continuously.

The perception of testing based on risk emphasizes continuity of testing, continuous testing, testing of a few lines of code, testing of a feature to implement, development and testing go hand in hand. This knowledge is particularly appropriate for testing in the agile development model — agile testing. In agile development, software testing can be interpreted as a continuous evaluation of software product quality. In agile methods, continuous integration is advocated, and continuous testing is actually advocated for continuous testing.

The risk-based perception of testing is also a constant reminder that we need to focus our efforts and balance risks and development cycle limits while doing the best we can. First assess the risk of testing. What is the probability that each feature will fail? According to the Pareto principle (also known as the 80/20 principle), which features are the top 20% of features that users use most? If a feature is broken, how much impact will it have on users? The test is then prioritized according to the risk. Tests are executed first for high-priority features. In general, tests on the 20% of features that users use most (high priority) will be fully and fully performed, while tests on lower-priority features (the other 80% of features that users do not use most) may be less demanding and less testing due to time or budget constraints.

Socially-based cognition

Software is different from hardware, software is generally an application system, and is often closely linked with people’s entertainment, transaction processing, business activities, community communication, so software has a strong social, so it is necessary to introduce psychology, anthropology and sociology into software testing. Software testing is not only a technical activity, but also a comprehensive social and psychological activity, according to a meeting of the Ministry of Education of the People’s Congress of the People’s Congress of the People’s Congress of the People’s Congress of the Ministry of Education. Assess the software quality through investigative and storytelling methods.

Through software testing of social cognition, emphasize the thinking ability and exploring ability of the tester, emphasizes the validity and reliability of the test, to understand the user behavior in the test, the background and purpose of human activities (context), continuous observation, continuous learning, discovery and quality related information (differences or questions), Protect the value of products from the interests of customers and business characteristics.

Because of the sociality of software testing, it is necessary to verify the ease-of-use, risk-free degree and context coverage of software products. In usability testing, people often do A/B testing, offering different solutions (UI layout, functional design, etc.) and releasing the product to different user groups to see which solution is preferred by users.

The perception based on economy

Generally speaking, a software product will not be released, deployed, or released without testing. Because under the current development mode and development technology situation, the existence of serious defects in the software developed by people is definitely a high probability event. If the software is released without testing, it may not work at all or not work well, or all kinds of problems may occur when it is used, resulting in low user satisfaction, negative impact on the product, and even serious economic losses to customers or life safety of users.

From an economic point of view, software defects will bring costs to enterprises, which is called Cost of Poor Quality (COPQ). Based on the understanding of economy, software testing is to reduce the cost of inferior quality and help enterprises gain profits by investing lower guarantee cost. High quality is not only competitive, but also brings good economic benefits. Apple, for example, earns a higher profit margin than other brands due to its high quality. According to relevant media statistics, Apple smartphone only accounts for a quarter of the high-end mobile phone market, but accounts for half of the profits.

The economic point of view of testing is how to get higher profit with minimum cost, which also requires software testing to start work as early as possible. The earlier defects are found, the less work will be reworked and the less loss will be caused. So, from an economic point of view, testing cannot begin after the software code is written, but rather from the first day of the project, testers are involved to find more defects as soon as possible and to urge and help developers fix them.

Standards-based cognition

Software testing is considered an integral part of two activities: “Verification” and “Validation.” If only one of these is done, the test is incomplete.

Validation is to verify that the software has correctly implemented the system functions and features defined in the product specifications. The validation process provides evidence that the software related products conform to the requirements of all lifecycle activities, that is, verify that the software implementation (that is, the product delivered to the customer) meets the software requirements definition and design objectives.

Validation is the activity of confirming whether the software developed meets the actual needs of users. Because the definition and design of software requirements may be wrong, the consistency mentioned above cannot guarantee that software products meet the actual needs of customers, and customers’ needs are also changing. When the definition of requirements is determined six months ago, such changes are more likely.

There are different interpretations of validation and validation. Simply put, unit testing, integration testing, and system testing can be understood as “validation” based on requirements definition documents and design specification documents; And acceptance testing, by the user at the user’s site to participate in, can be understood as “validated”, because before the requirements definition and design are likely to have error, the r&d team without understanding the meaning of the user (the user’s real expectations), only according to the requirements definition document and design specifications document to complete the test, It does not mean that the features implemented are what the user really wants. In acceptance testing, the user participates to confirm whether the implemented features are what the user really wants.

Another interpretation is that according to the V model shown in Figure 1-4, validation is architectural design review, detailed design review, and code review/unit test, respectively, to verify whether architectural design is consistent with requirements, detailed design is consistent with architectural design, and code is consistent with detailed design, represented by the thick dotted line with arrows on the left. Validation, on the other hand, is integration testing, system testing, acceptance testing, as indicated by thin dotted lines with arrows in the middle.

Figure 1-4 V model of software development

Full Software Testing (3rd Edition)

By Zhu Shaomin

This book is a systematic summary of the changes that have taken place in software testing over the past decade, and encapsulates the author’s valuable experience in software testing. This book first introduces different views on software testing, the whole process of software testing ideas, software testing infrastructure and TA framework, team capacity building; Then step by step into test planning, design, implementation, continuous feedback and improvement; Then, we discuss the idea of whole-process testing, including whole-process static testing, whole-process performance testing, whole-process security, whole-process modeling and whole-process visualization. The book concludes with a look at the future of software testing.

This book is suitable for software testers and can be used as a reference guide for related professionals