Qiao Liang shared a speech titled “Consistency is the Only way to Improve Research Efficiency” at THE MTSC Conference, which mainly focused on how to improve the efficiency of software development. Starting from Shannon’s law of entropy increase in information theory, he said that for a complex system like software, it is necessary to reduce the influence of uncertain factors as much as possible and strive to improve the consistency of software research and development so as to improve the efficiency of software research and development. Based on the law of entropy increase, this paper discusses whether the law has a guiding role in software testing industry. First, let’s review what the law of entropy is. This is originally a concept in thermodynamics, that an isolated system tends to increase entropy, and eventually it reaches its maximum entropy state, which is the most chaotic state of the system. In 1948, Shannon introduced entropy from thermodynamics into information theory. For an information system, if there is no external force, it will eventually reach the state of maximum information entropy, and the information system will become more and more chaotic, and the uncertainty will become stronger and stronger. There are many factors affecting the operation of the software system, and each factor will affect each other. Therefore, it is impossible to predict the final state of the software according to a single factor. This is similar to the stock market, in that you should never predict the stock market based on a single piece of information, because a single factor is not linearly related to the final outcome.

How do bugs arise

Once a piece of software is written, you deploy it and look at it after a while, it may not be what you expected, it may already have a gateway error, the page may display 500 errors, or it may simply be inaccessible. There are many reasons can cause this state of chaos, such as a user operation triggered a deeply hidden defects, the program caused a small change, other users and then to some other operation, it is possible that the malicious attack, etc., of course, may be your server due to not renew, or a cloud service provider, closed. So even if your software is nearly perfect and doesn’t need any modifications or updates, you can go a long way without maintaining it, and your software will get messier and messier. In the actual research and development process of a product, we also need to constantly add new functions and features to the software system, bringing more information (bits), resulting in a more complex system. These new information will further increase the information entropy of the software. The more complex a software system is, the more factors affect it, and the harder it is to control it manually. Almost all companies want to use the mature technology system to improve the efficiency of R&D, but in fact, the efficiency of R&D will only become lower and lower. The efficiency of r&d of a product is often high in the initial stage, and the efficiency becomes lower and lower in the later stage. One reason is that business growth inevitably leads to increased information entropy, higher uncertainty and more interference factors, thus slowing down the development process and generating more bugs. In information theory, information entropy is usually expressed as the reciprocal of probabilities. The lower the probability is, the higher the uncertainty of the program is, the higher the information entropy is, and the random events are more likely to form bugs.

What are the main factors that add to the uncertainty of the system?

The first factor is dependent on environmental uncertainty. For example, the operating system changes, database brand and version changes, and file access permissions change. The second factor is state management. Any data that can change is in some state, such as variables in a program, data stored in a database, data in a file. These data may be read and modified at different stages of the same system, but the outside world will never know when it was modified, thus creating uncertainty. Courier testing in roaming tests tests this uncertainty by tracking data to see which features in the system read and modify this data. The third factor is the branch path of the program. If you don’t have a well-developed GPS system, you’re going to get lost on a main road full of turnovers. For a complex software system, the increase of entropy is almost inevitable. With the addition of more and more new functions (information entropy), the quality of software will be increasingly insecure. What we can do is to invest more and more high cost to delay the formation process of its chaotic state. We can also use the causes of this disorder to improve our own work, just as you can make money from the stock market without being able to predict it. All systems of uncertainty cannot be controlled by a single factor, because you do not know whether the factor you control will affect the other factors. So you should give up control of an uncertain system and let it go, and that’s what we’re going to do: calculate the uncertainty, calculate the probability. If you calculate that the probability of a stock going up is 100%, that means nothing in the world is going to stop it going down, which is obviously impossible. You can enumerate the factors that influence the price, assign weights, and finally calculate the probability. It’s important to note that you can never list all the factors, but don’t panic, you can put all the other factors you haven’t considered together and assign a weight to them. In operation after operation, you will find that your model is not good enough and the calculated probability is not accurate. When you find other factors, you can modify and adjust the model.

The guiding significance of uncertainty to software testing

Whether you’re developing software or testing software, it’s no different than buying or selling a stock, or calculating uncertainty, you need to calculate what random events are going to affect the system. Then try to improve certainty and consistency. Now let’s write a very simple program:

def run() :
    print("hello world")
Copy the code

This is a program with low information entropy and high determinism. Every time you run this program, you almost always get a certain result: printing Hello World on the display device. But you can never say it’s certain, the probability of it happening is 100 percent. What if the environment on which the program depends changes? What if a new version no longer supports this print function? What if the usage of print has changed? Like your operating system doesn’t install correctly? Like your display device doesn’t work? Fortunately, through the DevOps research and development system, we can use deterministic tools like Docker to deploy and try to ensure the consistency of dependent environment. Therefore, we can reduce the weight caused by environmental uncertainty in the test, but it does not mean that we do not need to test the environment itself. For example, we can’t determine what kind of client and version the user chooses, so don’t test for different browsers and different APP versions. (The program can optionally force the user to select only one client.) Dependencies on the system, database, client, version number, interpreter, and so on are best covered by defect analysis. Then there is the uncertainty created by user input. In any program, the user’s input is not controlled, which can cause a very large random effect on the program.

def run(user_input) :
    print(user_input)
Copy the code

Does the print function handle all user input? Student: String ok? Integers are ok? Emoji? 100 gigabytes of data is ok? If the program calls another program do_something(user_input), can the program handle all user input?

def run(user_input) :
    do_something(user_input)
Copy the code

In statically typed languages like Java, typing is done at the language level to provide standardized and consistent methods for user input, whereas in dynamically typed languages like Python, verification must be done by the developer himself. The certainty guarantee of the program depends more on the logic ability and coding level of the developers, and different developers will form different processing methods, which increases the uncertainty of the program. Static languages do type detection for data types and provide consistency, but that doesn’t mean developers can leave user data alone. Because user input can go through other ways, such as input length exceeding processing power, such as entering malicious data sabotaging programs and so on. In short, development needs further control over user input, and branch statements and exception-handling statements are written to control the program, as if all sorts of strange paths must be dug on the main road for the convenience of all the residents. Again, with multiple branching paths, people tend to get lost.

def run(user_input) :
    if user_input > 10:...elif user_input > 100:...if isinstance(userinput, str): ...

    try:    
        do_something(user_input)
    except ValueError: ...
Copy the code

For random user data that cannot be determined at all, the program will write branch statements to form deterministic execution. Whether the multiple branches run is a probability problem. When testing a program, you must consider whether the developer has considered all the cases, and what if there is no corresponding branch execution for a user input? What if the developer’s branching isn’t working properly? When writing code, if statements forget to handle else, and incomplete branch coverage is common. The exception capture statement only considers the exception types within its ability, and a large number of unknown exceptions are not handled in the expected way. Developers will also use a large number of third-party libraries for parameter verification to simplify their own parameter verification code, but it is uncertain whether these third libraries meet the requirements, whether there are bugs and bugs, and whether they are used in the right way. In addition to the user’s input parameters, other data is used during the program’s execution, such as system-generated timestamps, randomly generated variables, data modified by other programs, and so on. These data are random events beyond the control of the programmer, and therefore difficult to discover, control, and test. For developers, writing deterministic code as much as possible can effectively suppress the generation of bugs. What is highly deterministic code? For example, write pure functions, minimize the dependence on other interfaces, do not modify global variables, do not modify attributes in the class, and minimize the dependence on state data such as files and databases. (This code is what I call highly testability, but there are several companies that write code this way.) For testers, you will need to analyze which features suited to write pure functions, where data is generated, which branch code, user input can form analysis are dependent on the database data, the file data, and other interface provides the data, at the same time, the tester to write your own code should also refer to these circumstances, reduce the uncertainty. Finally, the uncertainty created by calls between multiple programs, features, and modules is more difficult to control. The more complex a function is, the more data is involved, and the permutations and combinations can increase exponentially depending on the sequence of operations. It is feasible to use scenario test method and decision table in theory, but it is difficult to cover all of them in practice. At present, some testing problems may be alleviated by such technologies as intelligent testing and accurate testing, but the information entropy of these technologies is very high. Qiao Liang also proposed the following solutions to improve consistency and reduce uncertainty from a management perspective for reference. Process: Are the results and quality consistent with the same process executed by different people? Tools: How many tool platforms are there for the same task? Personal: To what extent does the same product code look like it was written by the same person? Organization: Does the structure and mechanism support the pursuit of “consistency”?

Now that you’ve read all of this, I’m sure you’re a patient, down-to-earth person, and a good friend to make. If you are interested, you can “click on me” to friend me and learn and improve together.