By Lao Hu, Great Effort intelligent QA team

Business background

Dali intelligent homework lamp has intelligent dual-photography design, which can help parents remotely tutor their children’s homework even when they are not at home. The innovative lamp form combined with Dali intelligent eye is designed for children’s desk homework scene, realizing intelligent recognition to explanation, helping children improve homework efficiency. Moreover, it can be linked with the parent-side App “Dali Tutoring” to create a balanced and happy learning tutoring relationship for students and parents.

Product characteristics

Smart hardware Traditional Mobile APP
The object under test Smart hardware

Smart hardware + mobile phone (parent end) interaction

Intelligent hardware + computer (teacher-side) interaction		 It’s usually a mobile App
Input mode Voice input

Signal input

Image input (pick up/click read)

Voiceprint recognition		 Click and slide
The leak test cost The cost of recalling equipment is extremely high It can be repaired by heat
Performance issues Pay attention to the App’s own memory /CPU/ frame rate

Check the remaining memory, CPU, and CPU temperature of the system

Pay attention to the remaining disk space in the system		 Just focus on the App itself

The priority will not be too high

Difficulties in functional Testing

  • Case maintenance problems caused by high-speed service changes
    • Due to the rapid change of business, the interface changes frequently. Due to the strict verification of the current interface, it brings great pressure to the maintenance of the old case.
  • Multimodal input
    • Traditional automated testing simulates the user clicking a button. We also need to process the voice input data and prepare the paper for the equipment to photograph.
  • Maintenance of the environment
    • Due to voice input, the environment requires no noise and multiple devices cannot be placed in one place.
  • Technology update & software refactoring
    • Currently, apps use WebView, Flutter, Lynx and other technologies to render pages, and the elements of these pages cannot be obtained using native methods.
  • Multi-application & multi-terminal detection
    • Because the project content is the interaction between multiple applications, the operation check needs to be carried out synchronously on the smart hardware and mobile phone, which makes it more difficult.
    • There are multiple apps running on the smart hardware, which need to switch back and forth between apps during the test. So bring difficulty to frame design.

Current earnings:

The company’s Shoots automation framework is very powerful and powerful. However, in the actual use of Shoots, combined with the actual needs of the business, we believe that UI automation should also have the following capabilities:

  1. Enrich testing methods, improve the ability to find more problems
  2. Reduce the cost of writing use cases
  3. Reduce the cost of consuming test results
  4. Can a code compatible with multiple versions of the test so as to branch inspection, so as to achieve the test front
  5. Ability to collect “artificial undetectable” data to raise bugs

General overview

The framework is supported around the following large function points

  1. Enrich testing methods, improve the ability to find more problems
  2. Reduce the cost of writing use cases
  3. Reduce the cost of consumption test results, data tracking and statistics, trend judgment
  4. Can a code compatible with multiple versions of the test so as to branch inspection, so as to achieve the test front
  5. Ability to collect “artificial undetectable” data to raise bugs

Use case writing: Cost reduction

Automated script generation

The traditional method of writing the particular name is:

The Window in the red box should be named, and the corresponding method should be given the appropriate function name. These results can’t be generated automatically.

The following is an example of automatically generated code: the class name is directly degraded to the corresponding Chinese name (manually entered during recording), and the corresponding function click will be automatically given in the form of ID name + text comment.

Def auto(self): self.wait(5) self.step(" start script, please rename it as soon as possible ") Return False self.touchocrKey (' self.checkenterwin ') if not self.checkenterwin (' self.checkenterwin '): Return False self.tapkey (' self.tapkey ') self.tapkey ('take\_photo') # text = self.checkenterwin (' self.checkenterwin '): Return False self.tapkey ('cropper\_iv\_retake') # text = full page check if not self.checkenterwin (' photo \_ main interface '): Return False self.tapKey('iv\_flashlight') # text = Fill light self.tapKey('iv\_flashlight') # text = Fill light Self. tapKey('iv\_close') if not self.checkEnterWin(' main interface '): return False return TrueCopy the code

Automatically generate page snapshots

Case modular design

As we write more use cases, we find that a lot of the logic is actually atomic operations, or a group of operations are used frequently. If we had it in multiple places, we would have to put out fires whenever there was a requirement change or a script problem. And the cost of writing a case will go up. Based on this consideration, we propose an ability to atomize a set of operations.

Each set of operations should be appropriately named and not be too long.

Custom popover elimination

Now, due to a variety of recommendation + push, some recommendation window will appear uncontrollably in various scenarios. These Windows are very difficult to deal with.

Based on this phenomenon, the recommendation window is specially optimized in each link.

POP \ _WINDOWS = {# 'students end result \ _ \ _ posture and inspections' : \ [' bt \ _pose \ _debug \], #' students end result \ _ \ _ posture and inspections' : \ [' BTN \ _start ', 2, 'KEYCODE \ _BACK', 1, 'bt \ _pose \ _debug', 1 \], students' end result \ _ \ _ posture and inspections' : \ [' BTN \ _start \ _finish ', 1, 'KEYCODE \ _BACK', 0.5 \], 'students end \ _ operation mode \ _ my homework remind' : \ [' \ _close iv, 1 \], 'end \ _ to upgrade students' : \ [' update \ _reboot \ _later ', 1 \], 'student end \ _ work began to tip: \ [' \ _close iv, 1 \],' MEDALS \ _ get ': \ [\ [320, 1300 \], 1 \],}Copy the code
def dealWithPopWindows(self, win): popWindows = self.getPopWindows() if popWindows == None or len(popWindows) == 0: return win for k,v in popWindows.items(): if window\_manager().isWindow(win, k): for item in v: if type(item) == int or type(item) == float: self.wait(item) elif type(item) == list and len(item) == 2: self.mainDevice().touch(item) elif item.startswith('KEYCODE'): Log.warning(" Pop window \[{}\] found, try to close by Key \[{}\] ".format(k, item)) self.mainDevice().keyevent (item) else: Log.warning(" Pop window \[{}\] found, try to close by key \[{}\] ".format(k, item) win.clickItem(item) self.wait(1) # Recursion is necessary here because there is such a scenario. Let's say the loop order is ABCD, C is done and it becomes A, but A is already looped. In other cases, clicking a button does not close successfully, so recursion can be used to check the interface repeatedly. So it's safer to iterate recursively. Return self.dealWithpopWindows (self.refreshWin()) return winCopy the code

When writing code, you only need to define popover information files and exit methods in constants (support for multi-step exit). You don’t need to worry about these details when writing code, because it’s all wrapped up.

Use case writing: Enrich means to find more problems

Automatically recommend page validation elements

Like the Response JSON tree on the server side, the Android display page is also a View tree structure. So like server result validation, page validation is page tree validation.

The idea is to evaluate the value of each node in the page tree. Android has container View, ButtonView, TextView. In our business experience, we tend to focus more on ButtonView (clickable properties) and TextView (whether or not it has text properties). And in a container View, if it’s a ListView or a RecycleView, we’re going to infer whether it’s generated in real time by determining whether the ViewiDs in it are the same. If it is generated in real time, the verification value is very weak.

Also, we will judge whether a node has verification value from the dimension of display area size. Views that display areas are too large or too small have low validation values.

Image search/text search

Supports the detection of the existence of keywords (can be multiple at the same time, support fuzzy matching).

You can also directly click:

You can also try again and again over a period of time for scenarios where the page load time is uncertain.

Multiple devices work together

The component supports multiple devices

Practical application in case

Play/record/record a video

The ability to use assistive equipment

  • Playback (Voice test)
  • Video recording (video framing scheme)
  • Recording (by detecting ripples to determine whether there is real sound, through ASR to identify whether the released sound is in line with expectations)

The audio Mock file fills/broadcasts the Mock

Both require research and development, which can reduce dependence on the environment. The stages we went through with audio testing were:

  1. MAC sounds
  2. Auxiliary machine playback

Because the computer cannot play the sound after bytest, can only use the auxiliary machine to play the sound. Disadvantage is very cost environment, need sound insulation equipment. The current mode is not completely obsolete, because to simulate the real scene also need to auxiliary machine playback.

  1. Radio simulation

A better way. Through the broadcast text, the broadcast text is equivalent to (the text result of the spoken ASR). The disadvantage is that there is a gap between the real business process and some buried points will be lost. It requires r&d collaboration.

  1. Voice files are added

Directly with voice files into the process and the real scene is basically consistent. It also requires research and development.

Picture a Mock

We started with an assistive device (PAD). But the follow-up effect is not very good, so use paper instead. However, the paper could not be dynamically switched. Finally, we discussed a mock scheme with the R&D team, which could directly mock the results of photos with the files we pushed.

Use case update maintenance

Automatic compatible AB experiment

Page jump logic changes.

Version A: A->B->C->D

Version B: A->B->D

If you want multiple versions of automated scripts to run, you must be compliant with process differences.

Similarly, if you need to do the AB test, as long as the configuration of the relevant page information, then you can also do a code.

Problems solved:

1: solve AB test

A plan:

entrance -> service_ui_A -> service_ui_A_1 -> end_page

Plan B:

entrance -> service_ui_B -> service_ui_B_1 -> service_ui_B_2 -> end_page

if helper.judgeAndAction({ 'entrance' : lambda win: helper.tapKey('enter'), 'service\_ui\_A' : lambda win: helper.tapKey('enter\_a'), 'service\_ui\_A\_1' : lambda win: helper.tapKey('enter\_a1'), 'service\_ui\_B' : lambda win: helper.tapKey('enter\_b'), 'service\_ui\_B\_1' : lambda win: helper.tapKey('enter\_b1'), 'service\_ui\_B\_2' : lambda win: helper.tapKey('enter\_b2'), 'end\_page' : None, # exitPage}, 'switch to main screen ', timeToWait=20, timeWaitPerTime=0.5, actionWhenNoMatch=lambda: exitPage(helper)):Copy the code

2: Some pages will show that you received XX medal after completing missions. But I don’t always get out, so this is the time

If not self.judgeAndAction({' MEDALS \_ get ': lambda win: self.tapKey('submit'),' self.tapKey ': None,},' self.tapKey '): return FalseCopy the code

3: Some result pages rely on external media as well as internal algorithms. For example, check the job, check the content may or may not be recognized. This is where enumeration and processing of possible result pages are needed.

Return self.judgeAndAction({' check job \_ not checked ': lambda win: self.tapKey(' BTN \_right'),' check job \_ result ': lambda win: Self. tapKey(' BTN \_right'), 'check job \_ pairs in the answer ': lambda win: self.tapKey(' BTN \_right'),' check job \_ all pairs ': lambda win: self.tapKey(' BTN \_right'), 'check job \_ all pairs ': lambda win: Self. tapKey('homework\_close'), 'check homework\_ to answer \_ end ': lambda win: Self. tapKey('homework\_correct\_close'), #' check homework\_correct\_ end \_225': lambda win: Self. tapKey('homework\_correct\_close'), 'check homework\_correct\_ calculation ': lambda win: self.tapKey(' BTN \_next'), 'Launcher\_ lower half ': None, 'Launcher' : None,}, 'check the job ', timeToWait = 60)Copy the code

4: To ensure a return to an interface. Then try to return to a new page by defining actionWhenNoMatch

Here is an example of returning to the main screen of any interface by repeatedly pressing the known exit button.

If helper.judgeAndAction({'Launcher' : lambda win: swipe\_X(helper.maindevice (), False), 'Launcher\_ lower half ': lambda win: Swipe \_X(helper.mainDevice(), False), 'Student side \_ NightMode ': lambda Win: dealWithNightMode(Helper, Win),' Home interface \_ Personal Customization ': Lambda win: Swipe \_X(helper.mainDevice(), True), 'Student \_ Video Call \_ Dialogue simple ': lambda win: Swipe \_X(helper.mainDevice(), True),' Student \_ Video Call \_ Dialogue simple ': lambda win: Helper. tapKey('control\_container'), 'student \_ Video call \_ dialog ': lambda win: Helper. tapKey('buttonCallCancel'),' Student \_ main interface ': None,}, 'switch to main interface ', timeToWait=20, timeWaitPerTime=0.5, actionWhenNoMatch=lambda: exitPage(helper)):Copy the code

On the phone, it’s even easier. Just press the back button

Def backToMain(helper): return helper. JudgeAndAction ({' helper ': None, 'helper ': None,' helper ': lambda win: Helper. tapKey(' home '), 'home interface \_ my \_ not logged in ': lambda win: Helper. tapKey(' home '),' Home interface \_ my \_ logged in ': lambda win: helper.tapKey(' home '), 'Home interface \_ my \_ logged in ': lambda win: Helper. tapKey(' home '), 'home page' \_ learning situation ': lambda win: Helper. tapKey(' home page '), 'parent end \_ Video Call \_ Confirmation type ': lambda win: Helper. tapKey('plugin\ _TV \_cancel'), 'parent \_ Video call ': lambda win: Helper. tapKey('tvCancelBtn'),' parent \_ Video call \_ call ': Lambda win: helper.tapkey ('tvRefuseBtn'),}, 'back to parent home ', actionWhenNoMatch=lambda: helper.mainDevice().press\_back())Copy the code

Multi-version page compatibility

Simply put, you can write multiple Windows and then automatically match them to the most appropriate one at run time

After any single page/multiple page validation, an attempt is made to bind the page to the corresponding version number.

Use case result consumption

Prints a report of the difference between the current page and the expected page snapshot

Check whether the current page matches the historical page snapshot

When our use case fails, with the native shoots, we can only know which key failed. And in fact, if we want to have more information on automatic attribution, or analysis, then we really want to know which page it is.

Of course, we can also obtain the current page through fault screenshots, but sometimes the page is the same as what we think, but the related keys may not be there or renamed, which will bring us difficulties in locating.

In conclusion, we need to output more information when failures occur to help us locate problems and automate attribution.

Multiple results are identified here, in order of highest to lowest match. Therefore, the current page is identified as student-side _ main interface _255.

Error analysis generates/opens bugs automatically

In order to reduce the labor cost of the result consumption, we try to adopt automatic consumption. We will use case steps + + + the reason for the error, the current page (expect page buttons | | image recognition to text) formation of the core reason for the error.

This core error cause is used to automatically attribute results. If the keyword is the same as the previous keyword, it is automatically attributed to the duplicate BUG. The status of the current fault is also attached. If the time of concern has been resolved, corresponding to the merged branch.

Failure scenario Restoration

Collect enough information when a use case fails to restore 99% of the failure scenario.

  • Screenshots of each step
  • Process operation (click the key, check the page passed, image recognition click the real coordinates)
  • Direction logs required by the RD
  • System logs and error level logs during the process

Therefore, once there is a problem, we do not need to go to see it immediately. The real-time requirements for consumption are not high, so as not to interrupt the rhythm of research and development and testing.

Log collection system

Configure the log keyword and output file. Logs are automatically collected during the test.

Key is the file name and value is the keyword. Multiple keywords are supported. If any keyword matches, the file will be written.

This is mainly used for special problems and StrictMode fault collection as mentioned later.

Automatic diagnosis

Automatic diagnosis currently takes many forms

  1. The plug-in

  1. Use case runtime detection

  2. Test according to test results

This is generally based on log detection. It may be specific or customized.

Consumption results are measurable

Align consumption standards with consumption results of the students. Indicate the processing time in the consumption process, because it has just been processed, the error of time estimation is theoretically very small. This way, after running for a while, you can see a breakdown of how long the test results have been consumed throughout the process.

UI automation is considered to have low ROI and poor sense of motion among business students. So what are the practical effects? The output of this data is actually a cognitive correction of conventional wisdom and provides accurate data for automation ROI assessment. With these reports, we can accurately measure the cost of test results, the cost of updating scripts due to development changes, and the cost of script imperfections, allowing us to calculate the actual benefits of client-side automation.

An alarm is generated if the test result is not consumed in time

The test left

Branch inspection

Bring the test forward! The front! The front again!

After r&d pulls the branch out, it is packaged for automated testing. Later, as soon as the development changes, we detect changes in GIT commit information and package the tests. In such an operation, we have a perception of the quality of the branch and can front-load risk.

Artificial and capricious

Performance Data Collection

The system collects performance data during the test and automatically checks for memory leaks or generates alarms for certain thresholds

Algorithm for acceptance

There are some algorithmic changes that require end-to-end pressure testing. At this point, you need to use the ability of the image/audio Mock to switch resources under test and then test for acceptance.

future

UI automation is still hard to do as a completely unattended test. And the development changed the page, so the testing had to follow up. It’s hard to be truly adaptive. Personally, I think the future development of UI automation will continue in two directions:

1. Fine UI verification

You can replace most of the labor by sampling UI controls + taking screenshots + memorizing layout + manual assertion base.

2. Intelligent traversal + automatic assertion

Calendar all UI controls, including new pages. Can do automatic assertion, automatic screening of problems to submit for manual review

ReZhao

The following positions are now open: Test development (senior) Engineer