Currently, under the lean development mode of Xianyu, the whole technical team faces many abilities and challenges, especially the 2-1-1 goal in efficiency (2 weeks demand delivery cycle, 1 week demand development cycle, and 1 hour to reach the release standard). Under this big goal, we must do the best in every link. The construction of automation is the key ability to determine the success or failure of CI (continuous integration). Today, I would like to share the practice of xianyu Android client performance automation.
1 Discovering problems
-
Tools missing:
At present, Taobao has a complete system for online performance level monitoring, but for performance testing of new functions, each business team has a corresponding performance team, and the generated tools are customized and developed according to their own business characteristics. Xianyu client currently uses Flutter as the main development language of the client. The acquisition of Flutter performance data and UI automation testing support tools are currently missing, and there is little industry practice on Flutter automation and performance.
-
N-fold test workload (N= number of branches in a release cycle) :
Together with the original development model is a function of integration testing, performance testing only after the need for integration of package for testing, now into a lane of development mode, a version will generally contain lane branch of about a dozen or more, we must ensure that the performance of the branch of each lane is up to standard, If there is a performance problem, it needs to be reported in the first time. If it is left over to the integration stage, the troubleshooting of the problem (screening in a dozen branches) will take a lot of time to solve the problem, and the efficiency is difficult to be greatly improved.
2 Analyzing problems
Systematic solution, to make each lane branch get effective test coverage, test pieces can be automated execution, continuous feedback results.
Based on the above problems, the following solutions are sorted out:
-
Capture of Flutter performance data (for example, Flutter has its own SurfaceView and the original FPS calculation method cannot be used directly)
-
Implementation of Flutter UI automation (Flutter/Native UI hybrid stack processing)
-
Automatic performance scripts and performance data are automatically collected and reported into the CI process
-
Notification/report presentation/analysis of performance issues
1 Performance Data
[FPS]
Adb shell Dumpsys SurfaceFlinger –latency data from adb shell Dumpsys SurfaceFlinger –latency data from ADB shell Dumpsys SurfaceFlinger — Latency data from ADB shell Dumpsys SurfaceFlinger — Latency data Handling SurfaceFlinger core code is as follows:
dumpsys SurfaceFlinger --latency-clear
#echo "dumpsys SurfaceFlinger..."
if [[ $isflutter = 0 ]]; then
window=`dumpsys window windows | grep mCurrent | $bb awk '{print $3}'|$bb tr -d '}'` # Get the current window
echo $window
fi
if [[ $isflutter = 1 ]]; then
window=`dumpsys SurfaceFlinger --list |grep '^SurfaceView'|$bb awk 'NR==1{print $0}'`
if [ -z "$window" ]; then
window="SurfaceView"
fi
echo $window
fi
$bb usleep $sleep_t
dumpsys SurfaceFlinger --latency "$window"|$bb awk -v time=$uptime -v target=$target -v kpi=$KPI $awkfike >>$file
Copy the code[CPU]
The command of top is used to obtain performance data (this method has the least loss caused by data collection).
export bb="/data/local/tmp/busybox"
$bb top -b -n 1|$bb awk 'NR==4{print NF-1}'
Copy the code[memory]
Dumpsys meminfo $package = Java Heap,Java Heap Average,Java Heap Peak,Native Heap Average,Native Heap Peak, Graphics, Unknown, Pss data
do_statistics() {
((COUNT+=1))
isExist="$(echo $OUTPUT | grep "Dalvik Heap")"
if [[ ! -n $isExist ]] ; then
old_dumpsys=true
else
old_dumpsys=false
fi
if [[ $old_dumpsys = true ]] ; then
java_heap="$(echo "$OUTPUT" | grep "Dalvik" | $bb awk '{print $6}' | $bb tr -d '\r')"
else
java_heap="$(echo "$OUTPUT" | grep "Dalvik Heap[^:]" | $bb awk '{print $8}' | $bb tr -d '\r')"
fi
echo "1."$JAVA_HEAP_TOTAL "2."$java_heap "3."$JAVA_HEAP_TOTAL
((JAVA_HEAP_TOTAL+=java_heap))
((JAVA_HEAP_AVG=JAVA_HEAP_TOTAL/COUNT))
if [[ $java_heap -gt $JAVA_HEAP_PEAK ]] ; then
JAVA_HEAP_PEAK=$java_heap
fi
if [[ $old_dumpsys = true ]] ; then
native_heap="$(echo "$OUTPUT" | grep "Native" | $bb awk '{print $6}' | $bb tr -d '\r')"
else
native_heap="$(echo "$OUTPUT" | grep "Native Heap[^:]" | $bb awk '{print $8}' | $bb tr -d '\r' | $bb tr -d '\n')"
fi
((NATIVE_HEAP_TOTAL+=native_heap))
((NATIVE_HEAP_AVG=NATIVE_HEAP_TOTAL/COUNT))
if [[ $native_heap -gt $NATIVE_HEAP_PEAK ]] ; then
NATIVE_HEAP_PEAK=$native_heap
fi
g_Str="Graphics"
if [[ $OUTPUT == *$g_Str* ]] ; then
echo "Found Graphics..."
Graphics="$(echo "$OUTPUT" | grep "Graphics" | $bb awk '{print $2}' | $bb tr -d '\r')"
else
echo "Not Found Graphics..."
Graphics=0
fi
Unknown="$(echo "$OUTPUT" | grep "Unknown" | $bb awk '{print $2}' | $bb tr -d '\r')"
total="$(echo "$OUTPUT" | grep "TOTAL"|$bb head -1| $bb awk '{print $2}' | $bb tr -d '\r')"
}
Copy the code[traffic]
Use Dumpsys Package Packages to parse out the current package to be tested to obtain traffic information
uid="$(dumpsys package packages|$bb grep -E "Package |userId"|$bb awk -v OFS=" " '{the if ($1 = = "Package") {P = substr ($2, 2, length ($2) - 2)} else {the if (substr ($1,1,6) = = "userId") print P, substr ($1, 8, length ($1) - 7)}} '| grep $package | $bb awk' {print $2} ')"
echo "Net:"$uid
initreceive=`$bb awk -v OFS=" " 'NR>1{if($2=="wlan0"){wr[$4]+=$6; wt[$4]+=$8}else{if($2=="rmnet0"){rr[$4]+=$6; rt[$4]+=$8}}}END{for(i in wr){print i,wr[i]/1000,wt[i]/1000,"wifi"}; for(i in rr){print i,rr[i]/1000,rt[i]/1000,"data"}}' /proc/net/xt_qtaguid/stats | grep $uid|$bb awk '{print $2}'`
inittransmit=`$bb awk -v OFS=" " 'NR>1{if($2=="wlan0"){wr[$4]+=$6; wt[$4]+=$8}else{if($2=="rmnet0"){rr[$4]+=$6; rt[$4]+=$8}}}END{for(i in wr){print i,wr[i]/1000,wt[i]/1000,"wifi"}; for(i in rr){print i,rr[i]/1000,rt[i]/1000,"data"}}' /proc/net/xt_qtaguid/stats | grep $uid|$bb awk '{print $3}'`
echo "initnetarray"$initreceive","$inittransmit
getnet(){
local data_t=`date +%Y/%m/%d" "%H:%M:%S`
netdetail=`$bb awk -v OFS=, -v initreceive=$initreceive -v inittransmit=$inittransmit -v datat="$data_t" 'NR>1{if($2=="wlan0"){wr[$4]+=$6; wt[$4]+=$8}else{if($2=="rmnet0"){rr[$4]+=$6; rt[$4]+=$8}}}END{for(i in wr){print datat,i,wr[i]/1000-initreceive,wt[i]/1000-inittransmit,"wifi"}; for(i in rr){print datat,i,rr[i]/1000-initreceive,rt[i]/1000-inittransmit,"data"}}' /proc/net/xt_qtaguid/stats | grep $uid`
echo $netdetail>>$filenet
}
Copy the code2 Performance automation scripts
-
Based on the automation cases of Appium, there are a lot of practices in this technology industry. I will not repeat them here. If you are not familiar with them, you can go to the official website of Appium, http://appium.io
-
Switch between Flutter and Native pages using Schema jump in App
-
Interactive scenarios such as text input for a Flutter page are manipulated using Integration tests based on the Flutter framework belt
An integration test
Generally, an integration test runs on a real device or an OS emulator, such as iOS Simulator or Android Emulator. The app under test is typically isolated from the test driver code to avoid skewing the results.
The application of THE UI automation of Flutter and the processing of the mixed page of Flutter/Native will be introduced separately in the following articles. For the relevant principles, please refer to “Thousands of Facets Recording and Playback Technology”.
3 Performance Automation CI process
4 Performance data report
[FPS]
-
Framediff: The difference between the start time and end time of drawing a frame
-
FPS: The number of frames displayed per second
-
Frames: All Frames in a refresh cycle
-
Jank: If a frame is drawn from the beginning to the end of more than 16.67ms, JanK will be recorded once. Jank is non-zero, which means that the hardware draws frames and is related to the screen hardware performance and related drive performance
-
Jank2: JanK2 is recorded once from the beginning of a frame to the end of a frame that exceeds 33.34ms
-
MFS: Maximum elapsed time per frame in a refresh cycle (the time difference between each two lines of vertical synchronization represents the frame interval between two frames drawn)
-
OKT: The number of times the frame elapsed exceeded 16.67ms in a refresh cycle
-
SS: Fluency, calculated by FPS, MFS and OKT, fluency = the ratio of actual frame rate to target frame rate 60 [the higher the target frame rate is, the better] + the ratio of target time to the difference between the two frames 20 [the lower the difference between the two frames is, the better] + (1- the number of times over 16ms/frame)*20 [the fewer times the better]
[CPU]
[Memory]
1 Monitor branch performance of a swimlane
The performance problems of the swimlane branch can be seen clearly on the report
2 Performance specific support
The Flutter merchandise detail page was reconstructed in 14 rounds of tests
Client image Unified resource test Four rounds of tests
Performance automation is only one part of the WHOLE CI process. To achieve the goal of maximum efficiency, Xianyu Quality team also produces many supporting tools, such as CI platform, traversal test, AI error identification, automatic use case generation and so on, which will be shared with you later.
https://testerhome.com/topics/2232
https://testerhome.com/topics/4775
The open source | the Flutter application framework Fish story behind the 200 million users
TensorFlow app “UI 2 Code”
Thousands of online problem playback technology revealed
How to filter data from any dimension of ten billion tables in milliseconds…
