Afore-mentioned: Some time ago, there was a demand for performance analysis, so we need to do a set of automatic testing tools for the performance of front page. Leaders have spoken, that let’s do it. It took a week to find the materials and match the good and suitable open source plug-ins. It took about a month (on and off) to build and develop the tool. A platform has been built to support such tools. VUE for the front end, Node for the back end (because I’m familiar with the front end), and mangoDB for the database. Well, not much to say, on talent ~

Design background & front-end principles

Design Background:

  1. Hard requirements: Perform performance analysis and optimization on the web and H5 pages
  2. Pain points:

Data collection requires manual intervention: The analysis through Perfomance of Chrome Dev-Tool requires manual refreshing and manual recording, which requires a lot of rework, especially in the case of controlling variable tests; Example:

2.2 Complexity of graphical continuous integration: Using code to log, that is, using TimeEvent time recorded in performance. Timing to calculate the page performance indicators, the integrity of indicators and graphical display still need work 3. Objective: To combine performance data collection with Selenium for automated collection, analysis and processing. Dynamic scripts are implanted to collect and analyze data in real time when performing UI automation operations.

Browser rendering principles:

  1. Browser rendering

  1. Browser rendering engine

2.1 The browser is the tool for Html parsing and final page presentation.

2.2 Main functions: To present web resources selected by users, the resources need to be requested from the server and displayed in the browser window. The format of resources is USUALLY HTML, including PDF, Image and other formats. In the browser component, the rendering engine is the part directly related to the user, rendering the page that the user wants.

2.3 Browser kernel [Rendering engine] : the function is to transform the page into visual image results. Rendering Engine workflow:

The first step is to parse HTML to build a DOM tree: start with the parsing of HTML tags, and parse the various tags into each node in the DOM tree. The node of the tag and the DOM tree is a one-to-one correspondence.

Step 2: Build the render tree: Parse the style information in the CSS and style TAB into a render tree. The render tree consists of rectangles containing attributes such as color and size that are displayed on the screen in the correct order.

Step 3, render the tree layout: Determine exactly where each node will appear on the screen.

Step 4, render tree drawing: Walk through the render tree and draw each node using the UI backend layer.

  1. H5 Page execution process:

[Page loading time :firstPaint, DomContentLoaded(first screen visible), OnLoad (first screen loaded)]

Parsing HTML structure

Load external script and style sheet files.

Parse and execute the script code. // Some scripts block page loading.

The DOM tree is built. / / DOMContentLoaded event.

Load external files such as images.

The page is loaded. / / the load event

Performance indicators + optimization method

Performance indicators:

HTTP related

1. Number of HTTP requests. Solution: CSS Sprite, picture map, JS CSS merge.

2. Check whether the component is compressed. Solution: compression method, compression object, image format and size appropriate, CSS at the top, JS at the bottom, JS & CSS compression, whether to add cache, avoid non-200 return values, use CDN.

Time correlation [time consuming]

1. White screen time: the time when the user first sees the content on the page, i.e. the time when the first rendering process is completed.

2. First screen time: When the user sees the first screen, that is, the area at the top of the page with the size of the current window, the complete time is displayed. Try to make it fit the one-second rule.

3. First resource download time: the download time from the start to the completion of the first resource, excluding the page drawing time.

4. Total resource download time: the time from the start of downloading to the completion of downloading all resources, excluding the page drawing time.

5. User operational time: the time from the page loading to user operational response.

WebView related [Memory/Traffic, CPU]

When testing H5 performance on Android and IOS, you should also pay attention to the general performance metrics of your app due to loading H5.

1. Memory: Memory changes before and after page loading can indirectly reflect the number and size of resources in H5, such as dom number and image size.

2.CPU: When the resource style of the page is complex and the visual effect is emphasized, the CPU usage can be observed to reflect the H5 rendering quality. If the CPU usage is high for a long time, you can consider reducing the visual effects of high computational effort.

3.FPS(Smoothness) : The frame rate, especially in H5 with video and animation effects, should be focused on to prevent severe lag.

Optimization method:

Reduce the number of HTTP requests:

1) Merge pictures: When there are many pictures, for relatively stable pictures, the number of pictures can be reduced and merged into a large one, so as to reduce the number of HTTP requests. Second, you can add caching to prompt skills. Merging large images can be handled using CSS Sprites technology.

2) Combine and compress CSS stylesheet and JS script: The tool can be used with Minify, YUI Compressor, etc.

3) Remove unnecessary requests: residual invalid connections that have no real effect on the page. You can see this with the Firebug tool.

4) Make the most of the cache: Focus on the client cache information.

If the current time is less than (<) Expires, the browser will retrieve the relevant data or HTML file directly from the cache; If the current time is greater than Expires, the browser sends a request to the server to retrieve the data.

Image optimization

  1. Use PNG images whenever possible, which are relatively small.
  2. For different image formats, do an optimization before going online. For example, PNG can be optimized by the tool Pngcrush, JPG can be optimized by Jpgtran, and GIF can be optimized by Gifsicle.
  3. Lazy loading of images, also known as lazy loading.

other

1. Use CDN: Content Delivery Network. Basic idea: avoid the bottlenecks and links that may affect the speed and stability of data transmission on the Internet as far as possible, so that the content transmission is faster and more stable. By preventing node server throughout the network on the basis of the existing Internet consisting of a layer of intelligent virtual network, CDN system can in real time according to the network flow quantity and each node connections, load condition and the distance to the user and comprehensive information such as response time, will the user’s request to guide users closest service node.

2. Enable GZIP: a common data compression format used to compress all text resources transmitted over the Internet, such as HTML, CSS, and JS.

3. Style sheet and JS file optimization: location storage. The CSS stylesheet file is placed at the top of the page; JS file at the end of the page.

4. Use a cookieless domain name: When a request is sent, a static picture is requested and a cookie is sent, the server will not do any use for these cookies, that is to say, these cookies have no use, there is no need to send along with the request.

5. Front-end code structure optimization: the logic of calculation is put into the back end, and the front end is only responsible for display. At the same time, the interfaces that provide data in the back end are split, not all squeezed into one interface.

6. Other optimization methods: 1) Avoid CSS@imort; 2) Optimize DNS lookup. Set HostnameLookups in the Apache httpd.conf configuration file to off to reduce the number of DNS queries. 3) Remove duplicate scripts; 4) Reasonable use of Etag; 5) the Favicon. Icon; 6) Avoid returns that are not 200;

Technology selection and project design

Front frame: VUE (Cn.vuejs.org/)

What is VUE:

Vue is a set of progressive frameworks for building user interfaces

Why VUE:

Lightweight, high-performance, with componentized MVVM libraries and easy-to-use apis.

Backend service: nodeJs (Nodejs. Cn /)

Database: MongoDB (www.mongodb.org.cn/).

Plugin: Browsertime + Coach + Pagexray

Browsertime (source:www.github.com/sitespeedio…

Browsertime uses Selenium NodeJS to drive the browser. Load a URL and execute configurable javascript to collect metrics for Browsertime’s four core functions: It uses the browser (Firefox/Chrome) to handle all content. It executes a set of default and configurable JS scripts when the URL completes loading in the browser. It records a video of the browser screen used to calculate visual metrics.

It allows the browser to run Selenium scripts (and dynamic scripts, for example: play a video) before and after accessing the URL (login user, and so on). Used in two different situations: run as a stand-alone tool to collect performance timings for a website. Integrates into the tool as JavaScript Runner to collect any JavaScript metrics/information. Running example:

Configuration Browsertime. Browsertime Starts Firefox/Chrome with Selenium (Webdriver: Chromedriver/Geckodriver) Browsertime start FFMPEG to record the video of the entire automation process open the browser and access the specified URL. When the page starts to load, Browsertime executes a JavaScript timing script to start measuring and collecting data:

  1. Navigation Timing metrics
  2. User Timing metrics
  3. First paint
  4. RUM Speed Index

Once the load is complete, data is collected to generate the HAR file, which records all requests/responses throughout the load. FFMpeg stops and the generated video file is analyzed. And Browsertime collects visual metrics.

Coach (source:Github.com/sitespeedio…

To analyze the page to execute JS script in the browser to check the rendering path, check whether the image is scaled in the browser, and so on, to score the page Performance, Web Best Practices, and availability, three core indicators. A score between 0 and 100 will be awarded. If you get 100, your page performance is great;

Why use this plug-in:

  1. Suggestions for making pages load faster;

  1. Use a real browser to view your page and simulate the real user behavior
  2. Each proposal has one or more unit tests
  3. It’s not just performance: it also gives you the accessibility of the current page and web best practices
  4. You can combine the knowledge in the DOM with HAR to give you powerful advice
  5. Can be used in conjunction with other Web performance testing tools (can be used with Browser Time)

Running example:

PageXray

PageXray converts HAR files into a JSON format that is easier to read and use. PageXray collects:

Size and number of requests for each content type. Size and request for each domain. Number of requests per response code. Base fields and httpVersion for base assets (the main HTML document). All assets with the following data (response) : Type, URL, size, expiration (normalized expiration converts the maximum lifetime/expiration to an expiration in seconds only), status (response code), timeSinceLastModified (using the last modified field in the response) header and standardizing it to seconds), HttpVersion and all request and response headers. If we use HAR from WebPageTest, we will also get SpeedIndex and other VisualMetrics. You’ll also get some additional metrics, such as SpeedIndex.

Graphics tool: Echarts(Echarts.apache.org/zh/index.ht…

Automation: Selenium (www.selenium.dev/) & WebDriver(www.selenium.dev/documentati…)

CI Integration: Graphite (Grafana graphiteapp.org/) (grafana.com/grafana/)

Project design flow chart:

Project landing (GitHub:Github.com/chenghuangu…

Supplement: Data comparison function of competing products:

Finally: Xiao Bai on the road, the first attempt to design and post. Have wrong, say wrong, improve, also please give directions ~