“Pixel Point Travel WebView”

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  3:      20053      20264      12264      11228       7765       7477       7543       9172       GIC  27 Edge      arch_timer
  5:        928       1570       1080        994        483        375        505        522       GIC  80 Level     timer
  6:          0          0          0          0          0          0          0          0       GIC  81 Level     acpu osa
  7:         91          0          0          0          0          0          0          0       GIC 103 Level     acpu om_timer
  8:          0          0          0          0          0          0          0          0       GIC 104 Level     hard_timer_irq
  9:       1748          0          0          0          0          0          0          0       GIC 186 Level     ipc_irq
 10:          0          0          0          0          0          0          0          0       GIC 187 Level     ipc_sem
 39:       3803          0          0          0          0          0          0          0       GIC 152 Level     ffd73000.i2c
 40:       1890          0          0          0          0          0          0          0       GIC 113 Level     fdf0c000.i2c
 41:       2006          0          0          0          0          0          0          0       GIC 114 Level     fdf0d000.i2c
 43:        103          0          0          0          0          0          0          0       GIC 107 Level   
 48:          0          0          0          0          0          0          0          0       GIC 145 Level     pl022
 49:          0          0          0          0          0          0          0          0       GIC 112 Level     pl022
 52:     100195          0          0          0          0          0          0          0       GIC 182 Level     asp_irq_slimbus
 53:      36182      49662          0          0          0          0          0          0       GIC 169 Level     mmc0
 54:       2414          0          0          0          0          0          0          0       GIC 290 Level     gpufreq
 55:          0          0          0          0          0          0          0          0       GIC 291 Level     gpufreq
 56:       6464          0          0          0          0          0          0          0       GIC 292 Level     gpufreq
 57:       3530          0          0          0          0          0          0          0       GIC 277 Level     irq_pdp
 58:          0          0          0          0          0          0          0          0       GIC 278 Level     irq_sdp
 59:          0          0          0          0          0          0          0          0       GIC 279 Level     irq_adp
 64:         56          0          0          0          0          0          0          0       GIC 171 Level     dw-mci

Copy the code

When we learned the principles of computer before, we knew that when an interrupt occurs, the CPU will stop the current running program, save the current running state, and then jump to the corresponding interrupt handler for processing, which is generally implemented by the three-party kernel driver. Such as Android in the input drivers/input/touchscreen ektf3k. C. The driver calls its input_report_abs and other methods to record the coordinates under the touch screen, and then writes them to the /dev/inpu/event1 device file via the input module. You can view this information using the Getevent tool.

adb shell su — getevent -lt /dev/input/event1

[   78826.389007] EV_ABS       ABS_MT_TRACKING_ID   0000001f
[   78826.389038] EV_ABS       ABS_MT_PRESSURE      000000ab
[   78826.389038] EV_ABS       ABS_MT_POSITION_X    000000ab
[   78826.389068] EV_ABS       ABS_MT_POSITION_Y    0000025b
[   78826.389068] EV_ABS       ABS_MT_SLOT          00000001
[   78826.389068] EV_ABS       ABS_MT_TRACKING_ID   00000020
[   78826.389068] EV_ABS       ABS_MT_PRESSURE      000000b9
[   78826.389099] EV_ABS       ABS_MT_POSITION_X    0000019e
[   78826.389099] EV_ABS       ABS_MT_POSITION_Y    00000361
[   78826.389099] EV_SYN       SYN_REPORT           00000000
[   78826.468688] EV_ABS       ABS_MT_SLOT          00000000
[   78826.468688] EV_ABS       ABS_MT_TRACKING_ID   ffffffff
[   78826.468719] EV_ABS       ABS_MT_SLOT          00000001
[   78826.468719] EV_ABS       ABS_MT_TRACKING_ID   ffffffff
[   78826.468719] EV_SYN       SYN_REPORT           00000000
Copy the code

When the coordinates are written to the /dev/input-event1 device file, the application only needs to listen for changes in the device file to know which touches the user has made. However, this is a tedious task that is handed over to the GUI framework of each system. WindowManagerService is a window management service in the Android system. It reads the location information and distributes it to the specified App. It calls back the response listener function (onTouch). At this point, the user’s touch event is passed to the page of the App, and the onTouch method is called back to respond to the user’s touch event. To open an H5 on Android, you typically start an Activity that has a WebView inside and calls the loadUrl method to load the URL.

1 Start and create an Activity

Note: This part involves more Android knowledge, students can skip the front end.

The startup flow chart of the Activity is as follows:

• Instrumentation: Monitor the interactions between applications and the system.
• AMS: Component management scheduling center, does nothing, but manages everything.
• ActivityStarter: a controller that starts an Activity. It handles the effects of intents and flags on the Activity. To be specific, it can: 1. 2 Verify the validity of startup parameters. 3 Returns the int parameter, indicating whether the Activity started successfully.
• The ActivityStackSupervisior: This class, as you can see from its name, is designed to manage the stack of tasks.
• ActivityStack: Used to manage activities in the task stack.
• ActivityThread: This class is used to start, switch, and schedule activities, services, and BroadcastReceivers.

Note: The ActivityStack container is designed to handle multiple Activitystacks. Earlier versions of the container had just one ActivityStack for the phone’s screen, and when the larger container was designed to handle multiple screens, We had multiple Activitystacks, and the Activitystackcontainer was introduced to manage multiple Activitystacks. The whole process mainly involves four processes:

• The caller process is the application Launcher process if the application is launched on the desktop.
• ActivityManagerService is the System Server process that runs System service components.
• Zygote process, which is mainly used to fork new processes.
• The newly started application process, which is used to host the application running process, is also the main thread of the application (the newly created process is the main thread), and handles the component life cycle, interface drawing, and other related matters.

With this understanding, the process can be summarized as follows:

1. Click the desktop application icon, and the Launcher process sends AMS the request to start the Activity (MainActivity) as a Binder.
2. AMS receives a start request, information such as delivery ActivityStarter processing Intent and Flag, and then to ActivityStackSupervisior/ActivityStack Activity into the stack related processes. At the same time, the Zygote process is asked to fork a new process in Socket mode.
3. Zygote forks a new process after receiving a request to create a new process.
4. Create the ActivityThread object in the new process, which is the main thread of the application. Start the Looper message loop in the main thread to process the creation Activity.
5. The ActivityThread uses the ClassLoader to load the Activity, create the Activity instance, and call back the Activity’s onCreate() method. This completes the start of the Activity.

Note: Many of these functions have the Locked suffix, which means that they are synchronized, reading and writing data shared by multiple threads.

As you can see above, Chromium has multiple processes, specifically:

There are two implementations of Chromium on Android.

• Chrome: Each process contains multiple threads, such as Main Thread and I/O Thread. Render Process and GPU Process are implemented based on Android Service components.
• Android WebView: Android View component, can be used in Android App, load H5.

• Single Process: Render Process and GPU Process become Browser Process threads.
• Multi-process: Android 7.0 added the developer option to enable multi-process. Android 8.0 enables multiple processes by default. Currently only Render Process, GPU Process is still a thread of Browser Process.

When we build a WebView instance, it will load the Chromium-related so library first, and then start Browser, Render, and other processes and the necessary components for the process to run. So the whole container starts up. The startup process is as follows:

• Browser: The AwBrowserProcess in // Android_webView is responsible for launching the Browser, which encapsulates the BrowserStartupController in // Content /public/ Android. The process of starting Browser is actually to create a Browser Main Loop in the UI thread of App. When Chromium needs to request Browser to perform an operation in the future, it can send a Task like Browser Main Loop. Of course the Task is executed in the drawing UI thread.
• // android_webView AwContents, which encapsulates //content/public/android ContentViewCore. Start Render differently in different versions.
  • Prior to Android O, there was a single-process architecture, which created a thread in the current App process in which the web page would be rendered. This is called the in-process Renderer.
  • After Android O comes the multi-process architecture, which creates a separate process in which web pages will be rendered. This is called an out-of-process Renderer.
• GPU: Chromium’s GPU implementation is in the Render Thread of the App, which is created by the App itself, since Chromium doesn’t need to launch it separately. But android_webview module will start a DeferredGpuCommandService service, when the CHromium Browser and Render need to perform operation on the GPU, Will send DeferredGpuCommandService service request, DeferredGpuCommandService services through the App UI Thread will GPU to Render operation Thread of execution.

After the container is started, loadUrl can be called to load the page.

Page rendering

The pictures in the rendering process are from the screenshot of PPT Life of a Pixel by Chromium engineer. The rendering process of a browser is to render a web page through a rendering pipeline into pixels that are eventually exported to the screen. There are three characters involved

• Input side: web page, which Chromium abstracts into Content.
• Rendering pipeline: Blink is mainly responsible for DOM parsing, style layout, drawing and other operations, converting web content into drawing instructions.
• Output: mainly responsible for the drawing instructions into pixels, display on the screen.

What is Content?

We see the concept of Content a lot in the Chromium project, so what is Content? Content is the area for rendering web Content, corresponding to AwContent in the Java layer and represented by WebContents at the bottom, as shown below:

Content is described in code by Content ::WebContents, which is created by Blink in a separate Render process. Specifically, Content corresponds to HTML, CSS, JS and image involved in front-end development, as shown below:

What is a Rendering Pipeline?

The rendering pipeline can be understood as a disassembly of the rendering process, like a factory assembly line, where the semi-finished product generated in one workshop is sent to the next workshop for further assembly. Disassembling the rendering process helps simplify the rendering process and improve rendering efficiency.

When rendering is dynamic, it will trigger rendering and update pixel points when the content changes. Like the Android rendering system, triggering rendering is also triggered by the invalidate mechanism. After triggering rendering, it is very expensive to execute the whole rendering pipeline.

• Triggering conditions are as follows:
  • scrolling
  • zooming
  • animations
  • incremental loading
  • javascript
• The triggering methods of each process are as follows:
  • Style: Node: : SetNeedsStyleRecalc ()
  • Layout: LayoutObject: : SetNeedsLayout ()
  • Paint: PaintInvalidator: : InvalidatePaint ()
  • RasterInvalidator::Generate()

After the rendering pipe converts the webpage into drawing instructions, it cannot directly turn the drawing instructions into pixels (raster) and display them on the screen (Window). At this time, it needs to use the ability of the operating system itself (the underlying graphics library). Most platforms follow the API standardized by OpenGL in terms of the graphical interface. DirectX on Windows, Vulcan on Android. As shown below:

Let’s start with structure

• Blink: The Render thread running in the Render process, which is the Chromium Blink rendering engine, mainly responsible for HTML/CSS parsing, jS interpretation execution (V8), DOM manipulation, typesetting, layer tree construction and update and other tasks.
• Layer Compositor: the Compositor thread running in the Render process is responsible for receiving the Main Frame generated by Blink, Layer tree management, Layer scrolling, rotation and other matrix changes, Layer partitioning, rasterization, texture uploading and other tasks.
• Display Compositor: UI thread running in the Browser process that receives the Compositor Frame generated by the Layer Compositor and outputs the final OpenGL rendering instructions to draw the web content to the target window via GL mapping operations.

It also mentioned the output Frame of each level. The Frame describes the encapsulation of the data related to the drawing content output from the lower module of the rendering line to the upper module.

• Main Frame: Contains a description of the content of a web page, mainly in the form of drawing instructions, or understood as a vector snapshot of the entire web page at a point in time.
• Compositor Frame: The Layer Compositor receives the Main Frame generated by Blink and converts it into an internal synthesizer structure. It is sent to the Browser and eventually to the Compositor Frame, which consists of two main parts:
  • Resource: This is an encapsulation of Texture, the Layer Compositor divides each Layer into blocks, assigns resources to each block, and arranges the rasterization task.
  • Draw Quad: This represents the Draw command (rectangle Draw command, specifying coordinates, size, transformation matrix, etc.). When the Layer Compositor receives Browser’s Draw request, it generates a Draw Quad command for each block of each Layer in the currently visible area.
• GL Frame: Display Compositor converts each Draw Quad Draw instruction of the Compositor Frame into a GL polygon Draw instruction that maps the target window using the corresponding Texture of the Resource wrapper. The set of GL drawing instructions constitutes a GL Frame, and finally the GPU executes these GL instructions to complete the drawing of the web page in the visible area of the window.

The scheduling of the entire rendering pipeline is based on requests and state machine responses. The scheduling hub runs in the Browser UI thread, which sends requests to the Layer Compositor to output the next frame according to the VSync signal of the display. The Layer Compositor determines whether to Blink the next frame according to the state of its state machine. The Layer Compositor and Display Compositor are producers and consumers. The Display Compositor holds a Compositor Frame queue that is constantly drawn and removed. The output frequency depends on the input Frame rate of the Compositor Frame and the drawing frequency of its own GL Frame. Let’s talk about the process

1. Parse/DOM: Content is parsed into a DOM tree, which is the basis for subsequent rendering processes.
2. Style: Parses and applies the Style sheet.
3. -Leonard: Layout.
4. Compositing Update: Split the entire page into separate layers according to certain rules for isolated updates.
5. Prepaint: Builds a property tree so that a node (transform, crop, effect, scroll) can be manipulated independently without affecting its children.
6. Paint: Paint. Verbs can mean paint, etc. Paint transforms a Layout Object in a Layout Tree into a drawing instruction (for example, draw a rectangle, draw a font, draw a color, which is a bit like calling a drawing API). These operations are then encapsulated in Dsipaly items, so the Display items are like paint, and it hasn’t really started to Draw yet.
7. Commit: Commit copies the paint data to the synthesizer thread.
8. Tiling: After raster receives the paint instructions, he blocks the layers first. The diagram block is the basic working unit of Raster.
9. Raster: Rasterization.
10. Activate: Rasterization is an asynchronous process, so the Layer Tree is divided into a Pending Tree (rasterizing the Layer that receives the Commit) and an Activate Tree (drawing the rasterized Layer from the Pending Tree). The process of copying a Layer from a Pending Tree to an Activate Tree is called Activate.
11. The Draw: After the blocks are rasterized, the synthesizer thread generates draw Quads for each block. These Draw Quads are encapsulated into a Compositor Frame and output to the GPU. The DRAW operation is the process of generating Draw Quads.
12. Display: After generating the Compositor Frame, Viz calls the GL command to output draw Quads to the screen.

Let’s look at the specific process.

01 Parse

The related documents

• The DOM standard

Related to the source code

• /blink/renderer/core/dom

When you download an HTML document from a server, the first step is parsing. The HTML parser takes in tags and streams of text (HTML is plain text) and parses the HTML document into a DOM tree. The DOM (Document Object Model), the internal representation of the DOM and the page, also exposes apis for JavaScript (V8 DOM API), allowing JavaScript programs to change the structure, style, and content of a Document. It is a tree structure, and we will see many more trees (layout trees, attribute trees, etc.) later in the rendering process because they are based on the DOM tree structure (HTML structure).

Note: HTML documents may contain multiple DOM trees, often referred to as Shadow trees, because HTML supports custom elements.

The process for parsing HTML to generate a DOM tree is as follows:

1. HTMLDocumentParser parses tokens in HTML and generates an object model.
2. HTMLTreeBuilder is responsible for generating a complete DOM tree. The same HTML document can contain multiple DOM trees. Custom Element elements have a shadow tree. Nodes passed in shadow Tree Slot will be found by FlatTreeTraversal traversal down.

The DOM Tree is used as the basis for the subsequent drawing process, and various types of trees are produced based on it. Specifically, the following transformations are performed:

Object conversion

• DOM Tree -> Render Tree -> Layer Tree
• DOM node -> RenderObject -> RenderLayer

DOM Tree (node is DOM node)

When an HTML is loaded, it is parsed to generate a DOM tree. Each node in the DOM tree corresponds to each element in the page, and the page can manipulate the DOM tree through JavaScript.

How Webkit Works

Render Tree (node is RenderObject)

However, the DOM Tree itself is not directly used for layout and rendering, so the kernel generates the Render Tree, which is a combination of THE DOM Tree and CSS, and the nodes of the two are almost one-to-one. Render Tree is a bridge between a typography engine and a rendering engine.

How Webkit Works

Layer Tree (node is RenderLayer)

Render engine is not directly using Render Tree for rendering, in order to more convenient processing operations such as positioning, clipping, industry scrolling, rendering engine will generate a Layer Tree. The rendering engine generates a RenderLayer for a particular RenderObject, but the child node of the RenderObject does not have a corresponding RenderLayer, so it is subordinate to the parent node’s RenderLayer. The rendering engine iterates through each RenderLayer, iterates through the RenderObjects that belong to that RenderLayer, and draws each RenderObject.

The Layer Tree determines the order in which the page is drawn, and the RenderObject that is subordinate to the RenderLayer determines what is drawn at that Layer.

What RenderObject is going to be the RenderLayer. GPU Accelerated Compositing in Chrome

• It’s the root object for the page
• It has explicit CSS position properties (relative, absolute or a transform)
• It is transparent
• Has overflow, an alpha mask or reflection
• Has a CSS filter
• Corresponds to element that has a 3D (WebGL) context or an accelerated 2D context
• Corresponds to a element

It doesn’t matter if you don’t know the above process, we will explain them all below.

Related to the source code

• /blink/renderer/core/css

Styles are usually the result of a combination of style renderers, which have complex priority semantics and a rendering process. The whole process is divided into three steps: 1. Collect, partition, and index all style rules in the style sheet.

Access each DOM element and find all the rules that apply to that element.

3 Combine these rules with other information (the style engine consists of partial default styles) to generate the final computed style.

|-------------------------------------------------| | | | margin-top | | | | |---------------------------------------| |  | | | | | | border-top | | | | | | | | |--------------------------|--| | | | | | | | | | | | | padding-top |##| | | | |  | |##| | | | | | |----------------| |##| | | | | | | | | | | | | ML | BL | PL | content box | PR |SW| BR | MR | | | | |  | | | | | | | | |----------------| | | | | | | | | | | | | | | padding-bottom | | | | | | | | | | | | | |--------------------------|--| | | | | | scrollbar height ####|SC| | | | | |-----------------------------| | | | | | | | | border-bottom | | | | | | | |---------------------------------------| | | | | margin-bottom | | | |-------------------------------------------------|Copy the code

The related documents

• Blink Layout
• CSS Box Model Module Level 3
• LayoutNG

Related to the source code

• /blink/renderer/core/layout

A Layout Tee is generated based on the DOM Tree to generate Layout information for each node. The process of Layout is to traverse the entire Layout Tree for Layout operations. The DOM Tree and Layout Tree are not always one-to-one. If we set dispaly: None in the tag, it will not create a Layout object.

05 Prepaint

What is an attribute tree?

In the hierarchical structure of the description attribute this piece, before the way is to use layer tree way, if the parent layer with matrix transformation (translation, scaling, or perspective), cutting or special effects (filter, etc.), need recursive applied to child nodes, time complexity is O (layers), which could be a performance problem in extreme circumstances.

Therefore, the concept of attribute tree is introduced, and the synthesizer provides transformation tree, clipping tree, special effect tree, etc. Each layer is composed of several node ids corresponding to matrix transformation nodes, clipping nodes and special effects nodes of different attribute trees. Such time complexity is O(the node to change), as shown below:

Related to the source code

• /blink/renderer/core/paint

The Paint operation converts Layout objects in a Layout Tree into drawing instructions (such as drawing rectangles, fonts, and colors, which are a bit like drawing API calls). These operations are then encapsulated in Dsipaly items, which are stored in PaintArtifact. PaintArtifact is the output of the Paint phase. So far, we’ve created a list of draw operations that can be replayed, but no actual draw operations have been performed.

Note: Recrod & Replay mechanism is used in most graphics systems now. The acquisition and execution of drawing instructions are separated from each other to improve rendering efficiency

• The background color
• floats
• The foreground
• outline

The Paint operation will eventually generate a Paint Tree based on the Layout Tree.

07 Commit

After the Paint phase is complete, enter the Commit phase. This phase updates a copy of the layer and property tree to the synthesizer thread to match the submitted main thread state. Copy layers and properties from the main thread to the synthesizer thread for use.

09 Raster

Once the layers are partitioned, Raster is performed. What is rasterization (rasterization)?

Raterization, also known as rasterization, is used to execute drawing instructions to generate color values of pixels. There are two rasterization strategies:

• Synchronous rasterization: Rasterization and composition in the same thread, or through thread synchronization to ensure light and composition
  • Direct Rasterization: Directly execute the drawing instructions for the visible areas in the eDisplayList of all visible layers to generate the color values of the pixels on the pixel buffer of the target Surface. Of course, if it is completely direct rasterization, there is no layer merge involved, and there is no need for subsequent composition.
  • Indirect rasterization: Allows additional buffers to be allocated for a layer that is rasterized to its own pixel buffer, The rendering engine then synthesizes and outputs the large ohmmeter Surface’s pixel buffers from these layers (view. setLayerType allows the application to assign pixel buffers to views). Android and Flutter mainly use direct rasterization of measurements, but also support indirect rasterization.
• Asynchronous block rasterization

1. In Proess Raster
2. Out of Proess Raster

1. In the old version, Skia runs in Renderer Process and is responsible for generating GL instructions. GPU has a separate GPU Process, and Skia cannot directly make rendering system calls in this mode. When initializing Skia, give it back a table of function Pointers (pointing to the GL API, but not the real OpenGL API, but the Proxy provided by Chromium), The process of converting function pointer tables to the true OpenGL API is called command buffers (GpuChannelMsg_FlushCommandBuffers). A separate GPU process helps isolate GL operations, improving stability and security. This pattern is also known as the sandbox mechanism (unsafe operations run in separate processes).

Display Compositor

After the Draw operation is complete, the Compositor frames are generated, which are output to the GPU Process. It receives Compositor frames from multiple Render processes from multiple sources.

• The Browser Process also has its Compositor to generate Compositor frames, which are typically used to draw the Browser UI (navigation bars, Windows, etc.).
• Each time a TAB is created or iframe is used, a separate Render Process is created.

Viz is short for VIsual, and it is an important part of Chromium’s overall architecture towards servitization, including Compositing, GL, Hit Testing, Media, VR/AR and many other functions.

VIz is also double-buffered, drawing Draw Quads in the background buffer and then executing swap commands to finally get them displayed on the screen. What is double buffering?

In the rendering process, if only a buffer is read and written, the screen will have to wait to read, while the GPU will have to wait to write, which will cause low performance. A natural idea is to separate reading and writing into:

• Front Buffer: The screen reads frame data from the Front Buffer for output display.
• Back Buffer: The GPU writes frame data to the Back Buffer.

The two buffer will not direct data copy (performance), but in the background buffer write complete, front desk buffer read complete, direct exchange of pointer, front desk becomes the background, the background at the front desk, so what time exchange, if the background buffer is ready to, the screen has not been processed at the front desk buffer, so there will be a problem, Obviously at this point you need to wait for screen processing to complete. After the screen is scanned, the device needs to go back to the first line to refresh. There is a Vertical Blank Interval during this period, which is the time for interaction. This operation is also known as VSync.

At this point, the rendering process is complete, and the front-end code becomes pixels that the user can interact with.