An overview of the

Android graphics system is a very important subsystem in Android, which involves many quite complex modules, such as SurfaceFlinger, Choreographer, HardWare Composer and other concepts that are rarely touched directly in daily development. After reading the source code of each component module of the graphics system one after another based on Android 10 version, combined with some blogs on the Internet, I have an overall understanding of the composition of the Graphics system of Android and the workflow between each component. This article intentionally makes a summary of the whole graphics system, and there may be some omissions. Later found and then complete.

There are some notes in the series that will be updated for omissions and errors:

  • Android-window mechanism source code interpretation: Mechanisms related to Android Window, including the start of WindowManagerService, the StartingWindow process in the startActivity process, And the process related to Windows in the startActivity process, such as when and how the DecorView is added, and so on, finally gives the WindowManager to add/remove/update View source parsing.
  • Android-window Token validation: Source code analysis of Android WMS in addWindow Token validation logic, and explain why the Application Context cannot be used to display the Dialog.
  • Android – View drawing principle: when receiving the Vsync signal, after the App process calls ViewRootImpl. PerformTraversals method to carry out the View of drawing (measure, layout, the draw) process.
  • Principles and Practices of Android-View: A few scattered knowledge points related to View are not important for understanding the overall graphics system. This paper mainly introduces the event distribution mechanism of View, sliding conflict, View.post and so on.
  • Android-surfaceflinger startup and how it works: Introduces the startup process of SurfaceFlinger and how it processes Vsync signals to synthesize layers.
  • How Android-Choreographer works: How App processes register with Choreographer and receive Vsync signals to start the drawing process of views.
  • Android-surface creation process and hardware and software drawing: In View drawing process, how the data drawn by View.draw can be synthesized by SurfaceFlinger, including Canvas, Bitmap, Surface, Layer, Buffer and hardware and software drawing workflow.
  • Android-surface double buffering and SurfaceView analysis: Introduce the double buffering technology in the process of View drawing and the basic use of SurfaceView and source code interpretation.
  • Overview of Android Graphics system: summarize the workflow of Android graphics system.

Let’s summarize the process with a picture before we begin:

After looking at the diagram, take a step-by-step look at how each component works.

Android Display Basics

Reference: “Finally Get it” series: Android screen refresh mechanism – VSync, Choreographer fully understand!

Relevant concepts

  • Screen refresh rate: The number of screen refreshes per second (how many frames of the image are displayed in a second) in Hz (Hertz), such as the common 60Hz. The refresh rate depends on a fixed parameter of the hardware (does not change).
  • Frame Rate: indicates the unit FPS. In The Android system, it is 60fps, that is, the GPU can draw 60 frames per second at most. The frame rate changes dynamically. For example, when the picture is still stationary, the GPU does not draw any operation, and the data in the buffer is refreshed on the screen, that is, the frame data of the last operation of the GPU.
  • Progressive scan: Instead of bringing an image onto the screen at once, the monitor scans the entire screen in successive lines, from left to right and top to bottom, showing each pixel in sequence, but too fast for the human eye to notice. In the case of a 60Hz refresh rate screen, this process is 1000/60≈16ms.
  • Tearing: a screen with data from two different frames will cause tearing.

The reason for the screen tear: the screen refresh rate is fixed, for example, every 16.6ms for each frame drawn from the buffer. Ideally, the frame rate and refresh rate are consistent, so that the display displays one frame for each frame drawn. However, CPU/GPU write data is not controllable, so some data in the buffer may be overwritten without being displayed at all. That is, the data in the buffer may come from different frames. When the screen is refreshed, it does not know the state of the buffer at this time, so the frame captured from the buffer is not a complete frame. That is, the picture is torn. That is, during the Display process, the data in the buffer is modified by the CPU/GPU, resulting in the screen tearing.

Dual cache: Since the same buffer is used for image drawing and screen reading, a screen refresh may read an incomplete frame (rip/blink). The dual buffer allows the drawing and display to have their own buffers. The GPU always writes the completed frame of image data to the Back buffer, while the display uses the Front buffer. When the screen is refreshed, the Front buffer does not change. They just swap. As shown in the figure below:

VSync: If two buffers are exchanged after the Back buffer is ready for one frame, there will be a problem if the screen has not fully displayed the contents of the previous frame. Therefore, the buffer swap can only be performed after the screen has processed a frame. After scanning a screen, the device needs to go back to the first line to enter the next loop, and there is a period of time called the VerticalBlanking Interval(VBI), which is the best time for buffer swapping. VSync, short for VerticalSynchronization, uses the vertical Sync Pulse that occurs during VBI to ensure that the double bufferexchange only at the optimal point in time. In addition, swapping refers to the respective memory addresses and can be considered instantaneous. The v-sync concept has been around since the early days of the PC world.

Prior to Android4.1: Before Android4.1, screen refresh also followed the above dual cache +VSync mechanism. The diagram below:

In chronological order:

  1. Display displays the data of frame 0. At this time, the CPU and GPU render the first frame and finish before Display displays the next frame.
  2. Display After frame 0 is displayed, that is, when the first VSync is displayed, the cache is exchanged and the first frame is displayed normally.
  3. Then frame 2 is processed, and for some reason it is not processed until the second VSync is coming. As a result, when the second VSync comes, the data of frame 2 is not ready, the cache is not exchanged, and frame 1 is still displayed. In this case, the frame is lost (Jank);
  4. When the second frame is ready, it is not displayed immediately, but waits for the next VSync to cache the data.

The screen will display frame 1 one more time because the CPU/GPU calculation of frame 2 failed to complete before the VSync signal arrived.

When Vsyn arrives, the screen will fetch new data from the Front buffer. In fact, the Back buffer is ready for the GPU to prepare the next frame. If the CPU/GPU was operating when Vsyn arrived, there would be a full 16.6ms, which would greatly reduce the presence of JanK (unless the CPU/GPU computations exceed 16.6ms).

After Android 4.1: Project Butter, Drawing with VSync, is implemented. After receiving VSync Pulse, the system will immediately start rendering the next frame. As soon as the VSync notification is received (triggered once at 16ms), the CPU and GPU immediately start computing and write the data to the buffer. The diagram below:

CPU/GPU can process data according to Vsync signal, so that the CPU/GPU has a complete time of 16.6ms to process data, reducing JANK. However, if the processing time of CPU/GPU exceeds 16.6ms, the figure below is shown:

  1. In the second period, because the GPU was still processing FRAME B, the cache failed to exchange, resulting in frame A being displayed repeatedly.
  2. After the B frame was completed, it had to wait for the next signal due to the lack of a VSync Pulse signal, which wasted a lot of time.
  3. When the next VSync appears, the CPU/GPU immediately performs the operation (frame A), and the cache is switched, and the corresponding display corresponds to B, and then it looks normal. However, because the execution time is still over 16ms, the next buffer swap that should be performed is delayed. This cycle repeats, and more and more Jank.

Since there are only two buffers, the Back buffer is being used by the GPU to process the data of FRAME B, and the contents of the Front buffer are used for Display. Both buffers are occupied, so the CPU cannot prepare the data of the next frame.

Tripple Buffer A Graphic Buffer Buffer is added on the basis of the double Buffer mechanism. CPU, GPU and display device can use their own Buffer to work without affecting each other. The advantage is that it can maximize the use of idle time, while the disadvantage is that it will occupy more memory of a Graphic Buffer size. The diagram below:

The first Jank is inevitable, but in the second 16ms period, CPU/GPU uses the third Buffer to complete the calculation of C frames. Although an extra A frame will be displayed once, the subsequent display will be smoother, effectively avoiding further aggravation of Jank.

Three buffering takes advantage of the time spent waiting for Vsync, reducing Jank, but bringing latency. So, normally there are two buffers, but when there is a Jank, there are three buffers.

summary

  • After scanning a screen, the device needs to return to the first line to enter the next loop, and the vertical Sync Pulse appears to ensure that the double buffering is switched at the optimal point. And after Android 4.1, CPU/GPU drawing starts when VSYNC arrives.
  • Double buffer is Back buffer, Front buffer, used to solve the picture tearing; Add a Back buffer for reducing Jank.
  • Frame drop (frame drop) means that the frame delay is displayed because the time for the cache swap can only wait for the next VSync.
  • Only when the App registers to listen for the next Vsync signal can it receive the Vsync callback. If the interface stays the same, the App will not receive the Vsync event every 16.6ms, but the underlying layer will still switch frames at that frequency (also by listening for the Vsync signal). That is, when the interface is unchanged, the screen will be refreshed every 16.6ms, but the CPU/GPU does not go through the drawing process.
  • When the View requests a refresh, the task does not begin immediately. Instead, it waits until the next Vsync signal arrives. After the measure/ Layout /draw process is finished, the interface will not be refreshed immediately, but will wait until the next VSync signal arrives for cache exchange and display.

Android Graphics Components

Reference: Android – Graphics

There are three ways to draw images to the screen: Canvas, OpenGL ES, or Vulkan. Whatever rendering API is used, everything is rendered to the Surface, which represents the producer in the BufferQueue, and the BufferQueue is usually consumed by the SurfaceFlinger. Every Window created on the Android platform is supported by the Surface, and all rendered visible Surfaces are composited into the display by the SurfaceFlinger.

The following figure shows how the key components work together:

Related components are as follows:

  • Image Stream Producers: The Producers can be anything that generates a graphics buffer for consumption, such as OpenGL ES, Canvas 2D and Mediaserver video decoders.
  • Image Stream Consumers: The most common consumer of Image Stream is SurfaceFlinger, a system service that consumes currently visible Surfaces and uses information provided in Windows Manager to synthesize them for Display. SurfaceFlinger uses OpenGL and HardWare Composer to compose the Surface. Other OpenGL ES apps can also consume image streams. For example, the camera app will consume a stream of camera preview images. Non-gl applications can also be users, such as the ImageReader class.
  • Hardware Composer: This is the Hardware abstraction layer for the display subsystem, and SurfaceFlinger can delegate some composition to Hardware Composer to share the workload on OpenGL and the GPU. SurfaceFlinger, after collecting all the buffers for the visible layer, asks the Hardware Composer how the composition should be performed.
  • Gralloc: The graphics memory allocator (Gralloc) is used to allocate memory requested by the image producer.

BufferQueue

Reference: Android – BufferQueue

Android graph data flow pipeline is shown below:

The object on the left is a renderer that generates graphical buffers such as the home screen, status bar, and system interface. SurfaceFlinger is the synthesizer, and Hardware Composer is the producer. The BufferQueue is a very important part of the Android graphics system. It is responsible for data transfer:

The producers and consumers in the figure are running in different processes. A BufferQueue is a data structure that combines a buffer pool with a queue. It uses Binder IPC to pass buffers between processes. Some important functions are:

  • Producers through BufferQueue request a free buffer (GraphicBuffer) : IGraphicBufferProducer. DequeueBuffer method
  • To cache area (GraphicBuffer) populated with data (after drawing, etc.), producers will cache area (GraphicBuffer) into the queue to BufferQueue: IGraphicBufferProducer. QueueBuffer method
  • Consumer from BufferQueue queue up a buffer (GraphicBuffer) : IGraphicBufferConsumer. AcquireBuffer method
  • After the consumer consumption (typically SurfaceFlinger synthetic data) will cache area (GraphicBuffer) back to the queue: IGraphicBufferConsumer. ReleaseBuffer method

The IGraphicBufferProducer is the producer interface of the BufferQueue, the implementation class is the BufferQueueProducer producer class; IGraphicBufferConsumer is the consumer interface of the BufferQueue, and the implementation class is the BufferQueueConsumer consumer class.

Check BufferQueue related source code, can be in frameworks/native/libs/GUI/include/GUI/BufferSlot h comments see GraphicBuffer there are several kinds of state:

  • FREE: The Buffer is not used by the producer/consumer and belongs to the BufferQueue
  • DEQUEUED: The Buffer was obtained by the producer and is owned by the producer
  • QUEUED: This Buffer is populated by the producer and QUEUED to the BufferQueue. It belongs to the BufferQueue
  • ACQUIRED: The Buffer is ACQUIRED by the consumer and the Buffer is owned by the Consumer
  • SHARED: The Buffer is in SHARED Buffer mode

These states are needed to maintain the structure of a buffer pool, rather than creating a shared memory each time a buffer is used and then releasing it when it is finished, which is inefficient.

The changing process of GraphicBuffer state: FREE -> dequeueBuffer() -> DEQUEUED -> queueBuffer() -> QUEUED -> acquireBuffer() -> ACQUIRED -> releaseBuffer() -> FREE.

As for the source code related to BufferQueue, you can read it in depth.

Relevant concepts

  • View: View, the content drawn to the screen, such as TextView, ImageView, etc.
  • Window: The carrier of a View. Windows can be added or deleted through Windows Manager. Windows do not exist. Views are the form of View in Android. Views cannot exist alone, they must be attached to the abstract concept of Windows.
  • WindowManager: Window in the management system. The actual function is implemented by WindowManagerService through Binder IPC.
  • Canvas: Provides some API for drawing Surface for actual drawing. If drawn by software, its drawXXX method draws the contents onto the Bitmap; If it is a hardware drawing, its drawXXX method is abstracted into a DrawOp operation, which is then added to the DisplayList for GPU rendering.
  • Surface: A Window corresponds to a Surface(except when there is a SurfaceView, Java Surface instances exist in ViewRootImpl, corresponding to native layer Surface objects). The Surface holds a BufferQueueProducer pointer (created in the Layer) that can produce an image cache for drawing, forming a producer-consumer model with the App and SurfaceFlinger.
  • Layer: When an App requests to create a Surface, SurfaceFlinger creates a Layer object. It is the basic operating unit of SurfaceFlinger’s composition, so a Surface corresponds to a Layer. It creates a BufferQueueProducer and a BufferQueueConsumer consumer, which are related to the storage and transfer of pixel data. The user ultimately sees the content of the screen as a result of many layers mixed in z-order.
  • SurfaceView: a View that is more special than TextView, Button, etc. It does not share a Surface with its host Window, but has its own Surface. And it can draw the UI in a separate thread. Therefore, SurfaceView is generally used to achieve more complex images or animation/video display.
  • Choreographer: Allows Choreographer to start drawing tasks only after receiving the VSync signal, ensuring that the drawing has a full 16.6ms. Usually the application layer does not use Choreographer directly, but rather more advanced apis such as View.invalidate() can be used to monitor the frame rate of an application through Choreographer.
  • SurfaceFlinger: Manages the consumption of the currently visible Surface. All rendered visible Surfaces are synthesized by SurfaceFlinger through information provided by The Windows Manager (using OpenGL and HardWare Composer, The synthesized data source, the GraphicBuffer in the BufferQueue mentioned above, is submitted to the back buffer of the screen and waits for the screen’s next Vsync signal to arrive before being displayed on the screen. SufaceFlinger connects to the screen through the buffer behind the screen, and connects to the upper layer through the Surface, acting as a link between the preceding and the following.
  • HWComposer: HardWare Composer defines a set of HAL interfaces. Chip manufacturers implement the interfaces based on HardWare features. Its main work is to synthesize the Layer display parameters calculated by SurfaceFlinger into the display Buffer. Of course, SurfaceFlinger is not the only input source of HWC. For example, the camera preview input Buffer can be directly written by the hardware device, and then as one of the HWC input and SurfaceFlinger output to do the final synthesis.
  • OpenGL: A 2D/3D graphics library that requires underlying hardware (GPU) and drivers to support. OpenGl ES(OpenGl for Embedded System) is commonly used on mobile devices.
  • Display: Abstract Display device, the traditional Display device is the mobile phone screen, in addition to Android also support other external input devices such as HDMI, Wifi Display, etc. Output the synthesized data from SurfaceFlinger, OpenGL, and HWComposer to the cache of the Display device for Display.

The SurfaceFinger workflow

See Android-SurfaceFlinger startup and how it works for more details

Surfaceflinger is a daemon process that is started by resolving the init.rc file at startup of the Android system. In the startup process of SurfaceFlinger:

  1. The SurfaceFlinger object is created, and the DispSync synchronization model object is created in the constructor.
  2. Then execute the logic that initializes the SurfaceFlinger:
    • Register and receive HWC related events.
    • Start APP and SF EventThread to manage two DispSyncSource delay source objects created based on DispSync. These are used for drawing (app–mEventThreadSource) and composing (SurfaceFlinger–mSfEventThreadSource). After the EventThread thread has started, the thread will remain blocked in waitForEventLocked (with listeners set as needed) until the Vsync signal is received and at least one connection is waiting for Vsync.
    • By MessageQueue. SetEventThread method creates a connection, and through the stars. The monitored data BitTube addFd method.
    • Create HWComposer object (through the HWComposer hardware module or software simulation of HAL layer to generate Vsync signal), the current Android system can basically be regarded as HWComposer hardware to generate Vsync signal. Since software simulation is not used, the following parsing will only refer to the Vsync signal in HWComposer;
    • Initialize the non-virtual display;
    • Start the boot animation service;
  3. Finally, we execute the SurfaceFlinger. Run logic. This method waits for the arrival of the message by executing the messagequewe. waitMessage method on the main SurfaceFlinger thread through an loop, calling looper.pollonce inside. This method reads data from the BitTube that looper. addFd listens on, and executes the corresponding callback method when the data arrives.

When a hardware or software simulation emits a Vsync signal:

  1. “DispSync” is called to process the Vsync signal (statistical and calculate the offset and period of the model), and determine whether to enable/disable the HWC Vsync signal according to the return value.
  2. DispSync calculates the occurrence time of the next Vsync signal according to the calculated offset and period, and informs the listener of the Vsync signal arrival event, and passes it to the DispSyncSource delay source. The delay source manages the sending and receiving of Vsync signal through EventThread.
  3. EventThread calls the Connection object to send data to BitTube, triggering the callback method set in the addFd function, which in turn calls the Sf. onMessageReceived function, and then syntheses the image.

Choreographer, on the other hand, listens for Vsync signals emitted by the synchronization simulation through the APP latency source mEventThreadSource object created above and its corresponding EventThread thread. Then perform the operation of measure/layout/draw. See How Android-Choreographer works for the detailed logic.

The workflow of SurfaceFlinger is summarized as follows:

Choreographer Workflow

See How Android-Choreographer works for more details

  • Choreographer: Enable CPU/GPU drawing to start when VSYNC arrives. Choreographer initializes by creating a connection that indicates interest in the Vsync signal, requesting the next Vsync signal through the postCallback method when there is a draw request, and then starting the draw task when the signal arrives.
  • Only when the App registers to listen for the next Vsync signal can it receive the Vsync callback. If the interface stays the same, the App will not receive the Vsync event every 16.6ms, but the underlying layer will still switch frames at that frequency (also by listening for the Vsync signal). That is, when the interface is unchanged, the screen will be refreshed every 16.6ms, but the CPU/GPU does not go through the drawing process.
  • When the View requests a refresh, the task does not begin immediately. Instead, it waits until the next Vsync signal arrives. After the measure/ Layout /draw process is finished, the interface will not be refreshed immediately, but will wait until the next VSync signal arrives for cache exchange and display.
  • There are two main reasons for frame loss. First, it takes more than 16.6ms to traverse the View tree and calculate the screen data. The second is that the main thread has been processing other time-consuming messages, causing the drawing task to be delayed (the synchronization barrier does not fully resolve this issue).
  • By Choreographer. GetInstance (). PostFrameCallback () to listen on frame rate.

It is recommended that you read The Android-SurfaceFlinger Launch and How it works article first, then combined with the Choreographer workflow, This gives you a good idea of how the Vsync signal coordinates the app-side drawing task and SurfaceFlinger’s composition task.

To summarize Choreographer’s workflow with a diagram:

View drawing Process

Detailed article see Android-Window mechanism source code interpretation and Android-View drawing principle

After Choreographer receives the Vsync signal, the drawing process of the View begins with the usual three steps: measure, Layout and draw.

Surface workflow and software and hardware drawing

For details, see android-Surface creation process and hardware and software drawing, Android-Surface dual buffering and SurfaceView analysis

The Surface logic is touched during the Draw of the View and can be used to request a cache from the SurfaceFlinger for drawing. Drawing tasks can be divided into software drawing and hardware drawing.

  • The mNativeObject property in the Surface object in the Java layer points to the Surface object created in the native layer.
  • Surface corresponds to the Layer object in the SurfaceFlinger, which holds the BufferQueueProducer pointer in the Layer, This producer object allows you to request a free GraphicBuffer from the BufferQueue at the time of drawing. The contents drawn on the Surface will be stored in this buffer.
  • SurfaceFlinger retrives the GraphicBuffer data from the BufferQueue via the BufferQueueConsumer consumer for compositing and rendering and sending it to the display.

Software rendering

Software drawing may draw to the view that does not need to be redrawn, and the drawing process is in the main thread, which may cause problems such as stalling. It writes the content to be drawn into a Bitmap, essentially filling the graphics cache of the Surface application.

Software drawing can be divided into three steps:

  1. Surface.lockCanvas — dequeueBuffer Removes a buffer from the BufferQueue.
  2. View.draw — Draws content.
  3. Surface. UnlockCanvasAndPost — — queueBuffer will populate the data buffer in BufferQueue queue, then notice to SurfaceFlinger Vsync signal (request) of synthesis.

Hardware rendering

Hardware rendering records the draw function as a draw instruction (DrawOp) in a list (DisplayList) and then gives it to a separate Render thread for hardware accelerated rendering using the GPU. It only needs to record or update the dirty part of the View object that needs to be updated, and the View object that does not need to be updated can reuse the previously recorded instructions in the DisplayList.

Hardware drawing can be divided into two phases:

  1. Construction phase: Draw the View (drawLine…) It’s abstracted as a DrawOp operation and stored in a DisplayList.
  2. Draw phase: first allocate the cache (same as software draw), then bind the Surface to the Render thread, and finally Render the DrawOp data through the GPU.

Hardware-accelerated memory requests, just like software drawing, use the BufferQueueProducer in the Layer where the producer queues out of the BufferQueue an idle buffer called the GraphicBuffer to render the data, SurfaceFlinger is also notified to synthesize. The difference is that hardware acceleration may make more sense than software rendering, and a separate Render thread is used to reduce the burden on the main thread.

double-buffering

In general, the two buffers used for double buffering are called front buffer and back buffer. The data displayed on the display comes from the cache in front of the front buffer, and the data of each frame is drawn to the cache behind the back buffer. After the arrival of the Vsync signal, the data in the cache will be exchanged (the pointer points to). The names and functions of front buffer and back buffer are reversed.

The Surface uses double buffering technology during the drawing of the View.

SurfaceView

SurfaceView is a special View that has its own Surface. Because of this, it is usually used to display more complex images or animations/videos.

To summarize the process of drawing Android hardware and software, use a graph:

Graphic system workflow summary

Here have Android graphics system work flow together, feeling after reading the source code for the process more clear understanding, rather than follow the conclusion on the net conformity and feel everyone said makes sense, but always feel that in some places have conflict again, their reading the source code to look at the others gives the conclusion after reading the source code, Read The Fucking Source Code. All that’s left is some of The details of The process, such as The difference between The requestLayout and invalidate methods. These related questions will be recorded slowly later when there is time. If there are mistakes in the content of the article, welcome to point out and make progress together! Feel good to leave a like to go again ha ~

Summarize the overall flow of the graphics system with a graph: