background

With the popularity of smart phones, we can’t get rid of our dependence on them in modern life. Travel, shopping, medical care, housing, social and other social needs are inseparable from the use of smart phones to achieve more efficient and convenient purposes. In just over ten years, relying on the development of smart phones, there have emerged endless Internet companies, mobile phone brand manufacturers and countless applications. As the relevant Internet industry market gradually reaches the population bottleneck, the competition based on the stock market has also ushered in a white-hot stage. One part of the industry is the competition among major mobile phone brands. In addition to the new strategy of “shell replacement” as the main one, major manufacturers have racked their brains to bring forth new ideas, such as folding screen, high refresh, fast charging, 100 million camera pixels and other software and hardware iterations are also frequently seen in various new phone releases. According to incomplete statistics, Android brands released 502 devices worldwide in 2021 alone (source: GSmarena). Let’s not discuss the inner volume behind the anxiety of reality, domestic mobile phone brands for new technology, new hardware exploration and slightly radical application, I think it has been very “aspirant” (here have to cue the shell is now too lazy to change Apple.

To return to the theme of this article, as an Android developer, in addition to adapting to a wide variety of highly fragmented models and systems, another priority is to enhance the user experience of your application with the new features featured by brand manufacturers. Like folding screen adaptation, the use of multi-screen small window characteristics to increase the user’s sense of multi-task interaction.

(Network diagram, intrusion and deletion)

The ultra-high pixel adaptation of the camera enricks the user’s experience of taking photos, and ADAPTS applications and games to the mobile phone screen that supports high refresh rate to increase the user’s smooth experience. This article takes a few of these new features and focuses on how the Refresh mechanism based on Android system can be adapted to some of the high refresh rate phones on the market to achieve a better user experience.

(Network diagram, intrusion and deletion)

As we all know, the screen refresh rate of most mainstream models on the market is still at 60Hz, which means that the screen refreshes at a rate of 16.6ms per 1000ms/60. However, now some high-end mobile phones can reach 90Hz or even 120Hz. As can be seen from the last GIF, the higher the refresh rate, the smoother the sensory experience will be for users. A quick tidbit here, you know that movies are made from so many frames, so what is the refresh rate of movies? 24 FPS! In other words, if you play a single sequence of images at a rate of 24 images per second, your brain will automatically associate it with a continuous image. So why did ang lee attempt “Billy Lynn’s Long Halftime Walk” at 120fps? From the point of view of the audience, the feelings brought by 120fps films are much more shocking and profound than 24FPS films, especially for some grand war scenes or narrative scenes, 24FPS high-speed pictures will appear motion blur in transition. But 120fps gives the viewer a clearer picture of these blurred transitions, which makes them feel more immersive. The same is true for games that support high refresh.

Screen refresh mechanism

In Android, UI rendering pipeline can be divided into 5 stages:

  • Phase 1: The application’s UI thread handles input events, calls the relevant callbacks that belong to the application, and updates the View hierarchy list that records the relevant painting instructions.
  • Phase 2: the application’s RenderThread sends post-processing instructions to the GPU;
  • Stage 3: GPU draws the frame data;
  • Phase 4: SurfaceFlinger is the system service responsible for displaying different application Windows on the screen. It composes what the screen should eventually display and submits frame data to the hardware Abstraction layer (HAL) of the screen.
  • Stage 5: The screen displays the frame content.

​

Android adopts a double buffering strategy, through the vsync signal to ensure the best timing of the cache exchange. There are two concepts mentioned above, one is frame rate, the other is refresh rate. An ideal state is that the frame rate is consistent with the refresh rate. As soon as the GPU finishes processing a frame, it brushes onto the screen and the next frame is ready. At this time, the data of the previous and previous frames is continuous and complete. However, the data transfer process from CPU to GPU is not controllable. When the screen is refreshed, if the frame buffer obtained is not fully ready, the problem of screen tearing often occurs in non-smart phones or old TV sets long ago (for example, the upper half of the screen is the previous picture and the lower half is the new picture). The double-buffered strategy solves this problem by maintaining two buffers. The front buffer is responsible for transporting the frame data to the screen while the back buffer prepares the rendering object for the next frame. The vsync signal is used to switch back buffer data to the front buffer.

We’ll look at some of the Android source code to achieve ($FrameDisplayEventReceiver Choreographer) :

private final class FrameDisplayEventReceiver extends DisplayEventReceiver implements Runnable {
        
    private boolean mHavePendingVsync;
    private long mTimestampNanos;
    private int mFrame;

    public FrameDisplayEventReceiver(Looper looper, int vsyncSource) {
            super(looper, vsyncSource);
    }
        
    @Override
    public void onVsync(long timestampNanos, long physicalDisplayId, int frame) {
        // Send vsync events to handler
        long now = System.nanoTime();
        if (timestampNanos > now) {
            Log.w(TAG, "Frame time is " + ((timestampNanos - now) * 0.000001 f)
                        + " ms in the future! Check that graphics HAL is generating vsync "
                        + "timestamps using the correct timebase.");
            timestampNanos = now;
        }
		// Determine if there are any pending signals that have not been processed
        if (mHavePendingVsync) {
            Log.w(TAG, "Already have a pending vsync event. There should only be "
                        + "one at a time.");
        } else {
            mHavePendingVsync = true;
        }

        mTimestampNanos = timestampNanos;
        // Replace the new frame data
        mFrame = frame;
        Message msg = Message.obtain(mHandler, this);
        msg.setAsynchronous(true);
        mHandler.sendMessageAtTime(msg, timestampNanos / TimeUtils.NANOS_PER_MS);
        }

    @Override
    public void run(a) {
        mHavePendingVsync = false;
        // Frame data processingdoFrame(mTimestampNanos, mFrame); }}Copy the code

Let’s use systrace tool to intuitively feel the Refresh mechanism of Android from the perspective of application:

In the figure above, each vertical gray area is a synchronization of Vsync signal, and each gray area is 16.6ms. In other words, as long as the UI Thread and the set of method references and executions of the RenderThread are executed in a grayscale area, the data rendering of this frame can be completed in 16.6ms. The rendering in the red box has obviously exceeded the boundary of a Vsync signal. That is to say, the rendering time of this frame is too long and the execution is not completed within a signal cycle. Then the execution of this frame is reflected in the code execution. Android typically analyzes the stack of related functions to locate the problem, for example, the problem in the red box is caused by the RPDetectCoreView custom view. If the application has been running for a while and this happens frequently, the resulting user perception is that the application is stalling and dropping frames.

High brush fit

Now that you know the basic refresh mechanism of Android, let’s take a look at how it can be adapted to current high-brush models. Apps or games can influence the refresh rate of the screen through the official Android SDK/NDK API. Why is it “influence” rather than “determine”? As mentioned earlier, the CPU/GPU write speed is not controllable, so these methods can only affect the screen frame rate.

Use the SDK that comes with Android

When adapting to high brush, we will register the sensor of the device like general, first know which sensors the device itself supports, and then carry out the specific collection action. Similarly, for screen refresh rates, we need to know what the screen supports and what the current refresh rate is, and then adjust it to the desired refresh rate. <br /

There are two ways to obtain the refresh rate, both of which are achieved by registering listeners:

  1. DisplayManager.DisplayListenerRun the display.getrefreshRate () command to query the refresh rate
// Register the screen change listener
public void registerDisplayListener(a){
     
    DisplayManager displayManager = (DisplayManager) mContext.getSystemService(Context.DISPLAY_SERVICE);
    displayManager.registerDisplayListener(new DisplayManager.DisplayListener() {
        @Override
        public void onDisplayAdded(int displayId) {}@Override
        public void onDisplayRemoved(int displayId) {}@Override
        public void onDisplayChanged(int displayId) {

        }
    }, dealingHandler);
}

// Get the current refresh rate
public double getRefreshRate(a) {
    
  return ((WindowManager) mContext
      .getSystemService(Context.WINDOW_SERVICE))
      .getDefaultDisplay()
      .getRefreshRate();
}
Copy the code
  1. You can also use NDKAChoreographer_registerRefreshRateCallback API
void AChoreographer_registerRefreshRateCallback(
  AChoreographer *choreographer,
  AChoreographer_refreshRateCallback,
  void *data
)

void AChoreographer_unregisterRefreshRateCallback(
  AChoreographer *choreographer,
  AChoreographer_refreshRateCallback,
  void *data
)
Copy the code

After obtaining the available refresh rate of the screen, you can try to set the refresh rate according to your business needs. The method is very simple, and the sample code is not used here:

  1. Use the SDKsetFrameRate()methods
    1. Surface.setFrameRate
    2. SurfaceControl.Transaction.setFrameRate
  2. Using the NDK_setFrameRatefunction
    1. ANativeWindow_setFrameRate
    2. ASurfaceTransaction_setFrameRate

FramePacingLibrary (FPL, Frame synchronization Library)

Here’s a bit more about FramePacingLibrary (alias Swappy). Swappy helps smooth rendering and frame synchronization for games based on OpenGL and Vulkan rendering API. Here is another frame synchronization concept that needs to be explained. Frame synchronization. As mentioned above, the entire rendering pipeline of Android is from CPU to GPU and then to HAL screen display hardware. Here, frame synchronization refers to the synchronization between the logical operation of CPU and rendering of GPU and the display subsystem and underlying display hardware of operating system. Why Swappy is good for games is that games involve a lot of CPU computing and rendering work, and these computing strategies are obviously expensive to develop from zero to one, So Swappy, like most game engines on the market (Unity, Unreal, etc.), offers a ready-made strategy mechanism to make development easier and better.

It can do:

  • Add a time stamp for each frame rendering to show the frame on time to compensate for the lag caused by the short game frame
  • The lock mechanism is injected into the program so that the pipeline of the display can keep up with the progress and not accumulate too much resulting in long frame lag and delay (synchronous fence)
  • Frame statistics are provided to debug and profile the program

To briefly describe how it works, here is an ideal 30Hz frame sync running on A 60Hz device, where every frame (A\B\C\D) is rendered properly on screen.However, this is not always the case in reality. For some short game frames, such as C frame in the figure below, due to the shorter time, C frame is preemption before B frame has fully displayed the due number of frames. Meanwhile, THE NB signal of B frame triggers the display of C frame again, just like a runner tripping over a pebble on the road. If you fall a few meters in a fixed position, a C frame will be displayed, resulting in a lag.

The Swappy library solves this problem by adding a rendering timestamp, which is like setting an alarm clock for each frame of data and not allowing it to appear on the screen until it goes off:

conclusion

Although adapting to high brush features does not require much code adaptation, it is important to consider the following aspects:

  1. Through thesetFrameRate()Method when setting the screen refresh rate, there are still some situations where the Settings cannot take effect, such as different frame rate Settings for Surface with higher priority, or the device in power-saving mode, etc. Therefore, we must take into account when developing programs that the program can run normally if the Settings fail.
  2. Avoid frequent callssetFrameRate()Method, in the transition of each frame, if the frequent call will cause the problem of frame drop, we need to obtain the due information through THE API in advance to adjust the correct frame rate once;
  3. Don’t stick to a specific frame rate. Instead, adjust your frame rate setting strategy based on your actual business scenario. The goal is to seamlessly transition between high and low screen refresh rates.

reference

High Refresh Rate Rendering on Android

Frame Pacing Library

Android Frame-rate ​

Author: ES2049 / Dawn

The article can be reproduced at will, but please keep this link to the original text.

You are welcome to join ES2049 Studio. Please send your resume to [email protected].