background

This year, Apple launched iPhone devices that support ProMotion screen, allowing the maximum refresh frame rate of App on iPhone 13 Pro and iPhone 13 Pro Max to reach 120Hz, greatly optimizing the smooth experience of App sliding/animation.

ProMotion is not a new concept. Back in 2017, Apple launched the second generation iPad Pro with this screen with a maximum refresh rate of 120Hz. On the iPad, high refresh rates are enabled by default for all apps. On the iPhone, Apple does not automatically enable this capability for all apps, perhaps because of power consumption concerns. Instead, developers need to manually add configuration items to enable it.

Recent reports pointed out that the iOS 15.4 beta revised this behavior (www.macrumors.com/2022/01/27/)… , the author verified that additional configuration items are still needed and the contents of this article are still applicable.

This paper introduces the observed phenomena and problems encountered in the adaptation of ProMotion dynamic frame rate on iPhone, tries to predict the principle behind it, and discusses the possible ideas to solve the problems. Finally, based on the research results, we optimize the plan for the launch of international short video business, and achieve the benefits of core business indicators.

What is frame rate

Before delving into the changes brought about by the ProMotion screen, let’s review a concept that seems familiar:

What is frame rate?

It is well known that monitors do not display truly moving pictures, and all animation is an illusion created by playing so many still frames at high speed that they deceive the human eye. The most basic definition of frame rate is how often the content on the screen changes, which is a physical indicator. The frequency of this change is determined by the following two values:

  • Refresh frame rate: It is controlled by the screen hardware specification, which is 59.94Hz for traditional display devices and determines the upper frame rate.
  • Render frame rate: The lower limit of the frame rate is determined by the execution rate of the CPU -> GPU rendering pipeline.

Ideally, the render frame rate and refresh frame rate should either match exactly, or be an integer multiple of the refresh frame rate, so that the actual presentation does not show any anomalies. But in reality, there are often mismatches between the two, and Caton is one of them:

caton

When the CPU -> GPU rendering pipeline runs into a bottleneck that causes a frame to take longer to render than the screen refresh interval, the previous frame will stay on the screen for several more frames. When this latency is too long, users perceive a delay in updating the picture, which is called stalling. This is one of the major performance issues encountered in iOS development.

The actual frame rate

Framerate is not the same as refresh rate, it has everything to do with what is being shown:

  • When displaying a static image, ideally only one render is required, even though the screen is still refreshed at 60Hz or more, and the contents of the FrameBuffer displayed with each refresh remain unchanged, the actual frame rate perceived by the user remains close to zero.
  • When presenting elements with a fixed frame rate, such as a 24FPS movie video, the user will naturally perceive the actual frame rate to be around 24FPS.
  • Display extremely high frame rate content, such as CS:GO unlocked frame rate >200 FPS, but due to the display device refresh rate limitation, the user’s perceived frame rate still does not exceed the upper limit of the hardware frame rate.

What is dynamic refresh rate

ProMotion is essentially an implementation of the Adaptive-Sync display standard.

Ref: en.wikipedia.org/wiki/Variab…

According to Apple’s official documentation, the refresh rate supported by the ProMotion screen is variable.

Specifically, for the iPhone:

The iPhone 13 Pro and iPhone 13 Pro Max ProMotion displays can present content on the display using the following refresh rates and timings:

120Hz (8ms), 80Hz (12ms), 60Hz (16ms), 48Hz (20ms), 40Hz (25ms), 30Hz (33ms), 24Hz (41ms), 20Hz (50ms), 16Hz (62ms), 15Hz (66ms), 12Hz (83ms), 10Hz (100ms)

And for the iPad Pro:

The iPad Pro’s ProMotion display can present content on the display using the following refresh rates and timings:

120Hz (8ms), 60Hz (16ms), 40Hz (25ms), 30Hz (33ms), 24Hz (41ms)

This is an implementation of Apple’s Adaptive Sync technology standard for VESA, which has been in the game industry for years, along with AMD’s FreeSync and Nivida’s G-Sync. This new display technology has the following advantages:

Reduce perceived lag

For screens with a fixed refresh rate, when the render time of a frame is abnormal and the render is not completed until after the VSync signal arrives, the current content will be stuck on the screen and the frame will have to wait for another VSync signal before being rendered to the user.

The adapt-sync technology avoids this and displays the frame as soon as possible after rendering, thereby reducing the display card’s immediate length:

Reduce screen power consumption on mobile devices

On a device with a fixed refresh rate screen, the GPU will render at a lower refresh rate than the actual frequency when displaying static content or content with a low frame rate (such as video). But a screen with a fixed refresh rate will still refresh at its maximum rate, redisplaying previous content, causing additional battery consumption.

In this case, you can actively reduce the refresh rate and reduce screen power consumption, which is especially important for mobile devices.

A representation of dynamic refresh rates

The iPhone 13 Pro, the iPhone 13 Pro Max, and the iPad Pro ProMotion displays are capable of dynamically switching between:

  • Faster refresh rates up to 120Hz
  • Slower refresh rates down to 24Hz or 10Hz

It is known that the refresh frame rate of ProMotion screen is not fixed, and the system dynamically switches the refresh frame rate of the screen in real time according to the type and state of the content currently displayed. In order to better understand the representation of this dynamic frame rate, the author respectively in

  • IPhone XR – None ProMotion
  • IPhone 13 Pro – ProMotion default lockout

It is found that when the App is run on devices equipped with ProMotion screen, frame rate indicators of various statistical diameters in different scenes do show interesting changes.

Specifically, the author conducted the following scenarios:

Test scenarios

  1. A static page

Static UIView with no animation/video elements

  1. Page in slide

UITableView that contains static cells that only observe the performance of slides

  1. Core Animation defaults to refresh rate Animation

Shows a simple displacement animation based on CABasicAnimation

  1. Core Animation 120Hz high refresh rate Animation

Simple displacement animation based on CABasicAnimation, tested only on ProMotion devices, Unlock the CADisableMinimumFrameDurationOnPhone and preferredFrameRateRange frame rate limit at the same time. (More on this limitation below)

  1. Metal Render 30Hz/60Hz video

Use mtKView-based rendering player to play video files with source frame rates of 30Hz/60Hz

And use the following frame rate indicators of statistical caliber to test:

Test indicators

  1. CADisplayLink calculates the frame rate

The main frame rate statistic in iOS.

According to the cadisplaylink. h header, CADisplayLink is “representing a timer bound to the display vsync.” By comparing the current frame/previous frame timestamp in the callback, the render time of the previous frame (ts) can be calculated, and the reciprocal (1/ts) is the current real-time frame rate.

  1. Xcode GPU Report Frame rate

Xcode -> Show Debug Navigator -> Frame rate displayed in FPS. This only counts the frame rate of the current application drawn directly by OpenGL ES or Metal, such as game rendering/video playback, but not the frame rate of Core Animation (as we all know, the latter is drawn by backboardd).

  1. Instruments Core Animation FPS

The frame rate displayed by the Core Animation FPS tool in Instruments. This measures the frame rate of Core Animation, i.e. the frequency of Render Server backboardd. Currently the tool has a BUG that prevents it from displaying frame rates higher than 60 FPS.

  1. Instruments Display/VSync signal frequency

The timestamp of the Surface/VSync signal displayed by the Display tool in the Instruments. As shown below:

  • Display: refers to the screen duration time of a single Surface corresponding to the Display, corresponding to the rendering frequency of cpu-GPU pipeline
  • VSync: indicates the VSync signal timestamp, corresponding to the refresh frequency of the screen hardware

On a 60Hz screen, iOS devices use a dual-buffer refresh mechanism by default, namely the front frame cache and the back frame cache. The GPU always draws the current frame on the post-frame cache. When the VSync signal arrives, the pointer to the before and after frames cache (Swap FrameBuffer) is swapped and the screen refreshes to show the new content.

When the screen displays content at 120Hz, iOS switches to a triple-buffered refresh mechanism (see Surface in three colors above), which reduces the pressure on the rendering pipeline but adds a certain amount of on-screen rendering delay.

Metal applications can avoid this delay by setting -[CAMetalLayer setMaximumDrawableCount:] to 2 to force dual buffering on 120Hz screens.

A Surface Display may last several VSync intervals, but the extra VSync signal still represents additional screen refresh on the hardware layer, resulting in additional battery consumption.

The ProMotion equipment

First let’s take a look at a traditional device with a fixed refresh rate.

VSync signal interval was fixed at 16.67ms

The XR’s screen refresh rate is fixed at 60Hz, which corresponds to the VSync signal interval, which is fixed at 16.67ms in any scenario.

In addition, when displaying static content, since the Layer Tree view does not change, Core Animation will not submit new transaction submissions and backboardd will not refresh, so the Surface corresponding to this frame has not been exchanged for a long time (tens of seconds). The value of Core Animation FPS is 0.

But because the VSync signal is still firing at 60Hz, the screen is showing the same Frame Buffer over and over again, consuming additional power.

CADisplayLink follows VSync signals almost entirely

Based on past knowledge of iOS, we know that CADisplayLink is driven by VSync signals:

The CADisplayLink callback configured by default should be roughly synchronous with the VSync signal.

This is verified on XR by using Instruments to record a main-thread holdup:

Among them:

  • The first linerunloopRecord each RunLoopAfterWaiting -> BeforeWaitingThe interval of
  • The second linetickRecords the default configured CADisplayLink callback interval
  • At the bottom is the hardware Display/VSync event sequence diagram

The following phenomena can be observed, which is consistent with our previous understanding of DisplayLink:

  • There is a strict one-to-one correspondence between the VSync signal and the wake up of RunLoop & the CADisplayLink callback.
  • The RunLoop is stuck, unable to process the Source 1 signal, and the DisplayLink callback is delayed until the stall ends.
  • During this process, the VSync signal interval remains constant.

ProMotion equipment

Let’s look at the test results for the ProMotion device.

VSync signal interval is variable

The VSync signal interval is variable on the ProMotion screen, specifically:

  • When static content is displayed, the screen frequency is lowered and refreshed at a minimum of 10Hz
  • When displaying Core Animation, the system changes the refresh rate to match the frame rate Settings of the Animation
    • * bypreferredFrameRateRangeYou can set thehintRequest high brush, but not necessarily in effect, see below”Application scenario of dynamic frame rate“Section.
  • When displaying the contents in sliding, the refresh rate fluctuates around 80Hz and changes with the sliding speed. The refresh rate increases on fast slides and decreases on slow slides.
  • When displaying a video, the refresh rate is consistent with the video frame rate

You can see that the VSync signal interval actively changes with the rendering frame rate of the displayed content.

Reduce display latency caused by lag

When a frame takes too long to render in the sliding process due to the main thread lag, the system will change the corresponding VSync signal interval of this frame (Surface 5 below) to reduce the delay from rendering to display, so as to slow down the immediate length of the card perceived by the user.

DisplayLink does not follow the VSync signal exactly

Here is a comparison record of the CADisplayLink callback and the Display/VSync event for the slide scenario. Different from before, there is no obvious following relationship between DisplayLink and VSync signals on the ProMotion device:

Specifically:

  1. The DisplayLink callback that the third arrow points to is not timely. The main thread had stalled before that, and two additional runloops were executed, but the DisplayLink callback was not called until the third time.
  2. Not only is there a timing mismatch, there are also cases where VSync is received but the DisplayLink callback is not triggered (and the main thread is idle), such as at ❓ in the figure above.

Remove the DisplayLink frame limit

We know that on iOS 15, Apple puts a default limit on the frame rate that third-party apps can display. Third party applications need to be in the Info. The plist add < key > CADisableMinimumFrameDurationOnPhone < / key > < true / > field can unlock a 120 hz refresh rate.

In iOS 15, CADisplayLink and other animation-related apis have also added an attribute for setting the preferred framerate:

/* Defines the range of desired callback rate in frames-per-second for this display link. If the range contains the same  minimum and maximum frame rate, this property is identical as preferredFramesPerSecond. Otherwise, the actual callback rate will be dynamically adjusted to better align with other animation sources. */ @property(nonatomic) CAFrameRateRange preferredFrameRateRange API_AVAILABLE(ios(15.0), watchos(8.0), tVOs (15.0));Copy the code

In order to further explore the relationship between DisplayLink and VSync signals on the new device, the author lifted the frame rate limit of the Core Animation of the test App, configured the corresponding API, and tested it again in different scenarios:

A scene that displays dynamic content

Animation scene

Show a medium speed displacement animation, resulting in the following image:

It can be intuitively found that the screen refresh rate after DisplayLink unlocks the frame rate is basically stable at 120Hz. And the relationship between VSync and DisplayLink seems to be re-matched.

However, by slowing down the animation speed, the author found that the corresponding relationship changed:

It can be observed that when playing slow motion pictures, the DisplayLink frequency is still the configured 120Hz, but the actual screen refresh rate is only 30Hz.

Sliding scenario

Let’s test it again in a different scenario. Quickly swipe the view to get the following image in Instruments:

It can be found that after DisplayLink unlocks the frame rate, the screen refresh rate is basically stable at 120Hz, and the frequency drops only when frames are lost.

  1. Note that there are no UI changes in CADisplayLink’s callback other than the OS_SignPOST call to report log.
  2. Even though the TableView presented by the author is extremely simple, frame loss can still be observed in the figure above, and it cannot be perfectly stable at 120Hz in sliding. This may indicate that UIKit’s rendering performance at 120Hz is somewhat of a native bottleneck.

Then the speed of swiping screen is reduced, and the result is similar to that of slow motion painting. Although the DisplayLink callback speed is not reduced, the VSync signal frequency is always kept at a low level:

Caton scenario

Both of the above tests were close to the ideal situation, where the entire Render Loop executed with almost no lag or lag. However, in the real world, there are all kinds of big and small lag problems.

In order to verify the relationship between DisplayLink and VSync signals in a more realistic situation, the author artificially added a tiny 20ms katton for testing under the condition of continuous sliding:

As can be seen from the above figure, the ProMotion screen has dealt with the stall well. Due to the existence of the three-buffer mechanism, during the Render Loop rendering of Surface 4, by changing the VSync interval, The system tries to delay the display of Surface 283 and Surface 250 in the buffer, so as to shorten the time for the user to see the still picture.

Then, the main thread resumes execution, and you can see that DisplayLink’s callback frequency quickly returns to its high pre-freeze level. At this time, the frequency of VSync signal actually decreases due to the aforementioned lag slowing mechanism. Now the frequencies don’t match.

This is similar to the situation of playing slow motion pictures/slow sliding before. Due to the lag and the existence of buffer mechanism, the system reduces the screen refresh frequency in a short period of time, but still maintains the high-speed callback of DisplayLink on the CPU side. This satisfies the Settings of the preferredFrameRateRange API.

To further analyze the nature of this mechanism, I will next attempt to reverse engineer the changes to the system library-related implementation in iOS 15.

Reverse analysis

DisplayLink driver mode changes

Set a breakpoint on the CADisplayLink callback method, running on iOS 14 and 15 ProMotion devices respectively, to get:

  1. On iOS 14, CADisplayLink is driven directly by receiving VSync signals via Source 1 mach_port

  1. On iOS 15 ProMotion devices, CADisplayLink is no longer driven by the VSync signal, but by a Source0 signal within UIKit

In 15, CADisplayLink registers a Source 1 signal when it is first created and added to RunLoop, as in 14.

The corresponding symbol of the callout callback address is display_timer_callback, which is also the same as in 14.

This also explains why the VSync signal on 15 does wake up a RunLoop, but this wake up does not necessarily trigger a DisplayLink callback, which means that the display_timer_callback behavior must have changed in some way compared to 14.

display_timer_callbackLogical variation

Analysis of the display_timer_callback implementation using Hopper shows no difference between the 15 and 14 implementations. Using LLDB debug, step by step analysis, observation to the subsequent calls function for CA: : Display: : DisplayLink: : callback, the key to the disassembly code as shown in the figure below:

If you look at disassembly code, you can see that if the CA::display_link_will_fire_handler block returns NO, Then the VSync signal correction will not trigger a subsequent CA: : DisplayLink: : dispatch_items calls.

This is actually verified in the LLDB:

Note that the _CFRunLoopCurrentIsMain in the figure above is close to the code in the red box above, The subsequent BLRAA directive clearly calls a block (LDR X9 [X8, #0x10] above stands for removing the invoke pointer from the block structure). The W0 register in the TBZ instruction is the return value of the block execution, and 0 (i.e., NO) jumps to 0x1848DBC08, which skips the call just after the dispatch_items call.

By looking at the blraa instruction step in above, we see that this block is actually registered by UIKitCore:

Find the symbol UIKit __UIUpdateCycleSchedulerStart private methods, the disassembly result verified it.

It is also found that the block returns a fixed value of 0x0.

The same symbol does not exist in the previous iOS version, that is to say, this should be a change in iOS 15. For non-promotion devices installed with iOS 15, retrace the above reverse flow and find that the CA:: display_link_WILL_FIRE_handler of this device is nil, unregistered:

Here CBZ performs a jump, indicating that x0 is nil, and x0 is obtained by LDR x0, [x8, # 0x1C8].

You can see that x0 is CA::display_link_will_fire_handler. Continue to analyze find private symbol __UIUpdateCycleSchedulerStart related implementation, before you can know that this is because in the ProMotion devices _UIUpdateCycleEnabled returned to the NO.

Under the condition of the returned NO __UIUpdateCycleSchedulerStart methods do not perform, CA: : display_link_will_fire_handler also will not be registered.

Changes to _UIUpdateCycleEnabled

Continuing with the code related to _UIUpdateCycleEnabled, I found that this change doesn’t just affect the DisplayLink driver.

When _UIUpdateCycleEnabled returns YES, UIKit can be carried in the UIApplicationMain _UIUpdateCycleSchedulerStart. The function, it is found that _UIUpdateCycleEnabled enabled will call [CATransaction setDisableRunLoopObserverCommits: YES].

Core Animation is the rendering engine of most iOS applications. Those familiar with the iOS rendering process must know that its execution is also driven by MainRunLoop, which is roughly as follows:

  1. MainRunLoopIssue an event/callback because the user operation /Timer/GCD etc was awakened
  2. Apply changes to the callbackLayer TreeTo triggersetNeedsLayout 或 setNeedsDisplay
  3. MainRunLoopThe execution is about to completeObserverdistributedBeforeWaitingThe event
  4. BeforeWaitingIn the triggerCore AnimationregisteredMainRunLoop Observer, triggers the transaction commitCA::Transaction::commit():
    • Trigger all kinds of Layout/Display logic from top to bottom to update the Layout/ content
    • Core Animation packages the updated Layer Tree and sends it to Render Server
  5. thenMainRunLoopEnter the dormant
  6. Render ServerPack upLayer TreeDecode, generate and submit the correspondingdraw calls
  7. GPUExecute render command, render outFrameBuffer, will be displayed on the screen when the subsequent VSync signal comes

Above + [CATransaction setDisableRunLoopObserverCommits: YES] this call gave the author prompts, Let’s verify the execution timing of CA::Transaction:: Commit () on iOS 15 ProMotion devices and see that it is indeed no longer driven by BeforeWaiting events:

In fact, the same Source 0 signal also drives the CADisplayLink callback:

If you look at the Source0 callback symbol runloopSourceCallback, you can see that Source0 is driven by signalChanges:

SignalChanges, in turn, are driven by multiple callbacks:

Among them:

  1. runloopObserverCallbackAs aBeforeWaiting 的 MainRunLoop observerDriver.
  2. runloopTimerCallback 由 mk_timerDriver, corresponding tomach_portUnknown. The test found that its callback frequency is around 1Hz, but it also changes constantly. Guess it is some kind of system timer.
  3. inputGroupSignaledCallback 由 mk_timerDriver, corresponding tomach_portIt’s the VSync signal.

  1. requestRegistrySignaledCallback 由 UIScrollViewDrive when you are about to start sliding.

Based on the above analysis, I have reason to believe that there has been a significant change in the rendering driver mechanism for iOS 15 apps. One of them is the change in DisplayLink’s driver.

conclusion

  1. In iOS 15, Apple has changed the driver mode of rendering event loop in ProMotion devices. Transaction submission of CoreAnimation is no longer driven by RunLoop completely, but involves multiple signal sources
  2. The system dynamic framerate selection mechanism takes into account the user set API (such as preferredFrameRateRange) and the frequency of changes in the actual display content. Specifically for CADisplayLink:
    • CADisplayLink unlocks a high refresh rate that only affects its own callback frequency when content changes at low speeds, and the system may still choose a lower screen refresh rate to reduce power consumption
    • CADisplayLink unlocks a high refresh rate for medium-speed content changes, allowing the system to choose a higher refresh rate and even lock the 120Hz refresh

As for how to define low/medium high speed, the author has done some experiments in setting dynamic frame rate of CAAnimation in the following part, which can be used as reference.

The default CADisplayLink callback frequency is up to 60Hz. Refresh events with higher frequencies cannot be monitored.

  1. In ProMotion devices, DisplayLink is no longer driven directly by the VSync signal, but is executed in the newly introduced render event loop. The new version of iOS implements a more sophisticated mechanism to call back as often as the user sets the preferred frequency, but does not guarantee strong correlation with the VSync signal. This means that the default CADisplayLink callback frequency does not match the actual frame rate, and the previous CADisplayLink based frame rate monitoring scheme is no longer viable on ProMotion devices.

Application scenario of dynamic frame rate

Monitor fluency performance at dynamic frame rates

CADisplayLink is generally used in the industry to monitor application fluency. Due to changes in CADisplayLink’s behavior on iOS 15, the original monitoring scheme was unable to assess the performance of the ProMotion screen beyond 60Hz.

Based on the above conclusions, the author currently assumes three compatibility modification schemes for ProMotion equipment:

Plan a (Pass)

For any device whose optimization goal is 60Hz, only the case where the refresh interval is longer than 16.67ms is considered. In other words, when the screen is refreshed at 120Hz, it is considered that no frame is lost even if 1 frame is lost, because the interval between the two frames is still less than 16.67ms at this time, theoretically, the user has little perception.

Advantages:

  • The solution is simple and only needs to be setpreferredFramesPerSecondIs a fixed value of 60
  • Compatible with previous metrics. The FPS index can still be calculated. If the refresh rate is higher than 60Hz, the refresh rate is considered as 60Hz

Disadvantages:

  • Since the maximum 60Hz can only be monitored, it is impossible to evaluate the impact of some small frame loss at a higher refresh rate on user experience, or the technical impact of some optimization for high brush screen
  • When the refresh rate is low, MainRunLoop will still run at 60Hz, which has a certain impact on power consumption

Scheme 2 (Pass)

By some means, you can replace the callback of the Source 1 signal that drives the display_timer_callback and use it to accurately listen for the VSync signal for accurate monitoring of the dynamic frame rate.

Advantages:

  • Theoretically the most accurate monitoring scheme
  • The impact on power consumption is minimal, and the callback frequency increases only when the screen refresh rate actually increases

Disadvantages:

  • Private apis are used
  • FPS metrics no longer apply
  • The VSync signal is currently not an exact match to the rendering flow, which is accurate but not necessarily practical

Plan 3 [Pick]

Calculate the real-time refresh rate of the current screen by confirming the Duration parameter in the CADisplayLink callback, and modify the preferredFrameRateRange to track it.

Advantages:

The solution is relatively simple, simply updating the preferredFrameRateRange property of the DisplayLink object with each callback

Disadvantages:

  • Due to the existence of dynamic frame rates, FPS indicators can reflect the real-time screen refresh, but the aggregated value is not significant, and consumption needs to be differentiated by specific models/scenarios
  • The current minimum callback frequency observed is 60Hz, which means that ProMotion screen performance at 48Hz, 30Hz or even lower refresh rate cannot be confirmed
  • When the refresh rate is low, MainRunLoop will still run at 60Hz, which has a certain impact on power consumption

Note that CADisplayLink’s preferredFrameRateRange needs to be set in a format similar to this:

NSInteger currentFPS = (NSInteger)ceil(1.0 / displayLink.duration); DisplayLink. PreferredFrameRateRange = CAFrameRateRangeMake (currentFPS 10.0, 0.0);Copy the code

CAFrameRateRange. Minimum 10.0 preferred 0.0 CADisplayLink is only used to monitor the current system frame rate without affecting the dynamic selection of frame rate.

Compared with the first two schemes, scheme 3 has small changes, does not use private API, and has higher monitoring accuracy. Its disadvantages are relatively acceptable.

Alternative metrics for FPS

Considering that the FPS metric on the ProMotion screen is no longer directly related to how smoothly the application runs, its aggregate value is of little reference value and it is necessary to find a new metric to replace it.

In WWDC20-10077 Eliminate Animation hitches with XCTest, Apple officially introduced the concept of Hitch Time Ratio. It is emphasized that it is more suitable for different refresh rate scenarios than simple FPS.

In the XCTest framework, Apple provides API XCTOSSignpostMetric to help developers get metrics instantly in a single test, but the API is only available in a single test and is not available online. MXAnimationMetric in MetricKit, although available online, is not real-time and cannot meet the monitoring requirements of large apps for different scenarios.

So, follow Apple’s definition of Hitch Ratio:

Hitch time:

  • Time in ms that a frame is late to display.

Hitch time ratio:

  • Hitch time in ms per second for a given duration.

The author tries to implement the (Scroll) Hitch Time Ratio calculation scheme based on CADisplayLink:

  1. The Hitch Time of the previous frame is calculated by calculating the timestamp of the previous frame and the target frame of the previous frame
  2. Determines if the frame is rendered in a slide
  3. The overall Hitch Frame is accumulated, and compared with the cumulative Frame interval, the Scroll Hitch Time Ratio is obtained

Improved frame rate for key scenes

During the test, the author found that the system App was stable and operated at the highest refresh rate of 120Hz when sliding:

And third party App even set up CADisableMinimumFrameDurationOnPhone to true cannot slide stability in full frame rate (validated, it remains on the iOS 15.4 beta system).

By taking advantage of the new apis introduced in iOS 15, we can proactively unlock higher/lower dynamic frame rates during key scenes such as swipes, transitions, and animations to optimize fluency or power and improve user experience goals.

Stable in sliding 120Hz

First of all, the author hopes that non-system App can also achieve stable 120Hz refresh in sliding as much as possible.

With the above analysis, this can be done with CADisplayLink. Here, the author puts forward two possible schemes for reference only:

  1. Create CADisplayLink and configure itpreferredFramesPerSecondIs 120, and then add it toUITrackingRunLoopModeIn the.
CADisplayLink *dp = ... dp.preferredFramesPerSecond = 120; / / or dp. PreferredFrameRateRange = CAFrameRateRangeMake (120.0, 120.0, 0.0); [dp addToRunLoop:[NSRunLoop mainRunLoop] forMode:UITrackingRunLoopMode];Copy the code

In a swipe, the CADisplayLink is activated, and the system locks the current frame rate to a maximum of 120Hz (effective only when the content changes at high speeds). The normal frame rate is restored when the slide is stopped.

  1. To add CADisplayLinkCommonModes, enable/pause CADisplayLink at start/stop sliding, and modify the correspondingpreferredFramesPerSecondAnd other properties trigger frame rate changes.
CADisplayLink *dp = ...
dp.paused = YES;
[dp addToRunLoop:[NSRunLoop mainRunLoop] forMode:NSRunLoopCommonModes];

CFRunLoopAddObserver(CFRunLoopGetMain(),
                         CFRunLoopObserverCreateWithHandler(kCFAllocatorDefault, kCFRunLoopEntry | kCFRunLoopExit, YES, 0,
                                                            ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) {
        if (activity == kCFRunLoopEntry) {
             dp.paused = NO;
             dp.preferredFramePerSecond = 120;
        } else {
             dp.paused = YES;
             dp.preferredFramePerSecond = 0;
        }
    }), (__bridge CFStringRef)UITrackingRunLoopMode);
Copy the code

In practice, scheme 2 has better versatility because it is also necessary to unlock frame rate upper limit in non-sliding state.

CAAnimation sets the dynamic frame rate

Apple provides only modify CAAnimation animation frame rate of the API, set CAAnimation. PreferredFrameRateRange can change its effects on the screen refresh rate.

  • For obvious user perception, such as transition animation, can be set to 120Hz.
  • For less perceptive, such as rotating animations, you can lower the frame rate, such as 30Hz.

However, as with DisplayLink, setting through the API above will “influence” the system’s dynamic frame rate choice, but this effect is not absolute. In practice, the author found that the screen refresh rate is related to the speed at which CAAnimation changes on the screen.

Regarding this point, taking iPhone 13 Pro as an example, the author uses a simple pan animation with a fixed frame rate of 120Hz as the illustration:

CABasicAnimation *anim = [CABasicAnimation animationWithKeyPath:@"transform.translation.y"]; CGFloat speed = 170.0/330.0; anim.toValue = @(100); anim.fromValue = @(0); Anim. Duration = 10.0; anim.repeatCount = FLT_MAX; anim.preferredFrameRateRange = CAFrameRateRangeMake(120, 120, 120);Copy the code

Where the speed variable is the translation speed, and the unit is pt/s. The test finds that:

  • speedWhen (0, 160] is set, the screen refresh rate is 60Hz
  • speedWhen [161, 320] is selected, the screen refresh rate is 80Hz
  • speedWhen [321, +∞), the screen refresh rate is 120Hz

The author only tested the panning animation scene on iPhone 13 Pro, and the above data is for reference only.

Finally, for the other common animation API, such as UIView. AnimateWithDuration, UIViewPropertyAnimator, etc., does not provide the corresponding API changes. It is theoretically possible to retrieve the CAAnimation objects created by the upper-level APIS to modify them.

Gestures/transitions and other scenarios unlock 120Hz

Other scenarios need to control the dynamic frame rate can also be manually modify CADisplayLink preferredFramePerSecond/preferredFrameRateRange properties, The implementation is basically the same as listening for the RunLoop to change the sliding frame rate.

UIGestureRecognizer is often used to implement interactive animations. After testing, it was found that enabling a CADisplayLink that unlocked the frequency while triggering the gesture callback also indirectly increased the callback frequency of UIGestureRecognizer, resulting in higher frame rate interactive animations.

For transition scenes, a simple solution is a swizzle UIViewController lifecycle message that enables/deactivates CADisplayLink frame rate unlocking on/off nodes, thus implementing a universal page transition animation frame rate unlocking scheme.

Flutter officials also plan to provide a similar API that allows apps to dynamically switch screen refresh rates for different scenarios (sliders, animations, etc.) : github.com/flutter/flu…

Online income

Based on the above ideas, the author’s team implemented an optimization project in the international short video business, which has been verified by experiments:

  • The large slide frame rate P50 rose to 112.2 from 81.57
  • Core business metrics also have some benefits

conclusion

In recent years, the development of software and hardware in The Apple ecosystem has been changing rapidly, including the continuous optimization of dyLD in the software layer and the introduction of Prewarm mechanism in iOS 15, as well as the new ProMotion screen. It can be seen that Apple has been committed to creating a more silky and smooth user experience.

The system-level optimization solutions provided by Apple are generally generic and not perceived. However, generality often implies certain limitations, which may leave additional optimization room for application developers to further study how to better adapt.

For example, in this paper, by studying the mechanism behind the newly introduced ProMotion screen, the author can get a glimpse of part of the essence through the appearance/in-depth assembly. Finally, the monitoring + optimization scheme is implemented, and the large sliding frame rate P50 increases from 80 to 112, achieving additional business benefits.

Finally, I believe that we developers, as part of the Apple ecosystem, should take the initiative to understand the underlying principles behind the above optimization while enjoying the benefits automatically generated by system-level optimization. On the one hand, understanding and learning Apple’s mature optimization ideas can improve our vision as engineers. On the other hand, the understanding of the underlying principle of the system can expand our “ammunition”. The wider and deeper we understand the whole link of business value delivery, the more likely we are to grasp the potential optimization points, so as to go further and better in the career path of performance optimization engineer.

The resources

  1. WWDC20 – 10077 Eliminate animation hitches with XCTest

    Developer.apple.com/videos/play…

  2. WWDC21 – 10147 Optimize for variable refresh rate displays

    Developer.apple.com/videos/play…

  3. Optimizing ProMotion Refresh Rates for iPhone 13 Pro and iPad Pro

    Developer.apple.com/documentati…

  4. What is Adaptive Sync?

    www.viewsonic.com/library/tec…

  5. Github.com/flutter/flu…

Join us

We are the basic technical team of byte international short video, and we are a team that pursues perfection in depth. We focus on performance, architecture, package size, stability, automated testing, basic library, compilation and construction, etc., to ensure the research and development efficiency of the super-large team and the use experience of hundreds of millions of users around the world. At present, there are a large number of talents in Shanghai, Hangzhou, Singapore and the United States. We welcome people with lofty ideals to build a global APP with 100 million users together with us!

You can click jobs.bytedance.com/experienced… , enter bytedance recruitment website to send your resume, you can also contact: [email protected] to consult relevant information or directly send your resume internal push!