introduce

The early development of Jingxi APP was mainly to replace the original H5 with rapid protobiochemical iteration to improve user experience. During this period, many performance problems also accumulated. After that, we began to do some work related to performance optimization. This paper mainly introduces the optimization strategies related to Beijing Xi photo library and some related knowledge about pictures.

Image performance issues

As an e-commerce APP, pictures are widely used in various business scenarios. We need to reduce network consumption/memory consumption/hard disk consumption as much as possible, while not reducing picture quality, improve picture loading speed, and bring better user experience. Based on these performance goals, we also tease out some performance issues through preliminary performance evaluations:

Slow image loading/high traffic consumption

Image links are sent by background interfaces, and the format and size of images are specified by each service background. Some businesses do not use smaller image formats such as WebP, or the image size is too large, which will lead to high network consumption. Especially when the network condition is poor, the image loading is slow, which brings bad experience to users. It also takes more time to read and write I/O and decode data, resulting in more power consumption.

Image memory usage is high

According to the preliminary evaluation of APP memory usage, the image memory consumption accounts for the highest proportion of the total memory consumption of APP, especially the large size image will occupy a lot of memory. On the one hand, the APP occupies too much memory and is easy to be killed by the system when it is withdrawn from the background. As a result, the experience will be affected when it is restarted next time. On the other hand, APP uses a lot of memory, which is easy to be killed by the system and generate OOM. In particular, we currently have a large number of low-end device users, and the device memory is relatively low.

To optimize the direction

Based on the above analysis of some performance problems, we carried out the overall reconstruction of the picture frame optimization. On the one hand isTo reducePicture network transmission, improve picture loading speed. On the other handTo reduceImage memory consumption.

Minimize network traffic

Jingdong picture server provides a variety of processing functions, such as picture format conversion, picture quality reduction, picture scaling, picture rounding and other functions. These functions are realized by adding specific parameters in the IMAGE URL, and the image server will complete the image processing and save it to the CDN server in advance according to the parameter Settings. We can reduce the image transfer size by adding image processing parameters.

Although the background can carry out URL preprocessing in advance, deliver the image URL that has added image parameters. However, because many background services are connected, image parameter Settings for each service vary greatly and may affect performance. For example, if the webP image format is not used, a large image size may be delivered. At the same time, considering the high cost of promoting the modification of each business background, and the number of front-end models, different models need to use different picture sizes. In addition, it is not convenient for grayscale degradation function, and the subsequent function modification is not convenient. Therefore, image URL preprocessing on the client is a better way, which can be unified control and convenient to update the function later.

Image URL preprocessing

Image library check whether the network picture is loaded beforejingdongImage of domain nameURL. ifThe domain nameMatch, picture frame to picture firstURLPreprocessing, preprocessing includesThe domain name and unified.Add scaling parameters.Add webP parameters.Add quality reduction parametersThe way to reduce picture network transmission size.

Tip: because the background pictures of return URL might with part of the image processing parameters, such as https://img11.360buyimg.com/img/pingou-head/25.jpg! Webp, directly append image parameters may result in invalid image processing parameters or loading failure due to incorrect format. Therefore, all image parameters will be calculated in advance during conversion, and then processed together to avoid adding repeated parameters.

The domain name and unified

Currently, the image server provides multiple image domain names to use, such as m.360buyimg.com, img10.360buyimg.com and other domain names. M.360buyimg.com is primarily for mobile. However, because various service backgrounds are connected, the interface delivers different domain name images. Images using different domain names may cause the following problems:

  • Bad for cache reuse– Image frames usually default toURLString generation pictureThe cache key, differentThe domain nameCause to produce differentThe cache key.Hard disk cacheFailure to reuse results in images being downloaded repeatedly,Memory cacheUnable to reuse the same image to occupy more than one memory.
  • Bad for HTTP/2 connection reuse– Most screens have a lot of images, many scenes will load more than one image at the same time, especiallyThe first screenDozens of images are usually loaded. When loading multiple images, each domain name needs to be re-establishedHTTPSConnect, experienceDNS resolution /TCP connection /TLS handshakeProcedure (Currently, it takes about time to create an HTTPS request50-150ms). If the use ofHTTP/2Link reuse only needs to be created onceHTTPSRequest, and subsequent image requests can reduce this part of the time.

Therefore, in the pre-processing, if the image is jingdong domain name, the image URL domain name is uniformly replaced with m.360buyimg.com.

Append image parameters

zooming

Many of the original image urls returned by the business background are larger than the actual size displayed by the client. On the one hand, it leads to waste by using more network traffic. On the other hand, it takes up more memory. At the same time, because the image size is inconsistent with the actual display size, the pixels are not aligned, and the GPU needs to do additional interpolation processing, which will also affect the rendering performance to a certain extent. So by adding the zoom parameter, we specify the image server to send smaller and more match the actual display size of the image.

Dynamic scale computes dimensions

Since iOS devices mainly use 2x/ 3X resolution, the business side needs to input the corresponding PTsize when using THE API, and the real pixel width and height are dynamically calculated according to the scale of the device in the image library.

Tip: Android devices are better suited to using a fixed scale because of their screen differences. Too many image sizes are not conducive to CDN cache. When there is no cache, relevant parameters of the image need to be processed, and image processing itself is a time-consuming operation.

Scale downgrade
  • Low-end downgrade– For parts3xScale low-end devices, because the machine itself is relatively low memory, use3xThe resolution of the image is calculatedpixelA large aspect ratio causes more memory consumption and more decoding/rendering performance consumption. So for images that exceed a certain width and height, downgrade to use2xTo calculate the resolutionpixelWide and high, reduce equipment performance consumption.
  • The downgrade– Because currently the APP does not targetiPadFor specific optimizations, so the default on the iPad is to zoom in. This can lead toiPadThe size of the picture below is calculated to be particularly large. Therefore, there are specific restrictions on the size of iPad pictures to prevent them from being too large.
  • Large image degradation– Pictures in normal conditionWidth/heightIt should not go beyond the screenWidth/height. To prevent some businesses from using oversized imagessizeSo add a limit to the final generated picturepixelThe size cannot exceed the screenWidth/height.
Drop mass

The image server supports the image quality parameter setting from 0 to 100. Reducing the image quality can reduce the image size, but too much quality reduction will also affect the image viewing experience. We set the image quality parameter to Q70 and specify the image server to deliver 70% quality images. For most businesses, the download size can be greatly reduced while the viewing experience can be guaranteed. Reduce image size by at least 30-40% by adding image degradation parameters.

Using WebP

According to Google, WebP lossless images have 26 percent fewer bytes than PNG images. WebP lossy images have 25-34% fewer bytes than comparable JPG images. Image server supports converting webP format, which can reduce image size. For the PNG/JPG image format, add webP parameter and specify the image server to deliver webP format. Although webP takes longer to decode than PNG/JPG images, the speed of network transmission is still greatly improved.

Tip: GIF is not processed because the current image server does not support GIF to webP.

URL preprocessing cache

Add lightweight cache to improve URL translation performance. This is because THE URL conversion itself is time consuming and a single image URL may be loaded/converted multiple times. The transformed URL is directly saved in the cache and can be returned the next time. The cache key is composed of URL+ related image conversion parameter stitching.

Image API Design

The image processing parameters are set through options. By default, q70 image quality and webP format are used. When the business side calls the loading image method, the following is the API of iOS:

imageView6.jx.setImage(url: URL(string: "https://img11.360buyimg.com/img/pingou-head/25.jpg"), 
                       placeholder: nil, options: [.imageSize(CGSize(width: 40, height: 40)))Copy the code

Reduce image memory consumption

PNG/JPG and other image formats need to be decoded to generate a bitmap before display, and then create texture according to the bitmap and pass it to GPU for rendering. The memory consumption of a bitmap is about pixels wide x pixels high x bits deep. Usually images are RGBA, 32 bits deep. A 500pX_500px sheet is about 1MB of ram. For GIF images, there are multiple frames, so the final memory consumption is x frames per frame.

On the one hand, our optimization direction is to reduce the memory consumption of image bitmap by the way of image scaling. On the other hand, limit the image cache limit to avoid excessive cache usage.

zooming

Using the URL preprocessing process above to get the image server to deliver smaller image formats already reduces memory. However, URL preprocessing only processed JPG/PNG images of JD domain name, and did not process images outside GIF or JD domain name, including some images that failed to load after URL conversion. Therefore, for this part of the picture, we will do image scaling on the end side to reduce memory consumption. For example, if a 300px_300px GIF contains 100 frames and the actual display area is only 50pX_50px, the total memory consumption can be reduced from 30MB+ memory to 3MB after optimization.

Set the image cache upper limit

Image caching is designed to reduce image decoding consumption. When the image is used for the first time, the bitmap of the image after decoding is saved in memory, so as to avoid repeated decoding in the next use. Although the image memory is high, it can avoid image decoding repeatedly, but occupying too much memory will also lead to the APP background being killed by the system or OOM. So we should keep the memory cache within a certain range.

For example, SDWebImage/Kingfisher, a third-party image library of iOS, uses NSCache to realize memory caching by default. Although NSCache reclaims memory when the device is running out of memory, there is no default limit on the maximum number of bytes of memory that can be saved, so memory can always be increased as long as the device is available. Therefore, the image cache upper limit is set to prevent the image cache from occupying too much memory. The image cache defines a default initial upper limit, and later increases the upper limit appropriately for 3x large screen devices and high-end devices (which have high memory).

Optimization results

Other income
  • The domain name and unified– to reduce the10% +Duplicate image downloads and memory consumption. And at the same time decreaseMany of the domain nameImages are created repeatedly as they loadHTTPSRequest process to reduce image load time.

Other strategies

Loading exception Handling

After a few images are converted through URL preprocessing, there may be abnormal scenes where images do not exist, leading to loading failure. So when the image load fails, we still need to load the original image URL. However, some special loading errors need to be shielded to avoid unnecessary loading, such as no network, network timeout, or active load cancellation errors. After that, the wrong IMAGE URL is reported to the background, so that the URL conversion policy can be adjusted later. In addition, some wrong image URL can be found to promote service modification. At the same time, this part of the connection is added to the error connection cache to avoid repeated preprocessing and repeated reporting.

Online configuration

Some existing functions, such as URL preprocessing/unified domain name /WebP usage, have been added online configuration to facilitate gray/degrade. Some functions can be degraded when problems occur, and grayscale tests can also be carried out when new functions come online.

A larger image detection

There needs to be a mechanism to detect non-conforming images in a timely manner. On the one hand, through online gray detection, we will report large pictures when they are found, and then promote the business side to optimize. On the other hand, in the daily test phase, we will enable the Debug detection tool. When detecting a large picture, we will remind business development students to optimize by flipping the picture or highlighting the background color.

Flutter image library optimization

Currently, jingxi APP has more than 10 secondary interfaces based on Flutter development, so we have also made some optimizations for Flutter image loading.

Connect to the native photo library

Because the image library of the Flutter framework only provides in-memory image caching and does not support hard disk image caching, the image will be downloaded repeatedly. So we rewrite the ImageProvider. When loading network images, we call the native image library through Channel. After downloading the images to the local disk, the native image library returns the image file directory. The Flutter then loads the decoded picture display through the file directory. On the one hand, we can make use of the optimization capabilities of the native image library, and at the same time, we can reuse the image hard disk cache to avoid repeated downloads.

Reduce memory consumption

When using the Image component, cacheHeight/cacheWidth decodes the Image to a bitmap size of the top pixel width to reduce memory consumption. Also, because the memory consumption of a Flutter is higher than that of the native Flutter, the memory consumption of a Flutter can be reduced by calling the imageCache method to clear the image memory consumption when the Flutter interface is closed.

GIF optimization

  • Animation optimization– Because it’s usually usedFlutterBoth are hybrid stack mechanisms,nativeandFlutterThe interfaces switch to each other in the navigation tree. So whenFlutterThe interface isGIFImages when jumping to native laterGIFThe animation will always be executed. So we got through theImageIntra-component listeningFlutter engineA lifetime notification sent to stop a Flutter interface when it is not at the top of the stackGIFAnimation execution reduces memory and CPU consumption.
  • Reduce decoding times– Internal pairs of the Flutter frameGIFRender processing in each frame of the screen to determine the current need to display the GIF frame, after thatGIFFrames are decoded and rendered. Because the decoded frames are not saved, frequent decoding will result in large memory fluctuations. After optimization, the decoded frame is saved to avoid the consumption of repeated decoding and to avoid memory fluctuations.

Memory fluctuation is obvious before optimization

After optimization, memory tilt is stable

Tip: Saving each frame also results in more memory consumption. Currently, giFs in the APP are usually small, so the overall control. Consider setting a buffer upper limit to control the number of frames cached and avoid memory overload.

Direction of subsequent optimization

Better caching algorithms

  • Remove the maximum memory first– the iOSNSCacheThe implementation. By setting the maximum memory size, you can remove the maximum memory size when the memory is insufficient.
  • The LRU cache– The oldest unused image memory is preferentially eliminated. For manyThe secondary interface“, the user will not open the interface again after opening. But because these image caches are used last, they are also removed last when the memory is cleared, which is not appropriate in this scenario.
  • Interface stack management– when the interfaceShut downRemoves all images from the screen, but results in frequent images for frequently opened screenscodecIt’s not a good fit.

Therefore, different recycling methods may be more appropriate for different business scenarios:

  • forShopping cart/my orderIn this type of interface, the image is basically fixed each time the user loads, so it is better to live in memory and recycle when memory consumption is too high.
  • forCheck/search the list of productsThis kind of interface is more suitable because it usually shows different pictures of the product list and the user does not frequently enter a particular shopGive priority toRemove this portion of memory.
  • For some popover functions, pictures will not be used again after display, so you can consider not adding them to memory.

Use better image formats

Using a better image format usually results in a smaller image byte size. At the same time, because of the increase in compression rate, you can reduce the size and improve the quality of the picture.

Tip: There are advantages to using a hard decoded image format. Hard decoding is the use of GPU decoding, compared with the use of CPU soft decoding performance better and save electricity.

  • APNG/ animated WebP instead of GIF– in accordance with theGoogleOfficially,GIFconvertBeneath WebPThe number of bytes is reduced by 64%, whileNondestructive WebPThe number of bytes has shrunk by 19%. So usingAnimation WebPMore network traffic can be reduced.APNGisMozillaDerived based onPNGGIF format and fully supportedRGBACompared to,GIFCan reduce the20% +Picture size. andGIFIt only supports 256 indexed colors and 1 bit alpha(with transparency, the edges will be clearly jagged)APNG/WebPCan also bring comparisonGIFBetter display.

Tip: WebP decoding consumes more CPU resources than GIF. Lossy WebP decoding time is 2.2 times that of GIF and lossy WebP decoding time is 1.5 times that of GIF.

  • HEIC – HEIC is a picture format based on H.265 video encoding format. Compared with WebP, HEIC can reduce the image size by 20% and has better codec performance. In terms of system compatibility, Android 9.0 or later supports HEIC. Apple only provides WebP hard decoding for iOS14 and later, the previous system can only use soft decoding, while HEIC has supported hard decoding for iOS11 and later machines, but it does not support browsers.

  • AVIF – AVIF is an image format based on AV1 encoding format. AVIF can reduce image size by 30%+ compared to WebP. It is currently only supported by Android 12 or later.

Note: WebP in VP8 encoding format is mainly used here. The overall performance of WebP in VP9 encoding format is not much different from THAT of HEIC.

But these image formats need to be supported by the image server before they can be used.

Flutter

Although we have made some improvements to the Flutter gallery, there is still a lot of room for improvement overall. Includes texture-based image schemes that are in use in the industry. After decoding the image on the native side, textures are created using the Flutter engine. The image texture ID is then passed to Flutter for rendering. This allows the image cache to be managed uniformly on the native side, optimising the way that both Flutter and native have a memory cache. In addition, the navigation stack for mixed stack can also better image memory reclamation. In addition, a more flexible image memory reclamation strategy should be provided for Flutter to avoid excessive memory consumption.

Tip: Textures can reuse the bitmap cache in memory, so they don’t cause more memory usage. The texturing method can reduce memory consumption by approximately 30% compared to the Flutter engine gallery, which is mainly caused by some other objects used.

Optimized H5 image loading

We can intercept the WebView image loading way, let the native image library to download the image and then pass the image binary data to the WebView display.

Reduce flow consumption

In this way, we can support the related functions of URL preprocessing of native photo library to H5 images, reduce the consumption of image traffic in the process of H5 loading, and improve the speed of image loading. Meanwhile, since the APP native and WebView image caching mechanisms are independent of each other, the repeated downloading of the same image can be reduced by uniformly managing the image caching on the native side.

Support for more image formats

For example, on iOS, WKWebView currently only supports PNG/JPG/GIF image formats. So we can download WebP/HEIC images through the native implementation, and then decode the images to WebView, so that we can support the display of other image formats.

Tip: Since WebView does not support direct bitmap binary data display, you need to convert to PNG/JPG binary data transfer in advance. So adding a PNG/JPG encoding process for other image formats can cause more performance overhead. However, for Android systems it should be possible to optimize the Web kernel layer to reduce this consumption.

Image resolution

For the same picture, because the resolution is different, resulting in repeated downloads. If optimized in the hard disk cache, images with small resolution can be directly stored in the hard disk cache, thus avoiding image duplication and generating bitmaps with smaller resolution when decoding will not take up more memory.

conclusion

This paper does not talk about the specific implementation of the bottom image loading library, the current image library is not directly used by the third party library or self-developed image library implementation methods are usually not very different. We are more focused on our business and how to maximize image loading performance on the network using image server capabilities. Therefore, some strategies may not be suitable for all apps, and optimization schemes should be carefully evaluated for their own business scenarios.

Extended link

  • WebP
  • Hand wash photo library HEIC use
  • Animated WebP vs. GIF
  • WebP support
  • APNG support
  • AVIF