Author: Wang Zhenhui (Shinjuku)

Last year, The Idle Fish Photo Library achieved good results in its large-scale application, but it also encountered some problems and appeals, and needs to be further evolved to adapt to more business scenarios and the latest FLUTTER features. For example, because the native ImageCache is completely abandoned, some low-frequency images will occupy the cache when mixed with native images. For example, we can’t display images on the simulator; For example, in the album, we need to build a picture channel outside the photo gallery.

This time, we cleverly combined the external texture with FFi solution to get closer to the original design and solve a series of business pain points. That’s right, there’s a new addition to the Power series, and we’re calling it “PowerImage”!

We will add the following core capabilities:

  • Support the ability to load UI.image. In last year’s scheme based on external texture, the user could not get the real UI.Image to use, which made the Image library powerless in this special use scenario;
  • Image preloading capability is supported. Just like the native precacheImage. This is useful in some scenarios where the speed of image presentation is high;
  • New texture cache, and native photo library cache through! Unified image cache to avoid memory problems caused by the mixing of native images;
  • Support for emulators. Emulator can’t show Texture Widget before flutter-1.23.0-18.1.pre;
  • Perfect custom picture type channel. Solve the business custom picture acquisition demands;
  • Complete exception capture and collection;
  • Support for GIFs.

Flutter native solution

Before we start our new solution, a quick reminder of the Flutter native image solution.

The native Image Widget first obtains the ImageStream using the ImageProvider and displays various states by monitoring its status. For example, frameBuilder and loadingBuilder, after the image is successfully loaded, the RawImage will be rebuilt, and the RawImage will be drawn by RenderImage. The core of the entire drawing is the UI.image in ImageInfo.

  • Image: responsible for the display of each state of Image loading, such as loading, failure, loading successfully display images, etc.
  • ImageProvider: is responsible for obtaining ImageStream, such as built-in NetworkImage and AssetImage.
  • ImageStream: The object that the image resource loads.

After reviewing the scheme of flutter native images, we find out whether there is an opportunity to connect flutter images and native in a native way in a certain stage.

The new plan

We cleverly combine FFi solutions with external texture solutions to solve a series of business pain points.

FFI

As mentioned at the beginning, the Texture scheme has a few things it can’t do and needs to be complemented by other schemes, the core of which is uI.image. We pass information such as native memory address and length to the flutter side to generate UI.Image.

Firstly, the native side obtains necessary parameters (take iOS as an example) :

    _rowBytes = CGImageGetBytesPerRow(cgImage);
    
    CGDataProviderRef dataProvider = CGImageGetDataProvider(cgImage);
    CFDataRef rawDataRef = CGDataProviderCopyData(dataProvider);
    _handle = (long)CFDataGetBytePtr(rawDataRef);
    
    NSData *data = CFBridgingRelease(rawDataRef);
    self.data = data;
    _length = data.length;
Copy the code

After the DART is taken sideways,

@override FutureOr<ImageInfo> createImageInfo(Map map) { Completer<ImageInfo> completer = Completer<ImageInfo>(); int handle = map['handle']; int length = map['length']; int width = map['width']; int height = map['height']; int rowBytes = map['rowBytes']; ui.PixelFormat pixelFormat = ui.PixelFormat.values[map['flutterPixelFormat'] ?? 0]; Pointer<Uint8> pointer = Pointer<Uint8>.fromAddress(handle); Uint8List pixels = pointer.asTypedList(length); ui.decodeImageFromPixels(pixels, width, height, pixelFormat, (ui.Image image) { ImageInfo imageInfo = ImageInfo(image: image); completer.complete(imageInfo); / / release native memory PowerImageLoader. Instance. ReleaseImageRequest (options); }, rowBytes: rowBytes); return completer.future; }Copy the code

We can get native memory through FFI to generate UI.Image. There is a problem here. Although FFI can directly obtain native memory, because decodeImageFromPixels has memory copy, the memory peak will be more serious when the decoded image data is copied.

There are two optimization directions:

  • Image data before decoding is given to flutter, which is decoded by the decoder provided by FLUTTER to reduce memory copy peaks;
  • Discuss with flutter officials to try to reduce this memory copy internally.

FFI is suitable for light use and special scenarios. This method can solve the problem of not getting the UI.Image can also be displayed on the emulator (FLUTTER <= 1.23.0-18.1.pre).

Texture

The Texture scheme has some difficulty with native, which involves having no UI.Image only textureId. There are several issues that need to be addressed:

The Image Widget needs the UI.Image to build the RawImage to draw. This was mentioned earlier in this article in the introduction of the Flutter native solution.

Problem 2: ImageCache relies on the width and height of the UI. Image in ImageInfo to calculate the cache size and verify the cache.

Question 3: Native side texture lifecycle management

There are solutions:

Problem 1: Solve the problem by customizing the Image, exposing the imageBuilder to allow external custom Image widgets

Create a custom UI. Image for Texture as follows:

import 'dart:typed_data';
import 'dart:ui' as ui show Image;
import 'dart:ui';

class TextureImage implements ui.Image {
  int _width;
  int _height;
  int textureId;
  TextureImage(this.textureId, int width, int height)
      : _width = width,
        _height = height;

  @override
  void dispose() {
    // TODO: implement dispose
  }

  @override
  int get height => _height;

  @override
  Future<ByteData> toByteData(
      {ImageByteFormat format = ImageByteFormat.rawRgba}) {
    // TODO: implement toByteData
    throw UnimplementedError();
  }

  @override
  int get width => _width;
}
Copy the code

In this case, the TextureImage is essentially a shell that is used only to calculate the cache size. In fact, ImageCache calculates the size, so you don’t have to touch the UI.Image directly, you can just call ImageInfo, so there’s no problem. This ISSUE can be specifically seen in @Hao Anissue [1] and PR[2].

Question 3: About the timing of the release of flutter image sensed by the native side

  • Flutter is released after 2.2.0. ImageCache provides the opportunity to reuse flutter directly without modification.
  • < version 2.2.0, ImageCache needs to be modified to obtain the cache discard time, and notify Native to release the cache when it is discarded.

The modified ImageCache release is as follows (part of the code) :

typedef void HasRemovedCallback(dynamic key, dynamic value); class RemoveAwareMap<K, V> implements Map<K, V> { HasRemovedCallback hasRemovedCallback; . } //------ final RemoveAwareMap<Object, _PendingImage> _pendingImages = RemoveAwareMap<Object, _PendingImage>(); //------ void hasImageRemovedCallback(dynamic key, dynamic value) { if (key is ImageProviderExt) { waitingToBeCheckedKeys.add(key); } if (isScheduledImageStatusCheck) return; isScheduledImageStatusCheck = true; //We should do check in MicroTask to avoid if image is remove and add right away scheduleMicrotask(() { waitingToBeCheckedKeys.forEach((key) { if (! _pendingImages.containsKey(key) && ! _cache.containsKey(key) && ! _liveImages.containsKey(key)) { if (key is ImageProviderExt) { key.dispose(); }}}); waitingToBeCheckedKeys.clear(); isScheduledImageStatusCheck = false; }); }Copy the code

The overall architecture

We have elegantly combined the two solutions:

We abstracted out the PowerImageProvider and created our own ImageInfo for external (FFI) and texture. It will provide uniform load and release capabilities through calls to PowerImageLoader.

The blue solid line ImageExt is a custom Image Widget that reflects the imageBuilder in the texture mode.

The blue dotted line ImageCacheExt is an extension of ImageCache that is required only in Flutter < 2.2.0. It will provide a callback to the moment when ImageCache is released.

This time, we have also designed super scalable capabilities. In addition to supporting network images, local images, flutter resources, and Native resources, we provide channels for custom image types. Flutter can pass any combination of custom parameters to Native as long as Native registers the corresponding type loader, such as “album” scenes. Users can customize imageType for album, and Native uses its own logic to load images. With this custom channel, even image filters can be displayed refreshed using PowerImage.

In addition to the image type extension, the render type can also be customized. As mentioned in FFI above, to reduce the peak problems caused by memory copying, the user can decode the flutter side. Of course, this requires the native image library to provide the data before decoding.

Data contrast

FFI vs Texture

Behavior: Manually scroll to the bottom and then to the top in listView. Native Cache: 100MB, Flutter Cache: 100MB

There are two phenomena:

  1. Texture: 395MB fluctuates, the memory is smoother
  2. FFI: 480MB fluctuation, memory burr

Texture scheme performs better than FFI in memory level and burr:

  • Memory level: Since the Texture scheme has no memory footprint in the cache on the flutter side and only one copy exists in the native image library’s memory cache, the memory cache on the flutter side is actually 100MB less than ffI’s.
  • Burrs: Because the FFI solution cannot avoid memory copy on the flutter side, there will be a copy and then release process, which causes burrs.

Conclusion:

  1. Texture works well for everyday scenes and is preferred;
  2. FFI is more applicable

A, Flutter <= 1.23.0-18.1. In the pre version, display the image on the simulator

B. Obtain uI. Image Image data

C. Decoded on the side of flutter, data copy before decoding has little influence. (E.g. Hummer’s external decoder library)

Scroll fluency analysis

Device: Android OnePlus 8T, CPU and GPU frequency lock.

Case: GridView 4 images per row, 300 images, sliding from top to bottom, sliding from 500,1000,1500,2000,2500,5 rounds. Repeat 20 times.

For I in {1.. 20}; do flutter drive –target=test_driver/app.dart –profile; Done run data, obtain TimeLine data and analyze it.

Conclusion:

  • UI Thread time texture is the best, PowerImage is slightly better than IFImage, and FFI has a large fluctuation.
  • Raster thread duration PowerImage is better than IFImage. The Origin native is good because it resizes the image, while the other methods load the original image.

More streamlined code

The DART side code is significantly reduced thanks to the technical solution that fits into the Native Design of the Flutter. We share a lot of code with the native images.

FFI scheme makes up the deficiency of external texture and follows the design specification of native Image. It not only enables us to enjoy the unified management brought by ImageCache, but also brings more simplified code.

In the future

I’m sure many of you have noticed that the GIF section was missing. The current figure parts are in development, internal Pre Release version, return when the load is actually OneFrameImageStreamCompleter, for dynamic figure, We will replace with MultiFrameImageStreamCompleter, how to do, is only a matter of some policy, is not difficult. Incidentally, there is another option: you can decode and render the GIF data to the Flutter side before decoding, but the supported format is not as rich as native.

We want to contribute PowerImage to the community, and to do that, we provide detailed design documentation, access documentation, performance reports, and we are also refining unit testing, which is done after the code is submitted or when the code is CR.

References

[1] ISSUE: github.com/flutter/flu… [2] PR: github.com/flutter/flu…

Pay attention to [Alibaba mobile technology] wechat public number, every week 3 mobile technology practice & dry goods to give you thinking!