preface

Flutter already has the ability to load images. The Image component can load network images, local images, and file images. So why do we need to implement other image loading schemes? The reason for this is that the Flutter image component has some flaws in its functionality:

  • The image cache does not have the persistence capability, and the image cannot be displayed in a netless environment.
  • File images are not shared with the native environment, causing duplicate image resource files.

Therefore, in order to meet the daily development needs and optimization points, something can be done to make the image component function satisfactory. Learn about the evolution of image components from Flutter native components to external textures.

The Flutter native image component

Flutter native images support multiple loading forms:

  • Image.net Work
  • Image.file (local Image)
  • Image.asset (file Image)
  • Image.memory (byte Image)

Image Loading Process (Network image as an example)

  • Step 1: the NetworkImage loading form is NetworkImage, and the internal form is network_image.NetworkImage.
Image.network(
    String src, {
. Omit unnecessary code  }) : image = ResizeImage.resizeIfNeeded(cacheWidth, cacheHeight, NetworkImage(src, scale: scale, headers: headers)),
. Omit unnecessary code super(key: key); .const factory NetworkImage(String url, { double scale, Map<String.String> headers }) = network_image.NetworkImage;  Copy the code
  • Step 2: NetWorkImage essentially inherits from ImageProvider, as do other load forms. ImageProvider is a basic abstract class for processing images. The load class inheriting it mainly implements load methods to perform different forms of loading process.
abstract class ImageProvider<T> {
  const ImageProvider();
.  @protected
  ImageStreamCompleter load(T key, DecoderCallback decode);
.} Copy the code
  • The third step: for example, the process of obtaining picture data in network form through the network, HttpClient requests the picture to obtain the final data Uint8List through Dart layer network request.
class NetworkImage extends image_provider.ImageProvider<image_provider.NetworkImage> implements image_provider.NetworkImage {
.    @override
  ImageStreamCompleter load(image_provider.NetworkImage key, image_provider.DecoderCallback decode) {
    final StreamController<ImageChunkEvent> chunkEvents = StreamController<ImageChunkEvent>();
  return MultiFrameImageStreamCompleter(  codec: _loadAsync(key as NetworkImage, chunkEvents, decode),  chunkEvents: chunkEvents.stream,  scale: key.scale,  informationCollector: () {  return <DiagnosticsNode>[  DiagnosticsProperty<image_provider.ImageProvider>('Image provider'.this),  DiagnosticsProperty<image_provider.NetworkImage>('Image key', key),  ];  },  );  }    Future<ui.Codec> _loadAsync(  NetworkImage key,  StreamController<ImageChunkEvent> chunkEvents,  image_provider.DecoderCallback decode,  ) async {  try {  final Uri resolved = Uri.base.resolve(key.url);  final HttpClientRequest request = await _httpClient.getUrl(resolved); headers? .forEach((String name, String value) {  request.headers.add(name, value);  });  final HttpClientResponse response = await request.close();  if(response.statusCode ! = HttpStatus.ok) { PaintingBinding.instance.imageCache.evict(key);  throw image_provider.NetworkImageLoadException(statusCode: response.statusCode, uri: resolved);  }   final Uint8List bytes = await consolidateHttpClientResponseBytes(  response,  onBytesReceived: (int cumulative, int total) {  chunkEvents.add(ImageChunkEvent(  cumulativeBytesLoaded: cumulative,  expectedTotalBytes: total,  ));  },  );  if (bytes.lengthInBytes == 0)  throw Exception('NetworkImage is an empty file: $resolved');  return decode(bytes);  } finally {  chunkEvents.close();  }  } }  Copy the code
  • Step 4: After obtaining the Uint8List data of picture, it is the decoding process. Image is obtained by DecoderCallback callback method after the original data delivered to the global singleton decoder PaintingBinding. Instance. InstantiateImageCodec, The data is then processed by the engine layer C++ ‘s instantiateImageCodec to return the Image data that can be rendered by the Flutter layer.
 final ImageStreamCompleter completer = PaintingBinding.instance.imageCache.putIfAbsent(
      key,
      () => load(key, PaintingBinding.instance.instantiateImageCodec),
      onError: handleError,
    );
 Future<ui.Codec> instantiateImageCodec(Uint8List bytes, {  int cacheWidth,  int cacheHeight, {}) assert(cacheWidth == null || cacheWidth > 0);  assert(cacheHeight == null || cacheHeight > 0);  return ui.instantiateImageCodec(  bytes,  targetWidth: cacheWidth,  targetHeight: cacheHeight,  );  } String _instantiateImageCodec(Uint8List list, _Callback<Codec> callback, _ImageInfo imageInfo, int targetWidth, int targetHeight)  native 'instantiateImageCodec'; Copy the code
  • Fifth step: engine layer c++ decoder specific in codec.cc, called Skia SkCodec to do image data processing. ToDart is executed after the internal processing of the decoder to return UI_COdec to the Dart layer.
// Dart layer code
_String _instantiateImageCodec(Uint8List list, _Callback<Codec> callback, _ImageInfo imageInfo, int targetWidth, int targetHeight)
  native 'instantiateImageCodec';
/// c++ layer code
static void InstantiateImageCodec(Dart_NativeArguments args) {
 UIDartState::ThrowIfUIOperationsProhibited();  Dart_Handle callback_handle = Dart_GetNativeArgument(args, 1); . Omit some code Dart_Handle image_info_handle = Dart_GetNativeArgument(args, 2);   std::optional<ImageDecoder::ImageInfo> image_info;  // if the image information is empty, no processing is done for empty  if(! Dart_IsNull(image_info_handle)) { auto image_info_results = ConvertImageInfo(image_info_handle, args);  if (auto value =  std::get_if<ImageDecoder::ImageInfo>(&image_info_results)) {  image_info = *value;  } else if (auto error = std::get_if<std::string>(&image_info_results)) {  Dart_SetReturnValue(args, tonic::ToDart(*error));  return;  }  }   sk_sp<SkData> buffer;   {  /// process image data  Dart_Handle exception = nullptr;  tonic::Uint8List list =  tonic::DartConverter<tonic::Uint8List>::FromArguments(args, 0. exception);  if (exception) {  Dart_SetReturnValue(args, exception);  return;  }  /// copy the image data  buffer = MakeSkDataWithCopy(list.data(), list.num_elements());  }   if (image_info) {  const auto expected_size =  image_info->row_bytes * image_info->sk_info.height();  if (buffer->size() < expected_size) {  Dart_SetReturnValue(  args, ToDart("Pixel buffer size does not match image size"));  return;  }  }  /// get the image target width and height  const int targetWidth =  tonic::DartConverter<int>::FromDart(Dart_GetNativeArgument(args, 3));  const int targetHeight =  tonic::DartConverter<int>::FromDart(Dart_GetNativeArgument(args, 4));   std::unique_ptr<SkCodec> codec;  bool single_frame;  if (image_info) {  single_frame = true;  } else {  /// The SkCodec decoder is used by the underlying decoder, which is also used by Android.  codec = SkCodec::MakeFromData(buffer);  if(! codec) { Dart_SetReturnValue(args, ToDart("Could not instantiate image codec."));  return;  }  single_frame = codec->getFrameCount() == 1;  }  /// The frame number information is obtained by the decoder at the same time to determine whether the picture is a GIF  fml::RefPtr<Codec> ui_codec;   if (single_frame) {  ImageDecoder::ImageDescriptor descriptor;  descriptor.decompressed_image_info = image_info;   if (targetWidth > 0) {  descriptor.target_width = targetWidth;  }  if (targetHeight > 0) {  descriptor.target_height = targetHeight;  }  descriptor.data = std::move(buffer);   ui_codec = fml::MakeRefCounted<SingleFrameCodec>(std::move(descriptor));  } else {  ui_codec = fml::MakeRefCounted<MultiFrameCodec>(std::move(codec));  }  /// Finally, the decoder results are returned to the Dart layer  tonic::DartInvoke(callback_handle, {ToDart(ui_codec)}); } Copy the code

External texture rendering images

The Texture component of Flutter has only one entry, textureId. The few lines of code required to implement the external Texture can be confusing. In the analysis of external texture principle before a simple understanding of external texture rendering picture function.

Use the Texture component

  • The Java layer creates the Surface and textureId via the Channel plugin pluginregistry.registrar.
/// The plugin interface gets the Texture registry
TextureRegistry textureRegistry = registrar.textures();
// create the Texture instance
TextureRegistry.SurfaceTextureEntry surfaceTextureEntry = textureRegistry.createSurfaceTexture();
long textureId = surfaceTextureEntry.id();
SurfaceTexture surfaceTexture = surfaceTextureEntry.surfaceTexture(); /// get the image address String url = call.argument("url"); . Omit the image request loading process// create a Surface instance to load surfaceTexture Surface surface = new Surface(surfaceTexture); /// The canvas draws a bitmap texture map Canvas canvas = surface.lockCanvas(rect); canvas.drawBitmap(bitmap, null, rect, null); bitmap.recycle(); surface.unlockCanvasAndPost(canvas); // Dart returns textureId Map<String, Object> maps = new HashMap<>(); maps.put("textureId", textureId); result.success(maps); Copy the code
  • The Dart layer creates a MethodChannel that passes the loading image path to the Native layer.
  static const MethodChannel _channel = const MethodChannel('texture_channel');
  /// the original loading image interface
  static Future<Map> loadTexture({String url}) async {
    var args = <String.dynamic> {      "url": url,
 };  return await _channel.invokeMethod("loadTexture", args);  }  // perform the load  Map _textureResult = await TexturePlugin.loadTexture(  url: _uri.toString(),  width: url.width,  height: url.height,  );  /// return the Native generated textureId  int id = _textureResult['textureId'];  /// Instantiate the Texture component to display the image  Texture( textureId: id); Copy the code

Code parsing

The Dart layer

Texture creates the render object TextureBox. The TextureBox will draw the TextureLayer, and the TextureLayer will add a texture to the Scene via uI.sceneBuilder, and finally render the texture by calling the engine layer SceneBuilder_addTexture method.

  • Texture
class Texture extends LeafRenderObjectWidget {
  const Texture({
    Key key,
    @required this.textureId,
  }) : assert(textureId ! =null),
 super(key: key);   final int textureId;   @override  TextureBox createRenderObject(BuildContext context) => TextureBox(textureId: textureId);   @override  void updateRenderObject(BuildContext context, TextureBox renderObject) {  renderObject.textureId = textureId;  } } Copy the code
  • TextureBox
class TextureBox extends RenderBox {
  TextureBox({ @required int textureId })
    : assert(textureId ! =null),
      _textureId = textureId;
. Omit code @override  void paint(PaintingContext context, Offset offset) {  if (_textureId == null)  return;  context.addLayer(TextureLayer(  rect: Rect.fromLTWH(offset.dx, offset.dy, size.width, size.height),  textureId: _textureId,  ));  } } Copy the code
  • TextureLayer
class TextureLayer extends Layer {
  TextureLayer({
    @required this.rect,
    @required this.textureId,
    this.freeze = false. }) : assert(rect ! =null),  assert(textureId ! =null);  . @override  void addToScene(ui.SceneBuilder builder, [ Offset layerOffset = Offset.zero ]) {  final Rect shiftedRect = layerOffset == Offset.zero ? rect : rect.shift(layerOffset);  builder.addTexture(  textureId,  offset: shiftedRect.topLeft,  width: shiftedRect.width,  height: shiftedRect.height,  freeze: freeze,  );  } } Copy the code
  • SceneBuilder
class SceneBuilder extends NativeFieldWrapperClass2 {
 
 void addTexture(
   int textureId, {
   Offset offset = Offset.zero,
 double width = 0.0. double height = 0.0. bool freeze = false.{}) _addTexture(offset.dx, offset.dy, width, height, textureId, freeze);  }  /// SceneBuilderThe corresponding scene_ _addTextureCc under the SceneBuilder::addTexture method  void _addTexture(double dx, double dy, double width, double height, int textureId, bool freeze)  native 'SceneBuilder_addTexture';  Copy the code
  • scene_builder.cc
void SceneBuilder::addTexture(double dx,
                              double dy,
                              double width,
                              double height,
                              int64_t textureId,
 bool freeze) {  auto layer = std::make_unique<flutter::TextureLayer>(  SkPoint::Make(dx, dy), SkSize::Make(width, height), textureId, freeze);  AddLayer(std::move(layer)); } Copy the code
  • texture_layer.cc
/// Create a texture layer object
TextureLayer::TextureLayer(const SkPoint& offset,
                           const SkSize& size,
                           int64_t texture_id,
                           bool freeze)
 : offset_(offset), size_(size), texture_id_(texture_id), freeze_(freeze) {} /// texture object rendering method void TextureLayer::Paint(PaintContext& context) const {  TRACE_EVENT0("flutter"."TextureLayer::Paint");  /// The texture object is drawn from the GetTexture method's map.  std: :shared_ptr<Texture> texture =  context.texture_registry.GetTexture(texture_id_);  if(! texture) { TRACE_EVENT_INSTANT0("flutter"."null texture");  return;  }  texture->Paint(*context.leaf_nodes_canvas, paint_bounds(), freeze_,  context.gr_context); } Copy the code

Java layer

  • FlutterRenderer
public class FlutterRenderer implements TextureRegistry {

  public FlutterRenderer(@NonNull FlutterJNI flutterJNI) {
    this.flutterJNI = flutterJNI;
    this.flutterJNI.addIsDisplayingFlutterUiListener(flutterUiDisplayListener);
 }  @Override  public SurfaceTextureEntry createSurfaceTexture(a) {  Create SurfaceTexture / / /  final SurfaceTexture surfaceTexture = new SurfaceTexture(0);  surfaceTexture.detachFromGLContext();  final SurfaceTextureRegistryEntry entry =  new SurfaceTextureRegistryEntry(nextTextureId.getAndIncrement(), surfaceTexture);   registerTexture(entry.id(), surfaceTexture);  return entry;  }  /// Register with Native JNI layer  private void registerTexture(long textureId, @NonNull SurfaceTexture surfaceTexture) {  flutterJNI.registerTexture(textureId, surfaceTexture);  } } Copy the code
  • FlutterJNI
  public void registerTexture(long textureId, @NonNull SurfaceTexture surfaceTexture) {
    // It is necessary to execute on the main thread or an error will be reported
    ensureRunningOnMainThread();
    ensureAttachedToNative();
    nativeRegisterTexture(nativePlatformViewId, textureId, surfaceTexture);
 } Copy the code

C + + layer

  • platform_view_android_jni.cc
{
          .name = "nativeRegisterTexture".          .signature = "(JJLandroid/graphics/SurfaceTexture;) V".          .fnPtr = reinterpret_cast<void*>(&RegisterTexture),
}
 static void RegisterTexture(JNIEnv* env,  jobject jcaller,  jlong shell_holder,  jlong texture_id,  jobject surface_texture) {  ANDROID_SHELL_HOLDER->GetPlatformView()->RegisterExternalTexture(  static_cast<int64_t>(texture_id), //  fml::jni::JavaObjectWeakGlobalRef(env, surface_texture) //  ); } Copy the code
  • platform_view_android.cc
void PlatformViewAndroid::RegisterExternalTexture(
    int64_t texture_id,
    const fml::jni::JavaObjectWeakGlobalRef& surface_texture) {
  RegisterTexture(
      std::make_shared<AndroidExternalTextureGL>(texture_id, surface_texture));
} Copy the code
  • texture.cc
// Register the texture method
void TextureRegistry::RegisterTexture(std: :shared_ptr<Texture> texture) {
  if(! texture) {    return;
  }
 // The inner map saves the texture instance  mapping_[texture->Id()] = texture; } // Get the texture object and extract it from the map std: :shared_ptr<Texture> TextureRegistry::GetTexture(int64_t id) {  auto it = mapping_.find(id);  returnit ! = mapping_.end() ? it->second :nullptr; } Copy the code
  • android_external_texture_gl.cc
/// add an external texture object instance
AndroidExternalTextureGL::AndroidExternalTextureGL(
    int64_t id,
    const fml::jni::JavaObjectWeakGlobalRef& surfaceTexture)
    : Texture(id), surface_texture_(surfaceTexture), transform(SkMatrix::I()) {}
 AndroidExternalTextureGL::~AndroidExternalTextureGL() {  if (state_ == AttachmentState::attached) {  glDeleteTextures(1, &texture_name_);  } } /// draw method void AndroidExternalTextureGL::Paint(SkCanvas& canvas,  const SkRect& bounds,  bool freeze,  GrContext* context) {  if (state_ == AttachmentState::detached) {  return;  }  if (state_ == AttachmentState::uninitialized) {  glGenTextures(1, &texture_name_);  Attach(static_cast<jint>(texture_name_));  state_ = AttachmentState::attached;  }  if(! freeze && new_frame_ready_) { Update();  new_frame_ready_ = false;  }  GrGLTextureInfo textureInfo = {GL_TEXTURE_EXTERNAL_OES, texture_name_,  GL_RGBA8_OES};  GrBackendTexture backendTexture(1.1, GrMipMapped::kNo, textureInfo);  sk_sp<SkImage> image = SkImage::MakeFromTexture(  canvas.getGrContext(), backendTexture, kTopLeft_GrSurfaceOrigin,  kRGBA_8888_SkColorType, kPremul_SkAlphaType, nullptr);  if (image) {  SkAutoCanvasRestore autoRestore(&canvas, true);  canvas.translate(bounds.x(), bounds.y());  canvas.scale(bounds.width(), bounds.height());  if(! transform.isIdentity()) { SkMatrix transformAroundCenter(transform);   transformAroundCenter.preTranslate(0.5.0.5);  transformAroundCenter.postScale(1.- 1);  transformAroundCenter.postTranslate(0.5.0.5);  canvas.concat(transformAroundCenter);  }  canvas.drawImage(image, 0.0);  } } Copy the code

  • The FlutterRenderer creates SurfaceTexture and textureId.
  • UrfaceTexture and textureId are registered with the engine layer via JNI
  • Finally, in the TextureRegistry of texture.cc, map caches instance objects as key-value pairs.
  • ④ Images will need to be rendered off-screen on the SurfaceTexture.
  • The textureId created by the Java layer is passed to the Dart layer as an input parameter to the Texture component.
  • The Texture component of the Dart receives the textureId input and instantiates the lower component.
  • ⑦ Execute engine layer to create TextureLayer when SceneBuilder calls addTexture.
  • TextureRegistry map gets SurfaceTexture instances from TextureId.

Image VS Texture

The Image and Texture components are essentially implementations of the RenderBox. We know that the Flutter rendering tree is actually the RenderObject that is merged into the Layer and eventually drawn onto the page by the Flutter engine. The Image implementation draws the UI.Image content on the UI.Canvas, and the Texture is the TextureLayer added to the UI. The difference is that Image does the drawing, Texture does the adding, and the Image loading and drawing of the Texture scheme is completely done by the native platform.

🚀 Texture complete code see here 🚀

reference

  • Never expected -Flutter to be textured externally
  • The frame architecture of idle Fish Flutter has evolved