Author – Oasis team – Tsukiki

Although WebGL doesTexture compression, there is no problem of decoding time when uploading GPU, but compressed image formats such as PNG/JPG/webP are still used in daily applications. These formats need to be converted to bitmaps for rendering in WebGL, i.e. each pixel is represented using RGB or RGBA. This process is calledImage decoding. Image decoding is a very important part of rendering. If you upload an Image object directly (texImage2D) to GPU, it usually takes a long time and blocks the main thread. For example, it will cause animation playback to stall and affect user experience. So, here we do some research and test on some WebGL image decoding schemes in browsers.The first picture is synchronous decoding, the second picture is asynchronous decoding, you can see the obvious relief of animation stuck

This article focuses on testing the image. decode method and createImageBitmap method.

Image.decode

Image.decode can decode images asynchronously without blocking main thread animation and interaction. The usage method is as follows:

const img = new Image();
img.src = '... ';
img.decode().then(function() {
  document.body.appendChild(img);
});
Copy the code

createImageBitmap

ImageBitmap is a data format specifically used for Canvas and WebGL rendering. CreateImageBitmap asynchronously returns a Promise with an ImageBitmap object. CreateImageBitmap can be used within workers, and ImageBitmap can be transferred between workers. CreateImageBitmap accepts multiple data sources, and this article focuses on testing Blob and HTMLImageElement, the two objects most commonly used in rendering engines.

// Use image as the source
createImageBitmap(image).then((imageBitmap) = >{
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, imageBitmap);
})

// Use blob as source
createImageBitmap(blob).then((imageBitmap) = >{
  gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, imageBitmap);
})
Copy the code

The performance test

After introducing the basic usage of the two asynchronous decoding apis, I tested 100 different 1024 * 1024 images (randomly generated by the script) in 5 ways, comparing the decoding time and texture upload time of the images. The five ways are as follows:

  1. Use the createImageBitmap method with Image as the source. (sample)
  2. Use createImageBitmap using Blob as source. (sample)
  3. Enable the createImageBitmap method for 5 workers. (sample)
  4. Decode using image.decode. (sample)
  5. Upload textures directly using image. (sample)

After entering the above several tests, the results are obtained (up and down about 100ms) :

1. MacOS (2.6 GHz I7 Chrome 87 6 times less performance)

Method of use Decoding time (ms) Texture upload time (ms) The total time note
createImageBitmap(Image) 2625 2967 5592 Asynchronous decoding
createImageBitmap(Blob) 559 2180 2739 Asynchronous decoding
createImageBitmap(Blob) + worker 210 2000 2210 Asynchronous + multithreaded decoding
Image direct upload 3020 3020 Synchronized decoding
Upload the image. After the decode 210 4978 5188 Asynchronous decoding

2. Android U4 (Mi 10 Pro U4 3.21.0.172)

Method of use Decoding time (ms) Texture upload time (ms) The total time note
createImageBitmap(Image) 1540 878 2418 Asynchronous decoding
createImageBitmap(Blob) 1096 129 1225 Asynchronous decoding
createImageBitmap(Blob) + worker 715 142 857 Asynchronous + multithreaded decoding
Image direct upload 905 905 Synchronized decoding
Upload the image. After the decode Decode error, The source image cannot be decoded. Asynchronous decoding

3. Android Chrome (Mi 10 Pro Android Chrome 87)

Method of use Decoding time (ms) Texture upload time (ms) The total time note
createImageBitmap(Image) 522 504 1026 Asynchronous decoding
createImageBitmap(Blob) 310 135 445 Asynchronous decoding
createImageBitmap(Blob) + worker 249 145 394 Asynchronous + multithreaded decoding
Image direct upload 510 510 Synchronized decoding
Upload the image. After the decode Decode error, The source image cannot be decoded. Asynchronous decoding

4. IOS safari (iPhone7 iOS 14.2)

Method of use Decoding time (ms) Texture upload time (ms) The total time note
createImageBitmap(Image) Does not support
createImageBitmap(Blob) Does not support
createImageBitmap(Blob) + worker Does not support
Image direct upload 1076 1076 Synchronized decoding
Upload the image. After the decode 2076 300 2376 Asynchronous decoding

conclusion

Through the above tests, the following conclusions can be drawn:

  1. Recommended for Android and Mac ChromecreateImageBitmapThe data source must be usedBlob, decoding can improve performance by about 10% :
    1. If the data source usesBlob, no decoding time; If the data source usesImageFirst, it takes a long time to create bitmap, and second, there is still decoding time in performance (it is expected that there should not be decoding time, which is a Bug in Chrome and has been mentioned to ChromiumissueChrome has officially confirmed the problem.
    2. Called in the workercreateImageBitmapMultithreading capabilities can be utilized to further improve performance by around 15%. Because worker threads are not particularly stable, whether to enable worker decoding is decided by user configuration. Users decide whether to use worker decoding according to the current CPU load, the number of required decoding and business scenarios.
  2. Do not use any asynchronous decoding schemes for iOS:
    1. Does not supportcreateImageBitmap;
    2. useImage.decodeThe total time is twice that of synchronous decoding;

According to the above test results and the derived conclusion, the best image request decoding scheme adopted in WebGL is as follows:

The above solution will be applied to OASIS-Engine, welcome to discuss it in PR.