preface

The thing is, in order to save traffic, there is a recent study on how to encode the JPEG output from the camera into WebP on iOS for subsequent image uploads. WebP, due to its advantages, can achieve both traffic savings and good image quality, which brings many benefits, such as speeding up load time and saving server bandwidth. For a detailed introduction to WebP, check out this article: WebP Extreme Compression and ios Implementation. In the process of research, it was found that WebP decoded most of the scenes on the mobile end, that is, its small volume was used to save the traffic of download resources, but there was not much data encoded on the mobile end. Therefore, I want to record this research through this article.

WebP coding on mobile

Android

As you know, WebP was invented by Google, so since Android4.0, the Bitmap API has provided WebP codec capabilities.

// Bitmap.java
public boolean compress(CompressFormat format, int quality, OutputStream stream)

// PIC is a Bitmap object
pic.compress(Bitmap.CompressFormat.WEBP, 90, outputStream)
Copy the code

The above code is a call to WebP coding on Android.

iOS

On iOS, there is no native support for WebP, so you need to use third-party libraries. Libwebp is a library from Google that provides WebP codec capabilities across platforms. According to the research on two popular image loading libraries in iOS: SDWebImage and YYImage, both of these libraries provide plug-ins that support WebP codec capability, and according to the declared Podspec, they both rely on libwebp.

  • SDWebImage provides the WebP plug-in: SDWebImageWebPCoder

  • Plug-ins provided by YYImage

Ps: It is worth noting that YYImage’s PodSpec declares a subspec that uses the Webp. framework, which will be addressed later.

Here are two simple library calls to WebP coding

// SDWebImage
SDImageWebPCoder.shared.encodedData(with: image, format: .webP, options: [.encodeCompressionQuality: 0.9])

// YYImage
let webpEncoder = YYImageEncoder.init(type: .webP)
webpEncoder?.loopCount = 0
webpEncoder?.quality = 0.9
webpEncoder?.addImage(with: jpegData, duration: 0)
let yyWebpData = webpEncoder?.encode()
Copy the code

WebP coding based on libWebp

Libwebp is a Google support library for codec the WebP format. Libwebp repository image address. Libwebp official API documentation. According to the document, the coded API can be divided into simple API and high-level API. By looking up the source code of libwebp, it is found that the implementation of simple API is actually the encapsulation of high-level API in the library, which can be understood as the customization of high-level API will be stronger.

Simple API

Let me post some of my code implementation using the advanced API

WebPConfig config;
if (!WebPConfigPreset(&config, WEBP_PRESET_DEFAULT.90.f)) {
   CFRelease(webPImageDatRef);
   return nil;
}

config.thread_level = 1;

WebPPicture pic;
if (!WebPPictureInit(&pic)) {
   CFRelease(webPImageDatRef);
   return nil;
}

pic.use_argb = 0;
pic.width = (int)webPImageWidth;
pic.height = (int)webPImageHeight;

WebPMemoryWriter writer;
WebPMemoryWriterInit(&writer);
pic.writer = WebPMemoryWrite;
pic.custom_ptr = &writer;

int result = WebPPictureImportRGB(&pic, rgb, (int)webPBytesPerRow);
if (!result) {
   WebPMemoryWriterClear(&writer);
   CFRelease(webPImageDatRef);
   return nil;
}

WebPEncode(&config, &pic);

NSData *webPFinalData = [NSData dataWithBytes:writer.mem length:writer.size];

WebPPictureFree(&pic);
CFRelease(webPImageDatRef);
WebPMemoryWriterClear(&writer);
Copy the code

  • You can see that the final WebPEncode function actually requires a WebPConfig object and a WebPPicture object.

  • WebPConfig object

    The object is mainly configured with some parameters of the compression algorithm during coding, please refer to the official document given by Google above for detailed explanation, the WebPConfigInitInternal function in the source code config_enc.c will be called finally, This is where the object is initialized and given a default value. The thread_level setting is used to enable libwebp’s multithreading capability. The main purpose of this setting is to see if it can improve the efficiency of the multithreading.

  • WebPPicture object

    This object can be understood as an exchange structure for WebP, i.e. input, output, and some information about the image.

    • The input is the RGB object, which is a Uint8_t * that is translated by the UIImage entity passed in, which we’ll talk about later. It is converted to WebPPicture information using the WebPPictureImportRGB function. Ps: It is important to note that before Import, you need to determine the channel type of UIImage, for example, if it is RGBA, you need to call WebPPictureImportRGBA, otherwise there may be encoded image distortion. Here we call WebPPictureImportRGB because we got it from the camera
    • The output is a WebPMemoryWriter object provided by libwebp, where the MEM property represents the converted data and the size property represents the size of the converted data. We can invoke native methods to initialize an NSData from these two properties.

  • WebPEncode

    Finally, the WebPEncode function is called to code WebP. After testing, this method was found to be the most time-consuming and CPU consuming. After the encoding is finished, we need to call the corresponding function to release memory and prevent memory leakage.

On the asymmetry between the number of channels in coding and the type of channels in the camera

The following is the parameter Settings of the camera

The test device was iPhone7 system: iOS 13.1. We tried to output a picture with the maximum resolution on the device. Here is the output

Here need to pay attention to the point is that although we think camera output will be RGB, but its still have 32 bits, and has kCGImageAlphaNoneSkipLast flags. So it actually has a channel type of RGBX. This clearly does not match the previous example of calling WebPPictureImportRGB before coding (of course, you could have called WebPPictureImportRGBX). This is not studied in depth, as it can be extended to consider the possible effect of going to 8 bits on coding speed. . Therefore, in this study, the processing scheme of SDWebImage is referred to. The following is the preprocessing before coding by referring to SDWebImage scheme (the relevant source code can be found in sd_encodedWebpDataWithImage method in SDImageWebPCoder. M).

CGImageRef webPImageRef = image.CGImage;
CGBitmapInfo bitmapInfo = CGImageGetBitmapInfo(webPImageRef);
size_t webPBytesPerRow = CGImageGetBytesPerRow(webPImageRef);

size_t webPImageWidth = CGImageGetWidth(webPImageRef);
size_t webPImageHeight = CGImageGetHeight(webPImageRef);

CGDataProviderRef webPDataProviderRef = CGImageGetDataProvider(webPImageRef);
CFDataRef webPImageDatRef = CGDataProviderCopyData(webPDataProviderRef);

uint8_t *rgb = NULL;
// Convert all other cases to target color mode using vImage
vImageConverterRef convertor = NULL;
vImage_Error error = kvImageNoError;

vImage_CGImageFormat srcFormat = {
    .bitsPerComponent = (uint32_t)CGImageGetBitsPerComponent(webPImageRef),
    .bitsPerPixel = (uint32_t)CGImageGetBitsPerPixel(webPImageRef),
    .colorSpace = CGImageGetColorSpace(webPImageRef),
    .bitmapInfo = bitmapInfo
};
vImage_CGImageFormat destFormat = {
    .bitsPerComponent = 8,
    .bitsPerPixel = 24,
    .colorSpace = CGColorSpaceCreateDeviceRGB(),
    .bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault // RGB888/RGBA8888 (Non-premultiplied to works for libwebp)
    };

convertor = vImageConverter_CreateWithCGImageFormat(&srcFormat, &destFormat, NULL, kvImageNoFlags, &error);
if (error ! = kvImageNoError) {
    CFRelease(webPImageDatRef);
    return nil;
}

vImage_Buffer src = {
    .data = (uint8_t *)CFDataGetBytePtr(webPImageDatRef),
    .width = webPImageWidth,
    .height = webPImageHeight,
    .rowBytes = webPBytesPerRow
};
vImage_Buffer dest;

error = vImageBuffer_Init(&dest, webPImageHeight, webPImageWidth, destFormat.bitsPerPixel, kvImageNoFlags);
if (error ! = kvImageNoError) {
    vImageConverter_Release(convertor);
    CFRelease(webPImageDatRef);
    return nil;
}

// Convert input color mode to RGB888/RGBA8888
error = vImageConvert_AnyToAny(convertor, &src, &dest, NULL, kvImageNoFlags);
vImageConverter_Release(convertor);
if (error ! = kvImageNoError) {
    CFRelease(webPImageDatRef);
    return nil;
}

rgb = dest.data; // Converted buffer
webPBytesPerRow = dest.rowBytes; // Converted bytePerRow
CFRelease(webPImageDatRef); // Use CFData to manage bytes for free, the same code path for error handling
webPImageDatRef = CFDataCreateWithBytesNoCopy(kCFAllocatorDefault, rgb, webPBytesPerRow * webPImageHeight, kCFAllocatorDefault);
Copy the code

The Accelerate framework of iOS is used here, and the final effect is to transform the original image output in the camera to generate a 24-bit image data with the bitmapInfo flag bit as kCGImageAlphaNone. This data stream is then encoded for WebP so that it is aligned with the WebPPictureImportRGB function at the time of encoding.

Encoding time consuming

According to official data, WebP is 10 times slower to encode than JPG. Here, the above scheme is used to select a result map (the size of the picture is 2448*3264) which is pre-output in the camera, and then the 10 times of coding cycle is carried out. Finally, the average time of 10 times is calculated. Ps: The code quality is set to 90% here

Model/System The average time per second for 10 cycles Average size /bytes
iPhone5s/iOS12 2.529741811752319 488398
IPhone5s/iOS11.2.6 2.5465808153152465 488398
IPhone8P/iOS 13.4 0.965844702720642 488398
The mini4 / iOS13.1.3 1.944874358177185 488398
IPhone7 / iOS13.1 1.0465802907943726 488398
  • The JPEG encoding time of this image on the iPhone7 device is 0.11402392387390137s, again this is the encoding quality of 90%. Here we can basically verify that the coding time is about 10 times.
  • In the above mentioned devices, there are two 5s, and the average time is more than 2.5s, which can preliminatively prove that the coding time has little to do with iOS system, and more to do with the hardware performance of the device itself.
  • As for the size of the encoding artifacts, the test here found that the basic average is 488398bytes, and the size of the artifacts may only be related to the encoding algorithm configuration (which may require a lot of compatibility testing).

digression

  • This interjus was a problem I encountered during my research: using the libwebp source code directly in debug mode was much more time-consuming, up to 10 times as much as above.

    The source code for libwebp can be pulled by pod (pod ‘libwebp’), as shown in the above SDWebImage podSpec declaration. Although I did not find the relevant PodSpec file in the image repository provided above, I did find the corresponding Git repository address in the JSON file after the pull, which corresponds to Google’s Git repository. libwebp

    If we use the source code directly, do the import. In debug mode, the coding time is dozens of times higher than the above statistics. This was one of the most important problems I encountered during the time-consuming research process, which was discovered by compiling libwebp’s iosbuild.sh into the framework. At present, it is suspected that in debug mode, libwebp will call some debug logic, resulting in the overall speed is slow, so far not found in the source code. This is probably why YYImage, mentioned at the beginning, provides two subspecs.

  • Another serious problem is CPU usage.

    It is found in the test that the CPU of the device will soar to more than 90% during the coding process. The tool is used to check that the main usage is from WebPEncode function, which may cause a large loss to the performance of the mobile phone.

extension

As mentioned at the beginning of this article, Android’s built-in WebP encoding method begs the question: Android itself uses libwebp to encode and decode, is it implemented through a high-level API? Can iOS refer to its WebPConfig Settings for effect alignment?

The compress method mentioned at the beginning of this article actually calls a native methodBy looking at the Android source code, this method is the final call on/external/skia/SRC/images/SkWebpEncoder CPP

bool SkWebpEncoder::Encode(SkWStream* stream, const SkPixmap& pixmap, const Options& opts)
Copy the code

Source code address: SkWebpEncoder, here reference is Android API 28.

As you can see, Android native customization for WebPConfig is also not high.

The last

This paper mainly introduces the research scheme of using libwebp for WebP coding in iOS, and some performance research of using this scheme.