Android implementation of arbitrary resolution video coding thinking and implementation

HardwareVideoCodec is an efficient Android audio and video coding library that supports both soft and hard coding. You can easily encode video at any resolution, regardless of camera preview size. It’s all so simple. At present, we have iterated several stable versions, welcome to study and use, if there are bugs or suggestions, welcome to Issue.

Anyone who knows anything about Android camera development knows that each device only supports a fixed set of resolutions, and each device is different. For example, some phones support 540×960, while others don’t. This forces us to get a list of supported resolutions every time we use the Android camera, and then pick a matching resolution. However, as mentioned above, each device supports different resolutions, so what if the resolution we want is not supported on a particular device? Many people will have to compromise and choose a universally supported resolution that supports the vast majority of devices. This minimizes error rates, but theoretically there will still be unsupported devices. For some people with obsessive-compulsive disorder, it is unbearable! The drudgery of camera compatibility, I believe, afflicts many fellow shoes, including me. So recently I’ve been thinking, is there a way to support any resolution camera preview and video encoding once and for all? Let’s analyze it first. The above questions can be broken down into two

1. How to select the most appropriate camera resolution according to the given target resolution 2. How to efficiently cut the camera output screen size to the target size

The best resolution option

Some of you might know that, so just pick the resolution that’s closest to it. So the question I’m going to ask here is, how do we define this closest? I believe that many of my classmates also went to the Internet to find some posts, I also did the same at the beginning, can find basically is to sort the camera resolution, and then choose a size closest to the target resolution. This method is also used in our company’s online projects. It is a pity that some models have made mistakes in selection. Why is that? What’s the problem? Has it occurred to you that resolution is a combination of width and height? For example, we need to output the resolution of 540×960, but a certain machine does not support this resolution, only the two resolutions of 720×960 and 540×1080 are supported. If we use sorting method to sort these two resolutions, should we prioritize the width or the height? Obviously, the width should be prioritized and 540×960 should be selected. In the same way, is it possible to prioritize height to get the best resolution? I can’t say for sure, but as responsible programmers, we need to consider that. Having said that, the bottom line is that sorting multiple keywords for resolution is not a guarantee that we will get the best resolution, because we don’t know which is the highest priority, width or height. Here, I’m going to give you the answer. First, the definition of optimal resolution has the following two points:

1. Width is greater than or equal to the width of the target resolution
2. Resolution (number of pixels) greater than or equal to the target resolution

It is easy to overlook the second point, which is the key to solving the above problems. Based on this point, we do not need to worry about which one comes first, width or height. Post the code directly below.

fun setPreviewSize(cameraParam: Camera.Parameters, context: CodecContext) {
    val supportSizes = cameraParam.supportedPreviewSizes
    var bestWidth = 0
    var bestHeight = 0
    for (size in supportSizes) {
        ifWidth >= context.video.width// Width >= context.video.width// width >= context.video.width// width >= context.video Size, height < bestWidth * bestHeight | | 0 = = bestWidth * bestHeight)) {/ / choose the resolution of the pixels in the least bestWidth = size. Width bestHeight = size.height } } context.cameraSize.width = bestWidth context.cameraSize.height = bestHeight cameraParam.setPreviewSize(context.cameraSize.width, context.cameraSize.height) }Copy the code

Is it very simple, a cycle comparison can come up with the best resolution, much simpler than the sorting method of the long horizon. Context. CameraSize is the final resolution selected, and the implementation logic is similar to what we analyzed above, so I won’t comment on it. Through the above analysis, although we can correctly choose the best resolution, it does not mean that the best resolution is our target resolution. In unusual cases, what we choose is a size larger than the target resolution. At this time, we need to crop the picture, which leads to the following problem.

How to efficiently cut the screen

#####1. Direct cropping this method needs to get every frame data in the camera callback, and then according to the invalid image format (YUV420P, YUV420SP…) Different algorithms are used for screen clipping. After clipping, the screen is rendered to the SurfaceView. Not to say how to cut performance, it needs to use different cutting algorithms according to different formats believe that a number of children’s shoes have been stumped. In practice, however, it is almost impossible to write your own clipping algorithm because it consumes too much resources. Of course, you can use Google’s open source libyuv, which is currently the best third-party library for cropping and formatting. It has been tested to achieve 720p cropping and formatting in 6ms with no problem. However, it would be naive to think that the solution is so simple. When you use libyuv to successfully scale YUV420SP to the desired size and try to draw it onto the SurfaceView, you will find that Canvas only supports Bitmap drawing, which means you will need to convert YUV420SP to Bitmap. However, the performance cost of image conversion is so high that it is almost impossible to use. Ha ha, don’t toss again, give up! #####2. Use hardware At this point you wonder if it would be nice to use hardware like video coding. Yeah, OpenGL ES you deserve it. For those of you who are familiar with OpenGL, texture positioning is done through vertices. It contains a set of input texture vertices and a set of target texture vertices. The former is used to indicate which part of the input texture is drawn to the target texture (or window), and the latter indicates where the input texture is drawn to the target texture (or window). It may be a little complicated, and at the beginning IT was hard for me to understand, but then I realized that it was the same as the code below.

Canvas.drawBitmap(Bitmap bitmap, Rect src, Rect dst, Paint paint)
Copy the code

SRC is the input texture vertex above and DST is the target texture vertex, which means that OpenGL supports clipping the input texture. Note that the range of the former is in the first quadrant, i.e., x,y∈[0, 1], while the latter is x,y∈[-1, 1]. All that said, what’s the point? Remember, Android has native support for converting camera images to SurfaceTexture (OpenGL supported textures). SurfaceTexture (SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture, SurfaceTexture The whole process takes up no CPU resources. For OpenGL ES, see the Android Audio and Video coding mess “chapter 2, using TextureView to render Camera images. This is directly pasted to the vertex calculation method.

private fun calculateBestLocation(previewWidth: Int, previewHeight: Int, videoWidth: Int, videoHeight: Int,
                                  location: FloatArray, textureLocation: FloatArray) {
    val previewScale = previewWidth / previewHeight.toFloat()
    val videoScale = videoWidth / videoHeight.toFloat()
    var destPreviewWidth = previewWidth
    var destPreviewHeight = previewHeight
    /**
     * if(previewScale > videoScale) previewHeight does not change, previewHeight is used to calculate previewWidth *elsePreviewWidth is not changed. PreviewHeight */ is calculated using previewWidthif (previewScale > videoScale) {
        destPreviewWidth = (previewHeight * videoScale).toInt()
        if (0 != destPreviewWidth % 2) ++destPreviewWidth
    } else {
        destPreviewHeight = (previewWidth / videoScale).toInt()
        if (0 != destPreviewHeight % 2) ++destPreviewHeight
    }
    val left = (previewWidth - destPreviewWidth) / 2f / previewWidth.toFloat()
    val right = 1f - left
    val bottom = (previewHeight - destPreviewHeight) / 2f / previewHeight.toFloat()
    val top = 1 - bottom
    System.arraycopy(floatArrayOf(-1f, -1f, //LEFT,BOTTOM
            1f, -1f, //RIGHT,BOTTOM
            -1f, 1f, //LEFT,TOP
            1f, 1f//RIGHT,TOP
    ), 0, location, 0, 8)
    System.arraycopy(floatArrayOf(left, bottom, //LEFT,BOTTOM
            right, bottom, //RIGHT,BOTTOM
            left, top, //LEFT,TOP
            right, top//RIGHT,TOP
    ), 0, textureLocation, 0, 8)
}
Copy the code

PreviewWidth and previewHeight are the camera preview resolution, which is the best resolution mentioned above. VideoWidth and videoHeight are the output resolution, which is the target resolution. Location is the target texture vertex and textureLocation is the input texture vertex. Note that in portrait, previewWidth and previewHeight are reversed, so you need to pass them in this way.

calculateBestLocation(previewHeight, previewWidth, videoWidth, videoHeight, location, textureLocation)
Copy the code

The above is about Android arbitrary resolution video coding thinking and implementation, it seems a bit complicated. By the time these two parts of logic are realized, they may be beaten to death by the product. Is there a third-party library that can preview and encode video at any resolution?

HardwareVideoCodec is one such efficient Android audio and video coding library. You can easily encode video at any resolution, regardless of camera preview size. And the interface is easy to use, you only need a few lines of code to achieve such a lofty function, but also give more than 20 special effects, and RTMP push module, whether you need short video or live demand, everything is so simple. At present has iterated several stable versions, hurry up to check learning bar! If you have bugs or suggestions, please welcome Issue.

---