In the group, someone mentioned this implementation: I have a video, I want to replace some content in the video with my own image, how to use OpenGL to implement this?

First of all, it should be made clear that video is composed of frame by frame, which makes use of the visual persistence effect of human eyes. Only when enough frames are played in one second can the image feel continuous.

And want to replace the content of the video, that is, to replace the content of each frame of the image, generally speaking, this should belong to the video post processing, with professional AE (Adobe After Effects) software to deal with it will be better.

Processing way of thinking

If you use OpenGL, there’s an idea:

First, each frame of image content is decoded through MediaCodec, and then the current frame of image is processed by OpenGL. A transparent mask layer is added to the original image. The requirement of the mask layer is that the content area to be replaced is non-transparent, while the other areas are transparent. The final result is a frame of replaced content image, and then the processed frame of image is encoded and re-encoded into the new video content.

The process of decoding -> processing -> encoding is repeated until the content of each frame of the video is processed, and the content replacement of the video is realized.

Of course, this is just an idea, the difficulty is how to find the appropriate mask layer, if the video image content is changing, the content to be replaced is not fixed, then the mask layer requirements are higher, each frame processing must have a suitable mask.

The following will be for the video of a frame of image content processing, how to replace a frame of image content.

Direct effect

The effect is as follows:

The result of the code is that the top left content is replaced by the top right content, resulting in the bottom right image.

The preparatory work

Can’t do design development is not good code farmers

It’s time to pull out my Big Gem Sketch:

Prepare a sheet of alternate content:

Then cut another image of the same size and replace the middle circular image with the desired one. Set the opacity of other surrounding contents to 0.

The next thing to do is to fuse the two images and introduce the replacement content based on shaders and color blends, respectively.

These two schemes have one thing in common, that is, they want to cover the picture with mask on the original picture. The difference is how to handle the coverage between the two pictures, and transparency is a good starting point.

Use shaders to replace

In OpenGL rendering pipeline, will first build a graph, and then to carry on the rasterizer, rasterizer to each fragment shader, after in the coloring process can be treated according to the need to piece of RMB, including abandon some fragment, such as simple said after is the first tangible in OpenGL non-ferrous, and in the process of shape can make some small operation ~ ~

Processing the slice is our slice shader script.

precision mediump float;
varying vec2 vTextureCoord; // Receives arguments from the vertex shader
uniform sampler2D sTexture;// Texture content data
void main() { 
   vec4 bcolor = texture2D(sTexture, vTextureCoord);// Sample the color value from the texture for this pixel
   if(bcolor.a<0.6) {
   		discard;
   } else {
      gl_FragColor=bcolor;
}}
Copy the code

Except for the content to be replaced, all other parts of our mask map are transparent. If the transparency value sampled is less than the threshold value, the element is discarded and will not be displayed directly.

The opacity that meets the requirement is displayed and overlays the original color when it is finally mapped to the viewport.

In this way, content replacement is achieved.

Use color mix to replace

Using color blending is not as straightforward as using shaders, either discarding some tiles or overwriting them.

It is based on certain calculation rules, to calculate the fusion between two colors.

To use color blending in OpenGL, set the blending factor properly.

        glEnable(GL_BLEND);
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)
        / / to draw
        glDisable(GL_BLEND)
Copy the code

The blending factor is set so that if the mask is transparent, use the color of the masked image, and if not, use the color of the mask image, so that some slices are not discarded directly.

Code implementation

In the specific code implementation, the USE of EGL to achieve off-screen rendering.

In the non-main thread, initialize the EGL environment, prepare the necessary work for drawing, perform the drawing, and finally read the drawing result through glReadPixels.

        Observable.fromCallable {
        Initialize the EGL environment
            return@fromCallable initEgl()
        }.map {
        // Set various matrices
            prepare(width, height)
            return@map it
        }.map {
        // Perform the drawing
            replaceContent(isBlend)
            return@map it
        }.map {
        // Read pixels
            val result = readPixel(width, height)
            it.destroy()
            return@map result
        }.subscribeOn(Schedulers.computation())
                .observeOn(AndroidSchedulers.mainThread())
                .subscribe({
                // Set the effect
                    mResultImage.setImageBitmap(it)
                }, {
                    showToast("replace failed")})Copy the code

The specific drawing process is relatively simple. If the color mixing is used, the color mixing is performed. Otherwise, the shader is used to draw, which also reflects the idea of directly covering the mask image on the original image.

 private fun replaceContent(isBlend: Boolean) {
        glClearColor(1f.1f.1f.1f) glClear(GL_COLOR_BUFFER_BIT or GL_DEPTH_BUFFER_BIT) mOriginImage? .drawSelf(mOriginTextureId)if(isBlend) { glEnable(GL_BLEND); glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) mReplaceImage? .drawSelf(mReplaceTextureId) glDisable(GL_BLEND) }else{ mAlphaTextureRect? .drawSelf(mReplaceTextureId) } }Copy the code

Note that the content read by glReadPixels is upside down and needs to be flipped upside down.

   for (i in 0 until height) {
            for (j in 0 until width) {
                pixelMirroredArray[(height - i - 1) * width + j] = pixelArray[i * width + j]
            }
        }
Copy the code

Specific implementation can refer to my Github project for a wave of Star.

Github.com/glumes/Andr…

Subsequent idea

For video content replacement, only a frame of image content replacement is presented here, and it is based on transparency.

When some Hollywood movie scenes are shot, a solid color curtain will be given at the back, and then the curtain content will be replaced with the background in the post-processing. This replacement can be realized by comparing the color range with shaders.

Of course, the gameplay is much richer with image recognition instead of content.

The last

Technical exchange welcome to pay attention to the public number audio and video development.