directory

  1. What is OES? What is SurfaceTetxure?
  2. Real-time filter flow
  3. The specific practices
  4. Problems encountered
  5. harvest

I. Introduction to basic knowledge

** What are the external textures? 支那

    public static native void glBindTexture(
        int target,
        int texture
    );
Copy the code

We have been using GL_TEXTURE_2D as the target in previous articles, which is a texture image that is rendered directly. The OES texture gles11ext.gl_TEXture_external_OES is used to render the video or Camera preview data after filtering and special effects instead of directly rendering to the screen

What is SurfaceTetxure?

In the previous section, we introduced Camera preview using SurfaceHolder.

The SurfaceTexture class was introduced in Android 3.0. Instead of rendering the image stream directly, the SurfaceTexture class transforms the image stream into a GL external texture for secondary processing of the image stream data (Camera filters, effects, etc.).

SurfaceTexture retrieves the frame data from the image stream (Camera preview, video decoding), calls updateTexImage(), updates the GL texture object corresponding to the SurfaceTexture based on the most recent image in the content stream, and then operates on it as if it were a normal GL texture.

Second, the process

After the Camera collects the data, it does not display it directly on the screen, but filters the image first, that is, rendering it on an external texture first and displaying it on the screen after processing.

OpenGL texture rendering basic flow.

  1. Create a Camera
  2. Create and load GLSL, create compile link program, get location
  3. Create external textures, bind textures, and set parameters
  4. Generates a SurfaceTexture based on the created texture ID
  5. Set the Camera to preview via SurfaceTeture instead of SurfaceHolder
  6. The Camera outputs the preview data to the SurfaceTexture, pushing a frame of preview data to the external texture
  7. Add a filter to this texture
  8. The processed data is plotted by OpenGL ES

Three, practice: Camera preview add real-time filter (original picture, black and white, warm and cold colors)

1. Create a Camera

private void initCamera(int cameraId) { curCameraId = cameraId; mCamera = Camera.open(curCameraId); Log.d(TAG, "initCamera: Camera Open "); Camera.Parameters parameters = mCamera.getParameters(); Camera.Size closelyPreSize = CameraUtil.getCloselyPreSize(true, SystemUtils.getDisplayWidth(), SystemUtils.getDisplayHeight(), parameters.getSupportedPreviewSizes()); Log.i(TAG, "initCamera: closelyPreSizeW=" + closelyPreSize.width + " closelyPreSizeH=" + closelyPreSize.height); parameters.setPreviewSize(closelyPreSize.width, closelyPreSize.height); MPreviewWidth = CloselyPresize.height; mPreviewHeight = closelyPreSize.width; mCamera.setParameters(parameters); }Copy the code

2. Load GLSL, create program, obtain location

omitCopy the code

3. Create external textures

private int genOesTextureId() { int[] textureObjectId = new int[1]; GLES20.glGenTextures(1, textureObjectId, 0); Gles20. glBindTexture(gles11ext.gl_texture_external_oes, textureObjectId[0]); // Set zoom in and out. Gles20. glTexParameterf(gles11ext.gl_TEXTURE_external_oes, gl10.gl_TEXture_min_filter, gles11ext.gl_texture_external_oes, gl10.gl_TEXture_min_filter, GL10.GL_NEAREST_MIPMAP_LINEAR); GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR); GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE); GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_WRAP_T, GL10.GL_CLAMP_TO_EDGE); return textureObjectId[0]; } int[] textureObjectId = new int[1]; GLES20.glGenTextures(1, textureObjectId, 0); Bind the external texture gles20.glbindTexture (gles11ext.gl_TEXture_external_oes, textureObjectId[0]); Set zoom in and out. Gles20. glTexParameterf(gles11ext.gl_TEXTURE_external_oes, gl10.gl_TEXture_min_filter, gles11ext.gl_texture_external_oes, gl10.gl_TEXture_min_filter, GL10.GL_NEAREST_MIPMAP_LINEAR); GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR); GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE); GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GL10.GL_TEXTURE_WRAP_T, GL10.GL_CLAMP_TO_EDGE);Copy the code

4. Generate a SurfaceTexture based on the created texture ID

mSurfaceTexture = new SurfaceTexture(mTextureId);
Copy the code

5. Set the Camera to preview via SurfaceTeture instead of SurfaceHolder

mCamera.setPreviewTexture(mSurfaceTexture);
Copy the code

6. The Camera outputs the preview data to the SurfaceTexture, pushing a frame of preview data to the external texture

Public void onDrawFrame(GL10 gl) {texture refresh if (mSurfaceTexture! = null) { mSurfaceTexture.updateTexImage(); }}Copy the code

7. OpenGL ES allows you to manipulate this texture and apply a filter

private void onBindFilter() { GLES20.glUniform1i(mUFilterIndex, mIndex); Switch (mIndex){case 0: // break; Case 1: / / black and white filter GLES20 glUniform3fv (uColor, 1, ImageBgData. GRAY_FILTER_COLOR_DATA, 0). break; Case 2: / / warm color filter GLES20 glUniform3fv (uColor, 1, ImageBgData. WARM_FILTER_COLOR_DATA, 0). break; Case 3: / / cool color filter GLES20 glUniform3fv (uColor, 1, ImageBgData. COOL_FILTER_COLOR_DATA, 0). break; }}Copy the code

8. The processed data is drawn by OpenGL ES

GLES20.glActiveTexture(GLES20.GL_TEXTURE0 + getTextureType()); GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, getTextureId()); GLES20.glUniform1i(mUTexture, getTextureType()); / / set point data GLES20. GlEnableVertexAttribArray (mAPosition); GLES20.glVertexAttribPointer( mAPosition, 2, GLES20.GL_FLOAT, false, 0, mVerBuffer); // GLES20.glEnableVertexAttribArray(mACoord); GLES20.glVertexAttribPointer( mACoord, 2, GLES20.GL_FLOAT, false, 0, mTextureCoordinate); Gles20.gldrawarrays (gles20.gl_triangle_strip, 0, 4); GLES20.glDisableVertexAttribArray(mAPosition); GLES20.glDisableVertexAttribArray(mACoord);Copy the code

The code is hosted on Github

4. Problems encountered

  1. How to set SurfaceTexure to Camera instead of SurfaceHolde
  2. The image preview is enlarged, and the aspect ratio of the view is reversed before the orthogonal projection matrix transformation, because the width of the camera is greater than the height
  3. Uniform VEC3 u_Color is defined in the slice shader. GlVertexAttrib3fv is only used for vertex shaders. If color is defined in a vertex shader, And then be transmitted by varying can also) correction for GLES20. GlUniform3fv (uColor, 1, ImageBgData. COOL_FILTER_COLOR_DATA, 0). Can switch the preview normally

Five, the data

Android Video Editor (4) Add different filter effects to video through OpenGL [Android Camera uses OpenGL ES 2.0 and GLSurfaceView to reprocess the preview in real time (black and white filter)]

Six, harvest

  1. Understand OES concepts
  2. Understand the concepts of FBO, VBO and PBO
  3. Familiarize yourself with the process by practicing real-time filters

Thank you for reading

Next we begin to enter the particle system, gradually realize netease cloud whale cloud effects. Welcome to pay attention to the public account “audio and video development journey”, learn and grow together.

Welcome to communicate