Major players now provide the ability to record GIFs. A GIF is a sequence of frames of images displayed in sequence, forming an animation. So GIF generation can be divided into two steps, first to obtain a group of consecutive images, the second step is to synthesize this group of images into a GIF file. There are many open source utility classes available on the web for GIF file synthesis. We’ll focus today on how to get a set of screenshots from the player. Without further ado, let’s take a look at the video playback process.

1.1. Image frame data is obtained from the image frame pool after decoding

As can be seen from the above flow chart, the screenshot only needs to obtain the decoded image frame data, that is, take out the specified frame image from the image frame pool. When we play with FFmpeg soft decoding, the image frame pool is in our own code, so we can take any frame. But when we use the system MediaCodec interface to decode and play the video, the video decoding is done by the system MediaCodec module. If we want to take out the image frame data from MediaCodec, we have to study the Interface of MediaCodec.

MediaCodec’s workflow is shown in the figure above. The MediaCodec class is part of Android’s underlying multimedia framework and is used to access the underlying codec components, typically along with MediaExtractor, MediaSync, Image, Surface, and AudioTrack classes.

In simple terms, the function of a Codec is to process raw input data into usable output data. It uses a set of input buffers and a set of output buffers to process data asynchronously. A simple data processing process can be roughly divided into three steps:

  1. fromMediaCodecTo get ainput bufferAnd fill in the raw data you unpacked from the data sourceinput buffer;
  2. Fill it with raw datainput bufferSent to theMediaCodec,MediaCodecThe raw data is decoded into image frame data, and the image frame data is placed intooutput buffer;
  3. fromMediaCodecGets a frame data with an available imageoutput bufferAnd then you can takeoutput bufferOutput to thesurfaceorbitmapCan be rendered to the screen or saved in an image file.
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, mWidth, mHeight); String mime = format.getString(MediaFormat.KEY_MIME); / / create a video decoder, configuration decoder MediaCodec mVideoDecoder = MediaCodec. CreateDecoderByType (mime); mVideoDecoder.configure(format, surface, null, 0); / / 1, to obtain the input buffer, the original video data packets into the input buffer int inputBufferIndex = mVideoDecoder. DequeueInputBuffer (50000); ByteBuffer buffer = mVideoDecoder.getInputBuffer(inputBufferIndex); // 2. Send the input buffer with the original video data to MediaCodec for decoding. Decoding data will be placed in the output buffer mVideoDecoder. QueueInputBuffer (mVideoBufferIndex, 0, the size, presentationTime, 0). // Get the output buffer with video frame data. The release of the output buffer will be data to render to the decoder in the configuration Settings on the surface of the int outputBufferIndex = mVideoDecoder. DequeueOutputBuffer (info, 10000); mVideoDecoder.releaseOutputBuffer(outputBufferIndex, render);Copy the code

Here is the basic flow for playing video with MediaCodec. Our goal is to get a video picture during this playback. As you can see from the above procedure, the output buffer method dequeueOutputBuffer returns not a buffer object but a buffer sequence number. Just pass the outputBufferIndex to MediaCodec when rendering, and MediaCodec will render the corresponding index into the surface where the initial configuration is set. In order to achieve a screenshot, we need to get the output buffer data. We need a method to get the output buffer through outputBufferIndex. Look at the MediaCodec interface actually has such a method, details are as follows:

/**
 * Returns a read-only ByteBuffer for a dequeued output buffer
 * index. The position and limit of the returned buffer are set
 * to the valid output data.
 *
 * After calling this method, any ByteBuffer or Image object
 * previously returned for the same output index MUST no longer
 * be used.
 *
 * @param index The index of a client-owned output buffer previously
 *              returned from a call to {@link #dequeueOutputBuffer},
 *              or received via an onOutputBufferAvailable callback.
 *
 * @return the output buffer, or null if the index is not a dequeued
 * output buffer, or the codec is configured with an output surface.
 *
 * @throws IllegalStateException if not in the Executing state.
 * @throws MediaCodec.CodecException upon codec error.
 */
@Nullable
public ByteBuffer getOutputBuffer(int index) {
    ByteBuffer newBuffer = getBuffer(false /* input */, index);
    synchronized(mBufferLock) {
        invalidateByteBuffer(mCachedOutputBuffers, index);
        mDequeuedOutputBuffers.put(index, newBuffer);
    }
    return newBuffer;
}
Copy the code

Return the output buffer, or null if the index is not a dequeued output buffer, or the codec is configured with an output surface. That is, if we set the surface when we initialize MediaCodec, then we get null output buffers through this interface. The reason is that when we set surface as the data output object for MediaCodec, the output buffer directly used the native buffer without mapping or copying the data into ByteBuffer, which would make the image rendering more efficient. The main function of the player is still to play, so it is necessary to set the surface. Then how can we achieve the function of screenshot under the condition that we can’t get the ByteBuffer that holds the decoded video frame data?

1.2. Obtain image frame data from View after rendering

At this time, we changed our thinking. Since it is not convenient to obtain the image frame data after hard decoding (Scheme 1), can we wait until the image frame data is rendered to the View and then obtain the data from the View (Scheme 2)?

SurfaceVIew
MediaCodec
SurfaceVIew
Why can’t I get images from SurfaceView?
SurfaceView
TextureView
MediaCodec
TextureView 

/** * <p>Returns a {@link android.graphics.Bitmap} representation of the content * of the associated surface texture. If  the surface texture is not available, * this method returns null.</p> * * <p>The bitmap returned by this method uses the {@link Bitmap.Config#ARGB_8888}
 * pixel format.</p>
 *
 * <p><strong>Do not</strong> invoke this method from a drawing method
 * ({@link #onDraw(android.graphics.Canvas)} for instance).</p>
 *
 * <p>If an error occurs during the copy, an empty bitmap will be returned.</p>
 *
 * @param width The width of the bitmap to create
 * @param height The height of the bitmap to create
 *
 * @return A valid {@link Bitmap.Config#ARGB_8888} bitmap, or null if the surface* texture is not available or width is &lt; = 0 or height is &lt; = 0 * * @see#isAvailable()
 * @see #getBitmap(android.graphics.Bitmap)
 * @see #getBitmap()
 */
public Bitmap getBitmap(int width, int height) {
    if (isAvailable() && width > 0 && height > 0) {
        return getBitmap(Bitmap.createBitmap(getResources().getDisplayMetrics(),
                width, height, Bitmap.Config.ARGB_8888));
    }
    return null;
}
Copy the code

So far a small step has been taken to get an image from the player. Now let’s look at how to get a set of images.

1.3 Obtaining a set of continuous images

If a single image is acquired successfully, is it difficult to obtain multiple images? Because the way we get the image is we wait until the image is rendered in the View and then we get it from the View. So the question is, if you want to generate a GIF with a playback duration of 5 seconds, is it really necessary to collect the set of images for 5 seconds and make all the data in the 5s render once on the View? This kind of experience is definitely not allowed. For this reason, we use a function similar to double speed playback to quickly render the image data within 5S on the View, so as to quickly obtain the image data of 5s class.

if(isScreenShot) {// GIF images don't need all frame data, define 5 frames per second, then render a frame every 200ms can render = (info.presentationtimeus - lastFrameTimeMs) > 200; }elseRender = mediaplayer.get_sync_info (info.presentationtimeus)! = 0; }if (render) {
    lastFrameTimeMs = info.presentationTimeUs;
}

mVideoDecoder.releaseOutputBuffer(mVideoBufferIndex, render);
Copy the code

As shown in the code above, the image rendering is not synchronized with the audio in screenshot mode, which enables fast image rendering. In addition, GIF images only have a few images per second, defined here as 5, so only need to select 5 images from the video source of 30 frames per second data to render. In this way, we can quickly obtain 5s image data.

After obtaining the required image data, all that is left is to synthesize the GIF file. This will fulfill the need to generate GIFs while playing video with MediaCodec hard decoding.