Original link: glumes.com/post/ffmpeg…

This is the third article in a series on the love-hate relationship between FFmpeg and MediaCodec.

I wrote FFmpeg calling MediaCodec for hard decoding.

  • FFmpeg calls Android MediaCodec for hard decoding

In addition, FFmpeg compilation script is also given, easy to solve the compilation problem.

  • Cliche -FFmpeg compilation problem is easy to fix

MediaCodec’s decoding capabilities are well known for not only decoding YUV data, but also directly onto the Surface.

In the short video world, MediaCodec’s ability to decode to the Surface is more commonly used, so that the image can be transferred to OES textures for subsequent rendering operations.

The FFmpeg call to MediaCodec for hard decoding just decodes the Buffer data without using the ability to decode it to the Surface.

After looking at some more data, FFmpeg calls to MediaCodec can be decoded to the Surface.

The specific reference is the content of this email:

Mplayerhq. Hu/pipermail/f…

This capability is described in detail here, and highlights the screenshots:

The picture content is very detailed, according to the steps of practice.

Code practice

If you are familiar with the flow of FFmpeg calling MediaCodec to decode Buffer data, then decoding to Surface is just a slight change in flow.

The first step is to prepare the Surface object, build the Surface object on the Java top layer and pass it to the Native layer through NDK, and then pass it to the jobject object.

If you’re not familiar with the NDK, check out this free course I recorded on MOOCs:

Android NDK free video online learning!!

Next are two new function methods:

The av_mediacodec_alloc_context and AV_Mediacodec_default_init methods associate Surface with AVMediaCodecContext and AVCodecContext.

So AVCodecContext holds AVMediaCodecContext, AVMediaCodecContext holds the Surface.

As for why to associate, because in FFmpeg source code, depending on whether the Surface is nullPTR, MediaCodec initialization and decoded data are processed differently.

Interested can read this piece of source code, the content is not easy to understand.

After decoding, FFmpeg will also return AVFrame data, but its data field is no longer the Buffer contents.

The AVFrame format is no longer NV12 (NV12 if decoding Buffer data), but a custom AV_PIX_FMT_MEDIACODEC, which represents the Surface mode to go.


if (s->surface) {

// If surface is not null:

// Use mediacodec_wrap_hw_buffer to process the data

  if ((ret = mediacodec_wrap_hw_buffer(avctx, s, index, &info, frame)) < 0) {

  av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec buffer\n");

       returnret; }}Copy the code

Data processing in Surface mode is mediacodec_WRap_hw_buffer, and Buffer mode is mediacodec_WRap_SW_buffer.

Meanwhile, the actual decoded data is stored in the AVFrame->data[3] field, which is an old employee.

In general, when decoding non-buffer data, special content will be saved to data[3], such as the hard solution on Window. Part of the source code is as follows:


static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx,

                                  MediaCodecDecContext *s,

                                  ssize_t index,

                                  FFAMediaCodecBufferInfo *info,

                                  AVFrame *frame)

// omit some source code

// Create an AVMediaCodecBuffer object

buffer = av_mallocz(sizeof(AVMediaCodecBuffer));

frame->buf[0] = av_buffer_create(NULL.0,

                       mediacodec_buffer_release,

                       buffer,

                       AV_BUFFER_FLAG_READONLY);

// Buffer attributes are assigned

buffer->ctx = s;

buffer->serial = atomic_load(&s->serial);

if (s->delay_flush)

    ff_mediacodec_dec_ref(s);

// index represents the index of mediacodec's buffer

buffer->index = index;

buffer->pts = info->presentationTimeUs;

// Assign buffer to the data[3] field

frame->data[3] = (uint8_t *)buffer;

Copy the code

With AVFrame data, the Surface still has no screen.

Recall that MediaCodec also calls a releaseOutputBuffer method to render data to the Surface, where the second parameter must be passed true.


public void releaseOutputBuffer (int index, boolean render)

Copy the code

Also, there is a method in FFmpeg.


int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render);

Copy the code

A buffer is the contents of frame->data[3] and render is the same as a releaseOutputBuffer.

The releaseOutputBuffer method’s first parameter, index, is already assigned to the buffer.

In this way, after decoding can be directly on the screen rendering display.


AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *) frame->data[3];

av_mediacodec_release_buffer(buffer, 1);

Copy the code

It is feasible after testing, but the speed of decoding the screen directly is very fast, not only the speed of 30ms video playback a frame oh, if you want to do the player, but also have to manage and control it.

In addition, the full code demo is directly decoded to the SurfaceView Surface.

The OnFrameAvailableListener callback of the SurfaceTexture can also be used to decode SurfaceTexture constructs. In addition, attachToGLContext can also be associated with the OpenGL environment, and every time when decoding, updateTexImage can be used to update the picture to achieve the target of decoding to OES texture. The specific operation is also easy and convenient.