MediaCodec is an impossible part of Android audio and video development, but it’s not very useful and has a lot of water holes. Today we’re going to go into science, oh, no, into MediaCodec.

Welcome to reprint, but please be sure to bring my information, thank you! Link: http://www.woaitqs.cc/2018/11/26/how-to-use-mediacodec-correctly/

MediaCodec Introduction

MediaCodec is a member of the Android multimedia support framework, which also includes MediaExtractor, MediaSync, MediaMuxer, MediaDrm, etc., responsible for codec related work. Let’s take a quick look at how MediaCodec works.

The working process

In broad terms, the codec processes the input data and produces the output data, while MediaCodec uses the input and output cache to process the data asynchronously. Briefly, the general procedure is as follows

  1. Request an empty input input buffer

  2. Fill in the data and give it to MediaCodec

  3. After MediaCodec processes the data, it puts the processed data into an empty output buffer

  4. Get the output buffer filled with data, get the data, and return it to MediaCodec.

Below is an illustration of the steps

The working process

Supported data types

Our brains can’t do everything, and MediaCodec can’t do everything with data. Generally operating within a limited range of systems, stable and reliable, hey, hey, hey. MediaCodec supports three data formats, which are described below:

  1. Compressed Data since it is a codec, it is bound to deal with the corresponding video, audio format of Compressed Data, that is Encode output Data, Decoder input Data. We call this kind of data compressed data. Compressed data format, depending on MediaFormat | Android Developers. In the case of video data, it is usually a frame of data; Audio data, generally a single processing unit (including the number of microseconds of data). Normally, unless BUFFER_FLAG_PARTIAL_FRAME is specified, half a frame will not occur.

  2. The Raw Audio Buffers codec, which encodes the corresponding Audio data, must process the Audio format data, that is, the PCM data. For audio encoding formats, only ENCODING_PCM_16BIT is confirmed to be supported by each System \ Rom.

Here is a brief excerpt from the Wiki definition of PCM data.

How PCM Work?

PCM is to transform a time continuous and continuous analog signal into a time discrete digital signal, which is transmitted in the channel. In short, PCM is the process of sampling analog signals, quantizing the amplitude of sample values and coding. For example, the sound heard is an analog signal, and then the sound is sampled, quantized, and encoded to produce a digital signal. Relative to the natural sound signal, any audio coding is lossy, in computer applications, can achieve high fidelity is PCM coding, so PCM convention has become common lossless coding, for sound, we usually use PCM coding.

  1. Raw Video Buffersa. Native raw video format. Namely MediaCodecInfo. CodecCapabilities. COLOR_FormatSurface, pay attention to this format, and not all Android phones support oh. B. YUV buffers. This parameter can be used in the Surface format or ByteBuffer format. Google says that after API 22, all models are guaranteed to support YUV 4:2:0. C. Other formats. This is to look at the manufacturer’s mood, can pass MediaCodecInfo. CodecCapabilities for query.

The state machine

MediaCodec state machine

MediaCodec is generally divided into three states: Stopped, Executing, and Released. The state machine diagram is shown above, followed by very specific call details.

The specific state switch will not be translated, you can through the above picture, combined with English take a look.

When you create a codec using one of the factory methods, the codec is in the Uninitialized state. First, You need to configure it via configure(…) , which brings it to the Configured state, then call start() to move it to the Executing state. In this state you can process data through the buffer queue manipulation described above.

The Executing state has three sub-states: Flushed, Running and End-of-Stream. Immediately after start() the codec is in the Flushed sub-state, where it holds all the buffers. As soon as the first input buffer is dequeued, the codec moves to the Running sub-state, where it spends most of its life. When you queue an input buffer with the end-of-stream marker, the codec transitions to the End-of-Stream sub-state. In this state the codec no longer accepts further input buffers, but still generates output buffers until the end-of-stream is reached on the output. You can move back to the Flushed sub-state at any time while in the Executing state using flush().

Call stop() to return the codec to the Uninitialized state, whereupon it may be configured again. When you are done using a codec, you must release it by calling release().

On rare occasions the codec may encounter an error and move to the Error state. This is communicated using an invalid return value from a queuing operation, or sometimes via an exception. Call reset() to make the codec usable again. You can call it from any state to move the codec back to the Uninitialized state. Otherwise, call release() to move to the terminal Released state.

MediaCodec Usage Tips

Here’s how to use MediaCodec with the right posture (knowledge), because it works differently on different phones and different systems. The correct use of it is not so much a test of skill as of experience and care.

Create Instance

The first is how to create MediaCodec. When you create MediaCodec, you need to figure out whether you want to create Encoder or Decoder.

When Media Types are known, the createDecoderByType, createEncoderByType, createByCodecName methods can be used to obtain an instance.

The question is how do you determine if your phone supports this MimeType? On the API 21, can use MediaCodecList. FindDecoderForFormat, MediaCodecList. FindEncoderForFormat these two methods to get, if you have to meet the requirements of the Codec, The corresponding MimeType is returned. Note that on LOLLIPOP, :: removes the KEY_FRAME_RATE setting from Format ::, which is a bit of a bug.

The underlying multimedia framework of Android adopts OpenMax standard, but the specific hardware codec function is responsible for by each manufacturer, which leads to a lot of differences between different mobile phones, which is also the root of the compatibility problem of Android multimedia framework. So how do we deal with the compatibility issues that we encounter even when we just create?

The following, including the content of the next chapter, mostly through official Compatibility Definition Document and Supported media formats | Android Developers as a reference, I won’t repeat the explanation.

Audio Codec

When things get tough, look at the picture below

MediaCodec Support Details

As you can see from the figure above, it is supported for most audio decoding. In actual development, when decoding an audio, there is no need to do any special processing. It is ok to deal with possible Crash and decode normally. On the contrary, you need to be careful when encoding audio. PCM/WAVE support is officially guaranteed for encoding for phones with recording Mic privileges, but we rarely use it. In addition, we can see that the familiar MP3 format is not supported. We use another encoding format that is supported by almost all devices, M4A, whose MimeType is “:: Audio/MP4A-latm ::”.

M4a sounds strange, but it’s actually short for MPEG-4 Audio. In order to distinguish MP4 files containing video and audio from audio only MP4 files, Apple has given the latter a separate name ::.m4a::, which is supported by most mobile devices today. M4A is a file extension for an audio file encoded with advanced audio coding (AAC) which is a lossy compression. M4A was generally intended as the successor to MP3, which had not been originally designed for audio only but was layer III in an MPEG 1 or 2 video files. M4A stands for MPEG 4 Audio.

Video Codec

Also in doubt, look at the picture below

Video Support Details

Vp8/9 applicability is not as wide as H.264, so we will skip it. 263 is available only after 7.0 and not as extensive as H.264; 265 Related, Android is not very supported. That leaves h.264 AVC. H.264 AVC Main Profile (MP) and H.264 AVC Main Profile (HP) will be supported on higher levels of Android (7.0 +), and there will be some compatibility issues, which will be discussed later. Overall, h.264 BaseLine encoder is a good choice to achieve the most complete compatibility, so that it can work on almost all models.

Configuration

Once you have created MediaCodec, you need to configure it so that the state of MediaCodec can change from uninitiated to configured.

configure

The most important parameter here is MediaFormat.

  1. If some MediaFormat is not set, MediaCodec will throw IllegalStateException. all keys not marked optional are mandatory !!

  2. If the parameters are not set correctly, an exception may be thrown. Like setting a resolution that’s out of the phone’s reach.

Format Setting required for Video.

Video Format Setting

Format Setting required for Audio

Audio Format Setting

Here are a few Settings that need special attention.

KEY_WIDTH and KEY_HEIGHT

When coding, make sure both are multiples of 16.

KEY_BIT_RATE & KEY_FRAME_RATE

Bits /seconds. The figure below shows the minimum resolution that the device is officially required to support for different resolutions.

FRAME_RATE, too, is almost always required to support at least 30 FPS. (It’s hard to say whether you can actually achieve 30 FPS, and the FPS Settings will depend on the brightness of the Camera preview. The combination of light And shadow secret | Coding And Living | Qisen Tang – Android Developer experience)

Minimum support for encoding
Minimum support for decoding

KEY_COLOR_FORMAT

CDD requires ::If Device Implementations include a video decoder or encoder, Video encoders and decoders MUST support YUV420 flexible Color format(COLOR_FormatYUV420Flexible):: COLOR_FormatYUV420Flexible was introduced in API 21 and is a magic format that can represent many formats.

Chroma planes are subsampled by 2 both horizontally and vertically. Use this format with Image. This format corresponds to ImageFormat.YUV_420_888, and can represent the COLOR_FormatYUV411Planar, COLOR_FormatYUV411PackedPlanar, COLOR_FormatYUV420Planar, COLOR_FormatYUV420PackedPlanar, COLOR_FormatYUV420SemiPlanar and COLOR_FormatYUV420PackedSemiPlanar formats. Although this format is supported by all mobile phones after 21 versions, but need to combine the Image when using | Android Developers to use getOutputImage/getInputImage.

In lower versions, you need to set the specific Color_Format. Basically, most devices support COLOR_FormatYUV420Planar or COLOR_FormatYUV420Planar. We only need to deal with these two color_formats.

Another pitfall is that on some devices, the format represented by ColorFormat may be reversed, such as NV12 and NV21. It is recommended that you use the Surface Input API mentioned in the previous section to circumvent these issues whenever possible.

KEY_I_FRAME_INTERVAL

The usual scheme is 1s, but for special cases such as picture movies, it can be set to 0, which means that you want each frame to be a KeyFrame.

However, in order to achieve accurate frame rate control is not realistic, try to use Surface, try not to occupy too much resources in the process, no obvious lag time, so that the frame rate is relatively better.

KEY_PROFILE & KEY_LEVEL

Set the encoding level here. Note that Profile must be paired with level. The following code is a simple example.

profile

The Profile level setting here is a pit.

  1. Prior to Android 7.0, NO matter how you set the Profile level, BaseLine was always used because it was dead in the source code…

  2. Pit NO.2 -> Main or High Profile can provide a higher compression ratio and other benefits, but it doesn’t mean compatibility is good. In certain cereal phones and some models of Oppo phones, the use of High Profile will cause the time inconsistency between PTS and DTS, and the picture will bounce back and forth during playback. (So far I don’t know how to climb out of this hole, or old practical back to BaseLine)

However, it is advisable to use profiles like High/Main as much as possible. New solutions and new technologies are only valuable if they are used.

As an extra note, when we develop the recording function, we use the AudioRecorder and send the corresponding data to the encoder. Here when configuring AudioRecorder, it is best to set the PCM format to ::ENCODING_PCM_16BIT::, sampling rate set to ::44100Hz:: (44100 is currently the only rate that is guaranteed to work on all devices), the channel setting is :: mono channel :: {@link AudioFormat#CHANNEL_IN_MONO} is guaranteed to work on all devices. Using an Android multimedia framework should be extremely demanding.

KEY_IS_ADTS

A key mapping to a value of 1 if the content is AAC audio and audio frames are prefixed with an ADTS header. The associated value is an integer (0 or 1). This key is only supported when decoding content, it cannot be used to configure an encoder to emit ADTS output.

This setting is only useful for decoding audio, but it stands out because it’s cratering. In general, we don’t need to ignore ADTS and just use Format in the callback. However, on some Meizu models, the Format may be wrong, in which case it is better to set KEY_IS_ADTS to 1 manually.

Other Tips

There are a few hiccups when it comes to actually using MediaCodec, so here’s a quick overview. (Specific how to use MediaCodec, many online information will not talk about)

  1. Make sure that the output data and muxer data are on the same thread, otherwise the screen may appear.

  2. CreateInputSurface () must be created after configure and before start, otherwise the creation may fail.

  3. When processing data, you need to handle position and offset.

There are some other Tips that you can update

The resources

  1. Android MediaCodec stuff

  2. Android Compatibility Definition Document

  3. MediaCodec  |  Android Developers

  4. tests/tests/media/src/android/media/cts – platform/cts – Git at Google

  5. GitHub – OnlyInAmerica/HWEncoderExperiments: Deprecated ( See https://github.com/Kickflip/kickflip-android-sdk for my current work). Experiments with Android 4.3’s MediaCodec and MediaMuxer

  6. GitHub – videolan/vlc: VLC media player – All pull requests are ignored, please follow https://wiki.videolan.org/Sending_Patches_VLC/

  7. tests/tests/media/src/android/media/cts – platform/cts – Git at Google

Document information

  • Copyright Notice: Freely reproduced – Non-commercial – Non-derivative – Remain signed (Creative Commons 3.0 License)

  • Published date: November 26, 2018

  • Social media: weibo.com/woaitqs

  • Feed subscribe: www.woaitqs.cc/feed.xml