preface
In Android audio and video development, online knowledge is too fragmented, self-learning is very difficult, but audio and video cattle Jhuster put forward “Android audio and video from the entry to improve – task list”. This article is one of the Android audio and video tasks list, corresponding to the content to learn is: learn MediaCodec API, complete video H.264 decoding. (This article is the most basic H264 decoding, advanced content will be explained later)
Audio and video task list
Audio and video task list: Click here to jump to view.
directory
(1) Basic concepts of coding and decoding
1.1 Why Video Coding
A video is made up of thousands of frames, just like a common GIF. If you open a GIF, you will find many images inside. In general, in order not to let the audience feel stuck, a video needs at least 24 frames per second (generally 30 frames). If the video is a 1280×720 resolution video, then the size of one second without coding: So unencoded video can’t be saved at all, let alone transmitted.
1.2 Video compression and coding standards
There are a lot of redundant information in video, such as strong correlation between adjacent pixels of images, similar content between adjacent images of video sequence, human visual system is not sensitive to some details, etc. The process of processing this part of redundant information is video coding. Video coding is mainly divided into H.26X series and MPEG-X series. This article mainly introduces the famous H.264 H.264: H.264/MPEG-4 Part 10, or AVC (Advanced Video Coding), is a Video compression standard, a widely used format for recording, compression, and publishing Video with high precision.
1.3 H.264 Basic Concepts
1.3.1 H.264 coding principle
For A period of little change image, we could start coding out A complete image frames A, then B frame is not all image coding, write only with A frame difference, so B frame size is only about 1/10 of the full frame or smaller, B C frame if change after the frame is not big, we can continue to code in the form of reference B C frame, this cycle. If it is found that frame D differs greatly from frame C, it first encodes a complete image frame D and continues the previous process.
1.3.2 H.264 three frames
There are three types of frames defined in H.264: I frame: the full encoding frame is called I frame, P frame: the frame generated by referring to the previous I frame and containing only the differential part encoding is called P frame, B frame: The frame before and after referring to the frame encoding is called B frame. In actual H264 data frames, the frame is usually preceded by 00 00 00 01 or 00 00 01 delimiter
1.3.2 H.264 decoding principle
Traverse through the H.264 data stream to get every frame of H.264. If each frame is filled into MediaCodec, MediaCodec will process it accordingly and then hand the processed data to the control for display or other operations
(2) H.264 decoding process
2.1 Briefly review the MediaCodec workflow
If you are not familiar with MediaCodec, please check out: Android Audio and Video Development basics (5) : Learn MediaCodec API, complete audio AAC hard programming, hard solution
MediaCodec workflow summary: Populate the data with MediaCodec, and MediaCodec processes it accordingly, The Client then presents the data to the control for display or other operations. (1) The Client requests the empty buffer [dequeueInputBuffer] from the input buffer queue. (2) The Client copies the data to be codeced into empty (3) MediaCodec takes a frame from the input buffer queue and decodes it. (4) After processing, MediaCodec sets the original buffer to empty and puts it back into the input buffer queue, (5) The Client requests the dequeueOutputBuffer from the output buffer queue. (6) The Client renders/plays the codec buffer (7) After rendering/playback, Client puts the buffer back into the output buffer queue [releaseOutputBuffer]
2.2 I understand the process of MediaCodec
The above process is provided by the official. Now I will explain the process of MediaCodec as I understand it
(three) H.264 decoding process source code
3.1 Preparing Resources
Source code res\raw\ HH264.h264 into the phone root directory, source code address at the end of the article
3.2 the layout
activity_main:
<? xml version="1.0" encoding="utf-8"? > <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<SurfaceView
android:id="@+id/SurfaceView"
android:layout_width="match_parent"
android:layout_height="match_parent" />
</androidx.constraintlayout.widget.ConstraintLayout>
Copy the code3.3 code
public class MainActivity extends AppCompatActivity {
private SurfaceView mSurfaceView;
private SurfaceHolder mSurfaceHolder;
private Thread mDecodeThread;
private MediaCodec mMediaCodec;
private DataInputStream mInputStream;
private final static String SD_PATH = Environment.getExternalStorageDirectory().getPath();
private final static String H264_FILE = SD_PATH + "/hh264.h264";
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
// Obtain permission
verifyStoragePermissions(this);
mSurfaceView = (SurfaceView) findViewById(R.id.SurfaceView);
// Get the file input stream
getFileInputStream();
// Initialize the decoder
initMediaCodec();
}
private static final int REQUEST_EXTERNAL_STORAGE = 1;
private static String[] PERMISSIONS_STORAGE = {
"android.permission.READ_EXTERNAL_STORAGE"."android.permission.WRITE_EXTERNAL_STORAGE"};
public static void verifyStoragePermissions(Activity activity) {
try {
// Check whether you have write permission
int permission = ActivityCompat.checkSelfPermission(activity,
"android.permission.WRITE_EXTERNAL_STORAGE");
if(permission ! = PackageManager.PERMISSION_GRANTED) {// There is no write permission. If you apply for the write permission, a dialog box will be displayedActivityCompat.requestPermissions(activity, PERMISSIONS_STORAGE, REQUEST_EXTERNAL_STORAGE); }}catch(Exception e) { e.printStackTrace(); }}/** * get the file stream to decode */
public void getFileInputStream(a) {
try {
File file = new File(H264_FILE);
mInputStream = new DataInputStream(new FileInputStream(file));
} catch (FileNotFoundException e) {
e.printStackTrace();
try {
mInputStream.close();
} catch(IOException e1) { e1.printStackTrace(); }}}/** * get the available byte array *@param is
* @return
* @throws IOException
*/
public static byte[] getBytes(InputStream is) throws IOException {
int len;
int size = 1024;
byte[] buf;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
buf = new byte[size];
while ((len = is.read(buf, 0, size)) ! = -1) {
// Write read data to byte output stream
bos.write(buf, 0, len);
}
// Convert this stream to a byte array
buf = bos.toByteArray();
return buf;
}
/** * Initializes the decoder */
private void initMediaCodec(a) {
mSurfaceHolder = mSurfaceView.getHolder();
mSurfaceHolder.addCallback(new SurfaceHolder.Callback() {
@Override
public void surfaceCreated(SurfaceHolder holder) {
try {
// Create an encoder
mMediaCodec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
} catch (IOException e) {
e.printStackTrace();
}
// Use MediaFormat to initialize the encoder and set the width and height
final MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, holder.getSurfaceFrame().width(), holder.getSurfaceFrame().height());
// Set the frame rate
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 40);
// Configure the encoder
mMediaCodec.configure(mediaFormat, holder.getSurface(), null.0);
startDecodingThread();
}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {}@Override
public void surfaceDestroyed(SurfaceHolder holder) {}}); }/** * start the decoder and start the thread that reads the file */
private void startDecodingThread(a) {
mMediaCodec.start();
mDecodeThread = new Thread(new DecodeThread());
mDecodeThread.start();
}
private class DecodeThread implements Runnable {
@Override
public void run(a) {
// Start decoding
decode();
}
private void decode(a) {
// Get a set of buffers (8)
ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
// The decoded data contains metadata information for each buffer
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
// How long to wait to get the buffer (in milliseconds)
long timeoutUs = 10000;
byte[] streamBuffer = null;
try {
// Returns an array of available bytes
streamBuffer = getBytes(mInputStream);
} catch (IOException e) {
e.printStackTrace();
}
int bytes_cnt = 0;
// Get the length of the available byte array
bytes_cnt = streamBuffer.length;
// There is no available array
if (bytes_cnt == 0) {
streamBuffer = null;
}
// Start position of each frame
int startIndex = 0;
// Define a variable to record the remaining bytes
int remaining = bytes_cnt;
// While (true) gets a frame, decodes it, displays it; Until the last frame is retrieved, decoded, and finished
while (true) {
// Continue reading when remaining bytes =0 or the initial read byte subscript is greater than the number of bytes available
if (remaining == 0 || startIndex >= remaining) {
break;
}
// Find the frame header
int nextFrameStart = findHeadFrame(streamBuffer, startIndex + 2, remaining);
// Can't find the header return -1
if (nextFrameStart == -1) {
nextFrameStart = remaining;
}
// Get the available cache
int inputIndex = mMediaCodec.dequeueInputBuffer(timeoutUs);
// There is a cache available
if (inputIndex >= 0) {
ByteBuffer byteBuffer = inputBuffers[inputIndex];
byteBuffer.clear();
// Pass the available byte array (a frame) into the buffer
byteBuffer.put(streamBuffer, startIndex, nextFrameStart - startIndex);
// Pass the data to the decoder
mMediaCodec.queueInputBuffer(inputIndex, 0, nextFrameStart - startIndex, 0.0);
// Specify the position of the next frame
startIndex = nextFrameStart;
} else {
continue;
}
int outputIndex = mMediaCodec.dequeueOutputBuffer(info, timeoutUs);
if (outputIndex >= 0) {
// The purpose of adding a try catch is to slow down the display. This step can be omitted
try {
Thread.sleep(33);
} catch (InterruptedException e) {
e.printStackTrace();
}
// Pass the processed data to the SurfaceView for display
mMediaCodec.releaseOutputBuffer(outputIndex, true); }}}}In actual H264 data frames, the frame is usually preceded by the 00 00 00 01 or 00 00 01 delimiter *@param bytes
* @param start
* @param totalSize
* @return* /
private int findHeadFrame(byte[] bytes, int start, int totalSize) {
for (int i = start; i < totalSize - 4; i++) {
if (((bytes[i] == 0x00) && (bytes[i + 1] = =0x00) && (bytes[i + 2] = =0x00) && (bytes[i + 3] = =0x01)) || ((bytes[i] == 0x00) && (bytes[i + 1] = =0x00) && (bytes[i + 2] = =0x01))) {
returni; }}return -1; }}Copy the code3.4 Code considerations
(1) Join permission
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
Copy the code(2) Decoding is a time-consuming operation, so it needs to be placed in the child thread
Android audio and video development foundation (six) : learn MediaCodec API, complete video H264 decoding