The content of this article is decoding bare stream, which is the process of reading THE H264 bit stream locally and decoding it into YUV pixel data.

Graph LR Video Format --> Video decoder --> Pixel Data

1. FFmpeg video decoding process

The figure above shows the process of video decoding through FFmpeg.

2. Code practice

2.1. Obtain the decoder

   enum AVCodecID audio_codec_id = AV_CODEC_ID_H264;
    const AVCodec *codec = avcodec_find_decoder(audio_codec_id);
    if(! codec) {fprintf(stderr."Codec not found\n");
        return;
    }
Copy the code

Find the registered decoder by calling the avcodec_find_decoder function based on the ID defined in the AVCodecID enumeration in the source libavcodec/ codecid. h file, We use the VIDEO H264 decoding ID to use AV_CODEC_ID_H264. You can also use avcodec_find_decoder_by_name to retrieve the decoder by passing in the name of the decoder, such as avcodec_find_decoder_by_name(“libx264”) to retrieve the decoder.

2.2. Initialize the Raw stream parser

AVCodecParserContext(data) + AVCodecParser(method)
AVCodecParserContext *parser = av_parser_init(codec->id);
if(! parser) {fprintf(stderr."Parser not found\n");
    return;
}
Copy the code

Call the av_parser_init function to initialize a bare stream parser, AVCodecParserContext. Pass in the ID of the parameter decoder.

2.3. Create context

// Assign the CODEC context
AVCodecContext *codec_ctx = avcodec_alloc_context3(codec);
if(! codec_ctx) {fprintf(stderr."Could not allocate audio codec context\n");
    return;
}

// Associate the decoder with the decoder context
int ret = avcodec_open2(codec_ctx, codec, NULL);
if(ret < 0) {
    fprintf(stderr."Could not open codec\n");
    return;
}
Copy the code

The avCOdec_alloc_context3 function initializes a context, allocates memory for the AVCodecContext, and then calls the avcodec_open2 function to open the decoder, associating the decoder with the decoder context.

2.4. Open the file

// Open the input file
FILE *infile = fopen(in_file, "rb");
if(! infile) {fprintf(stderr."Could not open %s\n", in_file);
    return;
}

// Open the output file
FILE *outfile = fopen(out_file, "wb");
if(! outfile) {fprintf(stderr."Could not open %s\n", in_file);
    return;
}

Copy the code

In_file is the path of the input file, that is, the path of the local H264 file, and out_file is the path of the file that stores the yuv420p video pixel data generated by decoding the H264 stream.

2.5. Create AVPacket and AVFrame

AVPacket *pkt = av_packet_alloc();
if(! pkt) {fprintf(stderr."Could not alloc avpacket\n");
    return;
}

AVFrame *decoded_frame = av_frame_alloc();
if(! decoded_frame) {fprintf(stderr."Could not allocate audio frame\n");
    return;
}
Copy the code

2.6. Read data and decode

// AV_INPUT_BUFFER_PADDING_SIZE decodes the number of additional allocated bytes required at the end of the input bitstream
    uint8_t inbuf[VIDEO_INBUF_SIZE + AV_INPUT_BUFFER_PADDING_SIZE];
    uint8_t *data = inbuf;
    size_t data_size = 0;
    // Read the VIDEO_INBUF_SIZE data into the cache
    data_size = fread(inbuf, 1, VIDEO_INBUF_SIZE, infile);
    while(data_size > 0) {
        ret = av_parser_parse2(parser, codec_ctx, &pkt->data, &pkt->size, data, (int)data_size, AV_NOPTS_VALUE, AV_NOPTS_VALUE,  0);
        if (ret < 0)
        {
            fprintf(stderr."Error while parsing\n");
            break;
        }
        data += ret;
        data_size -= ret;
        if(pkt->size)
            decoder(codec_ctx, pkt, decoded_frame, outfile);
        if(data_size < VIDEO_REFILL_THRESH) {
            memmove(inbuf, data, data_size);
            data = inbuf;
            size_t len = fread(data + data_size, 1, VIDEO_INBUF_SIZE - data_size, infile);
            if (len > 0) data_size += len; }}Copy the code

To read the local data and decode it, we created a data cache with the VIDEO_INBUF_SIZE + AV_INPUT_BUFFER_PADDING_SIZE. The AV_INPUT_BUFFER_PADDING_SIZE is added to prevent some optimized readers from crossing boundaries by reading too many at once. Then call the fread function to read the VIDEO_INBUF_SIZE data from the local file into the cache each time.

The av_parser_parse2 function is used to parse a complete Packet and is one of the core functions in the decoding process. Here are the official arguments for this function.

/** * Parse a packet. * @param s parser context. * @param avctx codec context. * @param poutbuf set to pointer to parsed  buffer or NULL if not yet finished. * @param poutbuf_size set to size of parsed buffer or zero if not yet finished. * @param buf input buffer. * @param buf_size buffer size in bytes without the padding. I.e. the full buffer size is assumed to be buf_size + AV_INPUT_BUFFER_PADDING_SIZE. To signal EOF, this should be 0 (so that the last frame can be output). * @param pts input presentation timestamp. * @param dts input decoding timestamp. * @param pos input byte position in stream. * @return the number of bytes of the input bitstream used. */

int av_parser_parse2(AVCodecParserContext *s,
                     AVCodecContext *avctx,
                     uint8_t **poutbuf, int *poutbuf_size,
                     const uint8_t *buf, int buf_size,
                     int64_t pts, int64_t dts,
                     int64_t pos);
Copy the code

1, s and AVCTx represent decoder and decoder context respectively. Output data size. If the output data is empty after the function is executed (poutbuf_size is 0), parsing is not complete. Frame 4, buf and buf_size represent the input data and the size of the input data respectively. 5. After the function is finished, return the data length of the binary stream that has been used

2.7, decoding

The decoder method has the following code:

static char err_buf[128] = {0};
static char* av_get_err(int errnum)
{
    av_strerror(errnum, err_buf, 128);
    return err_buf;
}

static void print_video_format(const AVFrame *frame)
{
    printf("width: %u\n", frame->width);
    printf("height: %u\n", frame->height);
    printf("format: %u\n", frame->format);
}

static void decoder(AVCodecContext *codec_ctx, AVPacket *pkt, AVFrame *frame, FILE *out_file){
    int ret = avcodec_send_packet(codec_ctx, pkt);
    if(ret == AVERROR(EAGAIN))
    {
        fprintf(stderr."Receive_frame and send_packet both returned EAGAIN, which is an API violation.\n");
    }
    else if (ret < 0)
    {
        fprintf(stderr."Error submitting the packet to the decoder, err:%s, pkt_size:%d\n", av_get_err(ret), pkt->size);
        return;
    }
    while (ret >= 0) {
        ret = avcodec_receive_frame(codec_ctx, frame);
        if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
            return;
        else if (ret < 0)
        {
            fprintf(stderr."Error during decoding\n");
            return;
        }
        static int s_print_format = 0;
        if(s_print_format == 0){
            s_print_format = 1;
            print_video_format(frame);
        }
        // frame->linesize[1] alignment problem
        Linesize [] represents the number of bytes per line, so the offset for each line is linesize[]
        for(int j=0; j<frame->height; j++)
            fwrite(frame->data[0] + j * frame->linesize[0].1, frame->width, out_file);
        for(int j=0; j<frame->height/2; j++)
            fwrite(frame->data[1] + j * frame->linesize[1].1, frame->width/2, out_file);
        for(int j=0; j<frame->height/2; j++)
            fwrite(frame->data[2] + j * frame->linesize[2].1, frame->width/2, out_file); }}Copy the code

2.7.1 Function analysis

Avcodec_send_packet and AVCODEC_receive_frame appear in a pair.

1. Call avcodec_send_packet() to pass the AVPacket object containing the raw compressed data to the decoder. Note that the avpkT-data buffer input must be larger than AV_INPUT_BUFFER_PADDING_SIZE. This is because the optimized byte stream reader must read 32 or 64 bits of data at a time, and the AVCodecContext must already be open via AVCoDEC_open2 when the packet is sent to the decoder. The input parameter, AVPakcet, is usually a single video frame or several complete audio frames. The caller retains the original properties of the package, and the decoder does not modify the contents of the package. The decoder may create references to packages. If the package does not have a reference count, a copy will be made. Unlike previous apis, the input package data is completely consumed, and if multiple frames are included, avcodec_recvive_frame is called multiple times until avCOdec_recvive_frame returns AVERROR(EAGAIN) or AVERROR_EOF. The input parameter can be NULL, or AVPacket’s data field is set to NULL or the size field is set to 0, indicating that all packets will be flushed and that the data stream has ended. The first refresh will always succeed, the second refresh is not necessary and returns AVERROR_EOF. If the decoder caches some frames and returns a refresh, all decoded packets will be returned.

The return value:

0: indicates success AVERROR(EAGAIN) : The current status does not accept input. Users must first read data frames using avCOdec_receive_frame (). AVERROR_EOF: Decoder refreshed and cannot send new packets to it; AVERROR(EINVAL) : No decoder opened, or this is an encoder, or requires a refresh; AVERRO(ENOMEN) : Failed to add packets to the internal queue

2. Call avCOdec_receive_frame to return decoded output data from the decoder. To receive the output of data in a loop, periodically call avcodec_receive_frame() to receive the output data until AVERROR(EAGAIN) or some other error is returned. Note that av_frame_unref(frame) is always called internally to free resources before performing any other operations.

The return value:

0: indicates success and returns a frame AVERROR(EAGAIN) : There is no frame output in this state. The new packet needs to be sent to the decoder using avCodec_SEND_packet. AVERROR_EOF: Decoder has been completely refreshed, no more output frames; AVERROR(EINVAL) : Codec is not open

2.8. Flush decoder

pkt->data = NULL; 
pkt->size = 0;
decoder(codec_ctx, pkt, decoded_frame, outfile);
Copy the code

2.9. Release resources

fclose(outfile);
fclose(infile);
avcodec_free_context(&codec_ctx);
av_parser_close(parser);
av_frame_free(&decoded_frame);
av_packet_free(&pkt);
Copy the code

2.10. Play yuV files

After decoding, we end up with a YUV file, which can be verified using the following instructions.

 ffplay -pixel_format yuv420p -video_size 768x320 -framerate 25 out.yuv
Copy the code

Here 768×320 is the resolution size of the video data, obtained by the above function method print_video_format.