1. Introduction

Decamping audio and video streams using FFmpeg is the initial step of processing audio and video input. FFmpeg provides a rich interface function for this operation. The following are some important functions that are described in detail.

2. Introduction to basic functions

1. av_register_all()

Function prototype:

void av_register_all(void);

Function function: initializes all components. Codecs and multiplexers are registered inside the function.

2. avformat_network_init()

Function prototype:

int avformat_network_init(void);

Function: Loads the socket library and the library related to network encryption protocols to support subsequent network-related functions

3. avformat_open_input()

Function prototype:

int avformat_open_input(AVFormatContext **ps, const char *filename, AVInputFormat *fmt, AVDictionary **options)

The AVFormatContext **ps function is used to open multimedia data and obtain related information. Parameter description: AVFormatContext **ps: If the function is successfully called, the audio and video information will be stored in the AVFormatContext ps structure. Const char *filename: indicates the URL of the open audio and video stream. AVInputFormat * FMT: You can forcibly specify the AVInputFormat in the AVFormatContext. This parameter is usually set to NULL, indicating that FFmpeg can automatically detect AVInputFormat. AVDictionary **options: Some options for attachments, usually set to NULL. Return value: successful call returns >=0

4. avformat_find_stream_info()

Function prototype:

int avformat_find_stream_info(AVFormatContext *ic, AVDictionary **options);

Function: read a part of audio and video data and get some related information about the video. The parameters are described as follows: AVFormatContext * IC: Saves the obtained stream information to the IC for subsequent use. AVDictionary **options: Indicates additional options, usually set to 0. Return value: successful call returns >=0

5. av_find_best_stream()

Function prototype:

int av_find_best_stream(AVFormatContext *ic, enum AVMediaType type, int wanted_stream_nb, int related_stream,
                        AVCodec **decoder_ret, int flags)

Function Function: Used to obtain audio stream, video stream index. Parameter description: AVFormatContext* IC: Obtained AVFormatContext. Enum AVMediaType Type: enter the stream information (audio or video) of a specific type to be sought. Int wanted_stream_nb: Generally, this parameter is set to -1. Related_stream: Normally set to -1. AVCodec** decoder_ret: Normally set to NULL. Int flags: The value is usually set to 0. Return value: successful call returns >=0

6. av_read_frame()

Function prototype:

int av_read_frame(AVFormatContext *s, AVPacket *pkt);

Function function: Used to obtain a frame of complete image data, or obtain multiple frames of complete audio data. Parameter description: AVFormatContext *s: The obtained shape format context AVFormatContext. AVPacket * PKT: This value must not be NULL. It must be an open space for receiving AVPacket data. Return value: successful call returns >=0

7. av_seek_frame()

Function prototype:

int av_seek_frame(AVFormatContext *s, int stream_index, int64_t timestamp,                   int flags);

Function: used to find a specific frame. Parameter description: AVFormatContext *s: The obtained shape format context AVFormatContext. Int stream_index: Basic stream index, indicating what kind of stream data (audio or video) seek is for. Int64_t TIMESTAMP: Indicates the time point to seek. The unit is time_base. Int flags: Indicates the flag bit used to set seek. Return value: successful call returns >=0

3. Basic types

1. AVFormatContext

When developing with FFMPEG, the AVFormatContext is a data structure that runs through many functions as an argument. It is the structure of FFMPEG decanting (FLV, MP4, RMVB, AVI) function. AVIOContext *pb: cache of input data unsigned int nb_stream: number of audio and video streams AVStream **streams: char filename[1024] : Name int64_t duration: duration (in microseconds) int BIT_rate: bit rate (in BPS) AVDictionary *metadata: metadata

2. AVStream

AVStream is one of the most important structures in FFmpeg. It is the encapsulation and abstraction of Stream. It describes the encoding format of video, audio and other streams and other basic Stream information. In addition, it is also an important carrier of audio, video and letter data stream. For a typical MP4 format media source, it needs to be decapsulated (demultiplexed), decoded and then output. While decamping streams that are separated from the container, the corresponding object in FFmpeg is AVStream. The number of avStreams will be nb_STREAMS +1 (total number of streams+1) in AVFormateContext and the AvStreams will be stored in the Streams array. Important variables: AVRational time_base: AVRational is a fractional type structure with two integer arguments representing the numerator and the denominator. It is used to indicate how many fractions of a second the duration of type int64_t represents in the next argument. Duration *(time_base.num/time_base.den) AVRational AVG_FRAME_Rate: Frame rate AVCodecParameters * CoDECPAR: Audio and video parameters, used in place of AVCodecContext

3. AVCodecParameters

Enum AVMediaType code_type: type parameter, indicating whether the media stream is audio or video enum AVCodecID Codec_id: indicates the encoding format, such as H264, MPEG4, MJPEG uint32_t codec_tag: Int format: indicates the pixel format of the video (YUV420,YUV422…) Or audio sampling format int width; Uint64_t channel_layout uint64_t channel_layout int channels int sample_rate int frame_size Audio only, the size of a frame of audio

4. AVPacket

AVBufferRef *buf: used to store reference count int64_t PTS: indicates the display time, converted int64_t DTS by PTS *(num/den) : Uint8_t *data int size AVPacket *av_packet_alloc(void) : Create AVPacket object and initialize AVPacket *av_packet_clone(const AVPacket * SRC) : Int av_packet_ref(AVPacket * DST,const AVPacket * SRC) : Void av_packet_free(AVPacket ** PKT) void av_packet_free(AVPacket ** PKT) Void av_init_packet(AVPacket * PKT) : Manually initialize an object int av_packet_from_data(AVPacket * PKT,uint8_t *data, int size) : Create and initialize the AVPacket object