I’ve found that there are a lot of misconceptions about video formats, and I wrote a previous post on the supported formats of AVPlayer, so HERE’s a separate post on that, so hopefully you’ll understand.

The basic concept

For an online video to play, it goes through the following steps:

Can be summarized as: pull data -> protocol -> decapsulation -> audio and video decoding -> audio and video synchronization -> play. Here is an explanation of each of these concepts:

Broadcast protocol

Generally, HTTP is used for vod, while RTMP or proprietary protocol is mostly used for live broadcast, because the delay is relatively small and RTMP itself is designed for live broadcast. Common streaming media protocols:

shorthand The full name Launch institutions Current Usage
HLS HTTP Live Streaming apple Mostly used in Apple
RTP Real-time transmission protocol IETF Has a wide range of
RTCP Real-time transmission control protocol IETF Has a wide range of
RTSP Real-time streaming protocol RealNetworks etc. Has a wide range of
RTMP Real-time messaging protocol Adobe More popular
MMS Streaming media protocol Microsoft Has a wide range of

The difference between RTP, RTSP/RTCP

RTSP initiates/terminates streaming media, RTP transmits streaming media data, and RTCP controls and synchronizes RTP.

HLS(Http Live Streaming) HLS is an Http based Streaming protocol introduced by Apple. Advantages: 1. The M3U8 index file can be used to select the play source intelligentarily for the current browsing device; 2. The M3U8 index file can be used to add backup index files to prevent the server from crashing and video playing failure; 3. 2. Video processing is required. 3. The request times for index files (TS video files) are relatively large

Video coding

The so-called video encoding means that through a specific compression technology, a video format file into another video format file way. It is a compression technology to reduce the video volume and ensure the picture quality at the same time. Common video coding includes:

The name of the Launch institutions Launch time Current Usage
HEVC(H.265) MPEG/ITU-T 2013 In the research and development
H.264 MPEG/ITU-T 2003 All fields
MPEG4 MPEG 2001 The tepid
MPEG2 MPEG 1994 Digital TV
XviD OpenDivX 2002 popular
VP9 Google 2013 In the research and development
VP8 Google 2008 Don’t spread
VC-1 Microsoft Inc. 2006 Microsoft

H.264

H.264 is also called AVC, is the international Organization for Standardization (ISO) and the International Telecommunication Union (ITU) jointly proposed after the MPEG4 of a new generation of digital video compression format, it combines the advantages of H.263 and MPEG4, has a higher data compression ratio. Under the same image quality conditions, the data compression ratio of H.264 is 2 times higher than THAT of H.263 and 1.5 times higher than that of MPEG-4. It is the best coding format supported by Apple.

Audio coding

Similar to video coding, audio coding is used to reduce the redundancy in the audio and to compress the audio while ensuring the quality of the audio.

The name of the Launch institutions Launch time Current Usage
AAC MPEG 1997 Streaming media
AC-3 Dolby Inc. 1992 DVD, digital TV
MP3 MPEG 1993 Various fields (old)
WMA Microsoft Inc. 1999 Windows

AAC AAC (Advanced Audio Coding technology), appeared in 1997, is based on MPEG-2 Audio Coding technology. Developed by Fraunhofer IIS, Dolby, Apple, AT&T, SONY and others to replace the MP3 format. In 2000, THE MPEG-4 standard was introduced, and AAC reintegrated its characteristics, so it is now called MPEG-4 AAC, namely M4A. Apple’s CoreAudio has good support for AAC.

Encapsulation format

Our common video formats MP4, AVI, RMVB, 3GP, MKV all refer to the encapsulation format of video, most cases are the suffixes of video files. Common combinations are:

The name of the Introduction to the Common encoding format extension
Flash Video An extension of Adobe Flash, a popular web video packaging format. With the rich video sites, this format has become very popular. H.264+MP3 flv
AVI (Audio Video Interleave) The earlier AVI was developed by Microsoft. It means Audio Video Interactive, which is a mix of Video and Audio encoded and stored. AVI is also the oldest format, having been around for over 10 years, and although there have been revisions (V2.0 was released in 1996), it is showing its age. AVI format is more limited, can only have one video track and one audio track (now there are non-standard plug-ins to add up to two audio tracks), and can have some additional tracks such as text. The AVI format does not provide any control functions. Xvid+MP3 avi
WMV (Windows Media Video) Also a group of digital video codec Format developed by Microsoft, ASF (Advanced Systems Format) is its packaging Format. WMV files encapsulated in ASF have digital copyright protection. VC-1+WMA wmv/asf
MPEG (Moving Picture Experts Group) Is an INTERNATIONAL Organization for Standardization (ISO) approved form of media packaging supported by most machines. It can be stored in various ways and adapt to different application environments. The mPEG-4 file container format is specified in Parts 1 (MUX), 14 (ASP), 15 (AVC), etc. MPEG controls are rich, with multiple videos (i.e., angles), audio tracks, subtitles (bitmapped subtitles), and more. A simplified version of MPEG, 3GP, is also widely used on pre-3G phones. H.264+AAC,H263+AAC dat(VCD),vob(DVD), mp4, 3gp
Matroska Matroska Media is a new multimedia package format that encapsulates multiple video encoding formats and 16 or more audio and language captions into one Matroska Media file. It is also one of the open source multimedia encapsulation formats. Matroska also offers great interaction and is more convenient and powerful than MPEG. A combination of various encoding formats mkv
Real Video Real Media (RM) Is a file container developed by RealNetworks. It usually only holds Real Video and Real Audio encoded media. The file has some interactive features that allow scripting to control playback. RM, especially variable bit rate RMVB format, has no complex Profile/Level, is simpler to make than H.264 video format, and is very popular among network uploanders. In addition, many people still have the misconception that RMVB is small in size and high in quality. This incorrect perception also leads many people to use RMVB. In fact, at the same bit rate, RMVB coding is larger than h.264, which is a highly compressed video coding. RealVideo+RealAudio rm/rmvb
QuickTime File Format It’s a container developed by Apple. On February 11, 1998, the International Organization for Standardization (ISO) recognized the QuickTime file format as the basis for the MPEG-4 standard. In addition to video and audio, QuickTime can store images, text (text captions), and more. H.264+AAC mov, qt

MP4 The MP4 format is the standard package format specified for use in H.264 encoding. 3GP is a simplified version of the MP4 format, reducing the storage space and lower bandwidth requirements, making the limited storage space available on the mobile phone.

In fact, the audio and video encoding formats corresponding to these package formats are not fixed, take MP4 for example, the common MP4 is h.264 +AAC package, but also Xvid+AAC encoding may be possible. If the decoder does not support Xvid, you may not be able to play, or you may be playing with sound and no pictures.

Which audio and video coding formats apple supports

We can find the supported video format iPhone-Compare Models-Apple in the phone introduction interface

Here we can find the corresponding video encoding format supported by iPhone7:

Video formats supported: HEVC, H.264, MPEG-4 Part 2, and Motion JPEG

HEVC is also called h.265, iOS11+A9 chip just started to support HEVC, iPhone6s and previous devices do not support HEVC decoding. Audio format with red box marked out, more content, can be compared.

Video formats supported by AVPlayer

Just because Apple devices support audio and video formats doesn’t mean AVPlayer supports many. To determine which formats AVPlayer supports, we can look at an API in AVKit:

// Display the currently supported audio and video formats
let asset = AVURLAsset.audiovisualTypes()
// Print the asset to get it.
asset type (
    "audio/aacp"."video/3gpp2"."audio/mpeg3"."audio/mp3"."audio/x-caf"."audio/mpeg"."video/quicktime"."audio/x-mpeg3"."video/mp4"."audio/wav"."video/avi"."audio/scpls"."audio/mp4"."audio/x-mpg"."video/x-m4v"."audio/x-wav"."audio/x-aiff"."application/vnd.apple.mpegurl"."video/3gpp"."text/vtt"."audio/x-mpeg"."audio/wave"."audio/x-m4r"."audio/x-mp3"."audio/AMR"."audio/aiff"."audio/3gpp2"."audio/aac"."audio/mpg"."audio/mpegurl"."audio/x-m4b"."application/mp4"."audio/x-m4p"."audio/x-scpls"."audio/x-mpegurl"."audio/x-aac"."audio/3gpp"."audio/basic"."audio/x-m4a"."application/x-mpegurl"
)
Copy the code

There is another way to determine if the current format is playable:

//An extended MIME type string such as video/3gpp2; Codecs = "mp4v. 20.9 mp4a. E1" or audio/aac; codecs="mp4a.E1".
let playable: Bool = AVURLAsset.isPlayableExtendedMIMEType("video/3gpp2; Codecs = \ "mp4v. 20.9, mp4a E1 \" ")
Copy the code

The MIMEType here can be found here system-declared Uniform Type Identifiers

AVPlayer supports video encoding formats: h.264, HEVC(iPhone7 and later), mpeg-4. Video format (package format) :.MP4,.mov,.m4V,.3GP,.avi, etc. If you want to support more video formats, you can use third-party frameworks. Common video encoding and decoding frameworks include VLC and FFMPEG.

reference

Video File Formats – Wikipedia, Free Encyclopedia Audio encoding format comparison – Wikipedia, Free Encyclopedia