The latest summit conference is just a quiz mode, but combined with live may bring a different effect. In general, there are several basic points that have become popular:

  • Low barrier: everyone can answer the questions
  • High bonus: almost all millions of above, but do not exclude the robot and your money 🙂
  • Low delay: set questions and answer questions at the same time. This is very high pressure for delayed live broadcast, almost the delay must be controlled within 2s. So, the HLS is basically cold.
  • KOL with rhythm, such as Wang Sicong sa coin
  • Host the United States eyebrow
  • .

As a front-end developer, I feel that the world of live streaming is far away from us. There are two big mountains, one is X5 and the other is Apple. Because the X5 and Apple don’t provide MSE for the front end, the result is either HLS live or you don’t. Now, I am working in Tencent, and recently on the X5 bar, ask them why not support MSE? The answer they gave was:

Arguably, MSE technology should allow for a significant expansion of front-end capabilities over the next year. But MSE is just a player-side technology, where does the stream come from? What’s with H5 live?

For the basics, I recommend going straight to this article:

Comprehensive advanced H5 live at https://www.villianhr.com/2017/03/31/

At present, the more popular protocols are: HLS, RTMP, HTTPFLV. The delay is the most important part of HLS, which is a live streaming player developed by Apple. The delay is about 10s+. RTMP and HTTPFLV are both related to Adobe, which makes it hard to live without Flash. Both of them have surprisingly good latency, only about 3s, but streams are in FLV format.

However, W3C does not agree, I do not want to have anything to do with Flash, I now have WebM, FMP4, TS why to accommodate your FLV?

To address this problem at its source, MPEG has launched the MPEG-Dash Live standard to unify awkward stream description files. It is mainly based on MPD files to do slice and download files. Let’s start with a simple MPEG-DASH file:

<? The XML version = "1.0" encoding = "utf-8"? > <MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:mpeg:dash:schema:mpd:2011" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011,http://dashif.org/guidelines/dash-if-simple" maxSegmentDuration="PT2S" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT10S"> <ProgramInformation> <Title>Media Presentation Description from DASHI-IF live simulator</Title> </ProgramInformation> <BaseURL>http://vm2.dashif.org/dash/vod/testpic_2s/</BaseURL> <Period id="precambrian" start="PT0S"> <AdaptationSet contentType="audio" mimeType="audio/mp4" lang="eng" segmentAlignment="true" startWithSAP="1"> <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/> <SegmentTemplate startNumber="1" initialization="$RepresentationID$/init.mp4" duration="2" media="$RepresentationID$/$Number$.m4s"/> <Representation Id = "A48" codecs = "mp4a. 40.2" bandwidth = "48000" audioSamplingRate = "48000" > < AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/> </Representation> </AdaptationSet> <AdaptationSet contentType="video" mimeType="video/mp4" segmentAlignment="true" startWithSAP="1" par="16:9" minWidth="640" maxWidth="640" minHeight="360" maxHeight="360" maxFrameRate="60/2"> <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/> <SegmentTemplate startNumber="1" initialization="$RepresentationID$/init.mp4" duration="2" media="$RepresentationID$/$Number$.m4s"/> <Representation Id ="V300" codecs=" avC1.64001 e" bandwidth="300000" width="640" height="360" frameRate="60/2" SAR ="1:1"/> </AdaptationSet>  </Period> </MPD>Copy the code

The basic principle is that the background slices a complete stream file and generates Initial Segment and Media Segments, such as.mp4,.ts, etc. More details will be given below.

What about the latency of the protocol?

Depending on your protocol, the maximum latency is faster than HTTP-FLV, approximately <= 2 to 3s. Of these, the most important is the template resolution mechanism it defines, which is directly based on the timestamp agreement.

MPD basic introduction

DASH is actually a series of concepts originally intended for Dynamic Adaptive Streaming over HTTP. It is mainly based on MPD files to do slice and download files. The whole mode is somewhat similar to HLS, but it can be applied to all video formats, such as MP4, webM. It slices the entire video file into specific HTTPFLV segments. However, it is superior to HLS, as explained above.

The playback mode of MPD is actually a slice URL that is negotiated and played according to the content of XML. A simple MPD file is:

<? The XML version = "1.0" encoding = "utf-8"? > <MPD availabilityStartTime="1970-01-01T00:00:00Z" id="Config part of url maybe?" maxSegmentDuration="PT2S" minBufferTime="PT2S" minimumUpdatePeriod="P100Y" profiles="urn:mpeg:dash:profile:isoff-live:2011,http://dashif.org/guidelines/dash-if-simple" publishTime="2018-01-06T06:57:00Z" timeShiftBufferDepth="PT5M" type="dynamic" ns1:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:ns1="http://www.w3.org/2001/XMLSchema-instance"> <ProgramInformation> <Title>Media Presentation Description from DASHI-IF live simulator</Title> </ProgramInformation> <BaseURL AvailabilityTimeOffset = "10.000000" > https://vm2.dashif.org/livesim-dev/ato_10/testpic_2s/ < / BaseURL > < Period id = "p0." start="PT0S"> <AdaptationSet contentType="audio" lang="eng" mimeType="audio/mp4" segmentAlignment="true" startWithSAP="1"> <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" /> <SegmentTemplate duration="2" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/$Number$.m4s" startNumber="0" /> <Representation AudioSamplingRate = "48000" bandwidth = "48000" codecs = "mp4a. 40.2" id = "A48" > < AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2" /> </Representation> </AdaptationSet> <AdaptationSet contentType="video" maxFrameRate="60/2" maxHeight="360" maxWidth="640" mimeType="video/mp4" minHeight="360" minWidth="640" par="16:9" segmentAlignment="true" startWithSAP="1"> <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" /> <SegmentTemplate duration="2" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/$Number$.m4s" startNumber="0" /> <Representation Bandwidth ="300000" codecs="avc1.64001e" frameRate="60/2" height="360" id="V300" SAR ="1:1" width="640" /> </AdaptationSet> </Period> </MPD>Copy the code

Every tag content in the XML file follows the MPEG-Dash LA standard. It contains N+1 Periods Tag. Each Period contains related media streams, such as video: different encoding, perspective, bandwidth, etc., and Audio: different language, type, and bandwidth, etc. Common parameters such as codec, frame rate, and audio-channel cannot be changed during the same Period. Therefore, it is important to provide as much descriptive information as possible within a Period. The information in the Period cannot be changed dynamically, but the user can manually select the desired resolution, bandwidth, and so on.

Periods can contain multiple streams, so they can also provide the ability to insert ads, or to shard view-streams. This is what MPEG-Dash was designed to do, negotiating the content of multiple media streams in one file. Its infrastructure content is:

Here, we will briefly introduce the specific contents of several key tags.

MPD

This is the outermost Tag of the file in MPD and has attributes to describe it. The attributes in this tag are extremely important, determining the file attributes described by the MPD and the order and content of the media stream.

Say it three times: MPD files are very important! MPD files are very important! MPD files are very important!

Its basic attributes are:

Next, let’s give a brief introduction one by one:

  • Id: Indicates the identifier of the MPD. This parameter is not required.
  • Profiles: Basic standards for setting UP MPD. Refer to the table Below for details.

  • Type: used to set the basic properties of the MPD file. There are two values

    • Static: The segment length must be between @availabilityStartTime and @availabilityEndTime.
    • Dynamic: This parameter is determined based on the availabilityStartTime.

  • AvailabilityStartTime: Sets the reference time for all segments. This attribute is required if type is dynamic.

  • AvailabilityEndTime: Range of static files for static files.

  • PulishTime: set the absolute time of MPD file generation (wall-clock)

  • MediaPresentationDuration: set the current mediaSegments the total time is long. For example, PT20M indicates that the duration is 20 minutes. This property can be used instead when mpD. minimumUpdatePeriod and period. duration are not set.

  • MinimumUpdatePeriod: Sets the update time of the current MPD file. This property should not appear when type = static.

  • MinBufferTime: Set the minimum segment length, for example, PT2S. This is usually used with Representation. Bandwidth (BPS) to calculate the minimum segment size.

  • TimeShiftBufferDepth: Used to set the segment’s valid range in MPD. Or you can view it as expiration time. Here, we’ll talk more about that later.

  • MaxSegmentDuration: Sets the maximum segment duration.

Now that we’ve seen the basic properties of the MPD tag, which are important throughout MPEG-Dash, we’ll briefly discuss MPD updates and file expiration points.

MPD file update mechanism

The update of the MPD file depends on the @minimumUpdatePeriod property setting, for example: minimumUpdatePeriod=”PT10S”. Indicates that the MPD file is updated 10 seconds after the last MPD file was obtained. If it does not exist or is 0, the MPD file will not expire. The entire update logic is:

There are a few things to be aware of when an MPD file is updated:

  • The value of the mpd. id attribute must be the same as that of the previous MPD
  • The period. id property must be the same as the previous Period
  • Mpd. publishTime needs to be consistent with the update time

For more details, see ISO/ITU standard 5.4 Media Presentation Description Updates.

Segment file expired

Especially for live slice files, the server capacity is limited, so they will not always be saved in the server. The best way to do this is to agree on the expiration time of the Segment. How is the expiration date determined?

Mainly according to the following formula:

[max(AST, now-timeShiftBufferDepth), now]
Copy the code
  • AST: is MPD availabilityStartTime properties.
  • TimeShiftBufferDepth: is the mpd. timeShiftBufferDepth property, which is used to determine the expiration time.

Because normally, Max (AST, now-timeshiftBufferDepth) = now-timeshiftBufferdepth. Therefore, the above scope can be reduced to:

now-timeShiftBufferDepth <= time <= now
Copy the code

The entire length range is controlled by the mpD. timeShiftBufferDepth property value. For example, the MPD tag is set to:

timeShiftBufferDepth="PT5M"
Copy the code

So, the file validity range is that links within 5 minutes before the latest MPD file publishTime are valid.

Period

Period is mainly used to contain data for specific streams, which itself only serves to set startTime and differentiate between multiple Period@id. It has few basic properties:

Here, I will directly express it in the form of text:

 
<xs:sequence>
	<xs:element name="BaseURL" type="BaseURLType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="SegmentBase" type="SegmentBaseType" minOccurs="0"/>

	<xs:element name="SegmentList" type="SegmentListType" minOccurs="0"/> 
	<xs:element name="SegmentTemplate" type="SegmentTemplateType" minOccurs="0"/> 
	<xs:element name="AssetIdentifier" type="DescriptorType" minOccurs="0"/>

	<xs:element name="EventStream" type="EventStreamType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="AdaptationSet" type="AdaptationSetType" minOccurs="0" maxOccurs="unbounded"/>
	<xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute ref="xlink:href"/>
<xs:attribute ref="xlink:actuate" default="onRequest"/>
<xs:attribute name="id" type="xs:string" />
<xs:attribute name="start" type="xs:duration"/>
<xs:attribute name="duration" type="xs:duration"/>
<xs:attribute name="bitstreamSwitching" type="xs:boolean" default="false"/> 
<xs:anyAttribute namespace="##other" processContents="lax"/>
Copy the code

For Period, let’s focus on the attribute content.

  • Id: Identifies the uniqueness of a Period. If mpd. type=dynamic, the ID does not change when MPD is updated.

  • Start: Sets the start time of the Segment reference under Period. Start (); start (); start ();

    • If the current Period does not have a start attribute, but the previous Period was normal, there are start and duration attributes. So the currentPeriod.start = prevPeriod.start + prevPeriod.duration
    • If the current Period does not have the start attribute,MPD.type = staticPeriod is the first Period tag to appear on the MPD, so the start value is set to 0 by default

  • Duration: Determines the duration of the current Period

  • BitstreamSwitching: Whether changing tangent flow is allowed. Default is false.

Such as:

  <Period id="p0" start="PT0S">
    // ...
  </Period>
Copy the code

AdapationSet

AdaptionSet is used to describe the basic information of the current included stream, which is equivalent to fTYPE + Moof Box information in MP4.

The main properties are as follows,

<xs:sequence>

	<xs:element name="Accessibility" type="DescriptorType" minOccurs="0" maxOccurs="unbounded"/>

	<xs:element name="Role" type="DescriptorType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="Rating" type="DescriptorType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="Viewpoint" type="DescriptorType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="ContentComponent" type="ContentComponentType" minOccurs="0" maxOccurs="unbounded"/>

	<xs:element name="BaseURL" type="BaseURLType" minOccurs="0" maxOccurs="unbounded"/> 
	<xs:element name="SegmentBase" type="SegmentBaseType" minOccurs="0"/>

	<xs:element name="SegmentList" type="SegmentListType" minOccurs="0"/>

	<xs:element name="SegmentTemplate" type="SegmentTemplateType" minOccurs="0"/> 
	<xs:element name="Representation" type="RepresentationType" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
	<xs:attribute ref="xlink:href"/>
	<xs:attribute ref="xlink:actuate" default="onRequest"/> <xs:attribute name="id" type="xs:unsignedInt"/> 
	<xs:attribute name="group" type="xs:unsignedInt"/> <xs:attribute name="lang" type="xs:language"/> 
	<xs:attribute name="contentType" type="xs:string"/> <xs:attribute name="par" type="RatioType"/>
	<xs:attribute name="minBandwidth" type="xs:unsignedInt"/> <xs:attribute name="maxBandwidth" type="xs:unsignedInt"/> 
	<xs:attribute name="minWidth" type="xs:unsignedInt"/> <xs:attribute name="maxWidth" type="xs:unsignedInt"/> 
	<xs:attribute name="minHeight" type="xs:unsignedInt"/> <xs:attribute name="maxHeight" type="xs:unsignedInt"/> 
	<xs:attribute name="minFrameRate" type="FrameRateType"/> <xs:attribute name="maxFrameRate" type="FrameRateType"/>
	<xs:attribute name="segmentAlignment" type="ConditionalUintType" default="false"/> 
	<xs:attribute name="subsegmentAlignment" type="ConditionalUintType" default="false"/> 
	<xs:attribute name="subsegmentStartsWithSAP" type="SAPType" default="0"/> 
	<xs:attribute name="bitstreamSwitching" type="xs:boolean"/>
Copy the code

Would it be clearer to look at the picture:

I’m not going to go into all of these properties, but let’s see a Demo to understand:

<AdaptationSet contentType="video" maxFrameRate="60/2" maxHeight="360" maxWidth="640" mimeType="video/mp4" minHeight="360" minWidth="640" par="16:9" segmentAlignment="true" startWithSAP="1"> <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main" /> <SegmentTemplate initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/t$Time$.m4s" timescale="90000"> <SegmentTimeline> <S d="180000" r="150" t="136377021600000" /> </SegmentTimeline> </SegmentTemplate> <Representation bandwidth="300000" Codecs =" avC1.64001e "frameRate="60/2" height="360" id="V300" sar="1:1" width="640" /> </AdaptationSet>Copy the code

This description is not limited to the AS property, but can also be used for Representation.

Representation

Representation IS nested in AS, which IS also the related description of the stream. In fact, it IS equivalent to an IS (Initial Segment).

The main information is:

The main usage is to provide basic descriptive information for the current stream:

<Representation bandwidth="300000" codecs="avc1.64001e" frameRate="60/2" height="360" id="V300" SAR ="1:1" width="640" />Copy the code

The Representation of a Rep (Representation) is defined within the scope of the current Period, i.e. : period.start-period.endtime. Description of a Rep To maximize usage, all streams contained by the Rep can use this content as description.

Alternatively, we can reuse the content of Rep by combining dependencyId and ID. DependencyId template Rep@id is specified to implement direct reuse of information.

<Representation bandwidth="300000" codecs="avc1.64001e" frameRate="60/2" height="360" id="1" SAR ="1:1" width="640" /> <Representation dependencyId="1" id="2" mimeType="video/mp4" />Copy the code

How does MPD express Segments

In MPD, description Segments consist of SegmentBase, SegmentTemplate and SegmentList. A complete Rep requires:

  • N + 1 SegmentList label
  • 1 SegmentTemplate
  • N+1 BaseURL tag, [0,1] SegmentBase, no SegmentTemplate and SegmentList tags.

Here, we briefly describe the contents of the corresponding Segment tag.

SegmentBase

SegmentBase is used only in static MPD files. The SegmentBase consists of several Segments tags.

<xs:sequence>
<xs:element name="Initialization" type="URLType" minOccurs="0"/>
<xs:element name="RepresentationIndex" type="URLType" minOccurs="0"/>
<xs:any namespace="##other" processContents="lax" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="timescale" type="xs:unsignedInt"/>
<xs:attribute name="presentationTimeOffset" type="xs:unsignedLong"/>
 <xs:attribute name="timeShiftBufferDepth" type="xs:duration"/>
 <xs:attribute name="indexRange" type="xs:string"/>
<xs:attribute name="indexRangeExact" type="xs:boolean" default="false"/>
 <xs:attribute name="availabilityTimeOffset" type="xs:double"/>
 <xs:attribute name="availabilityTimeComplete" type="xs:boolean"/>
 
 
 // extension prop
 <xs:extension base="SegmentBaseType">
   <xs:sequence>
	<xs:element name="SegmentTimeline" type="SegmentTimelineType" minOccurs="0"/>
	<xs:element name="BitstreamSwitching" type="URLType" minOccurs="0"/> 
   </xs:sequence>
	<xs:attribute name="duration" type="xs:unsignedInt"/>
	<xs:attribute name="startNumber" type="xs:unsignedInt"/>
</xs:extension>
Copy the code

If you’re confused, look at this:

! [image. PNG 194.7 kB] [108]

The Initialization Tag can be used as an element or directly as a Segments Tag attribute to set the URL of the Initial Segment. The basic Settings are:

 <BaseURL>v-0144p-0100k-libx264.mp4</BaseURL>
 <SegmentBase indexRange="678-1597" timescale="12288">
    <Initialization range="0-677"/>
</SegmentBase>
Copy the code

IndexRange sets the byte range of the media Segment, and range sets the byte range of the Initial Segment.

In addition, there are several attributes that are important:

  • PresentationTimeOffset: Sets the offset time of the current Segment relative to the start of Period.
  • TimeShiftBufferDepth: Overrides the MPD attribute by default and cannot be smaller than the MPD attribute.
  • AvailabilityTimeOffset: Sets the time offset available for the current Segment.

However, in actual use, SegmentBase is rarely used, so you can just remember it.

SegmentList

This tag is used to identify the contents of the URL list of a complete set of segments. For specific information, please refer to:

<xs:element name="SegmentURL" type="SegmentURLType" minOccurs="0" maxOccurs="unbounded"/>

<xs:attribute name="media" type="xs:anyURI"/>
<xs:attribute name="mediaRange" type="xs:string"/>
<xs:attribute name="index" type="xs:anyURI"/>
<xs:attribute name="indexRange" type="xs:string"/>
Copy the code

The main contents are:

This Tag is often used to contain the contents of a full List of videos.

 <SegmentList timescale="90000" duration="5400000">
       <RepresentationIndex sourceURL="representation-index.sidx"/>
       <SegmentURL media="segment-1.ts"/>
       <SegmentURL media="segment-2.ts"/>
       <SegmentURL media="segment-3.ts"/>
       <SegmentURL media="segment-4.ts"/>
       <SegmentURL media="segment-5.ts"/>
       <SegmentURL media="segment-6.ts"/>
       <SegmentURL media="segment-7.ts"/>
       <SegmentURL media="segment-8.ts"/>
       <SegmentURL media="segment-9.ts"/>
       <SegmentURL media="segment-10.ts"/>
</SegmentList>
Copy the code

SegmentTemplate

The SegmentTemplate is mainly used when the number of file slices is too large, and using the SegmentList tag is not worth the cost. For some live scenarios like live streams, mikes, etc. Using a SegmentTemplate should be very useful. It can be provided by startNumber and the corresponding template URL to the client for relevant link resolution. Such as:

<Representation mimeType="video/mp4"
                   frameRate="24"
                   bandwidth="1558322"
                   codecs="avc1.4d401f" width="1277" height="544">
  <SegmentTemplate media="http://cdn.bitmovin.net/bbb/video-1500/segment-$Number$.m4s" 
  initialization="http://cdn.bitmovin.net/bbb/video-1500/init.mp4" 
  startNumber="0" 
  timescale="24" 
  duration="48"/>
</Representation>
Copy the code

The specific attributes are as follows:

  • Index: For the Template field, replace the $Number$and $Time$identifiers.
  • Initialization: Specifies the address used to identify the initialization Segment. Only the $RepresentationID$identifier can be used in this attribute.

Using template can greatly reduce the size of the MPD file, but adds an additional amount of time for the following client to parse the MPD. In addition to the $Number$form used for decoding above, the $RepresentationID$identifier can be used for character substitution:

<SegmentTemplate duration="2" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/$Number$.m4s" /> <Representation bandwidth="300000" /> <Representation bandwidth="300000" sar="1:1" width="640" />Copy the code

So what are the identifiers you can use? In which tags should these identifiers be used? Let’s give a brief description:

Identifier USES

Identifiers are often used in SegmentTemplate tags, and only the following attributes can use identifiers: media, index, Initialization, and bitstreamSwitching attributes.

  • $RepresentationID $: corresponds toRepresentationThe ID attribute in the tag, as shown aboveV300.

Complete identifier explanation content, you can follow wechat public account: front-end little Jimmy, input: mpD. identifier to obtain.

SegmentTimeline

Multiple S labels are used in SegmentTimeline to identify Segment contents within the same MPD Duration. That is, the SegmentTimeline simply provides a template container, and the specific stream identity is determined by the S-tag content. Look directly at the simplest ST(SegmentTimeline) :

<SegmentTimeline timescale="1000">
    <S t="0" r="10" d="24000"/>
</SegmentTimeline>
Copy the code

Here, let’s first look at the basic attribute values under the S tag:

  • T: indicates the start time of the current Segment in the Period stream. Generally, you don’t need to set it. If set, the value isprevious S@t + @d * (@r + 1). The one before thatSThe end time stamp of the tag.
  • D: Identifies the duration of the current Segment
  • R: Sets the number of current segments.

Let’s look at a DEMO:

For detailed explanation and explanation of the DEMO, you can follow the wechat public account: Front-end Little Jimmy, and enter: MPd. timeline.

There will also be a follow-up article with an in-depth understanding of DASH live.