This chapter is some understanding of the present broadcast industry is more mature transparent MP4 gift implementation scheme, the content all comes from personal understanding of the whole tool chain, if there is not enough understanding, I hope you can discuss with me in the comments section.

1. Simple understanding of MP4

MP4 is a package format of streaming media. Internally, AVC is often used as the encoding mode of video track, while AAC is used as the encoding mode of audio track. When encoding AVC, the color sampling standard is YUV, which is a color sampling format of chroma + brightness, which can be converted to RGB by formula.

  MP4 = (Video Track) + (Audio Track) + (Other Track)
Copy the code

Specific MP4 concepts, you can go to Google search related information, this chapter will not focus on the discussion. Later, if you have time, write another essay to expand on these concepts,

MP4 Gift Effects

For designers, MP4 gift special effects, is a kind of what you see that is the animation program, fully emancipate the ideas of designers, can support all designers can design the animation, including 3D dynamic effect. And make full use of the advantages of avC’s high compression rate, when the client decoding, using hard solution to give full play to the ability of GPU, reduce the pressure of CPU. Is a good animation implementation scheme. However, due to the use of YUV color sampling standard, there is no alpha channel, which will cover the entire screen when playing full screen gifts, which is unacceptable for the product.

Transparent MP4 gift effects

As early as last year, the major live software on the launch of full screen MP4 gift dynamic effect, the author also through the analysis of sandbox, found that Tiktok, YY and other apps are transparent MP4 gift special effects scheme, to achieve full screen MP4 gift. The technical solutions implemented by each company are pretty much the same, except for the placement of the RGB +alpha area. Here we take AE as a resource export tool, from the design side of the resource output and client test rendering two aspects to make a simple introduction to the scheme.

3.1 Resource Output

After the design side uses AE to produce special effects, the exported rendering queue is an MP4 video as shown below. When the video is played in full screen, the background of the whole studio will be covered, which makes the experience bad.

Transparent MP4 effects: The exported render queue is an MP4 video shown below, which plays in full screen without the background of the studio being obscured, making it a great experience.

The layer on the right, that is, the layer on the left that has pixel values is not black, and the layer on the left that has no pixel values is black.

So the output of the second video makes the regular MP4 render transparent, because the value of the right region preserves the alpha channel of the left region.

3.2 Client Rendering

After the client gets the video track data, decodes each frame of the picture, and then mixes it with yuV on the left and YUV on the right before loading the screen. Gl formula can be understood as

Gl_FragColor = vec4(texture2D(texture, vec2(vuv.x /2, vuv.y)).rgb, texture2D(texture, vec2(0.5 + vuv.x /2, vuv.y)).r);