The profile

  1. What is Web multimedia technology

  2. Basic knowledge of audio and video

  3. Native multimedia capabilities provided by the browser

  4. The development and breakthrough of Web multimedia technology

  5. Summary and Prospect

What is Web multimedia technology

Web multimedia technology, divided into the following several directions

2. Basic knowledge of audio and video

Assume that 8bit represents a sub-pixel with sharpness of 1280 * 720, frame rate of 25fps and duration of 60s

Uncompressed video size = 8bit * 3 * 1280 * 720 * 25 * 60 = 3.9GB

Compression ratio 360:1

H264 compressed video size = 11MB

Video encoding and decoding process

Video compression

Space/frame compression: through a certain algorithm, the video information redundancy to eliminate, can compress the video file

Time/frame compression: save only the first image information, for the second image, save only the difference from the first image

Coding format development

This is the encoder developed by the International Standard Organization; The following is developed by Google

Container format

3. Native multimedia capabilities provided by browsers

<video> and <audio> elements

<! DOCTYPEhtml>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="Width = device - width, initial - scale = 1.0">
    <title>Multimedia capability</title>
</head>

<body>
    <video autoplay controls width="600" height="600">
        <source src="video.mp4">
    </video>
    <audio controls>
        <source src="audio.mp3">
    </audio>
</body>

</html>
Copy the code
  • Function: request audio and video resources, and then decoding audio and video resources, rendering play
  • Support video formats: MP4, MP3

methods

attribute

The event

<video> and <audio> element defects

  • Video formats such as HLS and FLV are not supported

  • Requests and loads of video resources cannot be controlled by code

    Segmented loading (save traffic)

    Clarity seamless switching

    Precise preloading

Major video sites use the <video> element

MediaSource

  • Extension of browser video playback capabilities, support video fragment loading (directly into fMP4 fragments), instead of Flash player
  • Support playback MP4 (streaming playback), HLS, FLV and so on
  • It can realize segmental loading, seamless resolution switching, adaptive bit rate, accurate preloading and so on

Mp4 Video file structure

Fmp4 video file structure

Use MSE to realize MP4 streaming playback

Iv. Development and breakthrough of Web multimedia technology

On demand

Encrypted audio and video playback

Adaptive bit Rate (ABR)

barrage

  • There is no collision
  • Interactive barrage
  • Reduced as

live

Software solution: the Web end play H.265 format and domestic browser anti hijacking

Push page flow

Image decoding

Cloud game

Cloud gaming principles and features

  • No installation required, low hardware requirements
  • Low latency and strong compatibility requirements

V. Summary and prospect

Bytedance Web multimedia technology development

W3C new standard for audio and video technology

  • WebCodecs
  • WebGPU
  • WebVR, WebXR