This article records the various potholes I stepped on when playing surveillance video using FLv.js. Although Getting Started is just a few lines of code, it’s easy to run a video demo, but there are all sorts of exceptions that will make you wonder what life is like.

For one thing, documentation on GitHub is obscure and the instructions are rudimentary. On the other hand, due to the influence of “video playing” thinking, there is not enough knowledge of convection and lack of experience in processing flow.

Here I will step on their own pit, as well as the process of stepping on the pit to supplement the relevant knowledge, a detailed summary.

Outline the preview

The content of this article includes the following aspects:

  • Live and on demand
  • Static data versus streaming data
  • Why FLV?
  • Protocol and base implementation
  • Detail handling points
  • Style custom

On-demand and live broadcast

What is live streaming? What is on-demand?

Not to mention livestreaming. With the popularity of Douyin, everyone knows what livestreaming is for. On demand is actually video playback, and let’s watch a video the same bi li bi li no distinction, in advance of the video is put out, call on demand.

For our front end, we take an MP4 link and put it in the video TAB. The browser takes care of things like video parsing and playback. We can drag the progress bar to choose any time we want to watch it.

But live broadcasting is different. Live broadcasting has two characteristics:

  1. We get stream data
  2. Demand real time

Let’s look at what streaming data is. For those of you who have never done audio and video front-end, the data we often contact is the JSON data that Ajax obtains from the interface, especially the file upload. The thing about this data is that it’s all disposable data. We get one request, one response, and we get the complete data back.

But streaming is different. Streaming data is captured frame by frame, you can think of it as a bit by bit. Like the live stream data, it is not a complete video clip, it is a small binary data, you need to be pieced together, it is possible to output a video.

Look at the real time. If it is on demand, we directly store the complete video on the server, and then return the link, and use the video or player to play. However, the real-time nature of live broadcast determines that the data source cannot be on the server, but on a certain client.

The data source is on the client, so how does it get to other clients?

For this question, take a look at the following flow chart:

As shown in the figure, the client initiating live broadcast is connected to the streaming media server, and the video stream generated by live broadcast will be pushed to the server in real time. This process is called push stream. Other clients are also connected to the streaming media server. The difference is that they are the players and will pull the video stream of the live streaming client in real time. This process is called pulling stream.

Push Stream -> Server -> Pull stream, this is the current popular and standard live streaming solution. See, the whole process of live broadcasting is streaming data transmission, and data processing is directly binary, which is several orders of magnitude more complex than vod.

Specifically, the real-time monitoring preview of the camera in our business is exactly the same as the one above, except that the client for initiating the live broadcast is the camera, and the client for watching the live broadcast is the browser.

Static data versus streaming data

We often contact with text, JSON, images and so on, all belong to static data, the front-end ajax request to the interface back data is static data.

As mentioned above, the video and audio generated by live streaming are streaming data. Stream data is frame by frame, which is binary data in nature. Because it is small, data flows continuously like water, so it is very suitable for real-time transmission.

Static data, which has corresponding data types in the front-end code, such as String, JSON, array, etc. So what is the data type of stream data (binary data)? How is it stored on the front end? How to operate?

To be clear, the front end can store and manipulate binaries. The most basic binary object is an ArrayBuffer, which represents a fixed length such as:

let buffer = new ArrayBuffer(16) // Create a 16-byte buffer filled with zeros
alert(buffer.byteLength) / / 16
Copy the code

An ArrayBuffer is only used to store binary data; if you want to manipulate it, you need to use view objects.

View objects, which do not store any data, are used to structure the data in an ArrayBuffer so that we can manipulate it. In other words, they are interfaces for manipulating binary data.

View objects include:

  • Uint8Array: 1 byte per item
  • Uint16Array: 2 bytes per item
  • Uint32Array: uses 4 bytes per item
  • Float64Array: Each item contains 8 bytes

According to the above criteria, a 16-byte ArrayBuffer can be converted into a view object and its length is:

  • Uint8Array: length 16
  • Uint16Array: Length 8
  • Uint32Array: Length 4
  • Float64Array: Contains 2 characters

This is just a brief introduction to how streaming data is stored on the front end, in case you see a long ArrayBuffer in your browser and don’t know what it is, remember it must be binary.

Why FLV?

As mentioned above, live broadcast needs real-time performance. Of course, the shorter the delay, the better. Of course, there are many factors that determine transmission speed, one of which is the size of the video data itself.

The MP4 format that we most commonly use for voD scenes is the most compatible on the front end. However, the larger size of MP4 makes parsing more complicated. This is the disadvantage of the MP4 in live situations.

FLV, on the other hand, has a very small header file with a simple structure and a large chunk to parse. It is very advantageous in the real-time requirements of live broadcasting, so it has become one of the most commonly used live broadcasting schemes.

Of course, there are other formats besides FLV, corresponding to the live broadcast protocol, let’s compare one by one:

  • RTMP: The underlying layer is based on TCP and relies on Flash in the browser.
  • Http-flv: Transmits FLV based on HTTP streaming I/O and plays FLV based on browser support.
  • Websocket-flv: Transmits FLV based on WebSocket and plays FLV based on browser support.
  • HLS: Http Live Streaming, an Http – based protocol proposed by Apple. HTML5 can play directly.
  • RTP: Based on UDP. The delay is 1 second. The browser does not support RTP.

In fact, RTMP was the most commonly used live broadcast solution in the early days, and it has good compatibility, but it relies on Flash. Currently, Flash is disabled by default in the browser, which has been eliminated by The Times, so it is not considered.

The HLS protocol is also common, and the corresponding video format is M3U8. It is made by Apple and supports phones very well, but the fatal drawback is high latency (10-30 seconds), so it is not considered.

RTP needless to say, browsers don’t support it, so that leaves FLV.

But FLV is also divided into HTTP-FLV and websocket-FLV. They look like brothers, so what’s the difference?

As mentioned earlier, live streaming is a real-time transmission, and a connection is created without breaking, requiring continuous push and pull. WebSocket is naturally the first solution we think of in such a scenario that requires long connection, because WebSocket is the technology of long connection real-time communication.

However, with the expansion of JS native capabilities, there are black technologies like FETCH that are stronger than Ajax. Not only does it support Promise, which is more friendly to us, but it’s also inherently capable of streaming data, performs well, and is simple enough to use to make it easier for us developers, hence the HTTP version of the FLV solution.

To sum up, FLV is the most suitable for browser live broadcast, but FLV is not a cure-all. Its disadvantage is that the front-end video tag cannot be played directly and needs to be processed.

The processing scheme is our hero today: flv.js

Protocol and base implementation

As mentioned earlier, FLV supports both WebSocket and HTTP. Fortunately, flV.js supports both protocols as well.

Choose TO use HTTP or WS, in fact, there is no difference in function and performance, the key depends on the backend students give us the protocol. My side of the choice is HTTP, front-end and back-end processing is more convenient.

Next, we will introduce the specific access process of FLv.js. The official website is here

Assuming that there is now a live streaming address: http://test.stream.com/fetch-media.flv, the first step we according to the website quickly began to build a demo:

import flvjs from 'flv.js'
if (flvjs.isSupported()) {
  var videoEl = document.getElementById('videoEl')
  var flvPlayer = flvjs.createPlayer({
    type: 'flv'.url: 'http://test.stream.com/fetch-media.flv'
  })
  flvPlayer.attachMediaElement(videoEl)
  flvPlayer.load()
  flvPlayer.play()
}
Copy the code

Install flv.js. The first line of code checks to see if the browser supports flv.js, which most browsers do. The next step is to get the DOM element of the video tag. FLV will output the processed FLV stream to the video element, and then realize video streaming on the video.

Next comes the key, which is to create the flvjs.player object, which we call the Player instance. The player instance is created using the flvjs.createPlayer function, which takes a configuration object and is often used as follows:

  • type: Media type,flv 或 mp4The default FLV,
  • isLive: Optional, whether the stream is live. The default value is true
  • hasAudio: Whether there is audio
  • hasVideo: Is there a video
  • url: Specifies the stream addresshttps(s) or ws(s)

Whether the above has audio, video configuration, or depends on whether the stream address has audio and video. For example, monitoring streams only have video streams and no audio, so even if you configure hasAudio: true, sound is not possible.

Once the player instance is created, there are three steps:

  • Mount elements: flvPlayer attachMediaElement (videoEl)
  • Load flow: flvPlayer. The load ()
  • stream: flvPlayer. The play ()

So much for the basic implementation process, let’s go over the details and highlights of the process.

Detail handling points

The basic demo is up and running, but there are some key issues that need to be addressed if you want to go into production.

Pause and Play

It is very easy to pause and play on demand. There is a play/pause button below the player. You can pause at any time you want. But on air, it’s different.

Normally live broadcast should have no play/pause button and progress bar. Because we’re looking at live information, you can’t pause the video and play it again from where you left it. Why? Because you’re live, so when you hit Play, you’re supposed to get the latest live stream and play the latest video.

In terms of technical details, the front-end video label has a progress bar and pause button by default. Flv.js outputs live stream to the video label. If the pause button is clicked at this point, the video will also stop, which is consistent with the vod logic. But if you hit Play again, the video will continue from the pause, which is not true.

So let’s take a fresh look at the play/pause logic of live streaming.

Why does the live stream need to be suspended? Take our video surveillance as an example, a page will put surveillance videos of several cameras, if each player has been connected to the server, continuous streaming, it will cause a large number of connections and consumption, the loss is white money.

Can we go to the web page, find the camera you want to see, click play and then pull stream? When you don’t want to watch, click pause to disconnect the player, which will save you useless traffic consumption.

Therefore, the core logic of play/pause in live broadcast is to pull/stop stream.

Understanding this, our solution should be to hide the pause/play button of the video, and then realize the logic of play and pause by ourselves.

The player instance (flvPlayer variable above) remains unchanged, and the play/pause code is as follows:

const onClick = isplay= > {
  // The isplay argument indicates whether the game is currently being played
  if (isplay) {
    // Stop the stream
    player.unload()
    player.detachMediaElement()
  } else {
    // The stream is disconnected
    player.attachMediaElement(videoEl.current)
    player.load()
    player.play()
  }
}
Copy the code

Exception handling

In the process of accessing live stream with FLv.js, there will be various problems, some are the problems of back-end data flow, some are the problems of front-end processing logic. Because the stream is fetched in real time, and the FLV is converted to output in real time, the browser console cycles out successive exceptions in the event of an error.

If you use React and TS, the screen is full and you can’t even develop. There are a lot of exceptions that can happen to live streams, so error handling is critical.

The official explanation for exception handling is less obvious, but LET me summarize it briefly:

First of all, flv.js exceptions are divided into two levels, which can be considered as first-level and second-level exceptions.

In addition, flv.js has a special feature that its events and errors are represented by enumerations, as follows:

  • flvjs.Events: indicates events.
  • flvjs.ErrorTypes: Indicates a level 1 exception
  • flvjs.ErrorDetails: indicates a level 2 exception

The exceptions and events described below are based on the above enumeration, which you can think of as a key value under the enumeration.

There are three types of first-level anomalies:

  • NETWORK_ERROR: Indicates a network error, indicating a connection problem
  • MEDIA_ERROR: Media error, formatting or decoding problem
  • OTHER_ERROR: Other errors

There are three common level 2 exceptions:

  • NETWORK_STATUS_CODE_INVALID: If the HTTP status code is incorrect, the URL address is incorrect
  • NETWORK_TIMEOUT: Connection timeout, network or background problem
  • MEDIA_FORMAT_UNSUPPORTED: The media format is not supported. Generally, streaming data is not in FLV format

With that in mind, we listen for exceptions on the player instance:

// Listen for error events
flvPlayer.on(flvjs.Events.ERROR, (err, errdet) = > {
  // The err parameter is a first-level exception and errdet is a second-level exception
  if (err == flvjs.ErrorTypes.MEDIA_ERROR) {
    console.log('Media error')
    if(errdet == flvjs.ErrorDetails.MEDIA_FORMAT_UNSUPPORTED) {
      console.log('Media format not supported')}}if (err == flvjs.ErrorTypes.NETWORK_ERROR) {
    console.log('Network error')
    if(errdet == flvjs.ErrorDetails.NETWORK_STATUS_CODE_INVALID) {
      console.log('HTTP status code exception')}}if(err == flvjs.ErrorTypes.OTHER_ERROR) {
    console.log('Other exceptions:', errdet)
  }
}
Copy the code

In addition to customizing play/pause logic, you also need to know the load status. You can use the following methods to listen to the completion of the video stream loading:

player.on(flvjs.Events.METADATA_ARRIVED, () = > {
  console.log('Video load completed')})Copy the code

Style custom

Why style customization? As mentioned earlier, the play/pause logic of live stream is different from that of vod, so we need to hide the action bar element of video and implement related functions through custom elements.

Hide the play/pause button, progress bar, and volume button with CSS:

/* All controls */
video::-webkit-media-controls-enclosure {
  display: none;
}
/* Progress bar */
video::-webkit-media-controls-timeline {
  display: none;
}
video::-webkit-media-controls-current-time-display {
  display: none;
}
/* Volume button */
video::-webkit-media-controls-mute-button {
  display: none;
}
video::-webkit-media-controls-toggle-closed-captions-button {
  display: none;
}
/* Volume control bar */
video::-webkit-media-controls-volume-slider {
  display: none;
}
/* Play button */
video::-webkit-media-controls-play-button {
  display: none;
}
Copy the code

The logic of play and pause is explained above, and a custom button on the style side can be used. We may also need a full-screen button. Let’s look at the full-screen logic:

const fullPage = () = > {
  let dom = document.querySelector('.video')
  if (dom.requestFullscreen) {
    dom.requestFullscreen()
  } else if (dom.webkitRequestFullScreen) {
    dom.webkitRequestFullScreen()
  }
}
Copy the code

For other custom styles, for example, if you want to make a barrage, just put a layer of elements on top of the video.

My official account

This article was first published in the front of the wechat public number cutting people, the public number promises not to accept advertising, and will be a long-term output of front-end engineering and architecture direction of the article, after practice and consideration, as always to ensure the quality.

If this article inspires you, please give me a thumbs up, which is my biggest encouragement 🙏🙏🙏

If you have any questions about the details of the article, welcome to add wechat consultation ~