WebRTC introductory tutorial essay | the nuggets technology and practical experience

Before the company is going to use webRTC to achieve video chat, research for a few days, masturbating a demo, (although the technology was not adopted in the end,囧), I heard that the Nuggets are recently engaged in webRTC prize essay activity, you can’t hold your own, brave to write a tutorial, incidentally also to organize their ideas.

Brief introduction to WebRTC

Web Real-Time Communication (WebRTC) is an API that can be used in Web apps such as video chat, audio chat, and P2P file sharing. -MDN

The Web technology is currently available in Chrome, Firefox and Safari.

WebRTC has three main apis

GetUserMedia – Collects local audio and video streams
RTCPeerConnection – API for creating peer connections and transferring audio and video
RTCDataChannel – Used to transmit binary data.

There is a lot of knowledge involved in these three apis. For a thorough and detailed understanding of this technology, I recommend visiting the official website. This tutorial will only cover the important points and write a chat room and multi-person video chat demo based on this premise

Livestream software (such as Douyu Livestream and Panda Livestream, etc.) collects and codes the audio and video of anchors on the client and transmits the data to the streaming media server. The streaming media server forwards the media data, and the client receives the video stream for decoding and plays the simplified architecture diagram

WebRTC provides end-to-end audio and video communication without the need for media servers to forward media data. The architecture is simplified as follows.

WebRTC collects and transmits audio and video data in three steps

Capture local audio and video streams in real time
Encodes audio and video in real time and transmits multimedia data to peers in the network
The peer receives the audio and video from the sender and decodes them in real time

Capture local media streams

This step is very simple, just call the navigator.getUserMedia API, the first parameter is the audio and video restrictions, the first parameter is the successful capture media stream callback, the callback is stream, the third parameter is the capture failed callback, The video is displayed by assigning stream to the srcObject of the video on a successful callback

Example code is as follows:


      
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="Width = device - width, initial - scale = 1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
</head>
<body>
    <div>
        <button id="start">Start recording</button>
        <button id="stop">To stop recording</button>
    </div>
    <div>
        <video autoplay controls id="stream"></video>
    </div>
    <script>
        // Just get the video
        let constraints = {audio: false.video: true}; 
        let startBtn = document.getElementById('start')
        let stopBtn = document.getElementById('stop')
        let video = document.getElementById('stream')
        startBtn.onclick = function() {
            navigator.getUserMedia(constraints, function(stream) {
                video.srcObject = stream;
                window.stream = stream;
            }, function(err) {
                console.log(err)
            })
        }
        stopBtn.onclick = function() {
            video.pause();
        }
    </script>
</body>
</html>
Copy the code

The local video has been collected and played, the next step is to solve how to conduct video conference with each other, that is, real-time transmission of audio and video. WebRTC provides end-to-end transmission of audio and video, that is, the encapsulated RTCPeerConnection API. Calling this API can create peer connection and transmit audio and video. But this API alone is not enough, we need to transmit signaling before we can transmit audio and video data.

What is signaling

Signaling is the process of coordinating communication. In order for a WebRTC application to establish a “call”, its client needs to exchange the following information:

Session control messages are used to turn on or off communication error message media metadata, such as codecs and codec Settings, bandwidth and media type key data, and network data, such as host IP addresses and ports, used to establish secure connectionsCopy the code

The signaling process needs a way to send messages back and forth between clients. The WebRTC standard does not specify signaling methods and protocols. We can use JavaScript session establishment protocol JSEP to implement signaling exchange. Assume that an RTCPeerConnection is set up between A and B

1. A creates an RTCPeerConnection object. 2. A generates an offer (an SDP session description) using the rtcPeerConnection.createOffer () method. 3. A calls setLocalDescription() with the generated offer and sets it to its own local session description. 4. User A sends the Offer to user B through the signaling mechanism. 5. B calls setRemoteDescription() with A's offer and sets it as its own remote session description so that its RTCPeerConnection knows A's setting. 6.b calls createAnswer() to generate answer 7.B sets its answer to the local session description by calling setLocalDescription(). 8. B then sends his answer back to A using the signaling mechanism. 9. User A uses setRemoteDescription() to set user B's reply to the remote session description.Copy the code

A and B need to exchange network information as well as session information. “Find candidates” refers to the process of finding network interfaces and ports using the ICE framework.

1. A uses onicecandiDate to create an RTCPeerConnection object and register A processor function. 2. The handler is called when the network candidate becomes available. 3. On the processor, USER A sends candidate data to user B through the signaling channel. 4. When B gets A candidate message from A, addIceCandidate() is called to add the candidate to the remote peer description.Copy the code

Exchange signaling requires a signaling server, which we will discuss later. Let’s now consider two RTCPeerConnection objects on the same page, one representing local and one representing remote, to illustrate the process and details of signaling exchange and network information exchange. This is the demo of the official document and the fifth laboratory tutorial. If you can access the Internet scientifically, you are advised to go to the tutorial directly to teach you how to build the WebRTC project from simple to profound. Here, we use this demo to explain the process of establishing the RTCPeerConnection

Start by creating index.html

The HTML structure is as follows:

 
      
 <html>

     <head>
         <title>Realtime communication with WebRTC</title>
         <style>
             body {
                 font-family: sans-serif;
             }
     
             video {
                 max-width: 100%;
                 width: 320px;
             }
         </style>
     </head>

     <body>
         <h1>Realtime communication with WebRTC</h1>
     
         <video id="localVideo" autoplay playsinline></video>
         <video id="remoteVideo" autoplay playsinline></video>
     
         <div>
             <button id="startButton">start</button>
             <button id="callButton">call</button>
             <button id="hangupButton">Hang up</button>
         </div>
         <script src="./main.js">
         </script>
 </body>

</html>
Copy the code

One video element shows the local video stream obtained through the getUserMedia() method, and the other shows the video stream transmitted through RTCPeerconnection. Here is the same video stream, but in a real application, one video shows the local video stream and the other shows the remote video stream

The main.js code is as follows

    'use strict';
    // Transmit video, not audio
    const mediaStreamConstraints = {
      video: true.audio: false
    };
    
    // Set only videos to be exchanged
    const offerOptions = {
      offerToReceiveVideo: 1};let startTime = null;
    
    // Set two videos to display local video streams and remote video streams
    const localVideo = document.getElementById('localVideo');
    const remoteVideo = document.getElementById('remoteVideo');
    
    let localStream;
    let remoteStream;
    // Establish two peer connection objects, with sub-tables representing local and remote objects
    let localPeerConnection;
    let remotePeerConnection;



function gotLocalMediaStream(mediaStream) {
  localVideo.srcObject = mediaStream;
  localStream = mediaStream;
  trace('Received local stream.');
  callButton.disabled = false; 
}

function handleLocalMediaStreamError(error) {
  trace(`navigator.getUserMedia error: ${error.toString()}. `);
}

function gotRemoteMediaStream(event) {
  const mediaStream = event.stream;
  remoteVideo.srcObject = mediaStream;
  remoteStream = mediaStream;
  trace('Remote peer connection received remote stream.');
}

function logVideoLoaded(event) {
  const video = event.target;
  trace(`${video.id} videoWidth: ${video.videoWidth}px, ` +
        `videoHeight: ${video.videoHeight}px.`);
}

function logResizedVideo(event) {
  logVideoLoaded(event);
  if (startTime) {
    const elapsedTime = window.performance.now() - startTime;
    startTime = null;
    trace(`Setup time: ${elapsedTime.toFixed(3)}ms.`);
  }
}

localVideo.addEventListener('loadedmetadata', logVideoLoaded);
remoteVideo.addEventListener('loadedmetadata', logVideoLoaded);
remoteVideo.addEventListener('onresize', logResizedVideo);


function handleConnection(event) {
  const peerConnection = event.target;
  const iceCandidate = event.candidate;

  if (iceCandidate) {
    const newIceCandidate = new RTCIceCandidate(iceCandidate);
    const otherPeer = getOtherPeer(peerConnection);

    otherPeer.addIceCandidate(newIceCandidate)
      .then((a)= > {
        handleConnectionSuccess(peerConnection);
      }).catch((error) = > {
        handleConnectionFailure(peerConnection, error);
      });

    trace(`${getPeerName(peerConnection)} ICE candidate:\n` +
          `${event.candidate.candidate}. `); }}function handleConnectionSuccess(peerConnection) {
  trace(`${getPeerName(peerConnection)} addIceCandidate success.`);
};

function handleConnectionFailure(peerConnection, error) {
  trace(`${getPeerName(peerConnection)} failed to add ICE Candidate:\n`+
        `${error.toString()}. `);
}

function handleConnectionChange(event) {
  const peerConnection = event.target;
  console.log('ICE state change event: ', event);
  trace(`${getPeerName(peerConnection)} ICE state: ` +
        `${peerConnection.iceConnectionState}. `);
}

function setSessionDescriptionError(error) {
  trace(`Failed to create session description: ${error.toString()}. `);
}

function setDescriptionSuccess(peerConnection, functionName) {
  const peerName = getPeerName(peerConnection);
  trace(`${peerName} ${functionName} complete.`);
}

function setLocalDescriptionSuccess(peerConnection) {
  setDescriptionSuccess(peerConnection, 'setLocalDescription');
}

function setRemoteDescriptionSuccess(peerConnection) {
  setDescriptionSuccess(peerConnection, 'setRemoteDescription');
}

function createdOffer(description) {
  trace(`Offer from localPeerConnection:\n${description.sdp}`);

  trace('localPeerConnection setLocalDescription start.');
  localPeerConnection.setLocalDescription(description)
    .then((a)= > {
      setLocalDescriptionSuccess(localPeerConnection);
    }).catch(setSessionDescriptionError);

  trace('remotePeerConnection setRemoteDescription start.');
  remotePeerConnection.setRemoteDescription(description)
    .then((a)= > {
      setRemoteDescriptionSuccess(remotePeerConnection);
    }).catch(setSessionDescriptionError);

  trace('remotePeerConnection createAnswer start.');
  remotePeerConnection.createAnswer()
    .then(createdAnswer)
    .catch(setSessionDescriptionError);
}

function createdAnswer(description) {
  trace(`Answer from remotePeerConnection:\n${description.sdp}. `);

  trace('remotePeerConnection setLocalDescription start.');
  remotePeerConnection.setLocalDescription(description)
    .then((a)= > {
      setLocalDescriptionSuccess(remotePeerConnection);
    }).catch(setSessionDescriptionError);

  trace('localPeerConnection setRemoteDescription start.');
  localPeerConnection.setRemoteDescription(description)
    .then((a)= > {
      setRemoteDescriptionSuccess(localPeerConnection);
    }).catch(setSessionDescriptionError);
}

const startButton = document.getElementById('startButton');
const callButton = document.getElementById('callButton');
const hangupButton = document.getElementById('hangupButton');
callButton.disabled = true;
hangupButton.disabled = true;

function startAction() {
  startButton.disabled = true;
  navigator.getUserMedia(mediaStreamConstraints, gotLocalMediaStream, handleLocalMediaStreamError)
  trace('Requesting local stream.');
}
// Create a peer connection
function callAction() {
  callButton.disabled = true;
  hangupButton.disabled = false;

  trace('Starting call.');
  startTime = window.performance.now();

  const videoTracks = localStream.getVideoTracks();
  const audioTracks = localStream.getAudioTracks();
  if (videoTracks.length > 0) {
    trace(`Using video device: ${videoTracks[0].label}. `);
  }
  if (audioTracks.length > 0) {
    trace(`Using audio device: ${audioTracks[0].label}. `);
  }
  // Server configuration
  const servers = null; 

  localPeerConnection = new RTCPeerConnection(servers);
  trace('Created local peer connection object localPeerConnection.');

  localPeerConnection.addEventListener('icecandidate', handleConnection);
  localPeerConnection.addEventListener(
    'iceconnectionstatechange', handleConnectionChange);

  remotePeerConnection = new RTCPeerConnection(servers);
  trace('Created remote peer connection object remotePeerConnection.');

  remotePeerConnection.addEventListener('icecandidate', handleConnection);
  remotePeerConnection.addEventListener(
    'iceconnectionstatechange', handleConnectionChange);
  remotePeerConnection.addEventListener('addstream', gotRemoteMediaStream);

  localPeerConnection.addStream(localStream);
  trace('Added local stream to localPeerConnection.');

  trace('localPeerConnection createOffer start.');
  localPeerConnection.createOffer(offerOptions)
    .then(createdOffer).catch(setSessionDescriptionError);
}
function hangupAction() {
  localPeerConnection.close();
  remotePeerConnection.close();
  localPeerConnection = null;
  remotePeerConnection = null;
  hangupButton.disabled = true;
  callButton.disabled = false;
  trace('Ending call.');
}

startButton.addEventListener('click', startAction);
callButton.addEventListener('click', callAction);
hangupButton.addEventListener('click', hangupAction);

function getOtherPeer(peerConnection) {
  return (peerConnection === localPeerConnection) ?
      remotePeerConnection : localPeerConnection;
}

function getPeerName(peerConnection) {
  return (peerConnection === localPeerConnection) ?
      'localPeerConnection' : 'remotePeerConnection';
}

function trace(text) {
  text = text.trim();
  const now = (window.performance.now() / 1000).toFixed(3);
  console.log(now, text);
}
Copy the code

Well, it’s a long one, so let’s see what main.js does

WebRTC uses the RTCPeerConnection API to set up a connection and transfer media streams between clients (what we call the end of the end). In this example, two RTCPeerConnection objects are on the same page. This example is of no practical use. But it is a good example of how the API works

It is recommended that a connection between two WebRTC users involve three tasks:

Each call requires the creation of an RTCPeerConnection object for each end to retrieve and share network information: possible connection endpoints, also known as ICE candidates retrieve and share local and remote descriptions: local media meta-information in SDP formatCopy the code

Suppose A and B want to set up A video chat room via RTCPeerConnection

First, A and B exchange network information, and searching for candidates is the process of finding network interfaces and ports using the ICE framework

A Creates an RTCPeerConnection object that binds A handler of the OnicecandiDate event (addEventListener(‘ icecandiDate ‘)), which corresponds to the following code in main.js:

    let localPeerConnection;
    // Servers are not used in this example. Servers are configured with STUN and TURN s servers, which we will cover later
    localPeerConnection = new RTCPeerConnection(servers);
    localPeerConnection.addEventListener('icecandidate', handleConnection);
    localPeerConnection.addEventListener('iceconnectionstatechange', handleConnectionChange);
Copy the code

WebRTC is designed to be directly end-to-end, allowing users to connect using the most direct route, without going through a server. However, WebRTC needs to deal with real-world networks: In this process, WebRTC uses the STUN server to obtain the IP address, and the TURN server as the rollback server to avoid the end-to-end connection failure

A call getUserMedia () to obtain video stream, and then addStream (localPeerConnection. AddStream) :

navigator.getUserMedia(mediaStreamConstraints, gotLocalMediaStream, handleLocalMediaStreamError).
function gotLocalMediaStream(mediaStream) {
  localVideo.srcObject = mediaStream;
  localStream = mediaStream;
  trace('Received local stream.');
  callButton.disabled = false;  // Video stream collected, ready to call
}
localPeerConnection.addStream(localStream);
trace('Added local stream to localPeerConnection.');
Copy the code

When the network candidate becomes available, the Onicecandidate handler is called from A to send serialized data to B. In A real application, this process would require A signaling server. Of course, in our example, The two RTCPeerConnections are on the same page and can communicate directly without additional signaling. When B gets candidate information from A, he will call the addIceCandidate to add the candidate to the candidate list

function handleConnection(event) {
  const peerConnection = event.target;
  const iceCandidate = event.candidate;

  if (iceCandidate) {
    const newIceCandidate = new RTCIceCandidate(iceCandidate);
    const otherPeer = getOtherPeer(peerConnection);

    otherPeer.addIceCandidate(newIceCandidate)
      .then((a)= > {
        handleConnectionSuccess(peerConnection);
      }).catch((error) = > {
        handleConnectionFailure(peerConnection, error);
      });

    trace(`${getPeerName(peerConnection)} ICE candidate:\n` +
          `${event.candidate.candidate}. `); }}Copy the code

The peer of WebRTC also needs to find out and exchange local and remote audio and video media information, such as encoding and decoding capabilities. Session description protocol (SDP) is used to exchange metadata, namely offer and Answer, to achieve the purpose of exchanging media configuration information

A runs the createOffer() method of RTCPeerConnection and returns A promise that provides A local session description of RTCSessionDescription: A

trace('localPeerConnection createOffer start.');
localPeerConnection.createOffer(offerOptions)
  .then(createdOffer).catch(setSessionDescriptionError);
Copy the code

If A Promise succeeds, A sets the successful description returned by the Promise to its own local description using the setLocalDescription method, B uses setRemoteDescription to set the description sent by A to him as its own remote description. B runs the createAnswer() method of RTCPeerConnection to generate A matching description. User B sets the generated description as A local description and sends it to USER A. User A sets it as its remote description

function createdOffer(description) {
  trace(`Offer from localPeerConnection:\n${description.sdp}`);

  trace('localPeerConnection setLocalDescription start.');
  localPeerConnection.setLocalDescription(description)
    .then((a)= > {
      setLocalDescriptionSuccess(localPeerConnection);
    }).catch(setSessionDescriptionError);

  trace('remotePeerConnection setRemoteDescription start.');
  remotePeerConnection.setRemoteDescription(description)
    .then((a)= > {
      setRemoteDescriptionSuccess(remotePeerConnection);
    }).catch(setSessionDescriptionError);

  trace('remotePeerConnection createAnswer start.');
  remotePeerConnection.createAnswer()
    .then(createdAnswer)
    .catch(setSessionDescriptionError);
}

function createdAnswer(description) {
  trace(`Answer from remotePeerConnection:\n${description.sdp}. `);

  trace('remotePeerConnection setLocalDescription start.');
  remotePeerConnection.setLocalDescription(description)
    .then((a)= > {
      setLocalDescriptionSuccess(remotePeerConnection);
    }).catch(setSessionDescriptionError);

  trace('localPeerConnection setRemoteDescription start.');
  localPeerConnection.setRemoteDescription(description)
    .then((a)= > {
      setRemoteDescriptionSuccess(localPeerConnection);
    }).catch(setSessionDescriptionError);
}
Copy the code

Done!

The final demo

Signaling server

The signaling server is also a Web server, but it transmits signaling instead of ordinary data. You can choose the server according to your preference (such as Apache, Nginx or Nodejs). Node and socket. IO are used to build the signaling server

socket.io

IO is used as signaling server. The back-end is Node, and the front end is Vue framework. It builds a chat room, and can initiate a video chat demo to the people in the chat room. TURN is the TURN server of our company, which is just for the demo to run. Real projects need to build STUN and TRUN servers by themselves, and github project address

Cloning project
NPM install Installs dependencies
Node run start Starts the back-end server on the local 9000 port
NPM run dev starts the front-end project on port 8080
Visit http://localhost:8080 to access the project, enter the username and password, enter the chat room, in the user list can select other users for video chat (you can open another window, enter the username and password again to simulate multiplayer)

Figure description: Open Google Browser, open three tabs, create three users A, B and C respectively, click to join the chat room and click to start collecting local videos, the pages of A, B and C are as follows.

Switch to the user list on page A, you can see that there are three users A,B and C.

Back to page A, you can see user A and user B interacting with user A

In this case, page A is as follows

Note: This project starts two local Node servers. Socket. IO is accessed from the backend server (http://localhost:9000). If deployed online, Replace the url of the socket in SRC/app. vue’s joinRoom method with the address of the online server.

The pcConfig of SRC/app. vue data is configured as STUN server and TURN server. In order to test the project, I used Google’S STUN server and our company’s TURN server (it was available in the previous test, but I don’t know whether it can still be used). The stuN server and TURN server will need to be replaced by the online server

Agora SDK experience essay contest essay | the nuggets technology, the campaign is underway

WebRTC introductory tutorial essay | the nuggets technology and practical experience

Brief introduction to WebRTC

Capture local media streams

What is signaling

The final demo

Signaling server

socket.io

Related Posts

WebRTC practical experience sharing: How to quickly build audio and video call APP?

Monitors CPU and GPU temperatures on Linux

Born out of BTC, BCH, where is the right path for you?