One, foreword

WebRTC technology has been used in a wide variety of industries and scenarios, but for most developers, real-time audio, video and related technologies are relatively untouchable.

As a Web developer, it took quite a bit of time to understand the concept of WebRTC. Firstly, WebRTC itself has many unique concepts. Secondly, although it has the word “Web”, it relies on the underlying concepts and network, which is rarely touched by Web development.

This article from the perspective of 0 experienced audio and video developers, analogous to common Web technologies, is expected to help you simply start WebRTC technology, patience through this article, you will:

  1. Know what WebRTC is
  2. Master WebRTC call principles
  3. Debug WebRTC applications with Chrome

Suitable reading target: Web development, js foundation, interested in WebRTC students

2. Use examples

Students who have not been exposed to WebRTC technology can first experience ZEGO’s GoEnjoy product, which contains the standard use plan of WebRTC in the browser, including but not limited to: Device detection, compatibility detection, weak network disconnection response strategy, etc., the application is free, can stamp –> sample Demo portal

Before getting into the main body, let’s have a basic impression of it.

 

Three, a brief introduction

After experiencing the Demo, it is necessary to take a look at the development history of the technology, application scenarios, etc., so that we can know why it is good, what is good, what is bad, etc.

Programmers often use 5W1H analysis, so this article will give you an introduction according to this idea:

What

WebRTC (Web Real-Time Communication) is a framework (APIs) that allows users to implement real-time audio and video Communication using their own traffic. It supports browsers (Firefox, Chrome, Safari), iOS, and Android native systems.

When

In December 2017, it became a W3C draft, and wechat browser in China was only supported in the second half of 19 years. There are still many compatibility problems with domestic mobile browsers. On January 26, 2021, it became an official W3C standard.

Who

In 2011, Google acquired several sub-projects (GIPS, On2, VPx) and established the current WebRTC project, which is currently an open source project of Google.

Where

It can be used in social/entertainment/education/tools where real-time audio and video communication is required, such as the recently popular metasverse.

Why

W3C standard, open source, plug-in, the overall effect is good.

How

This article is also in the middle, the ultimate goal is to let you know how to use.

There are a few concepts that need to be covered before we go into the actual code (you can also look at the code and then come back to this paragraph for further understanding).

  • MediaStream: a streaming media object, an encapsulated format for audio/video data, mounted to the video or audio tag for playback;

  • RTCPeerConnection: session control, sending and receiving network and media information, similar to HTTP objects.

  • SDP: used for media negotiation between two session entities, similar to a configuration item in HTTP.

It’s easier to understand with the following analogy:

 

4. Think ahead

Before going through the code, there are a few more things to think about, otherwise it’s not clear why you need to exchange SDP, cadidate, etc. (you can also look at the code first and then come back to this paragraph for further understanding).

The two parties use browsers to communicate. Inconsistency in browser capabilities and network conditions will have a great impact on communication. Let’s consider the following two questions:

1. Different video coding capabilities?

Peer-a and peer-b are browsers on both sides of the video interaction. Before communication, they must reach an agreement on video coding ability, as shown in the following figure, and finally reach A common H264. If no agreement is reached, communication fails.

2 computers, mostly in a local area Network, need NAT (Network Address Translation), so they cannot communicate directly.

The display is as follows:

Popular a bit metaphor: a house this year 30 (not me, not disorderly guess) by parents forced marriage, he can only ask for help matchmaker, just may be another a house understanding.

The matchmaker needs a way to bypass NAT to solve the problem, so that the two sides can establish communication, we need to use STUN and TURN.

Five, code explanation

Finally, we have arrived at the code explanation Part. The following code will explain the API needed for each step in the sequence of the stream section. If you are looking at the code directly, it is recommended to go back to the introduction of the third and fourth parts after reading the code, so that you can have a deeper understanding

Step 1: Create the data source

LocalStream as the sender’s local preview screen:

/ / create the data source const localStream = await the navigator. MediaDevices. GetUserMedia ({video: true, audio: true,}); // Display the data source, localVideo is the HTML video tag localVideo.srcobject = localStream;Copy the code

Step 2: Create the sending data instance

Used to send the data created in Step 1:

// Local instance const pc1 = new RTCPeerConnection(); // Const pc2 = new RTCPeerConnection();Copy the code

Step 3: Configure an example

The purpose of this step is to exchange information between icecandiDate and SDP

  • Icecandidate: contains communication protocols (TCP/UDP), communication IP, STUN and TURN protocols to describe the format of network information, to solve the problem of network connection between the two parties.

  • SDP: browser capabilities, including but not limited to audio and video encoding format, bandwidth, flow control strategy, etc. To solve the problem of mismatch between the capabilities of both sides in the pre-thinking, SDP browsers of both sides will automatically select the video encoding format supported by both sides through exchange.

// Tell the peer, Local address pc1.addEventListener('icecandidate', Async (e) => {// Send to peer // Peer add local address if (e.candidate) {await pc2.addIceCandidate(e.candidate); }}); pc2.addEventListener('icecandidate', Async (e) => {// Send to local end // Local end adds peer address if (e.candidate) {await pc1.addIceCandidate(e.candidate); }}); Const offer = await pc1.createOffer(); const offer = await pc1.createOffer(); pc1.setLocalDescription(offer); Const answer = await pc2.createAnswer(); // Create a remote SDP to tell the remote browser what capabilities to support pc2.setLocalDescription(answer); //... Send the remote SDP to the local end // Receive the remote SDP to tell the remote browser which capabilities are supported pc1.setRemoteDescription(answer); Pc2. setRemoteDescription(offer);Copy the code

Step 4: Send data

localStream.getTracks().forEach(
(track) => pc1.addTrack(track, localStream)
);
Copy the code

Step 5: Full compact Typescript code

Note that the typescript script used here needs to be converted to JS for actual execution.

const pc1 = new RTCPeerConnection(); pc1.addEventListener('icecandidate', async (e) => { if (e.candidate) { await pc2.addIceCandidate(e.candidate); }}); pc1.addEventListener('iceconnectionstatechange', (e) => { console.log('pc1: iceconnectionstatechange', e); }); const pc2 = new RTCPeerConnection(); pc2.addEventListener('icecandidate', async (e) => { if (e.candidate) { await pc1.addIceCandidate(e.candidate); }}); pc2.addEventListener('iceconnectionstatechange', (e) => { console.log('pc2: iceconnectionstatechange', e); }); pc2.addEventListener('track', (e) => { if (e.streams.length > 0) { remoteVideo.srcObject = e.streams[0]; }}); const remoteVideo = document.querySelector('#remoteVideo') as HTMLVideoElement; const localVideo = document.querySelector('#localVideo') as HTMLVideoElement; async function pushStream(answer: RTCSessionDescriptionInit) { pc1.setRemoteDescription(answer); } async function pullStream(offer: RTCSessionDescriptionInit): Promise<void> { pc2.setRemoteDescription(offer); const answer = await pc2.createAnswer(); pc2.setLocalDescription(answer); console.warn('answer', answer); pushStream(answer); } window.onload = async () => { const localStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true, }); localVideo.srcObject = localStream; localStream.getTracks().forEach((track) => pc1.addTrack(track, localStream)); const offer = await pc1.createOffer(); pc1.setLocalDescription(offer); console.warn('pc1 offer', offer); pullStream(offer); };Copy the code

Six, application examples

After learning the theoretical knowledge, let’s practice together to deepen our understanding of the knowledge — debug WebRTC application through Chrome browser. Learning to debug is an essential skill for further understanding and writing code, so don’t skip this step:

1. Click to open the DEMO

2. Open another TAB page and enter chrome://webrtc-internals/

3. Input relevant information in DEMO to start live broadcasting, and switch back to TAB page in 2, as shown below:

Blue: corresponds to the SDP process in the code

Green: corresponds to the network link status

4. As we continue, we can see the real-time data changes in the push stream:

Blue part: real-time data of pull stream, including resolution, bit rate, packet loss rate, etc

Green part: push stream real-time data, including resolution, bit rate, packet loss rate, etc

 

More field understanding, can be dug here: www.w3.org/TR/webrtc-s…

At the end,

With the update of hardware network, we have experienced the process of text -> picture -> video carrier change. With the popularization of 5G, the integration of audio and video technology is becoming a reality, WebRTC as one of the most important framework, browser support maturity is also rapidly improving, continue to learn WebRTC technology, pay attention to ZEGO is the construction of technology!

Attached is the author’s commonly used tools, recommended collection:

1 webrtc. Making. IO/samples /. Google’s official website demo, contains the latest functions

2 developer.mozilla.org/en-US/docs/… Interface Description

3 the w3c. Making. IO/webrtc – PC / #… (Introduction to W3C Standards)