WebRTC is Web real-time Communication. WebRTC consists of three components:

  1. Access getUserMedia for the user’s camera and microphone
  2. The PeerConnection of video sessions is established through the NAT and firewall
  3. DataChannels that establish point-to-point data communication between browsers

Corresponding to three API interfaces:

  1. The Network Stream API stands for media data streams
  2. RTCPeerConnection An RTCPeerConnection object allows users to communicate directly between two browsers;
  3. Peer-to-peer Data API A two-way Data channel between two nodes

Browser support for WebRTC

The latest support for Chengdu is available at caniuse.com

  1. PC browser support [PC browser support for WebRTC situation] (gtms04.alicdn.com/tps/i4/TB1w…).
  2. Android supports webRTC from version 29, but webRTC is disabled by default. If webRTC cannot be used properly, please check chrome://flags, Open the webRTC [to the mobile browser support situation] (gtms04.alicdn.com/tps/i4/TB1h…).
  3. IOS doesn’t support it yet, but Apple will soon

WebRTC API

  • MediaStream getUserMedia() is related to WebRTC because it is the portal to this set of apis. It provides access to the user’s local camera/microphone media stream. On mobile devices, only The WebView kernel of Android 36 or later is supported (that is, android L or later). Ios devices do not support webView kernel of Android 36 or later.
  • Function detection Different browsers support getUserMedia differently. Therefore, you need to check whether the user’s browser supports getUserMedia before using the API
function hasGetUserMedia() { return !! (navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia); }Copy the code
  • Request Permission Browser for security reasons, when requesting a webcam or microphone, a pop-up allows the user to choose whether to grant or deny access to their camera or microphone.
  • The method getUserMedia() takes three arguments, the first to specify the type of media you want to access, the second to call back the media stream if it succeeds, and the third to handle the media stream if it fails
navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.moZGetUserMedia || navigator.msGetUserMedia;
var video = document.querySelector('video');
navigator.getUserMedia({
    audio : true,
    video : true
    }, function (stream) {
            video.src = window.URL.creatObjectURL(stream);
    }, function (error) {
            console.log(error);
});
Copy the code

This allows you to successfully output the video stream to the page via the video TAB.

  • Video screenshots can be captured using the Canvas API: DrawImage (video, 0, 0) Draw a frame of the video to canvas, and then convert the canvas to an image through canvas’s Canvas. toDataURL(‘image/ PNG ‘).
var video = document.querySelector('video');
var canvas = document.querySelector('canvas');
var ctx = canvas.getContext('2d');
navigator.getUserMedia({
     audio: true,
     video: true
}, function(stream) {
     video.src = window.URL.creatObjectURL(stream);
}, function(error) {
     console.log(error);
});
video.addEventListener('click', function() {
     ctx.drawImage(video, 0, 0);
     var img = new Image();
     img.src = canvas.toDataURL('image/png');
     document.appendChild(img);
}, false)
Copy the code
  • Add a filter canvas to the picture to obtain the color value of each pixel (including four values of red, green, blue and alpha) through the CANVAS API: The color values of all pixels obtained by ctx.getimageData () can be redrawn to canvas by ctx.putimageData () after modifying the corresponding pixel values
var imgData = ctx.getImageData(); Grayscale: function(pixels) {var d = pixels. Data; for (var i = 0, len = d.length; i < len; i += 4) { var r = d[i], g = d[i + 1], b = d[i + 2]; d[i] = d[i + 1] = d[i + 2] = (r + g + b) / 3; } return pixels; Function (pixels) {var d = pixels. Data; for (var i = 0, len = d.length; i < len; i += 4) { var r = d[i], g = d[i + 1], b = d[i + 2]; D [I] = (r * 0.393) + (g * 0.769) + (b * 0.189); d[I] = (r * 0.393) + (g * 0.769) + (b * 0.189); [I + 1) = d (r * 0.349) + (g * 0.686) + (b * 0.168); [I + 2) = d (r * 0.272) + (g * 0.534) + (b * 0.131); } return pixels; }, // red: function(pixels) {var d = pixels. Data; for (var i = 0, len = d.length; i < len; i += 4) { var r = d[i], g = d[i + 1], b = d[i + 2]; d[i] = (r + g + b) / 3; d[i + 1] = d[i + 2] = 0; } return pixels; Function (pixels) {var d = pixels. Data; for (var i = 0, len = d.length; i < len; i += 4) { var r = d[i], g = d[i + 1], b = d[i + 2]; d[i] = 255 - r; d[i + 1] = 255 - g; d[i + 2] = 255 - b; } return pixels; }}; ctx.putImageData(filter[type](imgData));Copy the code
  • Audio processing Navigator.getUserMedia () can be combined with the Web Audio API to handle Audio effects
var range = document.querySelector('input'); window.AudioContext = window.AudioContext || window.webkitAudioContext; var audioCtx = new AudioContext(); navigator.getUserMedia({ audio: True}, function (stream) {/ / create audio stream var source = audioCtx. CreateMediaStreamSource (stream); / var/double second-order filter biquadFilter = audioCtx. CreateBiquadFilter (); biquadFilter.type = 'lowshelf'; biquadFilter.frequenc.value = 1000; biquadFilter.gain.value = range.value; source.connect(biquadFilter); biquadFilter.connect(audioCtx.destination); }, function(error) { console.log(error); });Copy the code

RTCPeerConnection

RTCPeerConnection, which is used to make calls between peers and establish connections to transmit audio and video data streams.

WebRTC is to realize real-time communication between two or more peers. Before communication can be achieved, a connection must be established between peers. This is the task of RTCPeerConnection. A Signaling Server is used to accomplish this. Signaling contains three types of information:

  • Session Control messages: Initializes and closes communication and reports errors
  • Network configuration: IP addresses and port numbers of both parties (internal IP addresses on the LAN must be converted to external IP addresses).
  • Media Capabilities: What Codecs and video resolution both browsers support.
var PeerConnection = window.RTCPeerConnection || window.mozRTCPeerConnection || window.webkitRTCPeerConnection; navigator.getUserMedia = navigator.getUserMedia ? "getUserMedia" : navigator.mozGetUserMedia ? "mozGetUserMedia" : navigator.webkitGetUserMedia ? "webkitGetUserMedia" : "getUserMedia"; var v = document.createElement("video"); Var PC = new PeerConnection(); pc.addStream(video); pc.createOffer(function(desc) { pc.setLocalDescription(desc, function() { // send the offer to a server that can negotiate with a remote client }); }) var PC = new PeerConnection(); pc.setRemoteDescription(new RTCSessionDescription(offer), function() { pc.createAnswer(function(answer) { pc.setLocalDescription(answer, function() { // send the answer to the remote connection }); }); })Copy the code

Once a peer is connected to a peer, audio and video data flows can be directly transmitted without using a third-party server. Specific documents can be found at developer.mozilla.org/en-US/docs/…

RTCDataChannel

RTCDataChannel can establish point-to-point communication between browsers. Common communication methods include webSocket, Ajax and Server Sent Events. Although webSocket is two-way communication, both webSocket and Ajax are communication between the client and the Server, which means that you must configure the Server to communicate. RTCDATAChannel takes a different approach

  • It uses another webRTC API: RTCPeerConnection, which provides point-to-point communication without going through the server, avoiding the server middleware
  • RTCDataChannel supports SCTP. SCTP is actually a connection-oriented protocol, but the concept of SCTP coupling is broader than that of TCP connection. SCTP improves the defects of TCP to make signaling transmission more reliable. The DESIGN of SCTP includes proper congestion control, protection against flooding and masquerade attacks, better real-time performance, and multi-attribute support.

WebRTC does not specify which signaling mechanism and message protocol to use. Technologies such as SIP, XMPP, XHR, and WebSocket can all be used for signaling communication in WebRTC. In addition to the signaling server, a peer needs to use another server (called STUN Server) to implement NAT/Firewall traversal when establishing a connection with a peer. Because many peers are in private Lans and use private IP addresses, they must be converted to public IP addresses to transmit data between peers. There are some technical terms involved, including STUN, TURN, ICE, etc. Actually, I don’t understand these concepts very well. WebRTC demos found online seem to use STUN Server provided by Google.

Refer to the article: www.html5rocks.com/en/tutorial… Developer.mozilla.org/en-US/docs/…

The purpose of WebRTC is to simplify the development work of browser-based real-time data communication, but the actual application programming is still a little complicated, especially the RTCPeerConnection call must have a deeper understanding of how to establish a connection, exchange signaling process and details. So we can use already packaged WebRTC libraries that further encapsulate the native WebRTC apis into simpler API interfaces. It also blocks differences between browsers. At present, there are two main WebRTC package libraries on the Internet:

  • IO github address: github.com/webRTC/webR…
  • SimpleWebRTC Github address: github.com/HenrikJoret…

Finally is written in WebRTC a little demo: ccforward. Making. IO/demos/webrt…

Relevant article recommended for WebRTC: www.html5rocks.com/en/tutorial…