HTML5: Real-time Video processing using Canvas

Look at this example.

The document content

The XHTML document used in this article is shown below.

  
    
      body {
        background: black;
        color:#CCCCCC; 
      }
      #c2 {
        background-image: url(foo.png);
        background-repeat: no-repeat;
      }
      div {
        float: left;
        border :1px solid #444444;
        padding:10px;
        margin: 10px;
        background:#3B3B3B;
      }
    
    
  
 
  
    
       

      
    
       
    
       

      
      
    
       
  
Copy the code

The key parts of the above code are as follows:

1. Create two Canvas elements with ids c1 and c2 respectively. C1 is used to display the original video of the current frame, c2 is used to display the video after performing chroma-keying effects; C2 preloads a still image that will replace the background color in the video. 2. Import JavaScript code from main.js; This script uses JavaScript 1.8 features, so the version is specified in line 22 when you import the script. 3. When the page loads, the processor.doload () method in main.js runs.

JavaScript code

The JS code in main.js contains three methods.

Initialize the chroma – key

The doLoad() method is called when the XHTML document is initially loaded. The purpose of this method is to prepare the required variables for the Chroma-key processing code, setting up an event listener that we can detect when the user starts playing the video.

doLoad: function() {
    this.video = document.getElementById("video");
    this.c1 = document.getElementById("c1");
    this.ctx1 = this.c1.getContext("2d");
    this.c2 = document.getElementById("c2");
    this.ctx2 = this.c2.getContext("2d");
    let self = this;
    this.video.addEventListener("play", function() {
        self.width = self.video.videoWidth / 2;
        self.height = self.video.videoHeight / 2;
        self.timerCallback();
      }, false);
  },Copy the code

This code gets references to the VIDEO element and the two Canvas elements in the XHTML document, as well as to the graphic context of the two Canvas elements. These will be used when we implement chroma-keying effects.

AddEventListener () listens for the video element and is called when the user presses the play button on the video. To cope with user playback, this code takes the width and height of the video, halving it (we will halve the size of the video when performing chroma-keying), and then calls the timerCallback() method to initiate the video capture and visual effects calculation.

Timer callback

The timer callback function is called when the video starts to play (when the “play” event occurs) and then takes care of its own periodic call, implementing keying effects for each frame of video.

timerCallback: function() {
    if (this.video.paused || this.video.ended) {
      return;
    }
    this.computeFrame();
    let self = this;
    setTimeout(function () {
        self.timerCallback();
      }, 0);
  },Copy the code

The callback first checks if the video is playing; If not, the callback does nothing and returns immediately.

The computeFrame() method is then called, which performs chroma-keying effects on the current video frame.

The last thing the callback function does is call setTimeout() to get itself recalled as soon as possible. In a real world, you might set the frequency of calls based on the frame rate of the video.

Processing video frame data

The computeFrame() method, shown below, is actually responsible for capturing the data for each frame and performing chroma-keying effects.

computeFrame: function() {
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);
    let l = frame.data.length / 4;
 
    for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
      if (g > 100 && r > 100 && b < 43)
        frame.data[i * 4 + 3] = 0;
    }
    this.ctx2.putImageData(frame, 0, 0);
    return;
  }Copy the code

When it is called, the video element displays the most recent video frame data, as shown below:

Review images

In line 2, the video frame is copied into the graphic context of the first Canvas CTx1, with the height and width values specified as half the size of the frame we saved earlier. Note that you can draw the current video frame through the drawImage() method, which passes the video element into the drawing context. The result:

Review images

Line 3 gets a copy of the current video frame of the original image data by calling the getImageData() method of the first Canvas context. It provides raw 32-bit pixel image data so we can manipulate it. Line 4 calculates the total number of pixels of the image by dividing the total length of the frame image data by 4.

Line 6 loops through all the pixels, getting the red, green, and blue values for each pixel, and compares them to the predefined background colors that will be replaced by the imported background image in foo.png.

For each pixel detected as a background, replace its alpha value with zero, indicating that the pixel is fully transparent. As a result, the final image background portion is 100% transparent, so that on line 13, when it is drawn into the context of the target, the effect is that the content is superimposed on the static background.

The resulting image looks something like this:

Review images

Do this repeatedly as the video plays, frame by frame, to create chroma-key effects.

Look at this example.