background

During the internship, there was a requirement that the front-end call algorithm model should be encapsulated into NPM package for the video conference group to use, so as to realize background blurring and background replacement functions in the video conference. Some fun features such as facial effects (beard, eyebrow), hair color replacement are likely to be added in the future.

The implementation should look something like this

Tencent conference interface:

​

In order to demonstrate the demand side, we first use the BodyPix model of Google TensorFlowing. Js to do a small demo, first to achieve the background blur and background replacement functions, the model effect is relatively satisfactory, the display screen is smooth.

TensorFlow.js is a JavaScript library. We can use it to create new machine learning models and deploy existing models directly in JavaScript. Very friendly for front-end personnel entry machine learning.

Js provides a number of pre-training models out of the box (see figure below): here we use the BodyPix model from the Image Processing category

This is the official presentation BodyPix demo https://storage.googleapis.co…

The features in the demo were a little too complex for our needs, and there was no background replacement. Therefore, I wrote a demo for the background blur, background replacement scene.

introduce

  • Train of thought: open the camera in the browser, get the video stream picture, calltensorflow.jsthebody-pixmodelMethod to plot the result. Among them, the background blurry is easier to realize and can be directly provided by the modeldrawBokehEffectMethods; There is no ready-made interface for background replacement in the model. Canvas is used to draw the modeltoMaskThe mask object returned by the method (a group of pixel points of the foreground color and background color, where the foreground color represents the portrait area and the background color represents the other areas) has some processing to achieve the background replacement (more on this later).
  • Techniques used: Vue + Element UI, TensorFlow-js (no need to learn, just use the examples) and some Canvas simple operations
  • The objective of the code has been on making https://github.com/SprinaLF/f…

Implementation effect

Let’s start with the final result:

1. Starting interface: the video will be displayed at the bottom after the camera is turned on, and the photos taken will be displayed at the bottom of the video

  1. Background blurry: you can choose medium, high and low blurry degrees

  1. Background Replace: When the mode is changed to Background Replace, the list of background images is displayed and the background can be switched

Core process

I. Introduction of the model

There are two ways to do it

  1. The introduction of the script
<! - the Load TensorFlow. Js - > < script SRC = "https://cdn.jsdelivr.net/npm/@tensorflow/[email protected]" > < / script > <! - the Load BodyPix -- > < script SRC = "https://cdn.jsdelivr.net/npm/@tensorflow-models/[email protected]" > < / script >
  1. To install, use the following command (My project has TensorFlow.js and BodyPix installed, run directlyyarn installInstall dependency.)
$NPM install @tensorflow/ TFJS or yarn add @tensorflow/ TFJS $NPM install @tensorflow-models/body-pix

II. Loading model

Body-Pix has two algorithmic model architectures, MobileNetV1 and Resnet50.

After the local attempt, the MobileNetV1 startup speed is very slow, the GPU requirements are relatively high, not suitable for general computers and mobile devices, only consider MobileNetV1

To preload the model, call the loadAndPredict method initially, with the parameters set as:

Model: {architecture: 'MobileNetV1', outputStride: 16, //8 The larger the value, the larger the layer, the more accurate the model, and the slower the speed. Each floating-point number has 4 bytes (unquantized). Maximum precision & original model size ', '2. 2 bytes per floating point number. Slightly lower accuracy, model size reduced by 2 times ', '1. 1 byte per floating point number. '*/} async loadAndPredict(model) {// async loadAndPredict(model) {this.net = await bodyPix.load(model); }

3. Blurry background

Examples on the official website:

Where, net.segmentPerson(img) returns the result of image pixel analysis, as shown in the figure below.

The existing BodyPix. drawBokeHeffect method is adopted to draw the result to the incoming canvas by passing in the image to be blurred, the canvas object to be drawn, the Segmentation and some parameters of the degree of blur.

Blurred background code:

Async blurBackground () {const img = this.$refs['video'] // Get video frame const Segmentation = await this.net.segmentPerson(img); bodyPix.drawBokehEffect( this.videoCanvas, img, segmentation, this.backgroundBlurAmount, this.edgeBlurAmount, this.flipHorizontal); If (this.radio===2) {if(this.radio===2) { Repeatedly call blurBackground requestAnimationFrame(this.blurBackground)} else with requestAnimationFrame this.clearCanvas(this.videoCanvas) // this.timer = setInterval(async() => { // this.segmentation = await this.net.segmentPerson(img); // bodyPix.drawBokehEffect( // this.videoCanvas, img, this.segmentation, 3, // this.edgeBlurAmount, this.flipHorizontal); / /}, 60)},

Supplement:

Here, video frames need to be constantly processed and drawn to Canvas to ensure a smooth experience.

Initially set a timer, every 60ms to execute the corresponding method, but the effect is not good, can feel obviously sluggish, the performance is not good. So I looked at the bodyPix demo code, with the window. The requestAnimationFrame instead of the timer. Switching the Timer to this approach provides a significant improvement in performance and fluency.

Four. Background replacement

The bodyPix does not provide a ready-made method for replacing the background, but one is to return a Mask object with the portrait as the passing foreground and the background as the passing background color (see figure below).

You can use the Canvas GlobalCompositeOperation property to set the type of compositing operation to apply when drawing a new shape, and to manipulate the mask to replace the background.

GlobalCompositeOperation has a number of types that allow us to set up operations (such as merge operations, drawing levels, color and brightness retention) for drawing a new image on the previous canvas. The default value is source-over, drawing a new image on top of the existing canvas context.

Source-in and destination-over are used

  1. Draw the background

The souce-in is used to draw a new background image to replace.

When the portrait portion (foreground color) is set to transparent in advance and GlobalCompositeOperation is Source-in type, the background image will be drawn only in the background color area, as shown in the figure below:

  1. Draw a portrait

Next, simply switch to Destination-Over and draw the portrait behind the existing contents of the canvas. The background will obscure the previous background and the portrait will be displayed.

Background replacement code:

async replaceBackground() { if(! this.isOpen) return const img = this.$refs['video'] const segmentation = await this.net.segmentPerson(img); Const ForegroundColor = {r: 0, g: 0, b: 0, a: 0} const BackgroundColor = {r: 0, g: 0, b: 0, a: 0} 255} let backgroundDarkeningMask = bodypix. toMask(segmentation, foregroundColor, BackgroundColor) if (backgroundDarkeningMask) {let context = this. VideoCanvas. GetContext (' 2 d) / / synthesis context.putImageData(backgroundDarkeningMask, 0, 0) context. GlobalCompositeOperation = 'source - in' / / new drawings in the overlapping area only context. The drawImage (enclosing backgroundImg, 0, 0, this.videoCanvas.width, Enclosing videoCanvas. Height) context. GlobalCompositeOperation = 'destination - over' / / new drawings not only in the overlapping areas context. The drawImage (img, 0, 0, this.videoCanvas.width, Enclosing videoCanvas. Height) context. GlobalCompositeOperation = 'source - over / / restore} the if (this. Radio = = = 3) { requestAnimationFrame( this.replaceBackground ) } else { this.clearCanvas(this.videoCanvas) } },

Others: Mirroring

Mirroring does not use the bodyPix method, although it does

Direct is implemented through CSS3, using VUE’s V-Bind to dynamically switch classes

<canvas v-bind:class="{flipHorizontal: isFlipHorizontal}" id="videoCanvas" width="400" height="300"></canvas>
  
.flipHorizontal {
    transform: rotateY(180deg);
  }

reference

Open the camera: https://www.cnblogs.com/ljx20… TensorFlow. Js model: https://github.com/tensorflow… canvas:https://developer.mozilla.org… JS statistical function execution time: https://blog.csdn.net/K346K34… Open the camera: https://www.cnblogs.com/ljx20… BodyPix real-time camera blur/background to replace https://www.tytion.net/archiv…