preface

New Year’s day, come to something interesting

Recently, I have been trying how to do DIY video through web, how to capture the behavior of video, and add some fun things to the video, such as adding a beard to the video?

Here is a list of recent toys

live demo

Click here to see your own effect (need to allow camera ah!! The service has been closed recently, the effect is no longer visible ~)

The effect

It looks something like this: Just add a beard to the face of the video (I have a blue light shield on my phone, you’ll have to deal with that).

Train of thought

First of all, I did the video dynamic mustache drawing thing, that must first support the video bar

Support video

Here the code is very simple, create a new HTML, put a video tag

navigator.getUserMedia =
	navigator.getUserMedia ||
	navigator.webkitGetUserMedia ||
	navigator.mozGetUserMedia

if (navigator.getUserMedia) {
	navigator.getUserMedia(
		{ video: { width: 500.height: 500}},function (stream) {
			var video = document.querySelector('video')
			video.srcObject = stream
			video.onloadedmetadata = function (e) {
				video.play()
			}
		},
		function (err) {
			console.log('The following error occurred: ' + err.name)
		}
	)
} else {
	console.log('getUserMedia not supported')}Copy the code

And that’s when the video is ready to play

And then we put this video in the canvas,

In fact, it is to periodically collect the content of the video, and then draw the image to the canvas

    setInterval((a)= > {
      v.drawImage(video, 0.0.500.375)
	  vData = v.getImageData(0.0.500.375)},20) 
Copy the code

This is where the contents of the canvas begin to move

Here’s what’s essential

Facial recognition

I’m gonna do facial recognition, so how do I do that

  1. Let me implement one myself (non-existent, although learning machine learning, but only a few days, far from enough)
  2. Find open source (Bingo, it’s you)

Open source TrackingJS for facial recognition

And then I found this trackingJS

Have a look at the effect, very good, although a little slow, but meet our function

face(image)-demo face(camera)-demo

Ok, so let’s run a local demo

. And then 5 minutes went by

Wait a minute, it looks like this recognition doesn’t recognize the contents of the camera (camera only supports face, mouth, eye, nose level).

mdzz

Ps: If you just add a hat or something, this library will do

The next few days I basically stuck in this link for a few days have no good ideas

Until one day I was swiping around and I saw this face-detection-Node-OpencV implementation

Oh, she also realized face recognition, it seems that she had seen contributor in TrackingJS before, and then she used OpencV. I remember that in this year’s D2 conference, some guests also shared ar theme things. It is also implemented by the server (that is, the canvas getImageData will be passed to the back end, and the back end will return the results after processing it). At that time, I thought the efficiency would be very poor (in the picture transmission aspect), but in fact, it seems that the effect is ok

Ok, next with the back end to write, it seems that the function will be much more. Then as the master of weapons would say: Fight, fight, fight

The node identification of opencv

So see opencV node has what good implementation

Then, node-opencV, this can recognize up to face level, seems not enough

Looking for

faced

Yooyoo, eyes and nose. Pretty cool. You’re the only one

Just ride the Node service and run

Then a simple 10-minute ride on koA2

The whole idea is:

  1. The front openingvideo
  2. video= >canvas= >base64= >ArrayBuffer(This library can only recognize image objects or ArrayBuffers)
  3. websocketBack and forth communication
  4. The back end processes the image and returnsFace and expressions usingLocation data on a picture
  5. The front-end renders the mouth position
  6. Combined with the beard

done

Start: 4.568ms start: 3.881ms start: 3.564ms start: 3.690ms start: 5.971ms start: 4.069ms start: 3.364ms start: 5.054ms start: 22.219ms start: 5.586ms start: 5.000msCopy the code

The local tests worked fine, but when deployed to the server, there were some problems

  1. Server InstallationopencvWhen there are deep pits
  2. videoAnd only inhttpswith
  3. httpsUnder thewebsocketHave to upgrade towss
  4. Static resource access problem again, damn it

Since this is a front-end article, I’m not going to talk about deployment.

That’s kind of the idea

The last

It’s probably the last one of the year. It’s probably haphazardly written and contextually incoherent (sorry, guys).

And then the final result looks like what you saw at the beginning of the article:

Click here to see what it looks like.

Detailed code words:

Both the front and back ends are here: face-decorator

Reference links/recommended reading

  • Yi? Can the browser do face detection?
  • Front end engineers do Face Detection
  • faced
  • node-opencv
  • HTML5-Face-Detection