background

Google has just released a MoveNet model that detects human posture and provides the corresponding tensorflow.js API. MoveNet is said to be able to detect 17 key points of the body quickly and accurately, and through a partnership with InclueHealth, the company will determine whether MoveNet can help patients with remote care.

Google Research launches MoveNet motion detection tool and Tensorflow.js API

Google Research launches MoveNet motion detection tool and Tensorflow.js API

steps

  1. Import the third-party plug-in TensorFlow
  2. Package. json configuration, configure some dependency packages
  3. Build NPM and import dependency packages
  4. Download the model from the Internet, put it on your own server, and configure a legitimate domain name
  5. With Movenet, the video stream is locally identified based on legitimate input parameters, and 17 points are returned, displayed using canvas
  6. Finally, the python server is used for identification

1. Third-party plug-ins

First go to the applet public platform and add TensorFlowJS of AppID: wx6AFed118d9e81DF9 to your applet. Steps: Settings -> Third-party Settings -> Plug-in Management -> Add plug-in -> Enter AppID

2. The configuration package

It is mainly tensorflow. js, tfJS-core, TFJS-Converter, TFJs-backend-webGL, communication fetch-wechat, and posit-detection of your dependent model (Movenet in this library). @mediapipe/pose (posit-detection)

{
  "name": "ai-action"."version": "1.0.0"."description": ""."main": "app.js"."scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": ""."license": "ISC"."dependencies": {
    "@mediapipe/pose": "^ 0.3.1621277220"."@tensorflow-models/pose-detection": "0.0.3"."@tensorflow/tfjs-backend-webgl": "^ 3.6.0"."@tensorflow/tfjs-converter": "^ 3.6.0"."@tensorflow/tfjs-core": "^ 3.6.0"."fetch-wechat": "0.0.3"}}Copy the code

Note: It is best to build the package and install the dependencies using the command line, because errors are possible.

3. Construction of NPM

Package. json is configured and installed using NPM or YARN

  npm install
Copy the code

With node_modules in place, as shown below, it allows projects to use NPM modules and build NPM

4. Download the model and legitimate domain name

Because the model link in the dependent package is external network, it can only be used by hanging VPN, so we had to download the model and put it on our own server. Fortunately, there is a website for downloading models on TensorFlow’s website. There are a lot of downloads and few references, indicating that people are doing this in secret.

Movenet singlepose/from a single gesture recognition model download download address (VPN), is a compressed package, compression out about 10 MB, 1 model. The json file and three bin file.

Note: don’t rename the bin file name on your own server, model.json doesn’t matter. After the model is downloaded and uploaded to the server, the corresponding legal domain name of the applet should be configured, as shown in the figure below



5. Use of front-end Movenet

1. Register the TensorFlow package

  // app.js
var fetchWechat = require('fetch-wechat');
var tf = require('@tensorflow/tfjs-core');
var webgl = require('@tensorflow/tfjs-backend-webgl');
var plugin = requirePlugin('tfjsPlugin');
App({
  onLaunch() {
    plugin.configPlugin({ Tf / / registration
      fetchFunc: fetchWechat.fetchFunc(),
      tf,
      webgl,
      canvas: wx.createOffscreenCanvas(),
      backendName: 'wechat-webgl-' + Date.now()
    });
  },
  globalData: {
    movenet: null,}})Copy the code

2. Use of load model page

var app = getApp()
var poseDetection = require('@tensorflow-models/pose-detection')
loadMoveNet() {
  if (app.globalData.movenet) return false
   var that = this,
     modelUrl = 'https://oss.lanniuh.com/actionRecognition/movenet/lightning/model.json',
     detectorConfig = {
       modelType: poseDetection.movenet.modelType.SINGLEPOSE_LIGHTNING,
       modelUrl: modelUrl,
     };
   poseDetection.createDetector(poseDetection.SupportedModels.MoveNet, detectorConfig).then(function (detector) {
     app.globalData.movenet = detector
   }).catch(function (err) {
     console.log(err)
   })
 },
Copy the code

3. Use the operation page

  <camera id="camera" class="camera" flash="off" device-position="front" resolution="medium" frame-size="medium"></camera>
  <canvas class="camera canvas" type="2d" id="myCanvas"></canvas>
Copy the code
cameraFrame() { / / video
    var that = this,
          store = [],
          startTime = new Date(),
          camera = wx.createCameraContext()
      that.listener = camera.onCameraFrame(function (frame) {
          if (frame && app.globalData.movenet) { // Frame rate control
              store.push(frame)
          }
      })
      that.listener.start({
          success: function () {
              that.flagTimer && clearInterval(that.flagTimer)
              that.flagTimer = setInterval(function () {  // Frame rate control
                  if (store.length == 0) return;
                  var object = {
                      data: new Uint8Array(store[store.length - 1].data),
                      height: Number(store[store.length - 1].height),
                      width: Number(store[store.length - 1].width)
                  }
                  that.actionSend(object)
                  store = []
                  that.setData({
                      resultFps: that.data.resultFps + 1.fpstime: parseInt((that.data.resultFps + 1) * 1000 / (new Date().getTime() - startTime))
                  })
              }, 1000 / that.data.fps)
          },
      })
  },
  
actionSend(){ / / recognition
   app.globalData.movenet.estimatePoses(object).then(function (res) {
         var ctx = that.ctx,
	         keypoimts = res[0].keypoints
	         ctx.clearRect(0.0, that.canvas.width, that.canvas.height)
	         that.drawSkevaron(keypoimts)
             that.drawKeypoints(keypoimts)
   }).catch(function (err) {
       console.log(err)
    });
},

drawSkevaron(keypoints, scale = 1) { // Connect key points
    / / to the head
    this.drawSegment(keypoints[0], keypoints[1]);
    this.drawSegment(keypoints[0], keypoints[2]);
    this.drawSegment(keypoints[1], keypoints[3]);
    this.drawSegment(keypoints[2], keypoints[4]);
    / / private parts
    this.drawSegment(keypoints[10], keypoints[8]);
    this.drawSegment(keypoints[8], keypoints[6]);
    this.drawSegment(keypoints[6], keypoints[5]);
    this.drawSegment(keypoints[5], keypoints[7]);
    this.drawSegment(keypoints[7], keypoints[9]);

    this.drawSegment(keypoints[6], keypoints[12]);
    this.drawSegment(keypoints[12], keypoints[11]);
    this.drawSegment(keypoints[11], keypoints[5]);
    this.drawSegment(keypoints[12], keypoints[14]);
    this.drawSegment(keypoints[14], keypoints[16]);
    this.drawSegment(keypoints[11], keypoints[13]);
    this.drawSegment(keypoints[13], keypoints[15]);
},

drawSegment(akeypoints, bkeypoints) { Draw a line / /
     var ax = akeypoints[0],
         ay = akeypoints[1],
         bx = bkeypoints[0],
         by = bkeypoints[1]
     this.ctx.beginPath();
     this.ctx.moveTo(ax, ay);
     this.ctx.lineTo(bx, by);
     this.ctx.lineWidth = 3;
     this.ctx.strokeStyle = '#ffffff';
     this.ctx.stroke();
     this.ctx.restore();
 },
 
 drawKeypoints(keypoints) { // Draw key points
     for (var i = 0; i < keypoints.length; i++) {
         var keypoint = keypoints[i];
         this.drawPoint(keypoint[1], keypoint[0]); }},drawPoint(y, x) { / / canvas painting
     this.ctx.beginPath();
     this.ctx.arc(x, y, 4.0.2 * Math.PI, false);
     this.ctx.lineWidth = 2
     this.ctx.strokeStyle = '#ffffff'
     this.ctx.fillStyle = '#00ff66'
     this.ctx.fill();
     this.ctx.stroke()
     this.ctx.restore();
 },
 
 canvasInit() { / / get the canvas
   var that = this
    wx.createSelectorQuery().select('#myCanvas')
        .fields({ node: true.size: true })
        .exec(function (res) {
            var canvas = res[0].node
            var ctx = canvas.getContext('2d')
            var dpr = wx.getSystemInfoSync().pixelRatio
            canvas.width = 480 * dpr
            canvas.height = 640 * dpr
            ctx.scale(dpr, dpr)
            that.ctx = ctx
            that.canvas = canvas
            that.res0 = res[0]})},onLoad: function (options) {
    this.canvasInit()
    this.cameraFrame()    
},
Copy the code

6. Python identification

Finally, pass the 17 unprocessed points to an interface, pass it to the Python server, return the parameters, and do the corresponding processing, and you can do something

For example, here is the action count function

apiRead(keypoints) {
    var that = this
      network.ajax_ai('/train/upload_landmarks', {
          user_id: app.globalData.userInfo.id,
          engine: "movenet".data: {
              keypoints: keypoints
          },
      }, function (res) {
          that.setData({
              poseCount: res.data.count || 0.poseScore: res.data.score || 0,})if (res.data.state == 'finish') {
              that.endPractise()
          }
      }, function (error) {
          console.log(error)
      })
  },
Copy the code

conclusion

Google’s new Movenet is more accurate and smoother than its predecessor, Posenet.

Posenet is trained according to each recognition, that is, image training, and the recognition accuracy may not be enough.

Movenet will have symmetry guess, cache the previous frame, predict the subsequent frame, that is, video training, recognition accuracy is relatively strong.

Since Movenet just appeared for a long time, only a month, the corresponding pit is still quite many, the key domestic mirror model address, have no, also have to download down… I don’t want to say anything else, okay

Unresolved issues

  1. Movenet Huawei mobile phone model compilation error, resulting in identification failure
  2. Because the model is over 10MB, the applets can’t be cached, so each load is a bit slow.