TensorFlow deeplearn.js is part of the TensorFlow project and is renamed tensorflow.js.

A browser-based JS machine learning library accelerated by WebGL.

Abstract:

This paper involves the understanding of the basic concept of TensorFlow and the practical application of transfer learning technology. The whole paper talks from technology to gameplay of the product. Only when designers/product managers understand the new features of technology, can they integrate new gameplay into the product. Designers should also pay attention to the changes in interaction modes brought about by new technologies and study which interaction modes are suitable for browser-based deep learning applications.

Basic knowledge of TensorFlow, javascript, and NodeJS is required to read this article.

To quickly practice code, we can use Chrome to go directly to the official website:

https://js.tensorflow.org/
Copy the code

Then open the Console panel and enter the code directly to use tensorflow.js. The official guide is very clear, with concise code examples covering the core tensorflow.js concepts, including Tensors, Operations, Models, Layers, and training. Here with hands-on links, deepen the understanding of these concepts.

Article 1 concept

1.1 tensors

Tensorflow. js refers to n-dimensional arrays as tensor. See the diagram below for your understanding.


Hands-on code:

Var shape = [2, 3]; Var a = tf. Tensor (,2,3,4,4,5 [1], the shape); a.print()Copy the code

It can also be written as:

Var a = tf. Tensor ([[1, 2, 3], [4, four, five]]). a.print()Copy the code

Tf.scalar, TF.tensor1d, TF.tensor2d, TF.tensor3D, and TF.tensor4d can also be written to improve the readability of the code.

var a=tf.scalar(4); a.print(); Var b = tf. Tensor1d ([0, 2, 3, 4]); b.print(); Var c = tf. Tensor2d ([[0, 3], [4, 5]]). c.print();Copy the code

Tensorflow. js also has the ability to create a tensor with 0 or 1.

tf.zeros([10]).print(); Tf. Zeros ([2]). Print (); Tf. Zeros ([2, 2, 3]). Print (); tf.ones([10]).print(); Tf. 'ones ([2]). Print (); Tf. 'ones ([2, 2, 3]). Print ();Copy the code

1.2 the Variables

Tensors are immutable and cannot change their values once they are created; Variables, on the other hand, can change its value dynamically and is primarily used to store and update values during model training.

var initalValues=tf.ones([8]); initalValues.print(); Var biases=tf.variable(initalValues); var biases=tf.variable(initalValues); biases.print(); Var updatedValues = tf. Tensor1d (,1,2,3,4,5,6,7 [0]); updatedValues.print(); biases.assign(updatedValues); biases.print();Copy the code

1.3 Operations (Ops)

Some mathematical operations, matrix transformation, convolution operations, logic operations and so on. The official API documentation is complete, write very clearly (js.tensorflow.org/api/0.6.1/), practice the square and the add below:

/ / square var d = tf. Tensor2d ([[1, 2, 3], [4 and 6]]). d.print(); var d_squared=d.square(); d_squared.print(); / / add var a = tf. Tensor2d ([[1, 2, 3], [4 and 6]]). Var b = tf. Tensor2d ([,1,9 [3], [14,25,16]]). a.print(); b.print(); var c=a.add(b); c.print(); d.add(b).square().print();Copy the code

1.4 Models and the Layers

Models are equivalent to the concept of JS functions that, given some input, use Ops to represent what the model does, producing some desired output. Tensorflow.js has two ways of creating a model.

Function predict(input) {return tf.tidy(() => {const x = {return t = a * x ^ 2 + b * x + c tf.scalar(input); const ax2 = a.mul(x.square()); const bx = b.mul(x); const y = ax2.add(bx).add(c); return y; }); } var a = tf.scalar(2), b = tf.scalar(4), c = tf.scalar(8); // predict function predict(1999993). Print ();Copy the code

As Tensorflow. js is calculated using GPU, memory of GPU needs to be managed when using Tensors and variables. Among other things, tf.Tidy’s methods help avoid memory leaks (and program crashes). Try Tidy:

Var y = tf.tidy(() => {// a, b, and one will be cleared when tidy finishes. const one = tf.scalar(1); const a = tf.scalar(3); const b = a.square(); Console. log('tensors number (in Tidy): '+ tf.memory().numtensors); return b.add(one); }); Console. log('tensors number (outside Tidy): '+ tf.memory().numtensors); y.print();Copy the code

In addition to Tidy, dispose can be used to manually manage GPU memory. Under the test:

Const x = tf. Tensor2d ([[0, 2, 3], [1, 2, 3]]). const x_squared=x.square(); x.print(); x_squared.print(); console.log(tf.memory().numTensors); x.dispose(); console.log(tf.memory().numTensors); x_squared.dispose(); console.log(tf.memory().numTensors);Copy the code

Tensorflow.js also has some model abstractions built in, and you can use tf.model to construct a layer-free model. The layers included in TF include tf.layers.simpleRNN, tf.layers.gru and tf.layers. LSTM. This is going to have to be done with a couple of small projects.

2 Official Example

We can download the official sample and run it locally to see the effect. The official tensorflow. js project uses YARN as the package management tool and Parcel as the Web application packaging tool.

http://www.parceljs.io
Copy the code

To use the official example, decompress the file, go to the project directory, enter YARN, install the dependency package, and enter YARN watch to run the project.

There are several examples on the website. The first is simply to build a small model from scratch to fit the curve. The second demonstrated CNN recognition of handwritten numbers. The third uses transfer learning, training a neural network to predict camera data. The fourth section describes how to import Keras or TensorFlow-trained models into tensorflow.js for use. If you are interested, you can study in detail.

3 webcam-transfer-learning


One of the official game example Webcam-transfer-learning, suggested to play a play, is based on MobileNet an example of transfer learning.

3.1 MobileNet

MobileNets:

Efficient Convolutional Neural Networks for Mobile Vision Applications

This is proposed in a paper by Google, can greatly compress the model file size, very suitable for mobile terminal use. This paper uses Keras pre-trained image classification model MobileNet_25_224. By loading the trained KerAS model, the new model can be trained using the transfer learning directly in the browser or again in the browser.

Download the trained model first:

https://github.com/fchollet/deep-learning-models/releases/download/v0.6/mobilenet_2_5_224_tf.h5
Copy the code

Then the terminal runs:

pip install tensorflowjs
Copy the code

Then run:

tensorflowjs_converter --input_format keras mobilenet_2_5_224_tf.h5 model
Copy the code

After converting to tensorflow.js callable model, we need to place the model on a server and set it to allow cross-domain requests. Here we can use a nodeJS library:

npm install http-server -g
Copy the code

Enter the model folder and run:

http-server -p 3000 --cors
Copy the code

Loading model can be used:

Const model = await tf. LoadModel (' https://localhost:3000/model.json ');Copy the code

The official also very close put the model on the storage.googleapis.com/tfjs-models… Ready to call.

Here’S how MobileNet works:


3.2 Transfer Learning

Webcam-transfer-learning is an image classification problem, associating the photos taken by the camera with the movements up, down, left and right. Mainly training data collection: camera shooting, each image is normalized into a tensor of shape [1,244,244,3], as training data; For this reason, tensorflow.js specifically encapsulates the relevant methods of calling Webcam, so that it can be used directly in tensorflow.js. Transfer Learning is used to reduce the amount of training data and achieve the purpose of classification.


3.2.1 Preprocessing Load the pre-training model MoblieNet, and intercept appropriate layers as output. We have already described how to convert keras-trained models into tensorflow.js model format. Here we get it directly from the model service provided by Google.

Code:

async function loadMobilenet() { const mobilenet = await tf.loadModel( 'https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_0.25_224/model.json'); Console. log(mobilenet.layers) const layer = mobilenet.getLayer(' conv_PW_13_relu '); console.log(layer.output.shape) return tf.model({ inputs: mobilenet.inputs, outputs: layer.output }); }Copy the code

By calling getLayer(‘ conv_PW_13_relu ‘), we enter the inner layer of the pre-trained MobileNet model and build a new model with the same inputs as MobileNet, But the output is the MobileNet intermediate layer named conv_PW_13_relu. We chose this layer by rule of thumb (it worked well for our mission). In general, the layer near the end of the pretraining model will perform better in the transport learning task because it contains higher-level semantic features of the input. Try selecting another layer and see how it affects the quality of the model! You can use the model. Layers layer to print the model.

Add:

console.log(layer.output.shape)
Copy the code

The printout is [NULL, 7, 7, 256], and every time the user takes a photo, MobileNet is immediately called to output the conv_PW_13_RELu layer as input to the Model (red box above).

We will use this layer of MobileNet output as input to our newly created model, which outputs four categories of predictions. (Red box above)

 model = tf.sequential({
    layers: [   
      tf.layers.flatten({
        inputShape: [7, 7, 256]
      }),
      tf.layers.dense({
        units: ui.getDenseUnits(),
        activation: 'relu',
        kernelInitializer: 'varianceScaling',
        useBias: true
      }),
      tf.layers.dense({
        units: NUM_CLASSES,
        kernelInitializer: 'varianceScaling',
        useBias: false,
        activation: 'softmax'
      })
    ]
  });
Copy the code

Create two fully connected layer models, independent of the Mobilenet model. Train the new model according to 4 pictures taken by the user.

Tf.layers.flatten is used here. As for the use of tF.layers. Flatten, it can be practiced as follows:

model=tf.sequential(); Model. The add (tf. The layers. Flatten ({inputShape: [12, 4]})); Nn = model predict (tf. 'ones (,12,4 [99])); console.log(nn.shape); nn.print()Copy the code

The input is a THREE-DIMENSIONAL tensor with shape [99,12,4], and the final output is a two-dimensional tensor with shape [99, 48]. Flatten compresses [12,4] into [12X4].

4. Products based on user personalized data

Webcam-transfer-learning games give us a way to play based on user personalization data. Users can train their own image classification models at very low cost for various classification problems. We can expand, for example, to recognize the user’s gestures to control the characters in the game; Recognize the user’s expression and control the expression of 3D characters; Recognize the number of faces in the image, automatically hide the contents of the browsing, prevent prying eyes… Even apps like AutoDraw, UI2Code and handwriting recognition can try to integrate personalized data retraining into the user’s gameplay and give the user control.

I think every new technology has a natural new way of interacting with it. The browser-based user personalized data retraining can extract the following basic interaction processes:

Set the category

– > Collect data

— > Start training

– > Use user data

– > Core functions

— > Complete the task/get a result.

Users use their own data, the application is more in line with the characteristics of users personalized, is a “personalized” product design method different from personalized recommendation.

The above is the full text, and this article is also published on wechat official account: Design-AI-Lab. Recently, I was thinking about polishing articles as products, and set a small tone: one article should try to cover two different fields of content, and cross-think about the relevance between the two fields. Readers are welcome to communicate in the wechat group and leave a message in the group.