Tensorflow.js is a new version of the popular open source library that brings deep learning to JavaScript. Developers can now use high-level library apis to define, train, and run machine learning models.
With pre-trained models, developers can easily perform complex tasks, such as visual recognition, generating music or detecting human posture, with just a few lines of JavaScript.
Starting with the front-end library for Web browsers, recent updates have added experimental support for Node.js. This allows tensorflow.js to be used in back-end JavaScript applications without having to use Python.
Reading about the library, I wanted to test it with a simple task…
Use tensorflow.js for image recognition through Node.js
Unfortunately, most of the documentation and sample code uses libraries in the browser. Node.js has not yet extended the project to simplify loading and using the pretrained model. Getting this done did cost me a lot of time reading the Typescript source files for the library. 👎
But, after several days of studying day and night, I finished! Cheers! 🤩
Before diving into the code, let’s give an overview of the different TensorFlow libraries.
TensorFlow
TensorFlow is an open source software library for machine learning applications. TensorFlow can be used to implement neural networks and other deep learning algorithms.
TensorFlow was released by Google in November 2015 and was originally a Python library. It uses CPU – or GPU-based computing to train and evaluate machine learning models. The library was originally designed to run on high-performance servers with expensive Gpus.
Recent updates have extended the software to run in environments such as mobile devices and Web browsers.
TensorFlow Lite
Tensorflow Lite is a lightweight version of the mobile and embedded device library, released in May 2017. The release comes with a new set of pre-trained deep learning models for visual recognition tasks called MobileNet. The MobileNet model is designed to run efficiently in resource-constrained environments, such as mobile devices.
TensorFlow.js
Following Tensorflow Lite, tensorflow.js was released in March 2018. This version of the library is designed to run in a browser and builds on an earlier project called deeplearn.js. WebGL provides GPU access to libraries. Developers use JavaScript apis to train, load, and run models.
Tensorflow.js recently ran on Node.js via the tfJS-Node extension library.
The Node.js extension is an Alpha release and is still under active development
Import existing models into tensorflow.js
The existing TensorFlow and Keras models can be executed using the tensorflow.js library. The model needs to be converted to the new format using this tool before execution. Github provides pre-training and transformation models for image classification, posture detection, and K-nearest neighbors.
Use tensorflow.js in nodeJS
Install tensorflow library
Tensorflow.js is installed in the NPM library.
Tensorflow/TFJS – Core tensorflow.js package
@tensorflow/ tfJs-node – Node.js extension of tensorflow.js
@tensorflow/ tfJS-node-gpu-tensorflow. js NodeJS extension supported by the GPU
npm install @tensorflow/tfjs @tensorflow/tfjs-node
// or...
npm install @tensorflow/tfjs @tensorflow/tfjs-node-gpu
Copy the codeBoth Node.js extensions use native dependencies, which are compiled as needed.
Loading tensorflow library
TensorFlow’s JavaScript API is available to users. Extension modules supported by Node.js do not expose other apis.
const tf = require('@tensorflow/tfjs')
// Load the binding (CPU computation)
require('@tensorflow/tfjs-node')
// Or load the binding (GPU computation)
require('@tensorflow/tfjs-node-gpu')
Copy the codeLoad the TensorFlow model
Tensorflow.js provides an NPM library (TFJS-Models) that can easily load pre-trained and transformed models for image classification, posture detection and K-nearest neighbors.
The MobileNet model for image classification is a deep neural network trained for 1000 different categories.
In the readme file of the project, the following sample code is used to load the model.
import * as mobilenet from '@tensorflow-models/mobilenet';
// Load the model.
const model = await mobilenet.load();
Copy the codeThe first challenge I encountered was that this didn’t work on Node.js.
Error: browserHTTPRequest is not supported outside the web browser.
Copy the codeLooking at the source code, the Mobilenet library is a wrapper around the base TF.Model class. When the load () method is called, it automatically downloads the correct model file from an external HTTP address and instantiates the TensorFlow model.
The Node.js extension does not yet support HTTP requests to dynamically retrieve the model. Instead, the model must be loaded manually from the file system.
After reading the source code for the library, I managed to create a workaround…
Load the model from the file system
If you manually create the MobileNet class, instead of calling the module’s load method, you can use the local file system path to override the automatically generated path variable that contains the HTTP address of the model. When this is done, calling the Load method on the class instance triggers the file system loader class instead of trying to use a browser-based HTTP loader.
const path = "mobilenet/model.json"
const mn = new mobilenet.MobileNet(1, 1);
mn.path = `file://${path}`
await mn.load()
Copy the codeGreat, it works!
But where do model files come from?
MobileNet model
The model of Tensorflow.js contains two file types, one is the model configuration file stored in JSON, and the other is the model weights in binary format. Model weights are typically split into multiple files for better caching by browsers.
View the auto-loaded code for the MobileNet model to retrieve the model configuration and weight fragments from the common store at that address.
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v${version}_${alpha}_${size}/
Copy the codeThe template parameter in the URL refers to the model version listed here. The page also shows the classification accuracy results for each version.
According to the source code, only the Mobilenet V1 model can be loaded using the TensorFlow-Models/Mobilenet library.
The HTTP retrieval code loads the model.json file from this location and then recursively retrieves all referenced model weight fragments. These files are in the format groupx-SHARd1of1.
Manually downloading models
You can save all model files to the file system by retrieving the model configuration files, parsing the referenced weight files, and manually downloading each weight file.
I want to use the MobileNet V1 module with an alpha value of 1.0 and an image size of 224 pixels. This gives me the following URL for the model configuration file.
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/model.json
Copy the codeAfter downloading this file locally, I can use the JQ tool to resolve all file names.
$ cat model.json | jq -r ".weightsManifest[].paths[0]"
group1-shard1of1
group2-shard1of1
group3-shard1of1
...
Copy the codeUsing sed, I can prefix these names with HTTP URLS, generated urls for each file.
$ cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's / ^ / https:\/\/storage.googleapis.com \ / TFJS - models \ / TFJS \ / mobilenet_v1_1 0 _224 \ / /'https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group1-shard1of1 https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group2-shard1of1 https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group3-shard1of1...Copy the codeThen, using the parallel and curl commands, you can download all of these files to a local directory.
cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's / ^ / https:\/\/storage.googleapis.com \ / TFJS - models \ / TFJS \ / mobilenet_v1_1 0 _224 \ / /' | parallel curl -O
Copy the codeImage classification Tensorflow.js provides this sample code to demonstrate the return classification of images.
const img = document.getElementById('img');
// Classify the image.
const predictions = await model.classify(img)
Copy the codeNot applicable to Node.js due to the lack of DOM.
The classify method accepts many DOM elements (canvas, video, image) and automatically retrieves image bytes from these elements and converts them to the TF.tensor3D class, which is used as input to the model. Alternatively, you can pass the TF.tensor3D input directly.
Instead of trying to emulate DOM elements in Node.js using external packages, I found it easier to build TF.tensor3D manually.
Read the source code for the method used to convert DOM elements into Tensor3D classes from image Generation Tensor3D. The following input parameters are used to generate the Tensor3D class.
const values = new Int32Array(image.height * image.width * numChannels);
// fill pixels with pixel channel bytes from image
const outShape = [image.height, image.width, numChannels];
const input = tf.tensor3d(values, outShape, 'int32');
Copy the codePixel is a 2D array of type (Int32Array) that contains a sequential list of channel values for each pixel. NumChannels is the number of channel values per pixel.
Create input values for JPEG
The JPEG-JS library is a pure JavaScript JPEG encoder and decoder for Node.js. Using this library, you can extract the RGB value for each pixel.
Const pixel = jpeg. Decode (buffer,true);Copy the codeThis returns a Uint8Array with four channel values (RGBA) for each pixel (width * height). The MobileNet model uses only three color channels (RGB) for classification, ignoring the alpha channel. This code converts the four-channel array to the correct three-channel version.
const numChannels = 3;
const numPixels = image.width * image.height;
const values = new Int32Array(numPixels * numChannels);
for (let i = 0; i < numPixels; i++) {
for (letchannel = 0; channel < numChannels; ++channel) { values[i * numChannels + channel] = pixels[i * 4 + channel]; }}Copy the codeMobileNet model input requirements
The MobileNet model used classifies images with a width and height of 224 pixels. For each of the three channel pixel values, the input tensor must contain floating point values between -1 and 1.
Before classification, it is necessary to readjust the input values of images of different sizes. Also, the pixel values from the JPEG decoder are in the range 0-255, not -1 to 1. These values also need to be converted before classification.
Tensorflow.js has some library methods to simplify this process, and fortunately, the TFJS-Models/Mobilenet library handles this automatically! 👍
Developers can pass the INT32 type and Tensor3D input of different dimensions to the classification method and then convert the input to the correct format prior to classification. This means super 🕺🕺🕺.
To predict
The MobileNet model in Tensorflow is trained to recognize entities in the top 1,000 categories in the ImageNet dataset. The model outputs the probability of each of these entities in the classified image.
A complete list of trained courses for the models in use can be found in this file.
The TFJS-Models/Mobilenet library exposes a classification method on the Mobilenet class to return the top X classes with the highest probability from the image input.
const predictions = await mn_model.classify(input, 10);
Copy the codePredictions is an array of class X and probability forms, of the following format.
{
className: 'panda', 0.9993536472320557} aim-listed probability:Copy the codeexample
After exploring how to use the tensorflow.js library and the MobileNet model on Node.js, the script will sort the images given as command-line arguments.
The source code saves this script file and the package descriptor to a local file.
//package.json
{
"name": "tf-js"."version": "1.0.0"."main": "script.js"."license": "MIT"."dependencies": {
"@tensorflow-models/mobilenet": "^ 0.2.2"."@tensorflow/tfjs": "^ 0.12.3"."@tensorflow/tfjs-node": "^ 0.1.9"."jpeg-js": "^ 0.3.4"}}Copy the code//script.js
const tf = require('@tensorflow/tfjs')
const mobilenet = require('@tensorflow-models/mobilenet');
require('@tensorflow/tfjs-node')
const fs = require('fs');
const jpeg = require('jpeg-js');
const NUMBER_OF_CHANNELS = 3
const readImage = path => {
const buf = fs.readFileSync(path)
const pixels = jpeg.decode(buf, true)
return pixels
}
const imageByteArray = (image, numChannels) => {
const pixels = image.data
const numPixels = image.width * image.height;
const values = new Int32Array(numPixels * numChannels);
for (let i = 0; i < numPixels; i++) {
for (letchannel = 0; channel < numChannels; ++channel) { values[i * numChannels + channel] = pixels[i * 4 + channel]; }}return values
}
const imageToInput = (image, numChannels) => {
const values = imageByteArray(image, numChannels)
const outShape = [image.height, image.width, numChannels];
const input = tf.tensor3d(values, outShape, 'int32');
return input
}
const loadModel = async path => {
const mn = new mobilenet.MobileNet(1, 1);
mn.path = `file://${path}`
await mn.load()
return mn
}
const classify = async (model, path) => {
const image = readImage(path)
const input = imageToInput(image, NUMBER_OF_CHANNELS)
const mn_model = await loadModel(model)
const predictions = await mn_model.classify(input)
console.log('classification results:', predictions)
}
if(process.argv.length ! == 4) throw new Error('incorrect arguments: node script.js <MODEL> <IMAGE_FILE>')
classify(process.argv[2], process.argv[3])
Copy the codetest
-
Follow the instructions above to download the model file to the Mobilenet directory.
-
Install project dependencies using NPM
npm install
- Download sample JPEG files for classification
wget http://bit.ly/2JYSal9 -O panda.jpg
- Run the script using the model file, then enter an image as a parameter.
node script.js mobilenet/model.json panda.jpgIf all is well, the following output should be printed to the console.
classification results: [ {
className: 'giant panda, panda, panda bear, coon bear', probability: 0.9993536472320557}]Copy the codeThe image was correctly classified as containing pandas with a 99.93% probability! 🐼 🐼 🐼
conclusion
Tensorflow.js brings deep learning to JavaScript developers. Using pre-trained models in the tensorflow.js library makes it easy to extend JavaScript applications with complex machine learning tasks with minimal effort and code.
Tensorflow.js has been released as a browser-based library and has been expanded to run on Node.js, although not all tools and utilities support the new runtime. After a few days of research, I was able to use the library with the MobileNet model for visual recognition of images in local files.
Having it run in the Node.js runtime means I now move on to the next step… Make it run in serverless functionality! Come back soon to read about tensorflow.js’ next adventure. 👋