EMQ X Kuiper is an edge lightweight iot data analytics/streaming software that runs on a variety of resource-constrained iot devices.

TensorFlow Lite is a set of tools that help developers run TensorFlow models on mobile, embedded, and iot devices, enabling machine learning predictions on devices with low latency and small binary capacity.

By integrating Kuiper and TensorFlow Lite, users can analyze the data in the flow with AI that contains a pre-built TensorFlow model. In this tutorial, we walk you through building a Kuiper plug-in that identifies the TensorFlow model with pre-trained images and flags the streaming pictures (binary data) generated by edge devices.

Conditions to prepare

To run the TensorFlow Lite interpreter, we need a trained model. This tutorial does not cover training and covering the model, but you can learn more by looking at tflite Converter. We can either train a new model or choose one online. In this tutorial, we will use the Label Image model from Mattn/Go Tflite. The REPo creates the Golang binding for the Tflite C API. We will also use it to implement our plug-in.

Developing a plug-in

To integrate Kuiper with TensorFlow Lite, we will develop a custom Kuiper function plug-in for use with Kuiper rules. For example, we will create a LabelImage function whose input is binary type data representing the image and output is a string representing the image label. For example, if there is a peacock in the input image, LabelImage(COL) will output “peacock”.

To develop function plug-ins, we need:

  1. Create the plug-in GO file. For example, in kuiper source code, createplugins/functions/labelImage/labelImage.goFile.
  2. Create a struct that implements the API. Function interface.
  3. Export the struct.

The key to implementation is the Exec function. The pseudocode is as follows:

func (f *labelImage) Exec(args []interface{}, ctx api.FunctionContext) (interface{}, bool) {
	
    / /... Initialization and validation

    // Decode the input image
    img, _, err := image.Decode(bytes.NewReader(arg[0]))
    iferr ! =nil {
      return err, false
    }
    var outerErr error
    f.once.Do(func(a) {		
      // Load the tag, tflite model, and initialize the Tflite interpreter
    })

		// Run the interpreter on the input image
		// Return the most likely tag
 		return result, true
}
Copy the code

You also need to pay attention to plug-in exports. This function is stateless, so we will export only one struct instance. All rules that use this function share an instance to avoid the overhead of creating instances and loading models. The model and label paths are specified at instantiation time.

var LabelImage = labelImage{
    modelPath: "labelImage/mobilenet_quant_v1_224.tflite",
    labelPath: "labelImage/labels.txt",}Copy the code

Consult this tutorial for detailed steps to create a Kuiper plug-in. See LabelImage.go for the complete source code.

Build and install the plug-in

To use the plug-in, we need to build it in an environment where Kuiper is running, and then install it in Kuiper.

Through a pre-built ZIP installation

If we use a Debian-based Kuiper Docker image with a 1.1.1 or 1.1-SLIM tag, we can install the pre-built labelImage plug-in. For example, to install the kuiper 1.1.2 plug-in in Docker Image emqx/ Kuiper :1.1.2-slim, the pre-built zip file is located at www.emqx.io/downloads/k… Rest command to install.

POST http://{{kuiperHost:kuiperRestPort}}/plugins/functions
Content-Type: application/json

{"name":"labelImage", "file": "https://www.emqx.io/downloads/kuiper-plugins/v1.1.2/debian/functions/labelImage_amd64.zip"}
Copy the code

Manual build

  1. If you do not run Kuiper with an official Kuiper Docker image, the pre-built labelImage plug-in will not apply due to the golang plugin’s limitations. You need to build the plug-in manually. There are three steps to manually create the plug-in zip file:
    1. Build the TensorFlowLite C API.
    2. Build the labelImage plug-in.
    3. Package the plug-in with the installation script.

Build the TensorFlowLite C API

There is a very simple tutorial on building a C API from the TensorFlow repo. We will expand in detail step by step in this section. Note that this plug-in is only being tested against TensorFlow V2.0-RC3, so we will build from that version. In the case of Ubuntu, here are the build steps:

  1. Install Python 3.

  2. Copy requirements.txt to the location you specify. Install the required Python library: pip3 install -r requirements.txt. Requirements from corresponding TensorFlow version TensorFlow/TensorFlow/tools/pip_package/setup. Py.

  3. Install TensorFlow’s build tool Bazel.

  4. Clone the Tesorflow repo and switch to the desired branch with git checkout v2.0-rc3 -b mybranch.

  5. Generate the target.so file, and the output will be in./bazel-bin. Copy the two so files into the tensorflow/lib folder.

    $ cd $tensorflowSrc
$ bazel build --config monolithic -c opt //tensorflow/lite:libtensorflowlite.so
$ bazel build --config monolithic -c opt //tensorflow/lite/c:libtensorflowlite_c.so
$ mkdir lib
$ cp bazel-bin/tensorflow/lite/libtensorflowlite.so lib
$ cp bazel-bin/tensorflow/lite/c/libtensorflowlite_c.so lib
Copy the code

  6. Install the so file.

    1. Update the LDConfig filesudo vi / etc / ld.so.conf.d / tflite.conf.
    2. The paths{{tensorflowPath}}/libAdd to tflite.conf, then save and exit.
    3. Run ldconfig:sudo ldconfig.
    4. Check the installation result:ldconfig -p | grep libtensorflow. Make sure two SO files are listed.

Build the labelImage plug-in

Ensure that the Kuiper Github repo is cloned. Plug-in source file is located in the plugins/functions provides/labelImage/labelImage. Go. Before building the plug-in, export the TensorFlow repo and build the library path.

$ cd {{kuiperRepoPath}}
$ export CGO_CFLAGS=-I/root/tensorflow
$ export CGO_LDFLAGS=-L/root/tensorflow/lib
$ go build -trimpath --buildmode=plugin -o plugins/functions/LabelImage.so plugins/functions/labelImage/*.go
Copy the code

By these commands, the plugin will be built into the plugins/functions provides/LabelImage. So. For development purposes, you can restart Kuiper to automatically load this plug-in and test it. Once the test is complete, we should package it as a ZIP file that can be used by the Kuiper plug-in installation API so that it can be used on other machines (such as production environments).

Packaging plug-in

The plugins/functions provides/labelImage catalogue of all the files and directories and build labelImage. So packaged together into a zip file. Zip files should have a file structure similar to:

  • etc
    • labels.txt
    • mobilenet_quant_v1_224.tflite
  • lib
    • libtensorflowlite.so
    • libtensorflowlite_c.so
  • install.sh
  • LabelImage.so
  • tflite.conf

Install the packaged plug-in to the target system, as shown in the pre-built ZIP installation.

Run the plugin

Once the plug-in is installed, we can use it in our rules. We will create a rule to receive image bytes from the MQTT topic and mark the image with the TFLite model.

Define the flow

Define flows through the Kuiper REST API. Let’s create a stream called TFdemo in binary format and subject tfdemo.

POST http://{{host}}/streams
Content-Type: application/json

{"sql":"CREATE STREAM tfdemo () WITH (DATASOURCE=\"tfdemo\", FORMAT=\"BINARY\")"}
Copy the code

Define the rules

Define rules through the Kuiper REST API. We will create a rule called ruleTf. We simply read the image from the TFdemo stream and run the custom labelImage function on it. The return result will be the label of the image identified by the AI.

POST http://{{host}}/rules
Content-Type: application/json

{
  "id": "ruleTf",
  "sql": "SELECT labelImage(self) FROM tfdemo",
  "actions": [
    {
      "log": {}
    }
  ]
}
Copy the code

The input data

Here, we create a GO program that sends image data to the TFDemo theme for processing by rules.

package main import ( "fmt" mqtt "github.com/eclipse/paho.mqtt.golang" "io/ioutil" "time" ) func main(){ const TOPIC = "tfdemo" images := []string{ "peacock.png", "frog.jpg", Opts := mqtt.newClientOptions ().addBroker (" TCP :// YouRownHost :1883") client := mqtt.newClient (opts) if token := client.Connect(); token.Wait() && token.Error() ! = nil { panic(token.Error()) } for _, image := range images { fmt.Println("Publishing " + image); payload, err := ioutil.ReadFile(image) if err ! = nil{ fmt.Println(err) continue } if token := client.Publish(TOPIC, 0, false, payload); token.Wait() && token.Error() ! = nil { fmt.Println(token.Error()) } else { fmt.Println("Published " + image); } time.Sleep(1 * time.Second) } client.Disconnect(0) }Copy the code

Run pub.go and it will start entering images into the TFdemo theme.

Check the result

Because our rule definition has only one target: log, the results will be written to a log file. We fill the stream with two images peacock. PNG and frog. PNG. Examining the log files, we find:

time="2021-02-05 16:23:29" level=info msg="sink result for rule ruleTf: [{\"labelImage\":\"peacock\"}]" file="sinks/log_sink.go:16" rule=ruleTf
time="2021-02-05 16:23:30" level=info msg="sink result for rule ruleTf: [{\"labelImage\":\"bullfrog\"}]" file="sinks/log_sink.go:16" rule=ruleTf
Copy the code

The image is labeled correctly.

conclusion

Through this tutorial, you can build a custom Kuiper plug-in to use a pre-trained TensorFlow Lite model. If you need to use another model, just follow the prescribed steps to create another function. Note that the built TensorFlow C API can be shared between all functions if they are running in the same environment. Hopefully, this article will help you better leverage AI in edge computing and empower your edge devices.

Copyright: EMQ

Original link: www.emqx.cn/blog/run-te…