preface

This example describes the application of YOMO in industrial Internet data collection. Taking the data collection of noise sensor as an example, it involves the whole process of data collection/processing/workflow/data display. In order to facilitate the experience of operation effect, it will be containerized and the experience version will be quickly deployed through Docker.

predicate

  • Xxx-source: Represents a data source collector
  • XXX – Zipper: Represents a workflow and control plane
  • XX-flow: Represents a workflow unit for actual business logic processing, scheduled by Zipper.
  • XX-Sink: Represents the destination of a data transmission. It usually lands in the database or is sent to the next agent and is scheduled by the zipper.

architecture

As can be seen from the figure, there are two independent regions, the edge end and the cloud end, which are connected through the weak network or the Internet. Here is a brief introduction of each service:

  • The sensor device (Noise) and data collection gateway (ZLAN) are deployed at the edge. The gateway will request the status data from the device at a fixed time, and convert it into MQTT protocol data and send it to the noise-source collector. The source plays the role of coding conversion and connecting with the YOMO workflow engine Zipper. The selection and configuration of hardware for sensor devices and data collection gateways is available for this article: https://yomo.run/zh/aiot.
  • For developers who don’t want to buy hardware, a noise-emitter simulator is also available to generate noise data.
  • Zipper is a powerful workflow engine. Through Workflow.YAML, multiple flows and sinks can be scheduled, allowing them to connect business logic in series in the way of flow to meet complex requirements. All communication and codec connected with it are carried out by QUIC+Y3, providing reliable real-time streaming processing, and the whole experience of streaming programming fun.
  • Noise-flow implements the simple process of dividing the noise value by 10 and monitors the output log for alarm if a certain threshold is exceeded.
  • The noise-sink does not actually output to the database. Instead, a WebSocket server is built to output the real-time noise state to any web page for display and consumption.
  • Noise-web is a web service that consumes WebSocket. It can be deployed anywhere, as long as it has access to the WebSocket service address provided by noise-sink. We assume that deploying back to the edge is OK.

code

The following table provides all the code of the case, for interested friends to see, refer to the code of this case body, you can easily develop the case of similar scenario.

project address instructions
noise-source yomo-source-noise-example Collect noise data in the MQTT message format
noise-zipper yomo-zipper-noise-example Orchestrate the workflow and data flow of the body
noise-flow yomo-flow-noise-example Preprocessing and warning of noise data
noise-sink yomo-sink-socketio-server-example WebSocket services are provided for data presentation
noise-web yomo-sink-socket-io-example Consuming WebSocket services shows noise status
noise-emitter yomo-source-noise-emitter-example The simulation generates noise data
quic-mqtt yomo-source-mqtt-starter Develop common components for XX-Source

Containerized deployment

On the download section of the project code can undertake local native deployment, experience the delight of YoMo development, but for the friend that want to eager to see the effect immediately, in a better way, of course, is the first to look at the effect of the fast running, so the last project has also done a container, the root directory of every project provides Dockerfile file, The official image is available for download at hub.docker.com:

project Mirror address The latest version
noise-source yomorun/noise-source yomorun/noise-source:latest
noise-zipper yomorun/noise-zipper yomorun/noise-zipper:latest
noise-flow yomorun/noise-flow yomorun/noise-flow:latest
noise-sink yomorun/noise-sink yomorun/noise-sink:latest
noise-web yomorun/noise-web yomorun/noise-web:latest
noise-emitter yomorun/noise-emitter yomorun/noise-emitter:latest
quic-mqtt yomorun/quic-mqtt yomorun/quic-mqtt:latest

Yomorun/QUIC-MQTT: Latest is the basic image for developing XX-Source and allows for a quick package of custom code, but can be ignored for now in this case.

Rapid deployment

For a quick run experience, this section describes how to run the entire container on the same host.

fast

With the above official image, it is much easier and you can experience the effect in simple steps:

  • Download the docker-compose.yml file.
  • rundocker-compose up -d
  • Have a cup of tea and wait. Visit http://localhost:3000/ to see the image below:

Note:

  • The Delay value here may not be very accurate, because it is deployed through the Docker container, and the clocks of each container are not perfectly aligned. If you want to view the most accurate Delay value, you need to natively deploy both noise-source and noise-web to the same host.
  • If the deployment is not local, you need to modify the environment variable Socket \ _Server \ _Addr in the docker-compose. Yml file to the host address where you deploy the service.

Check the status

View the status of the service by docker-compose ps

Name Command State Ports ----------------------------------------------------------------------------------------- noise-emitter sh -c go run main.go Up noise-flow sh -c yomo run app.go -p 4242 Up 4242/udp noise-sink sh -c go run main.go Up 4141/udp, 0.0.0.0:8000->8000/ TCP noise-source sh-C Go Run Main. Go Up 1883/ TCP noise-web./docker-entrypoint.sh yar... Up 0.0.0.0:3000->3000/ TCP noise-zipper sh-c YOMO WF Run Workflow... Up 9999/udp
  • Noise – Sink exposes 8000 WebSocket port to the noise-web display for consumption.
  • The noise-web exposes the HTTP port of 3000 for displaying real-time noise values and latency.
  • Both noise-zipper/noise-flow/noise-sink provide QUIC service on UDP port, and the whole process of QUIC communication.
  • The noise-source is the key to our connection to different devices. The 1883 MQTT port is provided, and of course it can be modified.

See the log

  • View noise-emitter docker-compose log-f noise-emitter

    noise-emitter    | 2021-04-26 10:11:03: Publish counter=12438, topic=NOISE, payload={"noise":12438}
noise-emitter    | 2021-04-26 10:11:04: Publish counter=12439, topic=NOISE, payload={"noise":12439}
noise-emitter    | 2021-04-26 10:11:05: Publish counter=12440, topic=NOISE, payload={"noise":12440}
noise-emitter    | 2021-04-26 10:11:06: Publish counter=12441, topic=NOISE, payload={"noise":12441}

This analog generator produces MQTT data: the subject is' NOISE 'and the value is an increasing ordinal number (in JSON format).

  • View noise-source docker-compose logs-f noise-source

    noise-source     | 2021/04/26 15:27:32 receive: topic=NOISE, payload={"noise":2638}
noise-source     | 2021/04/26 15:27:32 write: sendingBuf=[]byte{0x81, 0x1b, 0x90, 0x19, 0x11, 0x3, 0x45, 0x24, 0xe0, 0x12, 0x6, 0xaf, 0x90, 0xe8, 0xce, 0x83, 0x1c, 0x13, 0xa, 0x31, 0x37, 0x32, 0x2e, 0x31, 0x39, 0x2e, 0x30, 0x2e, 0x36}
noise-source     | 2021/04/26 15:27:33 receive: topic=NOISE, payload={"noise":2639}
noise-source     | 2021/04/26 15:27:33 write: sendingBuf=[]byte{0x81, 0x1b, 0x90, 0x19, 0x11, 0x3, 0x45, 0x24, 0xf0, 0x12, 0x6, 0xaf, 0x90, 0xe8, 0xce, 0x8b, 0x5, 0x13, 0xa, 0x31, 0x37, 0x32, 0x2e, 0x31, 0x39, 0x2e, 0x30, 0x2e, 0x36}

-receive: Represents the MQTT data received by Source. -write: The bytecode transcoded by Y3 is sent to the noise-zipper workflow engine.

  • View noise-zipper docker-compose logs-f noise-zipper

    noise-zipper | 2021/04/26 14:39:28 Found 1 flows in zipper config noise-zipper | 2021/04/26 14:39:28 Flow 1: Noise Serverless on noise-flow:4242 noise-zipper | 2021/04/26 14:39:28 Found 1 sinks in zipper config noise-zipper | 2021/04/26 14:39:28 Sink 1: Socket.io Server on noise-sink:4141 noise-zipper | 2021/04/26 14:39:28 Running YoMo workflow... Noise - zipper | 2021/04/26 14:39:28 ✅ Listening on 0.0.0.0:9999 noise - zipper | 2021/04/26 14:43:32 ✅ Connect to noise Serverless (4242), whose - flow: successfully. The noise - zipper | 2021/04/26 14:44:33 ✅ Connect to Socket. IO Server (noise-sink:4141) successfully.
The workflow engine is connected to the processing units of the two workflows, noise-flow and noise-sink.

  • View noise-flow docker-compose log-f noise-flow

    Noise - flow | ❗ value: 561.700012 reaches the threshold 16! 45.700012 noise - flow | [172.19.0.6] 1619425035923 > value: 561.799988 ⚡ ️ = 0 ms noise - flow | ❗ value: The threshold 16! 45.799988 noise - flow | [172.19.0.6] 1619425036923 > value: 561.900024 ⚡ ️ = 1 ms
Noise data is generated by the simulator, well above the preset threshold, and a warning message is printed.

Partition deployment

After the rapid deployment in the last section, we should know how to make containerization deployment on the same host and check the status of each service. However, it is not consistent with the actual architecture diagram of separation between cloud and edge proposed at the beginning. Serverless services such as Flow/Sink will be deployed in the cloud in the real scene. While the data receiver source will be deployed at the edge, this section will separate them and see how to orchestrate deployment.

The cloud deployment

Deploy the service

The goal is to deploy noise-zipper/noise-flow/noise-sink in the cloud. Check out the following configuration files and running steps:

  • Download the docker-compose-cloud.yml file.
  • rundocker-compose -f docker-compose-cloud.yml up -d

Check the status

Check the status of the service with docker-compose -f docker-compose-cloud.yml ps

Name Command State Ports ---------------------------------------------------------------------------------------- noise-flow sh -c yomo run app.go -p 4242 Up 4242/udp noise-sink sh -c go run main.go Up 4141/udp, 0.0.0.0:8000->8000/ TCP noise-zipper sh-c YOMO WF Run Workflow... The Up 0.0.0.0:9999 - > 9999 / udp
  • The noise-zipper exposes the service port of the cloud workflow engine, which is clearly a UDP port since the communication is a QUIC protocol.
  • The noise-sink exposes the port for data consumption through the WebSocket, allowing other Web services to display consumption.

Edge side deployment

Deploy the service

The goal is to deploy noise-source/noise-web/noise-emitter on the edge. Check out the following configuration file and run steps:

  • Download the docker-compose-edge.yml file.
  • rundocker-compose -f docker-compose-edge.yml up -d

Note:

  • The SOCKET\_SERVER\_ADDR variable in docker-compose-edge.yml needs to be set to the address and port exposed by the source-sink on the cloud side (default 8000).

Check the status

Check the status of the service with docker-compose-f docker-compose-edge.yml ps

Name Command State Ports ------------------------------------------------------------------------------- noise-emitter sh -c go run main.go Up noise-source sh -c go run main.go Up 1883/tcp noise-web docker-entrypoint.sh sh -c ... The Up 0.0.0.0:3000 - > 3000 / TCP
  • The noise-web exposes the port of the effect display site. Go to http://localhost:3000/ to see the same presentation.

Router deployment

This is where it ends, right? Also have no! Edge deployment is actually more complicated than cloud deployment, since many edge devices are older and do not necessarily support Docker containers, so you can choose to compile the source code into binary runtime files for different platforms, and just put the noise-source on the corresponding platform. Of course, a more convenient choice is to buy a router that supports Docker container, which can be directly deployed on the edge end through the container. At present, YOMO and IKUAI have reached deep cooperation to jointly promote the application and development of industrial Internet in various fields. You can check the detailed information through the following link:

  • Ikuai + Yomo accelerate the era of cloud origin of industrial Internet
  • IKUAI + YOMO Edge Computing Enable Industrial Internet

Refer to the reference

The above is the whole code and deployment process for the noise sensor case from data collection and processing to demonstration. For developers who need to expand or extend to other application scenarios, you can click on the link of each project to read in detail. Each project is micro-service-oriented construction, with clear service role and clear code. If you have any questions, you are welcome to raise Issues or discuss. Reference link:

  • https://yomo.run/
  • https://github.com/yomorun/yomo
  • https://github.com/yomorun/example-noise

Finally, add an architecture diagram with the exposed port numbers of each service, so that friends who encounter difficulties can refer to it.