preface

YoMo is an open source Programming framework, built for low-delay streaming data processing in the field of edge computing. Its bottom layer is based on IETF QUIC protocol communication in the core communication layer of HTTP 3.0, and Functional Reactive Programming is used as the Programming paradigm. It is convenient for developers to build reliable and secure real-time computing applications for time-series data, and is optimized for 5G and wiFI-6 scenarios to release real-time computing value.

Y3 is a Golang implementation of the YoMo Codec, which describes a fast and low CPU loss Codec that focuses on edge computing and stream processing. Check out Explainer for more information and more ways to combine with YoMo.

CluingOS is a cloud-native hyper-converged industrial iot platform with Kubernetes as the core. Its architecture can easily integrate, integrate and install third-party applications and cloud native ecological components, and support unified distribution, operation and maintenance management of cloud native applications in multi-cloud and multi-cluster.

In this case, we combined the YoMo + Y3 low-latency streaming with CluingOS distributed deployment features, show how to develop the deployment of a set of efficient industrial data collection application system, experience from the edge of the sensor data collection, low latency across more than 2000 kilometers to efficiently transport flow processing the whole process of the data to the cloud, Based on this example, you can follow this example and develop your own application scenarios to suit your own needs.

predicate

  • Xxx-source: indicates a data source collector that can directly receive MQTT data.
  • Xxx-zipper: represents a workflow and control plane.
  • Xxx-flow: Represents a workflow unit that is used for actual business logic processing and is scheduled by zipper.
  • Xxx-sink: indicates a data transfer destination. In this case, it is a WebSocket service that consumes data and is scheduled by zipper.
  • Xxx-web: represents a Web service that displays real-time sensor data.

architecture

As can be seen from the example in the figure, two separate service areas are divided between the edge and the cloud. The edge is located in Shanghai and the cloud is located in Guangzhou, which are more than 2,000 kilometers apart. We can see in the final test that its low-latency streaming processing provides surprising optimization for data collection and processing. In addition, the container distributed deployment of CluingOS hyper-converged industrial Internet of Things platform can effectively deploy and debug our application, so sweet! Here is a brief overview of each module and service:

The sensor

Vibration sensor, membrane key sensor

  • Vibration sensor. The raw data used to generate vibration-related data is converted to MQTT protocol data by Lora receiver in the following format:

    • TOPIC: shake / 20210627 _cluing/S07

    • Content:

      {
	// Tenant database instance
	"tenantId": "20210627_cluing".// Collection device terminal DEVEUI
	"devEui": "0850533277387820".// Collect raw data
	"data": "CwcMys+69Iks0As4YS4N6A==".// Data collection time
	"createDate": 1624937248919.// Collect the temperature
	"temperature": 75.// Z axis vibration strength
	"vertical": 81.//X axis vibration intensity
	"transverse": 53
}
Copy the code

  • Membrane key sensor. Used to generate key-related raw data, which is converted to MQTT protocol data by Loar receiver. The data format is as follows:

    • TOPIC: shake / 20210627 _cluing/S05

    • Payload:

      {
        // Tenant database instance
	"tenantId": "20210627_cluing".// Collection device terminal DEVEUI
	"devEui": "393235307d377504".// Collect raw data
	"data": "AAAQ5gAAAQIIAA==".// Data collection time
	"createDate": 1624937248919.// Key device key value
	"key": "0800"
}
Copy the code

5G CPE all-in machine

Lora Receiver, forwarding engine, Shake-source, shake-web

5G CPE all-in-one is a gateway device deployed at the edge. It can receive sensor data and convert it into MQTT protocol data. Our YoMo edge receiver Shake-source is deployed on this gateway device. One of its features is that it accepts the Container deployment and resource scheduling of The CluingOS hyper-converged Iiot platform, so you can easily distribute the application to the gateway device even if you are thousands of miles away, without the need for remote login.

  • Lora receiver is the default service for the gateway. Monitoring data from a variety of sensors can be received and converted to the MQTT protocol, data from different devices can be allocated to different topics for easy management, and data from the vibration sensor and membrane key sensor in the previous section can be converted to the data format shown in the definition.
  • The forwarding engine is the default service of the gateway. It is possible to forward MQTT data to different MQTT Broker services, including shake-source in our case.
  • Shake-source Indicates the data source. Data source receiving service developed based on YoMo framework, its function is to convert MQTT protocol data into Y3 data format and transmit it to the cloud shake-Zipper workflow engine in QUIC way.
  • Shake-web Data display. This is a display Web service showing the real-time data of the two sensors, mainly consuming the WebSocket data provided by Shake-sink, and also showing a complete real-time RTT round-trip delay.

SaaS services

Shake-zipper, Shake-Flow, Shake-Sink, and HOMEY Lean event management system

This is a complete set of cloud services, in which the container deployment is also scheduled and managed by CluingOS, just need to log in CluingOS with different users can switch management at the edge or in the cloud services.

  • Shake-zipper workflow engine. Choreography (workflow.yaml) allows you to schedule multiple flows and sinks to connect business logic as flows to meet complex requirements. All the communication and codec connected with it are carried out by QUIC+Y3, providing reliable real-time streaming processing, and experiencing the fun of streaming programming in the whole process.
  • Shake-flow Logical processing unit. In this case, the processing unit decodes the data transmitted from the source into Topic and Playload, and then processes the data of vibration sensor and membrane key sensor respectively, and calls HOMEY to call the US management system for alarm or send control instructions to control the edge devices after reaching a certain threshold.
  • Shake-sink data output unit. In this case, rather than output to a database, a WebSocket server is built to output real-time sensor data to any web page for display and consumption. The data here is presented as a data source for shake-Web.
  • HOMEY Management system. After receiving the event notification of Shake-Flow, the callus system will send an alarm message or send control commands to the edge to control certain devices to perform certain operations. However, in this case, we will use the time point when the event notification is received by the calling Us system as the statistical end point of delay, and analyze that our low delay based on YoMo+Y3 is indeed greatly optimized.

CluingOS hyper-converged industrial iot platform

You can log in as different users to switch the deployment and management of edge services or cloud services

CluingOS provides industrial Internet of Things big data intelligent platform services and an end-to-end integrated transparent factory system of “on-site collaboration + process control + data intelligence” based on container, subscription and micro-service architecture, and supports multiple ways and distributed deployment implementation of private cloud, public cloud or hybrid cloud.

code

Project Yomo-Cluing-Shake provides a full set of source code, the following table provides a brief description of each module, for interested friends to see, referring to the code of this case, can easily develop the case of class simulation scenario.

The module address Run locally instructions
zipper zipper make debug_zipper Orchestrate the workflow and data flow of this case
flow flow make debug_flow Preprocessing and alarm of sensor data
sink sink make debug_sink Provide WebSocket services for data display
source source make debug_source Collect sensor data in MQTT message format
emitter emitter make debug_emitter Simulation generates vibration and keystroke data
web web make debug_web Consuming WebSocket services to display sensor data
quic-mqtt yomo-source-mqtt-starter Develop generic components for XXX-source

Container deployment

By downloading the project code in the previous section, it can be run locally quickly and experience the fun of YoMo development. Meanwhile, we provide the corresponding Dockerfile file for each module to package the corresponding image and upload it to Hub.dockder.com for CluingOS to deploy.

The module Dockerfile Mirror address The latest version
zipper Dockerfile.zipper shake-zipper yomorun/shake-zipper:latest
flow Dockerfile.flow shake-flow yomorun/shake-flow:latest
sink Dockerfile.sink shake-sink yomorun/shake-sink:latest
source Dockerfile.source noise-source yomorun/shake-source:latest
emitter Dockerfile.emitter shake-emitter yomorun/shake-emitter:latest
web Dockerfile.web shake-web yomorun/shake-web:latest
quic-mqtt yomorun/quic-mqtt yomorun/quic-mqtt:latest

Yomorun/quic-MQTT: Latest is the base image for packaging XXX-source and can be used to quickly package custom code, but can be ignored for the time being in this case.

CluingOS deployment

The CluingOS Industrial hyperconvergence system can manage containers on the edge and cloud by different users logging in.

Cloud Service Deployment

User A deplores the Zipper/Flow /sink service to the cloud.

  1. Creating a custom application.

    Create a home-made app shake-Cloud to manage services in the cloud.

  2. Create a service.

    Go to the console of the Shake-Cloud application and select Add Service or Stateless Service to enter the process of creating a service.

    • Basic information: The service name is shake-sink

    • Container image:

      • Add container image: Select shake-sink from DockerHub

      • Port Settings: Specifies the service port exposed by the container, for example, 8000

      • Environment variables: For example SHAKE_ZIPPER_ADDR = shake-down zipper. Yomo cluing – shake: 9000, Here shake-zipper. Yomo-cluing-shake is the internal DNS name of the zipper obtained after shake-zipper is created.

    • Mount storage and advanced Settings: Neither need to be set in this case.

    • Edit external network access: Select the NodePort access mode to obtain the exposed port number 30095

  3. Service list

    At this point, the zipper/ Flow/Sink services are created respectively.

Deploy edge services

User B deploies source/web services to the edge 5GCPE fusioncube.

The deployment mode is the same as the cloud service deployment mode. First, create a Shake-Edge application, and then create stateless services in the application to obtain the service list:

Access sensor display effect

The forwarding engine forwards MQTT messages to the Shake-Souce service.

To access sensor data, you only need to change the forwarding address of the forwarding engine to the address port of the Shake-source service, and then the real-time data received can be displayed through shake-web.

Effect of contrast

In order to compare the delay of data reported by traditional HTTP, test cases are designed to compare the effect.

The integration with YOMO is carried out in real environment, and all relevant application components are deployed and installed in container using CluingOS. The scenario covers cloud – edge – end applications. In addition to verifying the acceleration effect of YOMO, yOMO is also integrated into the Lingsi platform system (CluingOS/AIOT/MOM). The following is the main introduction of the acceleration test effect of YOMO.

The actual scene

As you can see from the test architecture diagram, two sets of tests are used for comparison:

QUIC + Y3 channel

As shown in orange in the figure, the data transfer path is:

HTTP channel

As shown in the green process in the figure, the data transmission path is:

The environment that

  • 5G CPE all-in-one machine is placed in Shanghai Jinqiao office.
  • MOM intelligent manufacturing operation management system is deployed in Tencent Cloud in Guangzhou.
  • The test delay link is the network between Shanghai Jinqiao Office and Guangzhou Tencent Cloud.

The test method

  • Prepare sensors A (3932353062376611) and B (0850533277387820).
  • Sensor A uses the traditional HTTP protocol, and sensor B uses QUIC+Y3 protocol.
  • Each sensor transmits 600 records, and the average delay is calculated.

A formula to calculate

select device_sn,(timeCount/count)*1000 agv,count from(
  select device_sn,sum(unix_timestamp(end_time) -unix_timestamp(begin_time)) as timeCount,
  count(id) as count from sdm_device_logs_copy2 GROUP BY device_sn
) t
Copy the code

The test results

The serial number Protocol type Equipment coding Average delay (ms) Number of test records
1 HTTP Sensor A (3932353062376611) 100.268333 (ms) 600
2 QUIC+Y3 Sensor B (0850533277387820) 33.493333 (ms) 600

From the collected samples, the average delay of 600 data transmitted by each sensor is obtained by using the calculation formula. It can be seen that the average delay of HTTP protocol is 100.26ms, while that of QUIC+Y3 protocol is only 33.49ms, which has a very obvious acceleration effect.

conclusion

In recent years, the development of the new generation of information technology advances by leaps and bounds, the Internet from the consumer field to the industrial field accelerated expansion. From the perspective of digital industrialization, the industrial Internet wants to develop to a larger scope, deeper degree and higher level, and it is in urgent need of new technologies, products and solutions. YoMo open source programming framework can greatly improve the transmission efficiency from the edge to the cloud, improve the real-time performance and obtain the advantages of low latency. At the same time, the new streaming computing and programming paradigm provides developers with a new development experience, which enables them to develop streaming computing applications more naturally and efficiently. With the help of Lingsi’s CluingOS hyper-converged industrial Internet of Things platform, container services can be quickly and easily deployed on the edge and cloud, and service governance capability can be greatly improved.