The Internet of Things data acquisition involves the access of a large number of devices and the transmission of massive time series data. The combined technology stack of EMQ X MQTT server and TDengine big data platform is fully capable of the transmission, storage and calculation of massive time series monitoring data in scenarios.

After data is stored in the database, other methods such as data visualization system are often needed to statistics and display data according to rules to achieve data monitoring, indicator statistics and other business requirements, so as to give full play to the value of data. TDengine and open source software Grafana can quickly build a data visualization platform for the Internet of Things.

The whole package of solutions mentioned above does not require code development, and the products involved can provide different delivery modes of open source software, enterprise services and cloud SaaS services. According to project requirements, the free version or enterprise version can be privatized and cloud deployment can be realized.

Plan to introduce

Introduction of EMQ X

EMQ X is an open source MQTT messaging server based on the highly concurrent Erlang/OTP language platform, supporting millions of connections and distributed cluster architecture. EMQ X has a number of built-in functions out of the box, with both the open source EMQ X Broker and the Enterprise EMQ X Enterprise offering support for storing device messages to TDengine through the rules engine.

What is TDengine

TDengine is a big data platform designed and optimized by Taos data for Internet of Things, Internet of vehicles, industrial Internet, IT operation and maintenance. In addition to the timing database function, which is more than 10 times faster than the core, it also provides caching, data subscription, streaming computing and other functions to minimize the complexity of research and development, operation and maintenance. The core code, including cluster function, is all open source.

TDengine is available in community, enterprise, and cloud services editions. See the TDengine Usage documentation for installation/usage instructions.

Grafana profile

Grafana is a cross-platform, open source measurement and visualization tool that queries and processes data from a variety of data sources for visual presentation. It can quickly and flexibly create client charts, panel plug-ins have many different ways of visualization indicators and logs, the official library has a wealth of dashboard plug-ins, such as heat maps, line charts, charts and other display methods; Support Graphite, TDengine, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch, KairosDB and other data sources, support independent/mixed query and display of data items; You can create custom alarm rules and notify them to other message processing services or components.

The business scenario

This paper simulates the environment data collection scenario of the Internet of Things, assuming that there are environmental data collection points with certain data, and all collection point data are transmitted to the collection platform (MQTT Publish) through MQTT protocol. The theme design is as follows:

sensor/data
Copy the code

The sensor sends data in JSON format, including temperature, humidity, noise volume, PM10, PM2.5, sulfur dioxide, nitrogen dioxide, carbon monoxide, sensor ID, region, and collection time.

{
    "temperature": 30."humidity" : 20."volume": 44.5."PM10": 23."pm25": 61."SO2": 14."NO2": 4."CO": 5."id": "10-c6-1f-1a-1f-47"."area": 1."ts": 1596157444170
}
Copy the code

You now need real-time storage so that you can view the data at any subsequent time, and the following requirements are presented:

  • Each device reports data every five seconds. The database stores each piece of data for subsequent retrospective analysis.
  • Use the visual system to view indicator data, such as average value, maximum value, and minimum value, in any region or time interval.

Environment to prepare

All components used in this paper have Docker images. TDengine and Grafana are built with Docker except EMQ X, which needs to modify a few configurations for easy operation, download and install.

Refer to their official websites for installation package resources and tutorials:

  • EMQ X: EMQ official website www.emqx.io/cn/
  • TDengine: Taos data website www.taosdata.com/cn/
  • Grafana: Grafana.com/

Install EMQ X

If you are new to EMQ X, it is recommended to use the EMQ X documentation to get started quickly

Visit the EMQ X download page to download the installation package suitable for your operating system. The latest EMQ X open source version is V4.1.1 at the time of going to press. To download the ZIP package, proceed as follows:

Unzip the downloaded installation packageUnzip emqx macosx - v4.1.1. Zipcd emqx

Start EMQ X in console mode for easy debugging
./bin/emqx console
Copy the code

Log in to the EMQ X management console Dashboard using the default user name admin public and password for the first login.

EMQ X Enterprise edition 4.1.2 provides the native TDengine write plug-in for better performance and easier useRules Engine – Write data to TDengineTo view

Install TDengine

In order to facilitate the test, it can be installed through Docker (network ports need to be mapped), or it can be installed by installation package:

## Pull and start the container
docker run -d --name tdengine -p 6030-6041:6030-6041 tdengine/tdengine:latest

Check the container running status after startup
docker ps -a
Copy the code

Install Grafana

Install and start Grafana from Docker using the following command:

docker run -d --name=grafana -p 3000:3000 grafana/grafana
Copy the code

After successful startup, visit http://127.0.0.1:3000, visit Grafana visual panel, and log in to Grafana for the first time using admin admin’s default user name and password. After login, change the password as prompted and use the new password to log in to the main screen:

Configure EMQ X to store data to TDengine

TDengine creates databases and tables

Enter the TDengine Docker container:

docker exec -it tdengine bash
Copy the code

Create test database:

taos
create database test;
Copy the code

To create a sensor_data table, see TAOS SQL for the TDengine data structure and SQL commands:

use test;
CREATE TABLE sensor_data (
  ts timestamp,
  temperature float,
  humidity float,
  volume float,
  PM10 float,
  pm25 float,
  SO2 float,
  NO2 float,
  CO float,
  sensor_id NCHAR(255), 
  area TINYINT,
  coll_time timestamp
);
Copy the code

Configure the EMQ X rule engine

Open EMQ X Dashboared, enter rules Engine -> Rules page, click Create button to enter the Create page.

Rules of SQL

Rule SQL is used to filter EMQ X messages and events. The following SQL is used to filter payload data from sensor/data topics:

SELECT
  payload
FROM
  "sensor/data"
Copy the code

Use SQL test function, input test data for screening results test, test results and output content is as follows, indicating that SQL is written correctly:

{
  "payload": 30, "{\" temperature \ ": \" humidity \ ": 20, \" volume \ ": 44.5, \" PM10 \ ": 23, \" pm2.5 \ ": 61, \" SO2 \ ": 14, \" NO2 \ ": 4, \" CO \ ": 5, \" id \ ": \" 10 - c6-1f-1a-1f-47\",\"area\":1,\"ts\":1596157444170}"
}
Copy the code

In response to the action

To support the development of different types of platforms, TDengine provides apis that conform to REST design standards. RESTful Connector provides the simplest connection mode, that is, HTTP requests carry authentication information and SQL operations to be executed TDengine.

Use the Send-to-Web service in EMQ X open source to write data to TDengine via a RESTful Connector. The upcoming EMQ X Enterprise version 4.1.1 will offer a native write Connector with higher performance.

Sending to the Web service requires two pieces of data, one for the associated resource and the other for the message content template.

  • Associated resource: HTTP server configuration information, in this example, RESTful Connector of TDengine
  • Message content template: This is INSERT SQL with data. Note that we should specify the database name in SQL and enclose the character type in single quotes. The message content template is:
INSERT INTO test.sensor_data VALUES(
  now,
  ${payload.temperature},
  ${payload.humidity},
  ${payload.volume},
  ${payload.PM10},
  ${payload.pm25},
  ${payload.SO2},
  ${payload.NO2},
  ${payload.CO},
  '${payload.id}',
  ${payload.area},
  ${payload.ts}
)
Copy the code

The creation process

Click the Add button under the response action, select send data to Web service in the pop-up box, and click New Resource to create a New WebHook resource.

Choose Webhook resource type, fill in the request URL http://127.0.0.1:6041/rest/sql, select the POST request method, still need to add the Authorization request header as authentication information.

The Authorization value is Basic + TDengine {username}:{password} Base64 encoded character string, for example, root:taosdata encoded value is: Basic cm9vdDp0YW9zZGF0YQ==

On the response Action Creation page, select the new resource and fill in the message template content.

Generate simulated data

The following script simulates a scenario in which 10,000 devices report simulated data to EMQ X every 5 seconds for the past 24 hours.

  • Total data: 24 * 3600/5 * 100 = 1.72 million
  • Message TPS: 20

After installing Node.js and modifying the configuration parameters as needed, the reader can start node.js with the following command:

npm install mqtt mockjs --save --registry=https://registry.npm.taobao.org
node mock.js
Copy the code

Attached: simulation generates data and sends it to EMQ X code, please adjust related parameters according to cluster performance

// mock.js
const mqtt = require('mqtt')
const Mock = require('mockjs')

const EMQX_SERVER = 'mqtt://localhost:1883'
const CLIENT_NUM = 100
const STEP = 5000 // Analog acquisition interval ms
const AWAIT = 5000 // Sleep time after each send to prevent message rate too fast ms
const CLIENT_POOL = []

startMock()


function sleep(timer = 100) {
  return new Promise(resolve= > {
    setTimeout(resolve, timer)
  })
}

async function startMock() {
  const now = Date.now()
  for (let i = 0; i < CLIENT_NUM; i++) {
    const client = await createClient(`mock_client_${i}`)
    CLIENT_POOL.push(client)
  }
  // last 24h every 5s
  const last = 24 * 3600 * 1000
  for (let ts = now - last; ts <= now; ts += STEP) {
    for (const client of CLIENT_POOL) {
      const mockData = generateMockData()
      constdata = { ... mockData,id: client.options.clientId,
        ts,
      }
      client.publish('sensor/data'.JSON.stringify(data))
    }
    const dateStr = new Date(ts).toLocaleTimeString()
    console.log(`${dateStr} send success.`)
    await sleep(AWAIT)
  }
  console.log(`Done, use The ${(Date.now() - now) / 1000}s`)}/**
 * Init a virtual mqtt client
 * @param {string} clientId ClientID
 */
function createClient(clientId) {
  return new Promise((resolve, reject) = > {
    const client = mqtt.connect(EMQX_SERVER, {
      clientId,
    })
    client.on('connect'.() = > {
      console.log(`client ${clientId} connected`)
      resolve(client)
    })
    client.on('reconnect'.() = > {
      console.log('reconnect')
    })
    client.on('error'.(e) = > {
      console.error(e)
      reject(e)
    })
  })
}

/**
* Generate mock data
*/
function generateMockData() {
 return {
   "temperature": parseFloat(Mock.Random.float(22.100).toFixed(2)),
   "humidity": parseFloat(Mock.Random.float(12.86).toFixed(2)),
   "volume": parseFloat(Mock.Random.float(20.200).toFixed(2)),
   "PM10": parseFloat(Mock.Random.float(0.300).toFixed(2)),
   "pm25": parseFloat(Mock.Random.float(0.300).toFixed(2)),
   "SO2": parseFloat(Mock.Random.float(0.50).toFixed(2)),
   "NO2": parseFloat(Mock.Random.float(0.50).toFixed(2)),
   "CO": parseFloat(Mock.Random.float(0.50).toFixed(2)),
   "area": Mock.Random.integer(0.20),
   "ts": 1596157444170,}}Copy the code

Visual configuration

After the components are installed and simulated data is written successfully, the data visualization configuration required by the business is completed according to the operation instructions of Grafana visualization interface.

Add data source

Grafana’s TDengine data source requires manual installation of the plug-in in the TDengine documentation.

Add a data source, that is, the data source information that is displayed. Select a TDengine type data source and set connection parameters. By default, key configuration information is as follows:

Adding a New Dashboard

Obtain the Grafana dashboard export file import in the EMQ X Sample repository to see an example chart.

After the data source is added, add the data dashboard information to be displayed. The Dashboard is a collection of visual panels. After clicking New Dashboard, select + Query to add data panels by querying.

There are four steps to create the panel, which are Queries, Visualization, General, Alert, and creation time

Average panel

Using Grafana’s visual query building tool, the average was queried for all devices.

$form $to; $interval;

select avg(temperature), avg(humidity), avg(volume), avg(PM10), avg(pm25), avg(SO2), avg(NO2), avg(CO)  from test.sensor_data where coll_time > = $from and coll_time < $to interval($interval)
Copy the code

The name of the panel in General is changed to historical average. If you need to monitor alarms for services, you can format alarm rules in Alert. This function is not used for Visualization only.

After creating the Dashboard, click the back button in the upper left corner to add a data panel to the Dashboard. Click the top navigation bar to save the icon, and enter the Dashboard name to create the Dashboard.

Maximum and minimum panels

Continue clicking on the Dashboard’s Add Panel button to Add a Max/min chart. The operation procedure is the same as adding the average value. Only adjust the SELECT statistical method field in the query to MAX and MIN:

select max(temperature), max(humidity), max(volume), max(PM10), max(pm25), max(SO2), max(NO2), max(CO), min(temperature), min(humidity), min(volume), min(PM10), min(pm25), min(SO2), min(NO2), min(CO)  from test.sensor_data where coll_time > = $from and coll_time < $to interval($interval)
Copy the code

Dashboard effect

Save the dashboard, drag and drop to adjust the size and position of each data panel, and finally get a good visual data dashboard. You can select the time range and automatic refresh time in the upper right corner of the dashboard. At this time, the device continues to send data collection data, and the data value on the dashboard will change, achieving good visualization effect.

FAQ

Q: Why is there no icon data in Grafana?

  • Drag the time range to check that data is available for the selected time period

Q: What are the differences between EMQ X open source edition and EMQ X Enterprise Edition in writing TDengine functionality?

  • The open source version uses Webhook + TDengine RESTful Connector, which has a certain performance loss on both sides, with a maximum write speed of about 700 entries per second
  • The Enterprise edition uses the EMQ X native plug-in and is capable of 20,000 writes per second

Q: The rule is executed, but no data can be written?

  • Check whether the authentication information is correctly configured and whether the information about the request header, connection address, and port match the TDengin version

conclusion

So far, we have completed the system construction of the whole process of Internet of Things data transmission, storage and presentation with the help of EMQ X + TDengine. Readers can understand the application of EMQ X’s rich expansion ability and TDengine’s complete big data platform features in Internet of Things data collection. By learning more about Grafana’s other features, you can customize your own data visualization and alarm monitoring systems.

Copyright: EMQ

Original link: www.emqx.io/cn/blog/emq…