The Internet of Things (IIoT) and Elastic Stack Author Felix Rossel Marco De Luca

The Industrial Internet of Things (IIoT) gives manufacturing companies the opportunity to capture data from thousands of sensors and devices.

Connecting these data is important both in monitoring the production process and in obtaining appropriate measurements for further analysis. One of the key challenges is how to collect and standardize data so that it can be used in the areas of predictive analysis or workshop safety.

Even small manufacturing companies tend to operate multiple factories with hundreds of machines and thousands of sensors. New data is generated at the millisecond rate, easily adding up to a terabyte, but it is not stored and analyzed and therefore worthless.

With Elastic, you can collect, enhance, and analyze IIoT data and help production managers gain insight into the production process.


— Marco De Luca, Chief Solution Architect, Elastic

Being able to collect this data on a large scale will be an important watershed. It can optimize the production process, thereby increasing efficiency and improving product quality. Typically, manufacturers face at least one of the following challenges in addition to the general challenges mentioned above:

  • Manufacturing companies are very innovative, but there will still be a lot of old machinery that will probably continue to be used for years. The life cycle of machines is usually 10-20 years or more! How do you integrate these old machines into modern analytics platforms?
  • With sensors, autonomous vehicles, and a host of other devices that need to be monitored. Knowing the big picture is the foundation to help manage predictive maintenance, operations, and everything else. All kinds of devices use different technologies to communicate and are connected to the Internet in different ways.
  • There are many proprietary solutions that cannot communicate with each other. How do you get data from these systems and relate it to other machine data?
  • Some manufacturing companies are already thinking about how to build a machine data platform that can monitor all these old and new systems together. How do you leverage this data? Is it for operational/maintenance purposes, or is it building other/new business use cases and/or services?

For companies that specialize in collecting and analyzing machine data, there are many benefits. This not only optimizes the production process, but also improves product quality. The benefits of collecting and analyzing this data go far beyond that.

A well-designed monitoring solution combined with predictive maintenance can also help significantly reduce costs. Sensor data can provide a good indication of the condition of equipment or the quality of finished products. In addition, it helps to reduce resource consumption. MM Karton has demonstrated this potential by reducing raw material consumption by 20%.

Build the basis for IIoT data analysis

To be able to take advantage of these different kinds of data, you need to store the data in a separate environment that can be scaled on demand. ElasticSearch is a good fit for this use case. Not only does it allow you to store multiple types of data on a large scale, but it also lets you use Machine Learning capabilities to analyze data and visualize it in any way you like.

Once you are able to collect all the relevant data for each plant, imagine a comprehensive monitoring of how you can improve the efficiency of your production plant. Stacking all the necessary data in Elasticsearch and using Canvas to put data sets from multiple assembly plants into one screen is as easy as making a slideshow connected to live data for a presentation.

Here are some examples for you to look at:

Monitor every piece of equipment in multiple assembly plants from the macro to the micro

It’s great to have a big picture of a number of different assembly plants, but you also need to know how the different machines in each plant perform. In addition, you need to be able to combine and correlate different data models for each vendor.

Typical IIoT use cases using Elastic Stack

The following use cases are some examples of IIoT data analysis and monitoring using Elastic Stack. There are many other examples, such as automatically responding to questions by interacting remotely with a machine.

Predictive analysis introduces predictive maintenance

Using sensor data to analyze the health of IoT devices is very effective. Sensor data (such as air or oil pressure, temperature, voltage, speed, sound, frequency, or color/lightning changes) can be used as an early warning mechanism for faults. Knowing that a device will fail in the near future and reacting to that information will help reduce costs.

In addition, by knowing the threshold or data range of sensor data, production managers can set individual maintenance plans based on actual data, rather than following rigid maintenance cycles, thus avoiding a situation where components are replaced regardless of whether they are likely to fail. By leveraging this capability, a dedicated and meaningful maintenance plan can be developed to reduce costs and reduce production downtime.

Achieving this requires a considerable understanding of the environment that is causing the failure. Watching hundreds of sensors simultaneously in real time and comparing them with historical events isn’t easy for the rest of us, but for Machine Learning, it’s no small task. Finding anomalies for each sensor, or correlating data from all sensors with a single health score, can be very powerful.

The rate of defective or rejected products is reduced

Another important KPI is the defect rate/rejection rate. It is important to reduce the rate of defects, and this can only be done by understanding which parts of the production process lead to defects. The Elastic Stack outlier detection function based on Machine Learning can help find products that differ from the expected results. By combining this detection feature with sensor data, problems can be found quickly and effortlessly.

If machine learning jobs on the shop floor are properly designed, they may eventually be automated and optimized without human intervention.

The outlier detection function based on Machine Learning can detect product quality indicators that may violate common sense

Workshop safety

Today, the security of IT-related systems is a well-known challenge. Workplace safety is often more difficult to implement. At the moment, the general concept is to prevent intrusions by cutting off entire production facilities from the Internet.

This is becoming increasingly difficult. “Classic IT” (such as ERP such as SAP PP or PLM) has an increasing influence on SPS (such as Siemens Simatic) systems and will eventually lead to direct communication between good and bad content. Demand is generated by the business — MOD (Manufacturing on Demand) can already be implemented through 3D printing or laser cutting. For example, imagine a fully automated production process triggered by a custom order from any online store. The risk of sparks flying from interconnected IT to shop floor will become a new dimension of risk. Therefore, workshop safety will become a very important topic during the Industry 4.0 transition. Your smart factory can also monitor security events with Elastic Siem (Security Information and Event Management) and the ability to collect all relevant data.

Collect and analyze different IIoT data sources using Elastic Stack

The analytics required have been available in Elastic Stack for a long time. The hard work is putting data from different data sources on the stack.

Collecting data from all the various data sources available in the production plant cannot be done in just one way. You need to put multiple data sources together. Because Elastic is an open real-time data platform, it is easy to use its ecosystem to integrate all types of data sources.

In a manufacturing plant, there are many different systems that need to be monitored, from very old systems (which may run control software using Windows95 or earlier) to the latest technologies running Linux and OPC-UA. We will focus on the following types of systems:

  • Manufacturing control software (e.g. IBM MQ) used to control manufacturing processes within and between plants
  • Programmable logic controller (PLC) is a special form of computer equipment designed for application in industrial control system. It is a special-purpose “industrial computer” for controlling sensors and actuators; For example, it can control the machines that produce paper products in the paper industry, as well as box elevators and escalators. There are many use cases for equipment controlled by PLC.
  • Industrial robots, such as those made by KUKA (German/Chinese manufacturer). The Kuka robot is equipped with an OPC-UA server that can control the Kuka robot but at the same time interact with other robots and systems in the manufacturing workshop. In addition, OPC-UA also has the ability to interconnect systems over the Internet, so it will also face security threats. This is why security plays an important role in the OPC-UA specification, but it also needs to be monitored through a monitoring platform like Elastic.
  • Sensors and other autonomous vehicles or equipment are mostly used to measure temperature, humidity, speed, acceleration, positioning, vibration, or other metrics needed to correlate them to production process data. For example, changes in temperature and humidity can affect the operation of the robot and cause greater vibration, which can cause the robot to break down or produce poor quality products. So sensor and robot data is very important to customer operations.

Collect data from PLC, MQTT agent and OPC UA server

To collect all this near real-time data, we can use MachineBeat, a community that is able to collect data from MQTT agents and OPC UA devices. Its MQTT module is also capable of collecting data from different IoT cloud service providers, such as the AWS IoT Core or Azure IoT Center.

To collect data from PLC devices, it is integrated with Apache Project PLC4X as a Logstash plug-in, which enables customers to collect metrics from all PLC4X supported PLC4X. For more in-depth discussion of PLC4X in combination with Elasticsearch, see this blog post from our partner Codecentric.



An overview of the solution for collecting, expanding, and analyzing all data

Use Kibana for data visualization

Mastering all this information presents a new challenge. Storing large amounts of data is useful only if you can extract value from it. It is by gaining better insight into what is happening that value is generated. Visualization can help with this. In the following example, message passing between multiple queues in an IBM MQ-controlled manufacturing environment is visualized. In older monitoring systems, you need to have a specific understanding of what you are monitoring. Using Canvas, you can abstract specific knowledge and provide feedback on the situation in easily recognizable colors.

Real-time monitoring of IBM MQ metrics, including conditional color switching based on custom rule sets

Personally try

  1. Start a 14-day free trial of ElasticSearch to give you a basic understanding of IoT monitoring solutions.
  2. Go to GitHub and download MachineBeat for Linux or Windows.
  3. Follow the steps shown in the Configuration section of the README to configure the connection to the free OPC-UA server or enter the credentials to connect to our own OPC-UA server. Alternatively, you can follow the steps to configure the MQTT agent and use it to collect data from various sensors.
  4. Once the data starts flowing, you can build your own dashboards to visualize/analyze the data. Canvas is another great alternative to visual data, allowing you to create presentation slides using live data. For more information, check out the Canvas Getting Started blog.

To learn more about Elastic technology, please follow and sign up for the webinar. The upcoming schedule is as follows:

Wednesday, February 26, 2016 15:00-16:00 Kibana Lens Webinar

Wednesday, March 4, 2020 15:00-16:00 Elastic Endpoint Security Overview Network

Monitor website resources with Elastic Stack Wednesday, March 11, 2020 15:00-16:00