Hello, everyone. At the beginning of 2020, the old year is out and the New Year is in. This time we will share a super practical data integration tool for enterprises. In practical work, it is often necessary to show the value of data in the form of charts and further big data analysis; However, in the process of data processing, data sorting often takes 80% of the time. Especially, we often encounter the problem of inconsistent format of excel file data header, such as “certain department did not feedback data in accordance with the format”, “reports of each branch have various table headers, which is too painful to sort out”… Yes, excel data summary, often encounter data table header format is inconsistent, just want to loudly shout: I am too “south”!! Data format uniformity is so important.
Today, the editor will share with you a data integration tool: CEAMS integration tool, which generates a general Excel header, quickly realizes the standardization of data format, saves the labor consumption of daily data processing, and maximizes the realization of data realization. Suppose that we have several Excel files of food supervision data from different provinces on hand, among which the header format of each data table is inconsistent, and they need to be converted into a uniform and standardized format. Next, let’s take a look at how to standardize data formats through the CEAMS integration tool. Take hubei province food, Gansu province, Jiangsu province sampling unqualified data for example.
Existing Excel file data header:
① Screenshots of unqualified food sampling data in Hubei Province
② Screenshots of unqualified food sampling data in Gansu Province
③ Screenshot of unqualified food sampling data in Jiangsu Province
Problem description: take a closer look at these forms will find that their headers are not consistent, some of the key information format is sent a lot, such as “unqualified project | | inspection results | | standard” this column, hubei province, is in jiangsu province
Gansu province uses
If we directly use the original format to carry out big data analysis, the results may be biased and not refined enough.
After processing with the CEAMS integration tool, the data header in the standardized format:
The use of standardized data ensures the consistency of data, facilitates the integration of big data and visual data and powerful predictive analysis, and helps enterprises make real-time decisions. I. The overall workflow of CEAMS integration tool
2. Easy operation of CEAMS integration tool
1. Select the corresponding components from the toolbox on the left, place them in sequence as shown in Figure 1, and connect them with wires.
2. Double-click the component shown below to create its property Settings: corresponding services and interfaces; Click Save.
3. Double-click the component shown below to create its properties: select the corresponding service and interface; Click Save.
4. Double-click on the component shown below, place the mouse over a field in the input at Transform Message, and hold and drag to the matching field in the output to create the actual mapping between the input and output fields. Match all the same and similar names, such as sample number matching sample number, product name matching name.
5. Click the “Test” button in the upper right corner, select the Excel source file to be processed as prompted, and click save. CEAMS integration tool completes the test online.
6. The message “Test succeeded!” is displayed on the Console. , that is, the data has been successfully converted to the standardized format and the standardized format data has been synchronized to the database.
With these converted standard data, it is much more convenient to further carry out big data integration, modeling, analysis and management. There is no need to work hard for data sorting, realize the automation of data reports in operation and realize the visualization of big data analysis.
Three important components of the CEAMS integration tool
With the steps out of the way, let’s now look at three very important components of the CEAMS integration tool.
1. Component 1
CEAMS platform is responsible for customized development according to the specific needs of users. The main function of this component is to parse the input source data into JSON format output. Taking the above food supervision data as an example, the output JSON format content is as follows (there are many contents, only one line of data conversion JSON content is displayed) :
Test hubei province data component
Output:
{
“output”:[
{
“sid”:”GC19420000003335391″,
“pname”:”/”,
“paddress”:”/”,
“Cname “:” Yan ‘an Road Branch of Huangshi Zhongshang Supermarket Chain Co., LTD”
122 Yan ‘an Road, Huangshi City
“Name” : “leek”,
Bulk weighing
“trademark”:”/”,
“pdate”:”/”,
“Reason “:” Humus “,
“Test_results “:” humus “,
“Rlimit “:” propidium “,
“Result “:” unqualified “,
“Type “:” edible produce “,
“Notification_no “:” Issue 41 of 2019 “,
“time”:”2019-10-31″,
“Province”, “hubei province”,
“Level “:
“Notifying_unit “: hubei Institute of Food Quality and Safety Supervision and Inspection
Sampling Date: 2019-07-31
}
2. Component 2
By the user’s own operation, matching data fields; The main function of this component is to convert the JSON-formatted content parsed by Component 1 into data output in a uniform standardized format. Taking the above food supervision data as an example, the specific output of standardized data content is as follows (there are many contents, only one line of standardized format content of data conversion is displayed) :
Start Transform data
Output:
[
{
“sid”:”GC19420000003335391″,
“pname”:”/”,
“paddress”:”/”,
“Cname “:” Yan ‘an Road Branch of Huangshi Zhongshang Supermarket Chain Co., LTD”
122 Yan ‘an Road, Huangshi City
“Name” : “leek”,
Bulk weighing
“trademark”:”/”,
“pdate”:”/”,
“Reason “:” Humus “,
“Test_results “:” humus “,
“Result “:” unqualified “,
“Rlimit “:” propidium “,
“Type “:” edible produce “,
“Notifying_no “:” phase 41 of 2019 “,
“time”:”2019-10-31″,
“Province”, “hubei province”,
“Level “:
“Notifying_unit “: hubei Institute of Food Quality and Safety Supervision and Inspection
Sampling Date: 2019-07-31
}
3. Component 3
As with component 1, the CEAMS platform is responsible for development; It provides standardized data formats and synchronously imports the data generated by component 2 to the database. Four,
CEAMS integration tool, based on the concept of microservices, will apply modular encapsulation, through unified API call to achieve module interaction; To provide enterprises with cloud integration integrated development environment, no need to install and configure a separate development environment tools, login browser can handle data integration work anytime and anywhere online; Based on Node.js development, input data sources, automatically convert them into standard simple JSON data format, and arrange, reorganize, aggregate and integrate data structures; Provide visual and automated data integration processes to help enterprises reduce costs and maximize data value.
(after)
Reprint or excerpt from this article is prohibited without permission.