background
As an Internet e-commerce company, DeWo has many systems and scenes that need to do traffic monitoring and protection. Therefore, when we deeply customize the open-source flow control protection component Sentinel, we add many functions to help improve the flow control protection of various business systems.
During development, we found that the open source version of Sentinel did not support flow control data persistence, which we really needed: we needed a database that could host large amounts of traffic monitoring data, store and query that data efficiently.
In a production environment, we have hundreds of business systems and thousands of servers connected to Sentinel, which creates a huge amount of flow control data. So for this demand, choose a suitable database is undoubtedly very important, a good choice can achieve twice the result with half the effort.
Database selection
Let’s start with a rough estimate of the theoretical upper limit for the current amount of data: there are thousands of Sentinel Resources in the production environment today. And Sentinel surveillance data granularity according to the second time to statistics, one day can generate hundreds of millions of data in theory, the theory of writing data will reach the speed of the TPS, and there has been a rapid development of business, predictably data volume will be further explosions, it is obvious that the data scale is unable to use traditional relational database.
Because there are some internal applications using TIDB, so I first looked at the feasibility of using TIDB, but soon gave up, after all, as a distributed database, it is not aimed at monitoring data such as very strong timing characteristics of the scene. After that, we focus on the time series database.
There are advantages and disadvantages in the mainstream timing database:
- InfluxDB, probably the most widely used time-series database, also applies to scenarios. But the clustering feature requires the commercial version.
- OpenSDB, based on HBase, is too heavy for the current simple requirements.
- Cassandra, from the few comparative reports I’ve found, the performance is not quite up to par.
When I was ready to learn more about ClickHouse, I was introduced to TDEngine, a Chinese networked big data platform. A simple understanding on the Internet, found that the wind rating is good, the community is active is also high, and then went to the official website to check the TDengine and other database comparison report, found from the performance is also very excellent. So we wrote a demo that used Tdengine briefly. During the whole process, with the help of clear documentation, the learning cost was good, so we decided to use Tdengine.
Data structure and modeling approach
The data structure
First of all, let’s take a look at how Sentinel traffic data is presented. As can be seen from the figure above, the list of applications is on the left. In the menu of each application, there is an independent monitoring panel, and in the monitoring panel, the traffic data of all resources are collected according to the granularity of resources, such as QPS, QPS rejection, response time, etc. So from a front-end rendering perspective, the only key for data should be application-resource.
Then we’ll look at the structure of the data from an internal implementation perspective. The Sentinel client calculates traffic data for all resources on each server, aggregates it in seconds, and records it in a local log. The console acquires the collected traffic data by calling the interface exposed by the client, and then aggregates the traffic data of all stand-alone machines in the dimension of service and stores them in memory. So the data that we need to store falls into the database with application-resource as the only attribute.
Data modeling
The official documentation for Tdengine suggests the following data modeling methods:
In order to take full advantage of the timing and other data characteristics of its data, TDengine requires a separate table for each data collection point. The method of one data collection point and one table can ensure the performance of single data collection point insertion and query is optimal to the greatest extent. In the design of TDEngine, a table is used to represent a specific data collection point, and a super table is used to represent a set of data collection points of the same type. When creating a table for a specific data collection point, the user uses the definition of the super table as the template and specifies the label value for the specific data collection point (table). In contrast to a traditional relational database, a table (a data collection point) has static labels that can be added, deleted, or modified afterwards. A super table contains multiple tables with the same sequential data schema, but with different label values.
As you can see, the proposed data modeling approach in the official documentation perfectly fits the data characteristics of this scenario: one application-resource is a single table, and all application-resources are placed in a single super table for aggregate queries. So in the design of the table structure, the use of the official document recommended this way.
In addition, in terms of the selection of labels, although there is no requirement for aggregation operation at present, considering that the future aggregation operation is very likely to be done in the dimension of application, we decided to record some application information as labels in the table.
The overall architecture
As shown in the current architecture diagram, every business system connected to Sentinel sends periodic heartbeat requests to the console to maintain the health of its own machine. The console regularly polls all the machines, pulls the monitoring data recorded by the Sentinel client in the business system, and then writes to the Tdengine cluster in batches after the aggregation process.
Since the scenario is simple and not the primary monitoring system, the data can be lost in a small amount, so there are not too many failure handling mechanisms designed.
Technology selection
Connector
When it comes to Connector selection, the company’s primary development language is Java and the related ecosystem is more complete, so it was a natural choice to use the JDBC form of Connector. JDBC also provides better performance than HTTP, and the JDBC driver also supports automatic node switching when a node is unavailable. The only inconvenience is that the JDBC approach relies heavily on native library functions, requiring TDEngine to be installed on the client machine as well, which makes it a little more cumbersome during project deployment. But on the whole, the advantages outweigh the disadvantages.
Recently, the JDBC-RESTful approach has been updated to support cross-platform functionality. Since the operating system of the company’s servers is Linux, there is no cross-platform requirement, so we continue to use the Connector of JDBC-JNI.
Note: The figure is from the official website of TDengine
Database connection pooling with ORM
The database connection pool and ORM framework are also selected in the mainstream of the company Druid+ MyBatis, according to the official website of the Demo code can also be efficiently completed access. However, in the use of MyBatis, MyBatis is only used in the query to turn the ResultSet into a more convenient entity, and MyBatis is not used in the writing of data. For convenience, SQL is directly stitched in memory for execution.
In general, TDengine is very friendly to adapt to the mainstream framework, and supports Hikaricp, Druid, Spring JDBCTemplate, MyBatis, etc., and can be implemented quickly according to the Demo provided on the official website, saving a lot of time. Some considerations are clearly listed in the documentation.
The cluster structures,
Currently, there are three physical nodes in the Tdengine cluster, all of which are 16 cores /64G memory /1T storage. The official cluster set-up document is very detailed, and you can set up the TDEngine cluster directly by following the document for fool operation.
To build libraries
In the preliminary survey, it was found that if the cluster had only three machines, if the amount of data was too large, the number of copies was 3, which was equivalent to storing a complete data on each machine. Based on the possible amount of data, the pressure on storage and memory would be large, so the number of copies was set to 1 when building the database. Tdengine also supports dynamic modification of the number of replicas if the cluster size increases, making it easy to switch to a highly available cluster.
Also for query performance, set the blocks to 16 and the cache to 64MB, as follows:
CREATE DATABASE sentinel KEEP 365 DAYS 1 blocks 16 cache 64;performance
Currently, Tdengine is loaded with tens of billions of data and runs smoothly in production environments, with CPU usage less than 1% and memory usage stable at less than 25%. The following is a monitoring chart for a machine in the cluster:
In the earlier version of Tdengine (2.0.7.0), there were some memory footprint defects, but with the iteration of the version, the memory problem has been solved.
Write performance
The console machine is configured to be 4-core 16G, the maximum number of core threads set for batch write thread pool is 16, the maximum number of database connection pool threads is 20, and the actual usage is about 14.
The writing process is as follows:
The maximum number of writes for batch writes is set to 400, and the write time is as follows:
It can be seen that large quantities of writing can basically maintain the time consumption at 10ms, which belongs to the ideal range. The maximum length of SQL statements has not yet been adjusted, and it is possible to further optimize write performance by increasing the length of SQL statements.
Query performance
The following time is not included in the network overhead, etc., data from the client for the specified SQL statement of the query. The magnitude of query super table data is in the tens of billions. The following is the time consuming situation of several typical scenarios:
- Last_row function: 8.6ms 8.8ms 5.6ms
- Query all data from a single application + resource for 5 minutes: 3.3ms 3.3ms 3.3ms
- QPS: 1.4ms 1.3ms 1.4ms QPS: 1.4ms 1.4ms
- Average pass QPS for every two minutes of the day grouped by service dimension: 2.34s 2.34s 2.35s
- Querying the average pass QPS for each hour over three days by service dimension grouping: 2.17s 2.16s 2.17s
Whether aggregated queries over a large data range or specified queries for all data within a small range, the query efficiency is very good. In addition, compared with the data in the previous investigation, the query performance of the new version has been greatly optimized, and I believe it will be further improved in the future iteration.
Storage capacity
At present, Sentinel data does not use duplicas, and the full amount of data is distributed among three machines. According to the calculation, TDengine can compress Sentinel monitoring data by 10%, which is quite impressive.
Summary and follow-up planning
At present, TDEngine is only used as a small scale pilot of timing database, without some advanced functions such as stream calculation and built-in query function. As a timing database, its read-write performance and storage performance are satisfactory.
In addition, the difficulty of operation and maintenance and the cost of learning are surprisingly low. It is very easy to set up a set of usable clusters, which is also a huge advantage. In addition, the version iteration of TDEngine is very fast, some problems encountered in the old version are quickly fixed, and the performance optimization effect is also very significant.
During the period of investigation and use of Tdengine, there is also a very important feeling that the official documents are really very detailed. The article in the technical part explains the technical architecture and technical design of Tdengine in a simple way, and we can learn a lot. Guide articles are clear and simple steps, greatly reducing the cost of learning, so that developers can quickly complete framework adaptation, cluster building, SQL writing, etc.
In the future, we will continue to follow up the release notes of TDengine to see what new featrue, optimization points, bug fixes, etc., and update the version if necessary. It is expected that the performance and stability of TDengine will continue to improve. In the future, TDengine will also be used as one of the options for technology selection in other appropriate business scenarios. For example, in the future, it may be necessary to store not only aggregated data, but also flow control data in the single machine dimension.
Note: The data in this article are based on the 2.0.7.0 and 2.0.12.1 versions of TDEngine.
Article | Lynx
Pay attention to the technology of things, hand in hand to the cloud technology