About the author: Xue Chao, senior engineer of database operation and maintenance of China Mobile Internet of Things Co., LTD
In the Internet of things co., LTD. Is a wholly owned subsidiary of China mobile communications corporation investment established, the company in accordance with the strategy of China mobile’s overall layout, around the “Internet of things of business services provider support, special modules and chips, the Internet of things is special product agents” strategic positioning, professional operation iot private network, special modules and chips, the Internet of things It has developed and operated Internet of Things connection management platform OneLink and Internet of Things open platform OneNET, and promoted Internet of Things solutions, forming a comprehensive business layout in five directions and an Internet of Things “cloud-management-terminal” architecture.
This sharing mainly introduces the Internet of vehicles business, which mainly focuses on on-board location terminal and on-board video terminal to carry out business, including parking guards, road service individuals, road service industry, and unified filling business. By The end of May 2020, a total of 1.5 million terminals have been connected to the Internet of vehicles. Internet of vehicles users are mainly individual users and enterprise users. At present, a total of 1.51 million individual users and 1471 enterprise users have been registered.
Basic IOV architecture
Firstly, let’s talk about the infrastructure. According to the configuration of the business type, the OPD device mounted on the car will send the message to the input computing module and distribution engine in time, parse the message according to the pre-established protocol, and distribute different information to different downstream services. For example, trajectory service and alarm service. Different services use different storage media. For example, track is stored in TiDB, location service is stored in Redis cluster, driving video is stored in object storage of Qiniu, and complete message information is stored in HBase for data analysis.
IOV core scenario
Scenario 1: Device management module
Equipment management is mainly to record various state information data of vehicle-mounted equipment. The update frequency of some data is relatively high, and the peak value reaches 12,000 words/second. Before TiDB is used, device management is put in Redis Cluster, put in TiDB validation, mainly to see its efficiency in handling update statements.
Scenario 2: Driving track
The driving track scenario is mainly the writing of driving track data and a small amount of track query requests, with an average daily writing volume of 450 million lines of data. At present, the size of the validation cluster data is about 30 billion lines, and the final size will reach more than 160 billion lines, which is a relatively massive data at that time.
Trajectory storage evolution
In 2017, the track was running on Oracle’s two-machine RAC. In the wave of IOE removal, business development was limited, and Oracle’s hardware procurement requirements could not be approved by the group. Therefore, we began to consider migrating the storage of the whole track to the open source database. At that time, MySQL was chosen as the direction of migration, but the track module on Oracle had a large volume with 8 tons of data. In the early stage, MySQL was definitely unable to carry such a scale of business. Therefore, we considered to horizontally slice the data and combine with the Environment of Oracle. The peak QPS is about 10,000. At that time, the number of shards was set at three, and the track data of a specific device was controlled by the code and given to the specific shard. In our verification process, found that can’t deal with the three nodes, so we extend to eight nodes, basically can seat at this time the whole trajectory data to the service, but the side of the business logic and become quite heavy, maintenance cost is very high, so want to find a middleware to replace the piecewise function of the code.
So we chose MyCat, and after a few tweaks, the physical set of 16 X86 servers became 8 MySQL nodes, replacing Oracle. The process was not smooth. In the early stage of using MyCat, the writing performance was not good, and queues were often backlogged. We thought of some ways to optimize, for example, before writing data to MyCat, we calculated consistent hash for each track, grouped the data with the same hash value together, and then wrote to MyCat in batches. To reduce the overhead of distributing MyCat to individual data notes. In addition, sonwflake, Twitter’s distributed auto-increment ID algorithm, is used for ID components and transformed into auto-increment Big Int components to improve write performance.
After using MyCat for a period of time, we are also thinking about whether the cost of node expansion is high for the current cluster. Is it a big risk? The result was that it would take us 4 to 5 days to migrate data after node expansion, which was obviously quite expensive. This time, we focus on the TiDB, this level of product support elastic extension of official introduction, can easily cope with high concurrency, huge amounts of data scenarios, supports distributed transactions, and disaster recovery and automatic failover, it sounds very tempting, bosses talked of the things I’m looking for r&d, we hit it off, at the back of the things are going very well, Resource application and deployment environment, we quickly arranged 3 TiDB servers, 3 TiKV and 3 PD clusters, and started a series of scene verification.
Problems encountered
The first problem was that when verifying the device management module, it was found that the load of each node in the whole cluster was not high, but the processing efficiency was relatively low, resulting in the backlog of queues. In order to solve this problem, we also consulted official colleagues and made some optimizations, such as adjusting batch updates to reduce costs, expanding the capacity of a TiDB server node, and most importantly upgrading the TiDB version from 2.04 to 3.05.
Another problem is hot data, because the MySQL model component uses the increment int type, hot data effect is obvious after migration. To solve this problem, we change the primary key to uuid and use statements such as shard_row_ID_bits =N to split the data. After splitting the data, the efficiency of data writing is greatly improved. Now that we’ve heard that the 4.0 GA version of AutoRandom solves the same problem by eliminating the need for Uuid components, we can look forward to new features in this release.
What pain points TiDB solves
Firstly, its horizontal extension feature solves the problem of high maintenance cost brought by MyCat and other middleware sub-database sub-table. Improve the computing and storage capabilities of the system by seamlessly expanding TiDB and TiKV capabilities.
Second, TiDB is compatible with the existing MySQL syntax and protocol, and the migration cost is low. We migrated from MyCat cluster to TiDB with very little business code. In terms of data migration, the historical data was read from MyCat cluster and then written to TiDB cluster through the small migration tool developed. The data was double-written in the code layer, and we successfully migrated the data to TiDB.
Third, massive data, query efficiency is very good. Our track data is partitioned according to date, and 400 to 500 million data will be written every day. Therefore, in a data scenario of this magnitude, our device ID query can generally return results within 10 milliseconds, which can meet the requirements of our business scenarios. Fourth, capacity expansion and upgrades are very quick. TiDB is really good for version upgrades. After all the preparation, the entire upgrade was completed in less than 3 or 4 minutes, and the user experience was great.
Application prospect of TiDB in Internet of Things
Our company’s core product is the Iot card, the number of cards is more than 700 million; Another product is smart networking, which now has nearly 16 million home gateways; In terms of smart home and intelligent entertainment, we have about 7 million cameras and smart wearable devices. These scenes are all scenes with high concurrency and massive data. I think these scenes are suitable to be migrated to TiDB. As the application of our iot scenarios on TiDB becomes more mature, we will promote more businesses to migrate to TiDB in the future. At the same time, WE also hope that the students of PingCAP company can bring us more excellent products.