This article is based on the transcript of Huang Dongxu’s speech at PingCAP series D financing online press conference.

TiDB’s present and future

Hello, everyone. I am Huang Dongxu, co-founder and CTO of PingCAP. This is the first press conference of PingCAP since its establishment. Considering that many of the online audience may not have a strong technical background, I will do my best to make the technical part understandable.

Now, before WE get to the point, when we’re product managers in basic software and we talk to customers about requirements, they often say, For the database, my requirements are very simple, very basic, very simple, I do not require a lot of functions, security and stability is necessary, the best can be highly available, performance must be better, if the amount of data is large, can achieve elastic expansion is better; Also, it’s best not to make me learn too much new stuff. It’s pretty much the same product I used in the past. It’s a perfect database product.

Just like we use tap water at home, our demand for tap water is to turn the tap water to come out, but we do not need to know how to deal with the tap water factory behind, we just need to use cold water or hot water according to the actual situation. But from a technical point of view, a very simple basic requirement like hot and cold water, if you think about it in the database world, is a Turing prize level basic requirement, and just to explain a little bit, the Turing prize is the highest level in computing academia, the Equivalent of the Nobel Prize in computing.

We have two industry leaders here, Leslie Lamport on the left who won the Turing Award for his research in 2013, and Leslie Lamport on the right, who has a similar haircut to Liu Qi, who is a professor at UC Berkeley and the author of the famous CAP theorem, This is where the CAP in PingCAP comes from. Although this requirement seems to be a simple one, it is one that deserves to be studied for a long time, is technically challenging, and is a very cutting-edge research area.

Before talking about database, I want to take you to review a decade ago electronics, review our current life, everybody recall what are digital products, we have ten years ago we have nokia phone, for example, taking photos with a digital camera, there are also used for navigation independent of GPS equipment, listening to music with MP3, etc., are varied.

We’ll review these ten years, these things seem to gradually disappear in our life, a smartphone to unify many of these fragments of things up, I think that behind a very important point is our pursuit of a unified user experience driven the entire tech products occurred earth-shaking changes, Now a smart phone has basically solved 70 or 80 percent of the needs of digital life scenes in our lives.

TiDB is an HTAP system

Now, PingCAP is a database manufacturer. If we draw a number line, the business on the left is more real-time online. If the right side of this number line is offline, the product of database is probably on the left. Things like Hadoop or some data warehouses and reports are on the right.

Then look at a specific business scenarios, hypothesis is a company to create the electric business platform IT systems, comb inside electric business platform now has a variety of applications and scenarios, we according to the scene on the number axis, the left is online, on the right is offline, we see such as trading, order management, detailed query, IT could be a partial online business, Users can open their phones at any time; The offline business on the right is more like an internal operation, such as the boss checking how much money he made last year. This kind of report may be a more off-line business.

Have found that there are some among real-time reports, real-time pricing promotion, the recommendation of products sell like hot cakes, you are on the left is not appropriate, on the right side as if also is not quite right, so the middle section is a vague state, this is the language of a business, for example, we put the business in the shaft to go up and see, for example I am electric business platform and technical personnel, The business people tell me, what will these requirements look like when they are translated into the language of technology?

So it’s all kinds of OLTP databases and OLAP databases and data warehouses, things like ClickHouse, Greenplum, things like offline data warehouse Hadoop, HIVE, and for those of you who don’t know these terms, it’s okay, but I just want to show that business requirements are translated into technical language, It usually requires a complex set of data technology stacks to support it.

Many of you may have learned computer technology. I remember when I was in college, we had a course called database system. The teacher taught me that database is a system and software for storing and retrieving data, and several key commands are INSERT\SELECT\UPDATE\DELETE. I recall that there is no teaching which scenarios are OLTP scenarios and which are OLAP systems, it is not so complicated.

A database is supposed to store and retrieve data, just like a faucet. I also checked the definition of database, which does not say OLTP database or OLAP database in Wikipedia.

I know this may be a niche area, but from the roots of the word database, which is essentially a container, a system for storing and retrieving data, it doesn’t seem complicated.

Why is it that so many engineers today, so many users think that this database or this scenario must be an OLTP or an OLAP, that there needs to be a distinct classification. As in the e-commerce example, there are a lot of intermediate scenarios where it’s hard to say whether it’s an OLTP or an OLAP. However, the reality is that for many IT systems and business systems, the demand for real-time performance is getting higher and higher. In order to solve this problem, we have built all kinds of data islands and built all kinds of chimney-like systems.

So is the past categorization method suitable for the era of more and more real-time requirements?

When we look back and think about whether this classification is problematic, as a science guy or as an engineer with a science background, we particularly like to look for a definition or look for a classification. We have looked at a variety of definitions, from academia, industry, and consulting agencies, and found that HTAP is a definition that is more consistent or more suitable for the current TiDB application scenario.

TiDB is positioned as a real-time HTAP system. Many friends later asked me whether TiDB is a HTAP system, does that mean you are not an OLTP system, or whether you are an OLTP or OLAP?

Let’s go back to the example of smart phone. First of all, smart phone must be a 100% mobile phone. It must be able to make phone calls. I want to emphasize that TiDB is positioned as a real-time HTAP system, first of all a 100% OLTP system with some support for real-time OLAP Query.

When it comes to TiDB, I’m actually very impressed. I’ve been watching the product grow, especially in the last year, and I’m happy to see that TiDB 4.0 has become a mainstream release in the community.

When 4.0 was released, I made a passionate statement saying that it was a very milestone release. In fact, TiDB 4.0 has lived up to expectations, and now people like it so much that it has become mainstream.

Looking forward to TiDB 5.0

I want to give you a preview of TiDB 5.0. Before we get to 5.0, I want to emphasize a little bit about how TiDB makes products. We are all engineers, but also more down-to-earth, not what lofty, in plain English on the four points: steady, fast, easy to use and use the rest assured. As mentioned earlier, the user’s naive need for a database product is what we want to do with the product.

But in TiDB 5.0 we really put a specific goal into the direction of the product planning, that is, TiDB is going to be the core scenario of all industries. Previously, FROM 2.0 to 3.0 and 4.0, TiDB has gradually entered all walks of life, slowly infiltrating into some scenarios that require extreme stability and performance, including some core business systems of finance and banks.

With TiDB 5.0, we have for the first time explicitly stated that, at least at the product level, we need to meet the requirements of database performance and stability for core scenarios in various industries.

Let’s talk about the progress of TiDB 5.0 in these aspects.

First of all, stability. I often say that it is very difficult to make something right, but not difficult to make a database right. Therefore, we have built various correctness test systems. Now TiDB has been made, the next stage is to make TiDB more stable, but there is still a lot of work to do to make TiDB more stable, there is no shortcut in this aspect, everyone knows the truth, the devil is in the details, only to make deeper and deeper, more and more detailed, I am very happy to see, In TiDB 5.0 we have taken TiDB stability a step further with our users.

The following figure shows the test results of 5.0 in a brokerage institution. In the 48-hour high-pressure test scenario, the system jitter of TiDB 5.0 was always less than 5%, and the performance was significantly improved compared with 4.0. I would like to emphasize that we have already entered the deep water zone of reform in terms of stability. The low-hanging fruit has been almost picked, and there are still some scenarios and technical hard problems to be overcome. We are pleased to see that 5.0 has a better performance in stability than 4.0.

Second, fast performance. Especially for basic software such as databases, especially in core application scenarios such as some of the bank’s core transaction systems, an extra millisecond delay can affect the overall system and user experience.

In TiDB 5.0 we were pleased to see that latency was almost doubled down compared to 4.0. This reminds me of the joke I often make with the development team that each release of TiDB is a bit like Moore’s Law, with each release doubling performance, doubling latency, and maintaining the same cost, which will continue to fall in the future. I’m glad to see that 5.0 still maintains this pattern, so faster performance and reduced latency will be an important label in 5.0.

On the other hand, if you are familiar with TiFlash, you must smile at the title. We mentioned before that TiDB is a real-time HTAP architecture, and AP is provided by TiFlash analysis acceleration engine. In 5.0, TiFlash will support distributed aggregation and distributed computing (MPP), enabling the OLAP capabilities of TiDB to truly extend to more application scenarios and more complex JOIN scenarios.

Yes, there are two things I want to emphasize in this regard.

First of all, when you see the same deployment across data centers and across regions, many students who do distributed systems will feel very excited. TiDB can finally support geo-partition (multi-location multi-activity cross-region data distribution). Let me explain a little bit. TiDB is a distributed database, so there are a lot of users, a lot of developers who want TiDB to be truly deployed across long distances or across multiple data centers, forming a large network across the country or around the world.

Second, TiDB 5.0 will also be easier to use by integrating multiple popular stack ecosystems and providing a full-link tracking system.

Mention of Geo – Partion, in fact, there are a lot of customer demand have such scene, especially for some overseas customers, such as a European company’s business all over the world, at that time if there is a database system, able to support the global deployment across a long area, can be in a different country, different position at any time to provide services, Operations also eliminate the cost of deploying multiple stacks and database systems, making it an ideal choice for customers.

For example, we just mentioned how to reduce local latency in multi-location deployment. If a European company has a strict GDPR, it is known that the data of the enterprise cannot be exported. In this scenario, Geo-partition is a very practical feature.

If today’s TiDB were to do this, say, in Europe, China, and the US at the same time, the database performance and latency would be less than ideal in some scenarios, because you would need to go to a central server for processing, to get the timestamp, and then the service would be available.

In TiDB 5.0, we changed the central timing service to a distributed timing service, which enables the system to perform better locally or in multi-data center scenarios.

The second aspect is that TiDB has more and more “friends”. As a basic software, TiDB aims to unify many scenarios as much as possible, but I think the interconnection between TiDB and other ecological technology stacks in the industry is also very important.

TiDB is now able to communicate with Flink, Kafka, Spark, Hadoop, AWS S3, MySQL, Oracle and other traditional databases.

For example, this is an online real-time data warehouse business. The online business is constantly trading and writing, and the data is output to Kafka via TiDB TiCDC incremental subscription module, synchronized to Flink streaming computing engine for summary, aggregation and analysis, and then written back to TiDB. These data are written to TiDB to complement the online business. TiDB combines Kafka and Flink to form an easy-to-use real-time data warehouse architecture.

Speaking of security, security is actually very important for databases, and it is a very high priority feature in our product planning.

I would like to emphasize that in 4.0 TiKV storage layer already supports transparent data encryption for full link. In DBaaS platform, we are introducing RBAC, which is the authentication system based on user identity. At the same time, we will connect with AWS authentication system to provide users with more secure and reliable product and service guarantee.

In terms of security compliance, as TiDB is increasingly applied in some key business scenarios at home and abroad, different countries and different application scenarios have diversified requirements for compliance. TiDB has passed SOC2Type1 certification, which is a highly recognized compliance standard in the American financial industry. We will put more effort into compliance in the future.

What will databases look like in the future?

Above is the progress synchronization for TiDB 5.0 at this point in time. But where is the future?

In the last part of each share, we will talk about what the future should look like. I think that in the near future, an important direction of TiDB is to be more cloud. Why are clouds so important? Behind the database cloud, I think we can expand the following points:

  • One is Severless.

  • Second, intelligent scheduling ability;

  • The third is to take advantage of the cloud’s infrastructure to reduce costs.

Why are clouds so important? You can see the picture on the right, the horizontal axis is the amount of data and business requirements, the vertical axis is the enterprise cost of spending on IT, we are going to do in before the birth of the cloud, the uncertainty of business oriented in the amount of data, using traditional way of IDC deployment, the curve of the cost and investment can not perfectly match with the actual demand, There’s a lot of wasted resources.

First of all, only cloud can truly change the business model of the whole basic software into Pay as you Go, so as to fit the growth curve of the business as much as possible. For such important basic software as database, the elasticity of cloud can more reasonably meet the actual needs of the business.

Second, the cloud does a great job of shielding the complexity of the underlying infrastructure implementation, which makes sense for people who do basic software. Taking advantage of the cloud’s flexible scheduling capabilities and some new AI technologies, for example, allows the system to better understand the needs of the business and smarter plan where to put data and what indexes to build.

Thirdly, the cloud is a very good infrastructure. How to design the next generation of basic software for the infrastructure of the cloud era is a very important topic in my opinion. For those of you who follow the tech community, Snowflake’s news is very exciting, and Snowflake’s choice of technology has inspired me a lot about how to build a new generation of basic software using cloud infrastructure.

These are some of my prospects and views for the near future. Finally, I want to emphasize that what we want to do is to get back to the original database, to hide the complexity behind it, to provide users with a very low barrier to use, a great product that liberates productivity, thank you!