This article was first published on OPPO’s public account, WeChat ID: OPPO_tech
1. What is graph database
A Graph database is a database that stores and queries data as a Graph structure. Unlike other databases, relationships take precedence in graph databases. This means that applications do not have to use foreign keys or out-of-band processing, such as MapReduce, to infer data connections. Graph databases also have simpler and more expressive data models than relational or other NoSQL databases.
Graph database is widely used in social network, knowledge graph, financial risk control, personalized recommendation, network security and other fields.
2. Graph database survey
2.1 Research background
With the continuous growth of knowledge graph and other business data, the existing graph database JanusGraph has been difficult to cope with, and the import time has been unable to meet the requirements of business. Therefore, it is urgent to find open source property graph database with better performance.
The new graph database should meet the following requirements:
- Capable of supporting a large scale atlas of 1 billion nodes, 10 billion edges and 17 billion attributes
- The full import duration does not exceed 10 hours
- The average second-degree query response time is less than 50ms, and the QPS can reach 5000+
- Open source and support for distributed property graph databases
2.2 Research process
The first step is to collect common open source distributed attribute graph databases, as shown in the following table:
The second step, based on the test reports of Meituan, LightGraph, TigerGraph, GalaxyBase and other graph databases, the performance of several graph databases can be obtained as follows:
- Nebula Graph | HugeGraph | JanusGraph | ArangoDB | OrientDB
- Nebula Graph > HugeGraph > JanusGraph > ArangoDB > OrientDB
Nebula Graph excels at both import and query performance.
Third, to verify Nebula’s performance, a one-time comparison test was run against Nebula Graph and JanusGraph. The results are as follows:
In the diagram above, viewed as JanusGraph performance at 1, Nebula Graph imports performance is an order of magnitude faster than JanusGraph, and query performance is 4-7 times faster than JanusGraph. Moreover, as the number of concurrent queries increases, the performance gap widens, and JanusGraph starts at 20 threads and has error third-degree neighbor queries. Nebula Graph has no errors.
Nebula Graph is currently investigating the SST import, which could significantly increase the import speed.
A second-degree neighbor query test against the Nebula Graph using 120 threads resulted in a QPS of 6000+, a slight improvement over a standalone version. The success rate was close to 5 9s, and the response time was relatively stable, averaging 18.81ms, p95 38ms and P99 115.6ms, which met the demand.
2.3 Research conclusions
Nebula Graph delivers the required import performance, response time, and stability required by Nebula Graph. With data shard support, the distributed version is free and open source, used by many enterprises, with comprehensive documentation in Chinese and an active community, making it an ideal choice for an open source graphics database.
Nebula Graph: Nebula Graph
Image courtesy of Nebula Graph documentation
Nebula Graph is an open source, distributed, easily scalable native Graph database capable of hosting extremely large data sets with hundreds of billions of points and trillions of edges and providing millisecond queries.
Nebula Graph is a graph-based database written in C++ with a shared-nothing architecture that supports scaling without shutting down database services and provides a plethora of native tools. Nebula Graph Studio, Nebula Console, Nebula Exchange, and more can make it easier to use a Graph database.
Image courtesy of Nebula Graph documentation
Nebula Graph is composed of three services: Graph services, Meta Services, and Storage Services.
Meta services are responsible for data management, such as Schema operations, cluster management, and user rights management. The services are provided by the Nebula – Metad process. In a production environment, it is recommended to deploy three Nebula – Metad processes in a Nebula Graph cluster. Deploy these processes on different machines to ensure high availability. All of the Nebula – Metad processes form a cluster based on the Raft protocol, where one process is the leader and the others are followers.
The Graph service is responsible for processing query requests, including parsing query statements, validating statements, generating an execution plan, and executing the execution plan. The Graph service is provided by the Nebula – Graphd process and can be deployed in multiple formats.
The Storage service is responsible for storing the data provided by the Nebula Storaged process. All nebula Storaged processes form a cluster based on the Raft protocol. The data is stored in the Nebula Storaged partition. Each partition has a leader, and other replica sets form the followers of that partition.
4. Figure database platform construction
When I used JanusGraph before, I encountered some problems such as slow import, slow query, high concurrency OOM (JanusGraph thread pool adopted unbounded queue), FULL GC (business Gremlin statement contained Value caused by continuous expansion of meta space), etc. This is largely resolved when we cut to Nebula Graph.
JanusGraph does not have a user-friendly management interface. As shown in the figure above, we developed a set of management interfaces including multi-graph management, Schema management, graph visualization, graph import, and permission management.
Nebula Graph Studio offers multigraph management, Schema management, diagram visualization, and diagram import capabilities that save a lot of development effort and ease the barriers to use.
The entire Graph database platform is structured as shown above, with emphasis on full import, incremental import, Graph export, backup/restore, and query engineering capabilities based on Nebula Graph and the Nebula Graph official tools.
The official import tool needs to provide the import configuration file. In order to facilitate the business use, we designed a Schema configuration form. The business only needs to fill in the form. Execute the Compact task. Currently, the batch write import mode is still adopted. SST import will be investigated in the future, and the import performance can be further improved.
The official import tool uses an asynchronous client, so it is very difficult to control the import rate during import. If the import rate is too large, it is easy to cause the backlog of graph database requests and affect the stable operation of the cluster. If the setting is too small, the speed cannot reach the optimal value and the import is slow. We have modified the source code of the official import tool, changing the asynchronous client to the synchronous client, which can give consideration to performance and stability.
An export tool based on the nebula- Spark software is available to export data, as well as Schema configuration and index configuration for data migration.
To support data rollback, we developed a quick backup and restore function for specified atlas data. However, this function cannot back up atlas metadata. Full import will delete and rebuild the atlas. In the future, we will try to avoid this problem by clearing data and not deleting graphs in full import.
There are many edge types in knowledge graph service. Often, dozens or hundreds of edge types need to be queried in one query. In fact, only Top 10 results (sorted by rank) of each type need to be returned. This situation is difficult to implement with nGQL. Only all the data of these edges can be queried, or the Top N data of all edges combined. The former has performance problems, and the latter often only returns data of some types of edges, which cannot meet the requirements. In this case, we classify edges, and for those with a smaller number of edge types, a single statement queries all data. For a large number of edge types, multi-threading is used to query the Top 10 of each edge in parallel, which can be avoided to a certain extent.
In order to ensure the high availability of the service, we implemented a dual machine room deployment. In order to prevent the upper business from sensing the machine room switch, a query project (graph retrieval) is carried out in the upper layer of the graph database. The business directly calls the service of the query project, and the query project will select the appropriate graph database cluster to query according to the cluster status. In addition, the query project manages query statements for all businesses in order to shield upper-layer businesses from changes and version upgrades to the underlying graph database. When the graph database is incompatible with query statements due to version upgrade, it is only necessary to adjust the graph query language in the query project to avoid affecting the upper business. At the same time, the query result is also cached in the query project, which can greatly improve the throughput of graph query.
Of course, there were a few issues that were addressed with Nebula Graph’s help, such as rank’s size side sorting failure and query results that only returned side type IDS, which are not listed here for space reasons.
Note: The Nebula Graph issue mentioned above is for V1.2.0 only, and many of the issues have been fixed in subsequent releases.
5. Business landing
5.1 Knowledge Graph and intelligent Q&A
Before using the atlas, Assistant Bubo only supported DBQA based on documents. DBQA uses unstructured text and is suitable for answering explanatory and argumentative questions such as Why and How, while the accuracy and coverage of factual questions are not high.
After using the atlas, Bubo assistant supports knowledge-based q&A KBQA, which greatly improves the accuracy and coverage of factual questions such as What and When. For example, who is XXX’s wife? How much does XXX Ultraman weigh? What’s the area of Beijing?
In addition to factual questions, Assistant Bubo can also use the reasoning ability of the graph to achieve some complex questions: for example, what is the relationship between XXX and XXX? What was the first phone OPPO released? What films have XXX and XXX co-acted in? Who are the Gemini stars born in xx?
Because of the large semi-structured data of knowledge graph and the existence of many relationships among the data, relational database can not meet the requirements of storage and query, but graph database can just solve the challenges of large-scale map storage and multi-hop query.
5.2 Content Labels
In some recommended scenarios, it is necessary to understand video, audio, or text content and label it with relevant content. For example, in short video recommendation, understanding the content of the video helps users to make accurate recommendation.
For film and television video, actor, film and television programs, role structure into a film and television entertainment, when there is a new release of the film and television class short video, can face recognition in video actor, identified in the title or subtitle film role, reasoning out the corresponding map with the film and television works, called video content on the label, is recommended as the result.
5.3 blood source of data
In the data warehouse, various ETL jobs often need to be run. There are so many data tables and tasks that how to intuitively observe the relationship between the upstream and downstream of the data table and tasks becomes an urgent problem to be solved.
It is very troublesome to use relational database to deal with multi-level associated query, not only the development workload is large, but also the query performance is very slow. The use of graph database, not only greatly reduce the workload of development, but also can quickly find out the upstream and downstream relationship of the table, easy to intuitively observe the blood relationship of data.
5.4. Service Architecture Topology
In service resource management, business resources are divided into multiple levels, and each level has corresponding servers, services and managers. If a relational database is used to process, when multi-level resources need to be displayed, the query will be very troublesome and the performance will be poor. At this point, the relationship between resources, managers, servers, and business levels can be put into a graph database, which can be displayed in a single query statement. The query speed is also very fast.
6, summary
The transformation from JanusGraph to Nebula Graph through the Knowledge Graph and other business practices increased import performance by an order of magnitude, with a 3-6 fold increase in query performance and concurrency. Also, Nebula Graph is more stable than JanusGraph. Nebula Graph has had a lot of help with the Nebula Graph community’s support.
Graph database has developed rapidly in recent years. Neo4j raised 325 million USD in the first half of this year, which set a new record for database financing. According to a Gartner report, “By 2023, graph technology will facilitate rapid decision scenarios in 30% of the global enterprises. The annual growth rate of graph technology adoption is more than 100 percent.” With the spread of 5G and the Internet of Things, graph databases will become the infrastructure for handling relationships.
7. Reference documents
- 1. The data structure: what is the graph: blog.csdn.net/dudu3332/ar…
- Nebula Graph: docs.nebula-graph.com.cn/2.0.1/1.int…
- 3. What is the graph database that is gaining popularity? www.cnblogs.com/mantoudev/p…
- 4. The application of figure scene: help.aliyun.com/document_de…
- 5. The most complete knowledge graph technology review: www.sohu.com/a/196889767…
- 6.KBQA from getting started to giving up: www.sohu.com/a/163278588…
- 7. graphdb-benchmarks:github.com/socialsenso…
- 8. Mainstream open source distributed graph database Benchmark: discuss.nebula-graph.com.cn/t/topic/137…
- 9. Figure database LightGraph test report: zhuanlan.zhihu.com/p/79426763
- 10. TigerGraph official test: www.tigergraph.com.cn/developers/…
- 11.GalaxyBase official test: blog.csdn.net/qq_41604676…
Author’s brief introduction
Qirong is a senior backend engineer at OPPO, working on graph database, graph computing and related fields.
If there are any errors or omissions in this article, please go to GitHub: github.com/vesoft-inc/… Submit an issue to us in the Issue section or submit a suggestion under the suggestion feedback section of the official forum: discuss.nebula-graph.com.cn/ 👏; Ac graph database technology? To join the Nebula Exchange group, please fill out your Nebula card and Nebula Assistant will bring you into the group