Abstract: This installment will cover the migration of SSDB, LevelDB, and RocksDB to GaussDB(for Redis) in detail.

This article is shared by Huawei Cloud community “Huawei Cloud PB-level database GaussDB(forRedis) Uncovered issue 11: GaussDB(forRedis) Migration Series (Part 2)”, the original article is written by Gauss Redis official blog.

GaussDB(for Redis) is a cloud native NoSQL database based on computing and storage separation architecture and compatible with Redis ecology. Based on the strong consistency mechanism of multiple copies in shared storage pools, GaussDB supports persistent storage. The GaussDB(forRedis) team provides multiple data migration solutions for different database products to ensure high compatibility, cost-effective, high reliability, and non-destructive capacity expansion of the database. This installment details the migration of SSDB, LevelDB, and RocksDB to GaussDB(for Redis).

1. Migrate SSDB to GaussDB(for Redis)

SSDB is a high-performance NoSQL database developed using C/C++ language. It has similar API with Redis and supports KV, List, Map (Hash), Zset (sorted set), qlist(queue) and other data structures, so it has been widely used. SSDB is a persistent KV storage system, using LevelDB as the storage engine. Operations such as Compaction affect read and write operations. GaussDB(forRedis) is a cloud native NoSQL database compatible with the Redis ecosystem. It is based on the strong consistency mechanism of multiple copies in a shared storage pool to ensure data security and reliability. GaussDB(forRedis) uses RocksDB as a storage engine. The performance of RocksDB is significantly improved compared with LevelDB, and levelDB actively restricts writes. It also implements hot and cold separation, reducing the impact of storage layer operations on performance.

1.1 Migration Principles

Ssdb-port operates as a replica of the primary NODE of the primary SSDB database at the source end. Data is migrated in primary/secondary replication mode. The obtained data is parsed and converted into the format supported by Redis, and sent to the Redis instance specified in the configuration file. The migration process is shown in the following figure. After full synchronization is complete, new data in the SSDB is also synchronized to the Redis instance.

1.2 Prerequisites

  • Deploy the migration tool SSdb-port.

  • Ensure that the migration tool SSDB-port, source SSDB, and target GaussDB(for Redis) can communicate properly.

1.3 Procedure

1. Correct the configuration file conf of ssdb-port.

2. Run the./ SSDB -portssdb_port.conf command to start the process and migrate data.

3. Track migration logs, evaluate migration progress, and verify the correctness and completeness of migration data on Gaussian Redis instance after migration.

4. After the verification, switch services to GaussDB(for Redis).

1.4 Usage Instructions

  • Ssdb-port serves as the secondary node of the primary SSDB node. It reads only full and incremental data and has no data damage risk.

  • If the SSDB-port migration tool is used at the source end, the SSDB performance at the source end may be affected.

  • Full migration and incremental migration can be performed continuously. After all data is migrated to GaussDB(forRedis), the GaussDB service must be stopped for a short time.

1.5 Migration Performance Reference

  • Environment: the SSDB and ssdb-port are deployed on a huawei 4U16GB elastic cloud server at the source end, and the target end is 8U16GB elastic cloud server. There are three GaussDB(for Redis) instances.

  • Preset data: use memtier_benchmark to preset 100GB data.

  • Migration performance: about 3000qps.

2. Migrate LevelDB to GaussDB(for Redis)

LevelDB is an open source, persistent KV stand-alone database engine with high random write, sequential read/write performance, suitable for the application of the scenario of many writes and few reads. It is not designed as a C/S network structure and must be deployed on the same server as the service, which has great restrictions on the deployment and use of the service. Compared with RocksDB developed based on LevelDB, LevelDB has many disadvantages, such as poor computing performance of multi-core servers, inability to support TERabyte data storage, and inability to read data from HDFS.

GaussDB(for Redis) uses RocksDB as the storage engine. It is compatible with Redis and has rich data types, meeting LevelDB requirements. At the same time, GaussDB(for Redis) provides in-depth customization for RocksDB, enabling second-level split and elastic capacity expansion. Rapid and smooth expansion without data relocation, facilitating LevelDB services to be transferred to the Redis ecosystem.

2.1 Migration Principles

Leveldb-port is deployed on the same machine as LevelDB, the configuration file is ready, and full and incremental migration can be automatically completed after starting the migration. Full migration Snapshots LevelDB data, scans the entire database, packages the data into a format that GaussDB(forRedis) can identify, and sends the data to GaussDB(forRedis), providing high migration efficiency. Incremental migration parses LevelDB’s WAL files, parses LevelDB’s operations, shards the keys, and sends them through multiple threads.

2.2 Procedure

Deploy Leveldb-port on the levelDB server and connect to the Gauss Redis network.

Ensure that the GaussDB(for Redis) instance on the target end is cleared. To ensure security, the flushall command is not provided. Ensure that data in the instance is cleared before migration.

Modify the configuration file.

1. Preparations:

  • Deploy Leveldb-port on the levelDB server and connect to the Gauss Redis network.

  • Ensure that the GaussDB(for Redis) instance on the target end is cleared. To ensure security, the flushall command is not provided. Ensure that data in the instance is cleared before migration.

  • Modify the configuration file.

1. Run the./leveldb-port./leveldb-port.conf command.

2. Trace logs to determine the migration status:

  • If the migration is complete, “Fullmigrate Finished “is displayed.

  • Incremental migration is timed to print the number of wal files that have not been completed and the number of kV files that have not been sent.

1. Data verification: Sample LevelDB to ensure that GaussDB (forRedis) loads data correctly.

2. Service switchover: Switch services to GaussDB(for Redis) at the end of incremental migration.

2.3 Usage Instructions

  • The migration tool must be deployed on the source end, which consumes performance. You can modify the configuration file to control performance.

  • The migration process reads LevelDB’s source data files and is read-only, theoretically with no risk of data corruption.

  • The migration process does not need to stop service.

  • If a fault occurs during the migration, clear the GaussDB(forRedis) instance and restart the migration.

3. Migrate RocksDB to GaussDB(for Redis)

RocksDB is a persistent KV stand-alone database engine developed by FaceBook based on LevelDB, with strong sequential read and write performance and random write performance. RocksDB has made a number of optimizations over LevelDB, improved performance, and solved LevelDB’s problem of actively limiting writes. As a database engine, RocksDB is not designed as a C/S network structure, and needs to be deployed on the same server as the service directly, which has great restrictions on the deployment and use of the service.

GaussDB(for Redis) uses RocksDB as the storage engine and is compatible with the Redis protocol. It has rich data types and meets the requirements of RocksDB. At the same time, GaussDB(for Redis) provides in-depth customization for RocksDB to achieve elastic capacity expansion by splitting in seconds, enabling rapid and smooth expansion without data relocation, facilitating the transfer of RocksDB services to the Redis ecosystem.

3.1 Migration Principle

Use the self-developed migration tool Rocksdb-port, deploy it on the same machine as Rocksdb, prepare the configuration file, and start the migration to automatically complete full and incremental migration. Full migration Takes a snapshot of RocksDB data, scans the entire database, packages the data into a format identified by GaussDB(forRedis), and sends the data to GaussDB(forRedis), which has high migration efficiency. Incremental migration parses the WAL file of RocksDB, parses the operations of RocksDB, splits the keys, and sends them through multiple threads.

3.2 Prerequisites

  • Deploy the migration tool pika-port

  • Ensure that the source Pika instance, piKA-port, and target GaussDB(for Redis) instance can communicate properly.

3.3 Procedure

1. Preparations:

  • Deploy rocksdb-port on the same server as Rocksdb to ensure that the network is connected to Gauss Redis.

  • Ensure that the GaussDB(for Redis) instance on the target end is cleared. To ensure security, the flushall command is not provided. Ensure that data in the instance is cleared before migration.

  • Modify the configuration file.

1. Run the./rocksdb-port./rocksdb-port

2. Trace logs to determine the migration status:

  • If the migration is complete, “Fullmigrate Finished “is displayed.

  • Incremental migration is scheduled to print the number of wal files that have not been parsed and the number of kV files that have not been sent.

1. Data verification: Perform sampling verification on RocksDB to ensure that the GaussDB (forRedis) loads data correctly.

2. Service switchover: Switch services to GaussDB(for Redis) at the end of incremental migration.

3.4 Usage Instructions

  • The migration tool must be deployed on the source end, which consumes performance. You can modify the configuration file to control performance.

  • The migration process reads RocksDB’s source data files and is read-only, theoretically with no risk of data damage.

  • The migration process does not need to stop service.

  • If a fault occurs during the migration, clear the GaussDB(forRedis) instance and restart the migration.

4, endnotes

Gauss Redis is based on community Redis and combines with Huawei self-developed DFV Pool for strong consistency, second capacity expansion, over-availability, low cost and other advantages to ensure the accuracy and reliability of counting.

Author: Huawei Cloud Gauss Redis team.

For more technical articles, follow the official Gauss Redis blog:

Bbs.huaweicloud.com/community/u…

Click to follow, the first time to learn about Huawei cloud fresh technology ~