Abstract: This issue will cover in detail the migration of SSDB, LevelDB, and RocksDB to GaussDB(for Redis).

This article is shared from the Huawei Cloud Community “GAUSSDB (forRedis) migration series (2)”, the original author: Gauss Redis official blog.

GAUSSDB (for Redis) is a cloud native NoSQL database based on the computing storage separation architecture, compatible with the Redis ecology. It supports persistent storage based on the strong consistency mechanism of multiple copies in the shared storage pool. While ensuring the database’s high compatibility, cost-effective, high reliability, non-destructive capacity expansion and other characteristics, GAUSSDB (FORREDIS) team provides users with a variety of data migration solutions for different database products. This issue covers SSDB, LevelDB, and RocksDB migration to GaussDB(for Redis) in detail.

1, SSDB to GaussDB(for Redis) migration

SSDB is a high-performance NoSQL database developed in C/C++. It has a similar API to Redis and supports data structures like KV, List, Map (hash), ZSet (sorted), and QList (queue). So it’s widely used. SSDB is a persistent KV storage system that uses LevelDB as the storage engine. The business interacts directly with LevelDB, and operations such as CompAction have a direct impact on business reads and writes. GAUSSDB (ForRedis) is a Redis-compatible cloud native NoSQL database, based on a shared storage pool with strong consistency of multiple copies to ensure data security and reliability. GaussDB(Forredis) uses RocksDB as the storage engine, and its performance has been greatly improved compared with LevelDB, and solved LevelDB active limit write problem, while achieving the separation of cold and hot, reduce the impact of storage layer operation on performance.

1.1 Principle of Migration

SSDB-port is run as the replica node of the main node of the SSDB database at the source end, and data migration is carried out through master-slave replication. The obtained data will be parsed, converted to a format supported by Redis, and sent to the specified Redis instance in the configuration file. The migration process is shown in the following figure. After the full synchronization is completed, the newly added data in SSDB is also synchronized to the Redis instance.

1.2 Preconditions

  • Deploy the migration tool ssdb-port.

  • The migration tool SSDB-port, source SSDB and target GAUSSDB (for Redis) network should be interoperable.

    1.3 Operation steps

  • Correctly modify the configuration file conf for ssdb-port.
  1. Start the process with the command./ssdb-portssdb_port.conf for data migration.
  2. Tracking migration logs, evaluating migration progress, and verifying the correctness and completeness of migration data on Gauss Redis instances after migration completion.
  3. When validation is complete, the business is cut to GaussDB(for Redis).

1.4 Instructions for use

  • As the slave node of the SSDB master node, SSDB-port only reads full and incremental data, resulting in no risk of data damage.
  • Source-side SSDB performance is affected by the use of the SSDB-port migration tool on the source side.

  • Full migration and incremental migration can be non-stop service, all data into GaussDB(Forredis) after a short stop.

    1.5 Migration performance reference

  • Environment: The source SSDB and SSDB-port are deployed on Huawei Cloud 4U16GB elastic cloud server at the same time, the target end is 8U16GB, and the 3-node GausSDB (for Redis) instance.
  • Preset data: Preset 100GB of data using the memtier_benchmark tool.

  • Migration performance: about 3000Qps.

    LevelDB to GaussDB(for Redis) LevelDB to GaussDB(for Redis

    LevelDB is an open source, persistent KV standalone database engine with high random-write, sequential read/write performance. It is suitable for use in write more and read less scenarios. Its internal design is not C/S network structure, use and service must be deployed in the same server, for the deployment of services, there are greater restrictions. LevelDB has many disadvantages compared to the LevelDB RocksDB developed on the basis of LevelDB, such as it cannot make good use of the computing performance of multi-core servers, cannot support terabyte data storage, does not support reading data from HDFS and so on.

GaussDB(for Redis) uses RocksDB as a storage engine, compatible with the Redis protocol, with rich data types, can meet the use of LevelDB requirements. At the same time, GaussDB(for Redis) deeply customized RocksDB, to achieve second level split elastic expansion, expansion and contraction capacity without relocation data, fast and smooth, for LevelDB business to provide convenience to the Redis ecosystem.

2.1 Principle of Migration

Use leveldb-port, the leveldb migration tool, and deploy on the same machine as leveldb, prepare the configuration files, and start the migration automatically to complete the full and incremental migration. Full migration takes a snapshot of LevelDB data, then scans the entire database, packages the data into a format recognized by GausSDB (Forredis), and sends it to GausSDB (Forredis), which has high migration efficiency. The incremental migration resolves the LevelDB WAL file, resolves the LevelDB operations, and then shard the keys and sends them through multiple threads.

2.2 Operation steps

Deploy LevelDB-Port on the LevelDB server and make sure to connect to the Gauss Redis network.

Ensure that the target GAUSSDB (for Redis) instance has been cleared. For security reasons, do not provide the flushall command to the outside. Ensure that the data in the instance has been cleared before migration.

Modify the configuration file.

  1. Preparation:
  • Deploy LevelDB-Port on the LevelDB server and make sure to connect to the Gauss Redis network.
  • Ensure that the target GAUSSDB (for Redis) instance has been cleared. For security reasons, do not provide the flushall command to the outside. Ensure that the data in the instance has been cleared before migration.
  • Modify the configuration file.
  • /leveldb-port./leveldb-port.
  1. Trace logs to determine migration status:

[Fixed] “FullMigrate finished” will be printed when the migration is complete; Incremental migration is timed to print the number of wal files that have not been completed and the number of kV that have not been sent.

  1. Data validation: Sampling and verifying the LevelDB to make sure GaussDB (Forredis) loads the data correctly.
  2. Service switch: When the incremental migration comes to an end, switch the service to GausSDB (for Redis).

2.3 Instructions for use

  • The migration tool needs to be deployed on the source side, and there is a cost to performance that can be controlled by modifying the configuration file.
  • The migration process reads LevelDB’s source data files as a read-only operation, with theoretically no risk of data corruption.
  • The migration process does not require downtime.
  • If the migration process fails, you need to clean up the GausSDB (Forredis) instance and restart the migration.

3. Migration from RocksDB to GaussDB(for Redis

RocksDB is a Facebook levelDB based persistent KV stand-alone database engine, with strong sequential read and write and random write performance. RocksDB has done a number of optimizations over LevelDB, improved performance, and solved the LevelDB issue of actively limiting writes. As a database engine, RocksDB is not designed into C/S network structure, and it needs to be deployed on the same server as the service directly for use, which has great restrictions on the deployment and use of the service.

GAUSSDB (for Redis) uses RockSDB as a storage engine, compatible with Redis protocol with rich data types, can meet the use requirements of RockSDB. At the same time, GaussDB(for Redis) deeply customized RocksDB to achieve second-level splitting and elastic capacity expansion. The expansion and contraction capacity need no relocation data, which is fast and smooth, and provides convenience for RocksDB’s business to be transferred to Redis Ecology.

3.1 Principle of Migration

Use the self-developed migration tool rocksdb-port, deploy on the same machine as rocksdb, prepare the configuration file, and start the migration to automatically complete the full and incremental migration. Full migration takes a snapshot of the RocksDB data, then scans the entire database, packages the data into a format recognized by GaussDB(Forredis), and sends it to GaussDB(Forredis), which has high migration efficiency. Incremental migration parses the Wal file of RocksDB, parses the operation of RocksDB, and then slices the key in it, and sends it by multi-thread.

3.2 Preconditions

  • Deploy the migration tool pika-port

  • Ensure that the source Pika instance, Pika-Port, and the target GaussDB(for Redis) instance network interworks.

    3.3 Operation steps

  • Preparation:
  • Deploy RockSDB -port on the RockSDB same server and make sure to connect with Gauss Redis network.
  • Ensure that the target GAUSSDB (for Redis) instance has been cleared. For security reasons, do not provide the flushall command to the outside. Ensure that the data in the instance has been cleared before migration. Modify the configuration file.
  • Start the migration: execute the command./rocksdb-port./rocksdb-port.conf
  1. Trace logs to determine migration status:
  • [Fixed] “FullMigrate finished” will be printed when the migration is complete;
  • Incremental migration is timed to print the number of wal files that have not been parsed and the number of kV that have not been sent.
  • Data validation: RocksDB is sampled to verify that GaussDB (Forredis) is loading data correctly.
  1. Service switch: When the incremental migration comes to an end, switch the service to GausSDB (for Redis).

3.4 Instructions for use

  • The migration tool needs to be deployed on the source side, and there is a cost to performance that can be controlled by modifying the configuration file.
  • The migration process reads the RocksDB source data files, a read-only operation, and theoretically there is no risk of data corruption.
  • The migration process does not require downtime.
  • If the migration process fails, you need to clean up the GausSDB (Forredis) instance and restart the migration.

4, endnotes

Based on the community version of Redis and combined with Huawei self-developed strong consistent storage DFV Pool, Gauss Redis has the advantages of strong consistency, second capacity expansion, super availability, low cost and so on, which ensures the accuracy and reliability of counting.

By: Huawei Yungauss Redis Team.

Hangzhou, Xi ‘an, Shenzhen Resume Delivery: [email protected]

For more technical articles, follow the official Gauss Redis blog:

https://bbs.huaweicloud.com/community/usersnew/id_1614151726110813

Click on the attention, the first time to understand Huawei cloud fresh technology ~