[NoSQL Redis] RedisShake : synchronize data from one redis to another

Redis-Shake

Redis-shake is developed and maintained by NoSQL Team in Alibaba-Cloud Database department.

Redis-shake has made some improvements based on redis-port, including bug fixes, performance improvements and feature enhancements.

Main Functions

The type can be one of the followings:

• decode: Decode dumped payload to human readable format (hex-encoding).

• restore: Restore RDB file to target redis.

• dump: Dump RDB file from source redis.

• sync: Sync data from source redis to target redis by sync or psync command. Including full synchronization and incremental synchronization.

• rump: Sync data from source redis to target redis by scan command. Only support full synchronization. Plus, RedisShake also supports fetching data from given keys in the input file when scan command is not supported on the source side. This mode is usually used when sync and psync redis commands aren’t supported.

Please check out the conf/redis-shake.conf - https://github.com/alibaba/RedisShake/blob/develop/conf/redis-shake.conf to see the detailed parameters description.

Configuration

Redis-shake has several parameters in the configuration conf/redis-shake.conf, that maybe confusing, if this is your first time using, please visit this tutorial - https://github.com/alibaba/RedisShake/wiki/tutorial-about-how-to-set-up.

Verification

User can use RedisFullCheck - https://github.com/alibaba/RedisFullCheck to verify correctness.

Metric

Redis-shake offers metrics through restful api and log file.

• restful api: curl 127.0.0.1:9320/metric.

• log: the metric info will be printed in the log periodically if enable.

• inner routine heap: curl http://127.0.0.1:9310/debug/pprof/goroutine?debug=2

Redis Type

Both the source and target type can be standalone, opensource cluster and proxy. Although the architecture patterns of different vendors are different for the proxy architecture, we still support different cloud vendors like alibaba-cloud, tencent-cloud and so on.

If the target is open source redis cluster, redis-shake uses redis-go-cluster - https://github.com/chasex/redis-go-cluster driver to write data. When target type is proxy, redis-shakes write data in round-robin way.

If the source is redis cluster, redis-shake launches multiple goroutines for parallel pull. User can use rdb.parallel to control the RDB syncing concurrency.

The move slot operations must be disabled on the source side.

Usage

You can directly download the binary in the release package - https://github.com/alibaba/RedisShake/releases. Run through similar commands:

$ ./redis-shake.linux -type=sync -conf=redis-shake.conf # please note: user must modify redis-shake.conf first to match needs.

FAQs

restore key[x:x] error[MOVED xxxx when target to Alibaba Cloud KVStore

[PANIC] dbRumper[0] executor[0] restore key[a:a] error[MOVED 6060 172.16.1.100:6379]: pttl[15346234], value length[1030], rdb version[9], checksum[3651], check error[<nil>]

(TODO) May be the Redis version between source and target is not compatible。

Target key name already exists

[PANIC] dbRumper[0] executor[0] restore key[a:a] error[BUSYKEY Target key name already exists.]: pttl[15346234], value length[1030], rdb version[9], checksum[3651], check error[<nil>]

Set key_exists to rewrite to fix that issue.

# redis-share.conf

key_exists = rewrite

References

[1] alibaba/RedisShake: redis-shake is a tool for synchronizing data between two redis databases. Redis-shake 是一个用于在两个 redis之 间同步数据的工具，满足用户非常灵活的同步、迁移需求。 - https://github.com/alibaba/RedisShake

[2] CodisLabs/redis-port - https://github.com/CodisLabs/redis-port

[3] alibaba/RedisFullCheck: redis-full-check is used to compare whether two redis have the same data. redis-full-check用于比较2个redis数据是否一致，支持单节点、主从、集群版、以及多种proxy，支持同构以及异构对比，redis的版本支持2.x-5.x。 - https://github.com/alibaba/RedisFullCheck

[4] [wuxibin89/redis-go-cluster: redis cluster client implementation in Go - https://github.com/chasex/redis-go-cluster](https://github.com/chasex/redis-go-cluster