In this Valkey blog post, I will demonstrate how we can set up a simple replication process among the Valkey nodes and then finally use some auto-failover mechanism with the help of the Valkey-sentinel tool.
Basically, we have two setups with the information below.
|
1 |
IP valkey port sentinel port<br>172.31.21.83 6379 26379<br>172.31.58.227 6379 26379 |
Now let’s prepare some basic configurations in file [/etc/valkey/valkey.conf] for both servers.
|
1 |
bind IP ### Need to bind both IPs separately for each server.<br>requirepass valkey<br>masterauth valkey |
- requirepass – To add authentication in valkey
- Masterauth – To connect the replication securely.
Now restart the Valkey service on each node.
|
1 |
systemctl restart valkey |
Next, we will set up a basic replication setup. To establish the replication, we need to execute the command below from the slave node.
Syntax:
|
1 |
replicaof <masterip> <masterport><br> |
Command:
|
1 |
172.31.58.227:6379> replicaof 172.31.21.83 6379<br>OK |
The above setting can be persisted using the [config rewrite] command.
That’s all. The replication setup is successful now!
|
1 |
172.31.58.227:6379> info Replication<br># Replication<br>role:slave<br>master_host:172.31.21.83<br>master_port:6379<br>master_link_status:up<br>.... |
We need to make sure the firewall/IP whitelisting is done for both servers.
Now, let’s do some data writing on the master[ 172.31.21.83] to confirm if the replication works fine or not.
|
1 |
shell> valkey-cli -h 172.31.21.83 -a valkey |
|
1 |
172.31.21.83:6379> set key key1<br>OK<br>172.31.21.83:6379> get key<br>"key1" |
On slave[172.31.58.227], the changes were very well reflected.
|
1 |
172.31.58.227:6379> get key<br>"key1" |
The only caveat with this setup is if the master[172.31.21.83] goes down, then we have no auto-failover to sustain the incoming connections on other nodes unless we do some manual intervention.
However, there is one way of setting up an auto-failover mechanism in Valkey by using valkey-sentinel tool. In this way, the application can either use the direct sentinel port [26379] to connect to Valkey server or use some middleware/haproxy tool to distribute read/write. Well, we do not go into those details and are confined to the sentinel setup only here.
First, we can edit the sentinel configuration file [/etc/valkey/sentinel.conf] on all nodes as below.
|
1 |
sentinel monitor valkey-master 172.31.58.227 6379 2<br>sentinel down-after-milliseconds valkey-master 1500<br>sentinel failover-timeout va;key-master 3000<br>sentinel parallel-syncs valkey-master 1<br>sentinel auth-pass valkey-master valkey |
- sentinel monitor – It tells sentinel to start monitoring a new master with the specified name, ip, port, and quorum.
- sentinel down-after-milliseconds – Sentinel verify based on this if the instance is unreachable/down or not before doing any failover.
- sentinel failover-timeout – In case a sentinel voted another sentinel for the failover of a given master it will wait this many milliseconds to attempt to failover the same master again.
- sentinel parallel-syncs – The number of replicas that can be synced with the new master after a failover at the same time.
- sentinel auth-pass – To perform the secure authentication when requirepass is enabled in valkey configurations.
Then we can restart the valkey/valkey-sentinel services as below.
|
1 |
systemctl start valkey-sentinel<br>systemctl restart valkey |
We can see the sentinel process running on the default port [26379].
|
1 |
shell> ps aux | grep valkey-sentinel<br>valkey 14607 0.2 0.4 61332 15428 ? Ssl 06:54 0:13 /usr/bin/valkey-sentinel *:26379 [sentinel] |
Let’s do some failover now.
I stopped the Valkey service on the original master[172.31.21.83]. And then after a few seconds, slave[172.31.58.227] becomes the master.
|
1 |
172.31.58.227:6379> info replication<br># Replication<br>role:master<br>connected_slaves:0 |
Looking over the sentinel logs [/var/log/valkey/sentinel.log], we can see the master switchover events captured.
|
1 |
14607:X 04 May 2024 06:57:28.176 # +switch-master valkey-master 172.31.21.83 6379 172.31.58.227 6379<br>14607:X 04 May 2024 06:57:28.177 * +slave slave 172.31.21.83:6379 172.31.21.83 6379 @ valkey-master 172.31.58.227 6379<br>14607:X 04 May 2024 06:57:28.180 * Sentinel new configuration saved on disk<br>14607:X 04 May 2024 06:57:29.693 # +sdown slave 172.31.21.83:6379 172.31.21.83 6379 @ valkey-master 172.31.58.227 6379 |
Again, once we start the service on the old master [172.31.21.83], it automatically connects as a slave.
|
1 |
172.31.21.83:6379> info replication<br># Replication<br>role:slave<br>master_host:172.31.58.227<br>master_port:6379<br>master_link_status:up |
The status of the new master also reflects the changes.
|
1 |
172.31.58.227:6379> info replication<br># Replication<br>role:master<br>connected_slaves:1<br>slave0:ip=172.31.21.83,port=6379,state=online,offset=705402,lag=0 |
A basic replication setup in Valkey/Redis would be very handy to have multiple copies of data over the different machines and also act as a backup/disaster recovery option. Still, it’s better, especially for production setups, to have a sentinel layer in the topology to have a minimal impact on the application in case the master node goes down due to some reasons.
