In most database systems, like MySQL, PostgreSQL, and MongoDB, replication of some kind is used to create a highly available architecture. Valkey/Redis is no different in this regard. Replication is native functionality to Valkey, supporting multiple replicas, and even chains of replication.

To clear up any confusion, understand that Valkey replication is a different concept compared to Valkey clustering. 

Replication is a simple, asynchronous process in which writes on a single master are replayed on N-connected replicas. 

Clustering is a more complex architecture in which the data is fragmented/sharded across multiple Valkey servers, operating together as a single domain. Replication can and should be part of your clustered architecture, but it can also be used without clustering.

In this post, we will set up a basic master and two-replica configuration used for splitting reads and writes. To keep the post short and sweet, we will discuss failover using Valkey Sentinel in a later post.

Environment and configurations

In a typical production environment, each Valkey process would be installed on its own server. For this post, docker will be used to simplify things.

The working directory is laid out as follows:

In each data directory, hard links to the certificates were created, allowing all three Valkey containers to use the same keys. Example:

The configuration files for the master node, valkeyM1, are as follows:

The configuration file for the replicas is nearly identical, minus the user accounts and a few other master-only config parameters.

Now that we have our configuration files and TLS certificates ready, we can start launching containers.

Setting up the master

We launched our container, named valkeyM1, and specified our configuration file, located in the mounted volume path. We then grab the docker container’s IP address for later actions and then attempted to write a key-value pair.  This produced an error as no replicas are connected, and we specified in the config that at least one replica is required. The ROLE command is a simple status check that says we are currently a ‘master’, the replication offset is zero bytes (since there have been no writes yet), and the empty array indicates no replicas are connected.

Since we cannot do anything until at least one replica is connected, let’s set one up.

Setting up replicas

Note above that we did not specify the REPLICAOF parameter in the config file for this replica, valkeyR1. As such, this Valkey server is functioning as an independent master. Let’s connect this server to our M1 above.

If we look at the log for valkeyM1, we can see the synchronization attempt:

Notice that there are two replication IDs on the master. These are referred to as the ‘current ID’ and the ‘old ID’. The old ID is used by replicas which get promoted to master in the event of failover.

When a replica takes over as master during a failover event, it will change its ‘current ID’ to a newly generated ID, indicating the start of a new dataset/topology change. The ‘old ID’ will become the failed master’s ID. This allows for other replicas, which have reconnected to this replica as the new master, to continue receiving events from that failed stream. Once other replicas have received all old events, they will switch to the new ID of the newly promoted master. This is an automatic process.

The next several lines of the log output above show the snapshot creation of the RDB file, and streaming a copy to the replica. Recall that in our config of R1, we specified diskless sync.

Now that we have at least one replica connected to our master, we can write to it:

Additional replicas

We can start an additional replica. Two changes have been made to R2’s config. We added ‘REPLICAOF 172.17.0.6 6379’, and changed ‘repl-diskless-sync no’.

Because the ‘REPLICAOF’ command was in the configuration file, R2 immediately connected to M1 and began a sync of the RDB file. The RDB file was copied to R2’s disk initially and, after completion, loaded into R2’s memory.

Looking at the master’s ROLE, we can now see two connected replicas, which are both in sync with the ID offset of the master (2832):

Utilizing replicas and failover

In order to utilize the R1 and R2 for reads, your application must be configured to open separate connections to these instances and execute commands. Valkey does not have a single endpoint for routing connections. You must code the read/write split yourself.

Also, the above architecture has no failover capabilities. If valkeyM1 goes offline, R1 and R2 will continue to function as readers returning, potentially, stale data.

Notice the ‘-1’ value for “connect” in the ROLE output, indicating that we are not connected to a master. You can also see the successful read of the ‘foo’ key.

If you do not want replicas to serve stale data when the connection to the master is lost, you can set ‘replica-serve-stale-data no’, as shown here:

The replicas will continue reconnection attempts to the master every one second.

In order to have R1 or R2 automatically take over as the new master, Valkey Sentinel must be used. Setup of the sentinel is beyond the scope of this post, but will be the scope of a future post.

In the meantime, you can run ‘REPLICAOF NO ONE’ on R1, and create the replication user on R1, then tell R2 to become a REPLICAOF R2:

R1 will generate a new ‘current ID’ and will set its secondary ID to the failed M1’s ID. R2 will request partial resync from R1, using M1’s original ID. After the backlog is synced, R2 will switch to R1’s new ID.

Conclusion

In this post, we learned how simple it is to configure asynchronous replication in Valkey in order to create replicas for simple read/write load balancing and to get us started on our journey toward high availability.

Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments