In this blog post, we are going to implement the concept of sharding in a Valkey setup. This is a built-in feature and can be implemented by enabling clustering in the Valkey configuration.

Sharding, in general, helps in distributing/scaling application writes over multiple nodes. In a similar fashion, it works in Valkey. Here, it uses the hash slots, which automatically distribute the keys over the different master nodes.

Let’s start with some practical stuff.

Our environment consists of six Valkey nodes.

Below is our basic Valkey configuration file [/etc/valkey/valkey.conf] required for clustering/sharding. 

Here, “nodes.conf” file will automatically generate and update the information about the cluster nodes when required.

Cluster creation

The below command will create three shards, each containing one master and one slave. 

Output:

That’s it. The sharded environment is ready now.

Below we can see the slot distribution with respect to each shard. The slaves are pointing to each of their masters. 

Output:

For a production environment, we can better control the failure/timeout handling by using the below additional cluster settings as well.

Write activity in the cluster

We can interact with the nodes in cluster mode with the “-c” option:

And then perform the writes. The keys are automatically distributed to different nodes, based on the modulo-hash of the key, mapped to the 16K slots. (Ex: hash(key) % 16384 = slot 3245, which lives currently on server X).

In a similar fashion, we can get the details as well.

Adding  new nodes 

If we wanted to add additional shards to our cluster, we can do that by the “–cluster add-node” option with the below  command. There is no requirement that each shard have both a master and slave, and masters can have multiple slaves.

Add new slave or DR nodes:

Here, the first node is the new slave node and the second node is the master on which this node needs to be added.

Add new Master nodes:

 Here, the first node is the new master node, and the second node name can be any live master node for reference.

Cluster resharding

Resharding involves moving hash slots from one shard to another shard. This also helps in balancing/distribution of hash slots when adding a new master node.

Let’s see the below scenario of adding a new master node[172.31.56.37].

Here, we add the new master node via any of the existing master nodes as a connection source.

Output:

The node [172.31.56.37] was successfully added. Now, if we check the status of this new master, we observe it doesn’t contain any hash slots yet. Yes, the auto-rebalancing is not automatic with Valkey/Redis sharding, and we have to rely on a manual resharding approach.

Output:

The output above consists of the below tokens:

 

  1. Node ID
  2. IP: Port (redis/cluster port)
  3. Flags: Master, Replica etc
  4. If showing Replica, the Node ID of the Master
  5. Time of the last pending PING waiting for an acknowledgement/reply
  6. Time of the last PONG received
  7. Configuration epoch for this node
  8. Status of the node
  9. Slots allocated.

 

Let’s do resharding now.

  • Connect to any of the existing nodes and then follow the instructions over the interactive option.

  • Now, if we check the status again, we can observe that around ~1k slots moved from existing masters to this new master [172.31.56.37].

Output:

Also, we can use the non-interactive way as below to move/re-distribute the slots.

Failover in sharding/clustering

If one of the master’s in a shard goes down, the “Slave” will be promoted to “Master” automatically by the cluster. We don’t need any sentinel services for the failover; The cluster is itself capable of taking care of the failure under cluster mode.

Let’s see an example here.

We have the below Master which has one connected slave [172.31.56.12].

Now let’s stop the service on this Master node[172.31.46.16].

The slave[172.31.56.12] is now promoted to Master based on the node timeout settings.

If we start the service on the old Master [172.31.46.16], it will automatically be added as a slave of [172.31.56.12].

Summary

Here, we cover the basics of a sharding/clustering setup in the Valkey and Redis environment, with built-in master failover capabilities. Along with that, we also see how the key/slots distribution works in the topology and the slot migration/resharding when adding a new master node. For a production environment we should have at least three nodes per cluster to better cover the hash slots. The setup also should include the Slave nodes to have a highly available deployment.

Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments