Valkey (a community-driven fork of Redis) uses a high-performance replication model to synchronize data between a primary node and its replicas. One critical component of this system is the client-output buffer, especially the configuration of its limits for replicas.
This blog explores the client-output buffer, how its limits work in the context of replicas, and why tuning it properly is essential for maintaining healthy replication in Valkey.
The client-output-buffer-limit setting directly affects memory usage on the primary node. All outgoing data to connected clients, such as replicas or pub/sub subscribers, is temporarily stored in memory until acknowledged or consumed. If a client (e.g., a replica) is slow to read this data, the buffer on the master can grow up to the configured limit. This means that higher buffer limits can increase the risk of memory pressure on the master node, especially in scenarios with many replicas or high-throughput workloads.
Understanding how these limits work and tuning them to your deployment’s needs can help prevent replication issues and improve the resilience of your Valkey cluster.
What is the client-output buffer?
In Valkey, each connected client, including replicas, is assigned an output buffer. This buffer temporarily holds data that Valkey intends to send to the client. For replicas, the buffer is used to stream commands or RDB/AOF data that keep them in sync with the primary.
Since replicas can sometimes fall behind due to multiple causes, Valkey needs to manage how much data it allows to accumulate in memory before considering the replica too slow to keep up.
When configuring the client-output-buffer-limit, it’s critical to understand its interaction with the repl-backlog-size setting.
The replica client-output buffer holds the data sent from the primary to each replica. However, Valkey uses a replication backlog buffer—a circular buffer that stores the most recent write operations—to support partial synchronization, allowing replicas to catch up without needing a full resync. If the output buffer limit for replicas is set lower than the replication backlog size (repl-backlog-size), the configuration becomes illogical and is effectively ignored. In such cases, Valkey automatically uses the backlog size as the effective output buffer limit.
This behavior ensures that partial syncs can be completed successfully before the replica is disconnected due to buffer overflows. Importantly, this fallback scenario has no additional memory implications because the replication backlog and the replica’s output buffer share the same underlying memory allocation, both point to the same object in memory. This shared reference model avoids duplication and helps maintain efficient memory usage during replication.
For practical purposes, the replica output buffer should always be configured at least as large as the repl-backlog-size to ensure stable and efficient replication, particularly under high-throughput conditions. It’s also worth mentioning that Valkey will ignore the client-output-buffer-limit for replicas if the value is lower than repl-backlog-size
Buffer limits: A safety mechanism
Valkey allows administrators to set soft and hard limits for the client-output buffer through the following configuration directives:
|
1 |
client-output-buffer-limit replica <hard limit> <soft limit> <soft seconds> |
- Hard limit: If the replica’s buffer exceeds this value (in bytes), Valkey immediately disconnects the replica.
- Soft limit: If the buffer exceeds this value and stays above it for more than <soft seconds>, the replica is disconnected.
- Soft seconds: The amount of time the soft limit is tolerated.
For example:
|
1 |
client-output-buffer-limit replica 256mb 64mb 60 |
This configuration allows a replica to have up to 256MB of output buffer. If the buffer grows past 64MB and stays there for over 60 seconds, Valkey will disconnect the replica.
If no limits are explicitly configured, Valkey applies default values, which are a 256mb hard limit, a 64mb soft limit, and a soft limit duration of 60 seconds.
You can configure the client-output-buffer-limit for replicas either at startup or during runtime.
Configuration file (at startup):
Add the following line to your valkey.conf file:
|
1 |
client-output-buffer-limit replica 256mb 64mb 60 |
Runtime (without restarting):
Use the CONFIG SET command to apply the change dynamically:
|
1 |
CONFIG SET client-output-buffer-limit “replica 256mb 64mb 60” |
To ensure this change persists after a server restart, run:
|
1 |
CONFIG REWRITE |
This will save the current in-memory configuration to the config file.
Why does this matter?
These limits prevent memory exhaustion on the primary node. If one or more replicas become too slow and the primary keeps queuing data for them, memory usage could grow uncontrollably.
A disconnected replica will attempt to reconnect, and, depending on the situation, perform either a partial sync (PSYNC) or a full resync (which is more resource-intensive). Therefore, buffer limits also affect replication resilience and failover performance.
Best practices
- Monitor buffer usage using INFO replication and CLIENT LIST.
- Set realistic buffer limits based on expected write throughput and replica capacity.
- Use metrics and alerts to catch replication lag before disconnections occur.
- Prefer stable, low-latency networks between nodes.
Conclusion
The client-output buffer limit for replicas in Valkey is a key mechanism to safeguard the primary’s memory and ensure system stability. Higher buffer limits can increase the risk of memory pressure on the master node, especially in scenarios with many replicas or high-throughput workloads. Understanding how these limits work and tuning them to your deployment’s needs can help prevent replication issues and improve the resilience of your Valkey cluster.
Learn more about Percona software and services for Valkey
