Replication Lag on AWS FSx: The Hidden EC2 Single-Flow Bandwidth Limit

A recent case in our Percona Support team started with a familiar complaint. A PostgreSQL standby lagging behind its primary. Although the problem was simple, it brought a specific flavor that’s worth sharing.

The customer had already reached out to AWS Support about the storage layer behind the database, an Amazon FSx filesystem mounted over NFS on the standby’s EC2 instance. Let’s not stigmatize the infrastructure choice. It met the customer’s requirements, and they paid for provisioned resources to guarantee a performance baseline, a better starting point than many similar cases we see.

AWS confirmed that both the EC2 instance and the FSx filesystem looked healthy, with the filesystem sitting at roughly 50% of its provisioned capacity. From that angle, storage looked like a non-issue, and the investigation could easily have moved elsewhere in PostgreSQL.

We see this pattern often enough to call it out. A component reporting comfortable headroom relative to its provisioned limit does not mean the component isn’t the bottleneck. This post walks through what we found and why the standard utilization numbers didn’t show it.

The problem with looking at utilization alone

Provisioned IOPS and throughput numbers describe what a storage backend can deliver, regardless of what the path between the database host and that backend can actually carry.

An EC2 instance talking to FSx over NFS goes through several layers, including the instance’s own network bandwidth, the network path to the filesystem, and the filesystem’s own limits. Usually, monitoring dashboards only consider the last of those. If any of the other layers cap out below the provisioned limit, the system can become saturated even as every capacity dashboard still shows green.

That’s exactly what we found here.

Step 1. vmstat shows processes stuck waiting on I/O

The first signal came from vmstat. The “b” column reports the number of processes blocked waiting for I/O to complete. In a healthy system, “b” sits at zero most of the time, with occasional small blips, but not this time. The “wa” column was also consistently high across all samples, pointing to storage/network wait, while si/so stayed low, an early sign that memory wasn’t the initial suspect.

That was not an occasional blip. Across all the samples, we classified every reading:

“b” was above zero in 99.4% of samples, and above 5 more than half the time. The CPU columns in the same output showed a consistent I/O wait (wa) value across all lines. Both point to a storage subsystem bottleneck, somewhere between the disk, the network, or the NFS server.

Step 2. nfsiostat sends mixed signals

nfsiostat displays NFS traffic by read and write operations, including latency. Looking at averages:

Average write exe time is 136.52 ms, most of which (134.53 ms) is queue time. That’s already a red flag, as 136.52 ms is long enough for any modern storage system. Since most of the time is spent in the queue rather than in the actual round trip, the bottleneck doesn’t seem to be the underlying disks themselves.

Comparing peak and p95 usage against the provisioned limits looked fine at first:

49% of provisioned throughput and 42% of provisioned IOPS at peak. On paper, this storage subsystem has plenty of room left. This is the same conclusion AWS Support reached, and it’s a reasonable one to draw from these two numbers alone.

Step 3. The distribution tells a different story from the average

Peak and p95 only describe the top of the range. They don’t say how often the system sits there. So we built a distribution of every 1-second read throughput sample instead of collapsing it into a single peak or percentile:

The single most common bucket among all samples is 4.75-5.00 Gbit/s, and no sample surpassed 5.01 Gbit/s, not a random peak but a ceiling. Every time the workload tried to push beyond 5 Gbit/s, it was capped at that rate instead of climbing further, which explains the queue time we saw in nfsiostat. The requests were piling up behind a virtual wall unrelated to the FSx filesystem’s provisioned throughput.

Step 4. The limit on the network path

A number that consistently caps out at 5 Gbit/s pointed us to networking limits rather than disk performance issues. According to AWS documentation, an r5n.8xlarge instance provides 25 Gbps of bandwidth. However, an EC2 instance can become a victim of what AWS calls “single-flow traffic” (a single TCP connection). This limitation caps network transfer bandwidth at 5 Gbps regardless of the EC2 instance’s total aggregate network bandwidth, and it applies to most of today’s EC2 instances.

One look at the mount confirmed a single flow was carrying everything:

No nconnect option, which means a single TCP connection was carrying all NFS traffic between the EC2 instance and FSx. That single connection was subject to EC2’s single-flow bandwidth limit, not the instance’s aggregate limit, and definitely not the throughput FSx itself was provisioned for. The 25 Gbps the EC2 instance could theoretically use, and the 10.2 Gbps throughput FSx could theoretically deliver, were both irrelevant as long as everything moved through a single flow capped at roughly 5 Gbps.

This is why the AWS Support answer and the customer’s own checks weren’t wrong, but were just looking at the wrong layer.

The fix

NFS supports the nconnect mount option, which allows a client to multiplex operations across multiple TCP connections to the same NFS export rather than a single one. It’s supported on NFS v3, v4.0, v4.1, and v4.2, and available by default on modern Linux kernels (5.3+), with backports available on some enterprise distributions. AWS itself documents and suggests using nconnect when possible:

We recommended reviewing the nconnect configuration with their own infrastructure team or AWS Support, since this is a network and infrastructure configuration decision, not a PostgreSQL one. We did not commit to a specific nconnect value, not because we didn’t want to, but because properly benchmarking it would require resources and access our team didn’t have. 

The customer came back after setting nconnect=16 and reported that the lag immediately started to close. That’s a good outcome, but the value itself probably deserves a second look. Any value greater than the default of a single connection would have shown improvements right away in this capped environment. However, sixteen connections (16*5 Gbps each) is a theoretical maximum of 80 Gbps, far beyond both endpoints in this setup, the EC2 instance’s own 25 Gbps aggregate limit, and the 10.2 Gbps FSx filesystem’s own provisioned throughput.

Two connections would theoretically already reach close to the FSx throughput ceiling, and three would cover it with a comfortable buffer. Going straight to sixteen removed the bottleneck, but it also means the OS is maintaining far more resources with no measurable benefit.

Parallelism doesn’t scale as a clean multiplication of a single connection’s throughput. Other resources, on both the client and server sides, become part of the equation as concurrency increases. As with any other piece of infrastructure or software, finding the right number requires proper benchmarking.

As a database support team, our role was to identify and prove the cause behind the lag, provide the evidence, and propose a possible path forward.

Conclusion

A component reporting low utilization against its provisioned limit doesn’t rule out an I/O bottleneck elsewhere in the path, it only rules out that specific limit. In this case, FSx itself was never the problem. The actual constraint was the single-flow network bandwidth cap between the EC2 instance and its FSx filesystem, throttling every request that went through it.

Two things are worth carrying into the next investigation, like this one. First, vmstat’s “b” column and a sustained “wa” value are cheap and fast signals that something downstream of PostgreSQL is the bottleneck, before diving into storage-specific tooling. Second, when checking a metric against a provisioned limit, consider the full distribution of samples, not just the peak or a single percentile. A value that repeatedly hits the same ceiling (even one well below the provisioned maximum) is a stronger signal than a single peak value.