Percona ClusterSync for MongoDB (PCSM) replicates data between MongoDB clusters to keep migrations with near-zero downtime. Prior to version 0.9.0 it required the source and target to run the same major version, which ruled out the lift-and-shift move most migrations want: going from an older major like 6.0 straight onto a newer one like 8.0.
When we first pointed PCSM at a target running a newer major version of MongoDB, we quickly discovered a problem. Cross-version replication failed because a change stream was applying only a small fraction of the events it read. Because so much of what we read never made it to the target, we assumed the problem was format. We braced for a layer that would rewrite change stream events between major versions.
That assumption was wrong, and being wrong about it is the most interesting part of this story.
The same-major requirement in earlier versions stemmed from a simple assumption: matching majors maintained metadata and API compatibility. The tool never had to reason about a version gap. The catch is that getting both clusters onto the same major version first is exactly the step a low-downtime migration aims to avoid.
PCSM already knew the versions it was talking to. On startup, it asked each cluster for its build version and logged the results. It just did nothing with that information. No downgrade guardrail, no compatibility gate, no feature-compatibility-version check. Cross-version replication worked when it worked and broke when it broke, and nothing in the tool had an opinion either way.
So we ran a spike to find out what cross-version replication actually needed. Two findings reshaped the whole effort.
First, we found that for the operations PCSM cares about, MongoDB 6.0 and 8.0 produce the same change-stream events. There was no format to translate. The layer we expected to build did not need to exist.
Second, the cross-version failures were not many separate problems. They were one problem. A DDL operation would fail, the error got swallowed, and PCSM would attempt to continue applying changes to a target whose state had already diverged from the source. That single swallowed error cascaded into failed selective replication, missing indexes, and capped-collection mismatches downstream. The gap between events read and events applied was just the tool reading events it could no longer apply, because the collection state had already drifted.
The goal shrank from building a complex translation layer to maintaining better discipline about handling the operations we already understood.
We set out with a longer list of goals than we shipped, and several of them turned out to matter less than we thought going in. Here is what actually landed.
The tool now:
Two things we had considered did not ship and were moved to the known limitations list: patch-level version enforcement and fetching the feature compatibility version at startup. More on those below.
The replication path that applied DDL operations (creating and dropping collections, building indexes, changing collection options) used to ignore errors and keep running. We made those failures surface and stopped the run instead. On its own, that fixes no compatibility logic, yet it took the cross-version suite from about half-failing to clean.
The reason is that almost none of those failures were independent. One DDL operation would fail, the error went nowhere, and PCSM kept applying later events against a target that had already drifted from the source. Every subsequent mismatch was counted as a separate failure. Once the first error stopped the run instead of vanishing, what was left was a handful of operations that genuinely behave differently between majors. Surfacing these failures made them findable. Handling those few remaining cases is what got the release as a whole to a clean run.
Before doing anything else, PCSM now reads the build version from both clusters and compares them. A downgrade is a hard stop. A cross-version upgrade logs an informational line and proceeds. This check is the contract PCSM 0.9.0 that is shipping:
|
1 2 3 4 5 6 7 8 |
var ErrDowngrade = errors.New("downgrade not supported") func CheckVersionCompat(source, target ServerVersion) (bool, error) { if source.Major() > target.Major() { return false, errors.Wrapf(ErrDowngrade, "source %s > target %s", source, target) } return source.Major() < target.Major(), nil } |
The comparison is limited to the major version. Equal majors are treated as the same version, a lower source major as a supported upgrade, and a higher source major as a refused downgrade. That is the whole gate.
This was the one genuine cross-version-only bug. It’s a good example of the kind of thing that does not show up until you actually run the mismatch. MongoDB 6.x rounds capped-collection sizes up to the nearest 256 bytes internally. When a capped-size change flows through the change stream, the event carries the size the user asked for, say 3333 bytes, not the size 6.x actually stored, 3584. A newer target stores exactly what it is told, so applying the requested size produces a collection that does not match the source.
The fix rounds the size up to the nearest 256-byte boundary when the source is 6.x. It showed up in just three of the suite’s two hundred tests, all in the capped-size checks. That was the entire substance of what we had feared would be a sprawling 6.0-to-8.0 compatibility effort.
While PCSM is still catching up to the source, some operations arrive referencing objects that do not exist yet or are already on their way out. Missing namespaces, missing indexes, invalid options, and mid-drop databases are all treated as non-fatal during this window, because the final state converges regardless.
The TTL-index case is the one where the version difference actually shows its face. PCSM clones a TTL index with its expiry set so far in the future that the target will not start deleting documents while the initial sync is still running, then restores the real expiry once the sync is done.
A 7.0 target had no problems when restoring the real expiry. The 8.0 target we tested rejected that operation as an index-options conflict. Same replayed sequence, same options, accepted by one major and refused by the next. PCSM handles it structurally: when the restore is refused, it drops the index and recreates it from the specification it already has. That happens once per affected TTL index, at the end of the sync.
PCSM 0.9.0 covers every pair in the supported matrix, on both replica-set and sharded topologies: 6.0, 7.0, and 8.0 to themselves, and the lower-to-higher pairs 6.0-to-7.0, 6.0-to-8.0, and 7.0-to-8.0. This is what our CI tests cover.
The contract has edges, and you should know them before you plan a migration around it:
That is the shape of what shipped, and it is enough to plan a migration around safely.
Percona ClusterSync for MongoDB 0.9.0 ships cross-version replication. The documentation has the setup and the supported matrix, the 0.9.0 release notes cover what landed, and the source is on GitHub. If you put it through a migration and something behaves unexpectedly, Percona Forum is the place to report it.
Resources
RELATED POSTS
