This blog post was born of a discussion with a colleague in Percona Support, who also co-authored it with me here on the use of the ARBITER feature in MongoDB installations nowadays.
Turns out, after analyzing past real cases we’ve faced in support, maybe it’s time to say goodbye to ARBITERS.
We are not saying the feature is broken. Instead, the confusion it generates and the trade-off rarely match how people run MongoDB today.
This article is our feedback on why ARBITERS were an interesting feature in the past but are not anymore, and why you should stop using them in your production environment.
In the early releases of MongoDB in 2010, the ARBITER feature was introduced as we know it.
The goal was always to help with elections. As per the recommendation, you should always aim for an odd number of members and therefore, votes.
But teams wanted an odd number of votes without paying for another data-bearing node. Enter the ARBITER: it is cheap, it votes, it does not store data, it does not apply the oplog; it’s lightweight.
That design answers a voting problem. It never answered a data durability problem, and it never will.
That was fine until MongoDB started to correctly focus on durability as the product evolved.
It was in the MongoDB 3.x and 4.x eras that we saw durability hardening come from different angles:
All of that uses majority in its own way.
While an ARBITER helps with elections and the majority of votes, the cluster now also depends on other kinds of majority: write majority, majority-committed reads, and features built on top of them.
If you are a MongoDB DBA or specialist, those distinctions should be clear. For the majority of users (no pun intended), they are not. You get lost, and worse: the cluster stalls or hits performance issues because of the topology you chose.
Let’s forget elections for a moment and look at scenarios where an ARBITER has no meaning. These are pain points we keep seeing. Most of them would have been easy to avoid with a data-bearing node instead.
A sharded cluster with three shards, plus a config server replica set.
Each shard had three data-bearing members in one data center (DC1), three in the other (DC2), and an arbiter on a third site (DC3).
The Config Servers were built differently.
One data center went down. Elections finished, and every shard had a new primary. A few minutes later, write throughput fell from roughly 3-4k ops/s per shard to about 50-75 ops/s.
If the application was using { w: 2, j: true }, and four members were still reachable (three with data, plus the arbiter), why did throughput collapse?
3 DCs, 6 data bearing-nodes, 1 arbiter. On paper, a strong topology. It took one failure for availability to go downhill, because majority still required four data-bearing acknowledgments, and the arbiter was not one of them.
A replica set with four data-bearing members and one arbiter: two data-bearing nodes in one data center, two in another, and the arbiter on a third site.
Heavy writes were happening due to an ongoing project, making oplog generation huge on the primary, and that load cascaded to the secondaries.
Guess what happened?
Flow control. With four data-bearing members, the majority commit point moves only when at least three of them have applied the write.
Two is not three, so the commit point stopped and flow control throttled writes on the primary. The arbiter was up the whole time. It holds no data, so it never counted toward that point.
A Primary-Secondary-Arbiter set.
The primary had election priority 2. The application was writing with w: 1.
The primary crashed(N1). The secondary(N2) became primary(which is what the arbiter is for).
For about an hour, the application kept writing, and those writes were acknowledged. Then the crashed node(N1) was started again. The replica set looked healthy.
“Two is one, and one is none.” – Those writes had existed only on the temporary primary(N2). The arbiter does not store data, and the old primary had been down, so nothing else had a copy of it.
However, during the catch-up, the temporary primary(N2) went down, and the old primary(N1) had not yet caught up.
When the node (N2) holding that hour of writes rejoined, it was ahead, and past the common point its oplog was a different history. It rolled them back to match a primary (N1) that had never seen them.
The reverted documents were still on disk, as bson, under the rollback directory in dbPath. That is how they were recovered. The set had not replicated them.
With a third data-bearing node rather than an arbiter, the case could have played out differently. Even with w: 1, a third node that was up and replicating would already have that data. It would be ahead of N1, so it would become primary. N1 would catch up from it, and only then take over on priority. A tail that never reached that third node could still roll back.
The list could go on, but the point is already clear. Part of the job, when you are doing a root cause analysis or joining incident calls, is to untangle those misunderstandings.
The arbiter was in the voting majority. It was never in the majority that writes, commits, and flow control actually wait for.
Even if those cases are not enough for you to drop ARBITERs in production, the manual has been stacking the same caveats.
Three-member sets: P-S-S is marked recommended. PSA is the cost fallback, with weaker redundancy and fault tolerance because you only have two full copies of the data.
Do not run an arbiter on a host that already runs a primary or a secondary. If that host dies, you lose the data and the vote together. More than one arbiter undermines the majority write concern.
If the secondary is unavailable or lagging, “majority” write concern can cause performance problems, as in the cases above. In a P-S-A deployment, it only takes one secondary going down or having issues.
Starting in MongoDB 5.3, support for multiple arbiters in a replica set is disabled by default.
That happens for the issues we described; Multiple arbiters prevent the reliable use of the majority write concern. Multiple arbiters allow replica sets to accept writes even when the replica set doesn’t have sufficient secondaries for data replication.
You cannot change a shard key using a transaction if the replica set has an arbiter.
Arbiters cannot participate in the data operations required for multi-shard transactions.
Transactions whose write operations span multiple shards will error and abort if any transaction operation reads from or writes to a shard that contains an arbiter.
Last but not least, it is worth mentioning that the use of ARBITER is not supported in any MongoDB Atlas tier.
With thousands of clusters deployed, using ARBITERs would be an easy way to reduce the cost of data-bearing nodes. But when we weigh the pros and cons, that saving would come at a price: the risk of the product failing in front of its users.
The use of ARBITER was a cheap way to get an odd number of votes. It still is.
It has never been a way to get durability, and the product spent the last decade making durability the thing your cluster actually waits on.
Once you analyze the risks and the traps you can fall into, the cost reduction topic starts to shift to another side.
Until you figure it out, it might be later, and the data-bearing node you skipped was the cheaper line item.
If you are a seasoned MongoDB DBA and you already know these caveats, an arbiter in production might not be one of your top problems right now.
For everyone else, if you want one less flaw that can turn into an incident, avoid arbiters in production. They can cause failures that do not look like election problems, and those are the ones that take the longest to detect.
Resources
RELATED POSTS
