Just Because You Can, Doesn’t Mean You Should: Re-evaluating Galera Node Weights

We all know that when using a cluster based on Galera (Codership) such as Percona XtraDB Cluster (PXC), it is recommended to deploy an odd number of nodes.

The generic answer to this recommendation is that otherwise the risk of having an inconsistent cluster (like split-brain) is higher.  (see Get started with Percona XtraDB Cluster – Percona XtraDB Cluster ) 

As a reminder, when split-brain occurs, a cluster will stop accepting queries, which is what a safe solution should do while waiting for the DBA to review and decide which of the two nodes is the leader. I do not even want to describe the infamous wsrep_provider_options=”pc.ignore_sb = yes”, which has been the cause of many, many nights spent trying to reconcile the corrupted datasets. If you have it, remove it.

However, in Galera, we also have the setting wsrep_provider_options=”pc.weight=N”. With it, we can define a specific weight in the quorum calculation for each node.
The question naturally arises: if we can assign a different weight to each node, why can’t we have an even number of nodes and assign them different weights? 

The answer to the question is simple, yes, we can. 

What is not simple is to identify who should have a different weight and how to distribute it.

To understand the problem, we need to review how the quorum is calculated (I have touched on the topic long ago here TusaCentral – Geographic replication and quorum Calculation in MySQL/Galera) and what a View is.

In Percona XtraDB Cluster (PXC) and Galera, a View (or cluster view) is the agreed-upon membership of the cluster at any specific moment in time. It represents the exact list of nodes that are currently connected, synchronized, and actively communicating with each other.

Whenever a node joins the cluster, leaves gracefully, or drops out due to a network failure, the remaining connected nodes negotiate a new View and assign it a unique View ID.

Its relationship to the Primary Component (PC) is based on quorum:

• The Quorum Check: Every time a new View is formed, the nodes inside it calculate whether they hold the majority of the cluster’s total votes.
• The Primary Component: If the new View holds the majority, it is designated as the Primary Component. Only the Primary Component is allowed to accept and process transactions.
• Non-Primary Views: If a network split occurs, isolated nodes will form their own separate View. Because this smaller View lacks a majority, it becomes a non-primary component and immediately halts database operations to prevent split-brain data corruption.

Essentially, a View defines who is currently connected to each other, while the Primary Component is the specific View that has enough members to safely keep the database running.

The formula for calculating the quorum is:

  Σ p·w  <  2·(Σ m·w + Σ l·w)

Where:
p = members of the last PRIMARY view
l = members that left gracefully (LEFT)
m = members of the component being evaluated
w = pc.weight (0–255).

In short, the total weight of the previous Primary view must be less than twice the weight of the members of the new component plus the weight of the members that left gracefully.  To be clear, a crashed / unreachable node (DOWN or in another partition) still counts in Σ p·w, so it pushes the threshold up.
A graceful leave (LEFT), instead, is credited to the new component. Also, any node that was not in the last PC is dropped from calculation, so a node that has just joined or rejoined contributes nothing to this check.

Look for the function:

(repo https://github.com/percona/galera/tree/a6b54d887d58bae43ae3f46f5696195329405b7e) 

Now let’s assume that we have a cluster of 4 nodes, there are many possible combinations where things will work and where instead they will hit us badly.

The most common is the split-brain scenario:

To prevent this, we may decide to use one node with different weight:

Here we go, problem solved… really?

No, because changing weights may cause other issues, for example if we calculate the weights wrongly with respect to the number of nodes:

In this case one single node could bring down the whole cluster. 

Of course, if we perform standard operations, or crashes happen in cascade and the cluster has time to rebuild the Primary Component, all will work well:

The problem is that:

• Cluster dimensions are dynamic and change in relation to the active Primary Component
• In environments managed by operators (e.g., Kubernetes), scaling the number of nodes up or down is even more dynamic and difficult to predict.
• In real life we may have an incredible variety of scenarios, I have built just a few (see PDF)  

The real question is not if we can do it but whether it is worth it: does using an odd number of nodes help or not?

The answer is partly. 

Odd nodes get rid of the exact-tie case in the simple situation. They don’t completely stop the cluster losing PRIMARY state.

What matters is the parity of the total weight, not the node count.
A tie happens when present = partitioned, which means the counted weight (Σw(PC) − Left) splits exactly in half. 

That is only possible when the counted weight is even. So:

• 5 nodes of w=1, any two-way split: 3|2 or 4|1. One side always strictly wins.
• 6 nodes with one node at w=2 (total 7) behave exactly like an odd cluster. Adding the Galera Arbitrator (garbd) as a 7th voter does the same thing.
• 3 nodes weighted 2,1,1 (total 4) is an odd node count with an even total, so it can tie again.

Where odd nodes still lose PRIMARY:

1. The PC shrinks after each failure, and its parity changes. 5 nodes lose 1 and stay PRIMARY, but the new PC has 4 nodes. A 2|2 split after that ties. A rolling failure still ends at 2 nodes and losing one of those ties.
2. Graceful leaves change the parity. If 1 of 5 shuts down cleanly, the counted weight is 4, so a 2|2 split ties.
3. Splits into three or more groups. A 2|2|1 split of 5 has no tie, but no group is bigger than the rest combined (2 < 3), so every group goes NON-PRIMARY.
4. Losing a majority is still losing. 5 nodes survive 2 failures but not 3, so odd doesn’t add fault tolerance. 5 and 6 nodes both tolerate only 2 failures. The 6th node adds cost and a tie risk, and no extra resilience.
5. Two data centers. With 3+2 nodes, the 3-node site survives a link cut, but if that site goes down, the 2-node site stops. To survive the loss of either site we need a third location, even if it only runs garbd.

In practice: aim for an odd total weight in the PC and use a third site or an arbitrator for multi-DC setups. Even then, a cluster that has already shrunk can become even again, so the last-two-nodes tie can’t be designed away. 

Only pc.ignore_sb or a manual bootstrap gets past it. But you do not want to go there.

Given this, and given that what counts is NOT the number of nodes but the balance in the weight calculation, is using weights the right way to compensate?

Well, this may be my personal opinion, but in my experience, weights make the cluster harder to manage, and the stability they add is narrow.

A weight-2 node acts like two nodes that always fail together. Four nodes weighted 2,1,1,1 behave like a 5-node cluster where two of the nodes share one box.
We trade a split-brain tie for a single point of fate:

• Gain: a clean 2|2 split now has a fixed winner, the side holding the heavy node.
• Loss: losing the heavy node plus one other is fatal. Over-weighting makes it worse: with w=3, losing that node alone causes a tie.

The set of survivable failures shifts rather than grows. We still tolerate only one arbitrary failure plus some specific splits.

What weights add to day-to-day operations:

1. Nodes are no longer equal. Everyone must know which node is “special”. Runbooks, monitoring and on-call decisions depend on it.
2. Maintenance order matters. Shut the heavy node down cleanly so its weight counts for neither side.
   If it crashes during maintenance on another node, the cluster can stop. Patching or rebooting it may first need its weight moved to another node, a step that’s easy to forget.
3. Weight is a configuration that can drift. It has to be consistent across config files, rebuilds and node replacements. A rebuilt node that comes back with the default w=1 silently changes the quorum maths.
4. The tie-breaker site is fixed. With two data centers, the weighted side always wins. That’s no different from simply putting 3 nodes in one DC and 2 in the other, and in both cases losing that DC stops the cluster.
5. Shrinking still breaks it. After failures the PC changes, so a setup tuned for 4 nodes may tie at 3 or 2.

Simple to most complex for the same goal:

1. An odd number of equal nodes: symmetric, nothing special to manage.
2. An even number of nodes plus the Galera Arbitrator (garbd): the data nodes stay equal, and the arbitrator holds the tie-breaking vote. If it runs at a third site, the cluster survives losing either main site, which weights alone can’t give you.
3. An even number of nodes with weights: use this only when you can’t add a node or an arbitrator and you deliberately want one side to always win a split.

Weights don’t make the cluster generally more stable, and they add operational complexity. They make sense as a deliberate tiebreaker, not as a fix for even node counts.

Conclusions

I stand by the evergreen KISS motto: Keep It Simple and Straightforward.
Having a PXC cluster with more than 3 nodes makes sense in very few cases, and only to scale reads or if you plan to kill 2 nodes at the same time on a cluster of 5.
Given the significant complications the increase of nodes or the use of weight brings, I do not really see why someone in a healthy state of mind should look for trouble.

Just because it can be done, it doesn’t mean “do it”, especially if the gain is minimal and the complexity increases. 

Do yourself a favor: keep your cluster to the minimum, to what is really needed and do not try to stretch it. You will save yourself from spending miserable nights doing cluster recovery or worse data reconciliation.

Great MySQL to everyone.