Bypassing SST in Percona XtraDB Cluster with binary logs

Bypassing SST in Percona XtraDB Cluster with binary logs

PREVIOUS POST
NEXT POST

In my previous post, I used incremental backups in Percona XtraBackup as a method for rebuilding a Percona XtraDB Cluster (PXC) node without triggering an actual SST. Practically this reproduces the SST steps, but it can be handy if you already had backups available to use.

In this post, I want to present another methodology for this that also uses a full backup, but instead of incrementals uses any binary logs that the cluster may be producing.

Binary logs on PXC

Binary logs are not strictly needed in PXC for replication, but you may be using them for backups or for asynchronous slaves of the cluster.  To set them up properly, we need the following settings added to our config:

As I stated, none of these are strictly needed for PXC.

  • server-id=1 — We recommend PXC nodes share the same server-id.
  • log-bin — actually enable the binary log
  • log-slave-updates — log ALL updates to the cluster to this server’s binary log

This doesn’t need to be set on every node, but likely you would set these on at least two nodes in the cluster for redundancy.

Note that this strategy should work with or without 5.6 asynchronous GTIDs.

Recovering data with backups and binary logs

This methodology is conventional point-in-time backup recovery for MySQL.  We have a full backup that was taken at a specific binary log position:

We have this binary log and all binary logs since:

Recover the full backup

We start by preparing the backup with –apply-log:

The output confirms the same binary log file and position that we knew from before.

Start MySQL without Galera

We need to start mysql, but without Galera so we can apply the binary log changes before trying to join the cluster. We can do this simply by commenting out all the wsrep settings in the MySQL config.

Apply the binary logs

We now check our binary log starting position:

We can compare the Xid on this binary log position to that of the backup. The Xid in a binary log produced by PXC will be the seqno of the GTID of that transaction. The starting position in the binary log shows us the next Xid is one increment higher, so this makes sense: we can start at this position in the binary log and apply all changes as high as we can go to get the datadir up to a more current position.

This action isn’t particularly fast as binlog events are being applied by a single connection thread. Remember that if the cluster is taking writes while this is happening, the amount of time you have is limited by the size of gcache and the rate at which it is being filled up.

Prime the grastate

Once the binary logs are applied, we can check the final log’s last position to get the seqno we need:

This is indeed the seqno we put in our grastate.dat. Like in the last post, we can copy a grastate.dat from another node to get the proper format. However, this time we must put the proper seqno into place:

Be sure the grastate.dat has the proper permissions, uncomment the wsrep settings and restart mysql on the node:

The node should now attempt to join the cluster with the proper GTID:

This, of course, still does not guarantee an IST. See my previous post for more details on the conditions needed for that to happen.

PREVIOUS POST
NEXT POST

Share this post

Comment (1)

  • Mann Reply

    Hi Jay,
    We’ve been using different server-id for different PXC nodes, in your note above, it recommend PXC nodes share the same server-id. Is that a over oversight, what is the problem if we’ve using different for each nodes?

    September 6, 2017 at 4:41 pm

Leave a Reply