Crash Recovery

What does the full cluster crash mean?

A full cluster crash is a situation when all database instances where shut down in random order. Being rebooted after such situation, Pod is continuously restarting, and generates the following errors in the log:

It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates.
To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1

Note

To avoid this, shutdown your cluster correctly as it is written in Pause/resume Percona XtraDB Cluster.

The Percona Operator for Percona XtraDB Cluster provides two ways of recovery after a full cluster crash.

Bootstrap Crash Recovery method

In this case recovery is done automatically. The recovery is triggered by the pxc.forceUnsafeBootstrap option set to true in the deploy/cr.yaml file:

pxc:
  ...
  forceUnsafeBootstrap: true

Applying this option forces the cluster to start. However, there may exist data inconsistency in the cluster, and several last transactions may be lost. If such data loss is undesirable, experienced users may choose the more advanced manual method described in the next chapter.

Object Surgery Crash Recovery method

Warning

This method is intended for advanced users only!

This method involves the following steps:

  • swap the original PXC image with the debug image, which does not reboot after the crash, and force all Pods to run it,
  • find the Pod with the most recent PXC data, run recovery on it, start mysqld, and allow the cluster to be restarted,
  • revert all temporary substitutions.

Let’s assume that a full crash did occur for the cluster named cluster1, which is based on three PXC Pods.

Note

The following commands are written for PXC 8.0. The same steps are also for PXC 5.7 unless specifically indicated otherwise.

  1. Check the current Update Strategy with the following command to make sure Smart Updates are turned off during the recovery:

    $ kubectl get pxc cluster1 -o jsonpath='{.spec.updateStrategy}'
    

    If the returned value is SmartUpdate, please change it to onDelete with the following command:

    $ kubectl patch pxc cluster1 --type=merge --patch '{"spec": {"updateStrategy": "OnDelete" }}'
    
  2. Change the normal PXC image inside the cluster object to the debug image:

    $ kubectl patch pxc cluster1 --type="merge" -p '{"spec":{"pxc":{"image":"percona/percona-xtradb-cluster:8.0.20-11.1-debug"}}}'
    

    Note

    For PXC 5.7 this command should be as follows:

    $ kubectl patch pxc cluster1 --type="merge" -p '{"spec":{"pxc":{"image":"percona/percona-xtradb-cluster:5.7.31-31.45-debug"}}}'
    
  3. Restart all Pods:

    $ for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do kubectl delete pod cluster1-pxc-$i --force --grace-period=0; done
    
  4. Wait until the Pod 0 is ready, and execute the following code (it is required for the Pod liveness check):

    $ for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do until [[ $(kubectl get pod cluster1-pxc-$i -o jsonpath='{.status.phase}') == 'Running' ]]; do sleep 10; done; kubectl exec cluster1-pxc-$i -- touch /var/lib/mysql/sst_in_progress; done
    
  5. Wait for all PXC Pods to start, and execute the following code to make sure no mysqld processes are running:

    $ for i in $(seq $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do pid=$(kubectl exec cluster1-pxc-$i -- ps -C mysqld-ps -o pid=); if [[ -n "$pid" ]]; then kubectl exec cluster1-pxc-$i -- kill -9 $pid; fi;  done
    
  6. Wait for all PXC Pods to start, then find the PXC instance with the most recent data - i.e. the one with the highest sequence number (seqno):

    $ for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do echo "###############cluster1-pxc-$i##############"; kubectl exec cluster1-pxc-$i -- cat /var/lib/mysql/grastate.dat; done
    

    The output of this command should be similar to the following one:

    ###############cluster1-pxc-0##############
    # GALERA saved state
    version: 2.1
    uuid:    7e037079-6517-11ea-a558-8e77af893c93
    seqno:   18
    safe_to_bootstrap: 0
    ###############cluster1-pxc-1##############
    # GALERA saved state
    version: 2.1
    uuid:    7e037079-6517-11ea-a558-8e77af893c93
    seqno:   18
    safe_to_bootstrap: 0
    ###############cluster1-pxc-2##############
    # GALERA saved state
    version: 2.1
    uuid:    7e037079-6517-11ea-a558-8e77af893c93
    seqno:   19
    safe_to_bootstrap: 0
    

    Now find the Pod with the largest seqno (it is cluster1-pxc-2 in the above example).

  7. Now execute the following commands in a separate shell to start this instance:

    $ kubectl exec cluster1-pxc-2 -- mysqld --wsrep_recover
    $ kubectl exec cluster1-pxc-2 -- sed -i 's/safe_to_bootstrap: 0/safe_to_bootstrap: 1/g' /var/lib/mysql/grastate.dat
    $ kubectl exec cluster1-pxc-2 -- sed -i 's/wsrep_cluster_address=.*/wsrep_cluster_address=gcomm:\/\//g' /etc/mysql/node.cnf
    $ kubectl exec cluster1-pxc-2 -- mysqld
    

    The mysqld process will initialize the database once again, and it will be available for the incoming connections.

  8. Go back to the previous shell and return the original PXC image because the debug image is no longer needed:

    $ kubectl patch pxc cluster1 --type="merge" -p '{"spec":{"pxc":{"image":"percona/percona-xtradb-cluster:8.0.20-11.1"}}}'
    

    Note

    For PXC 5.7 this command should be as follows:

    $ kubectl patch pxc cluster1 --type="merge" -p '{"spec":{"pxc":{"image":"percona/percona-xtradb-cluster:5.7.31-31.45"}}}'
    
  9. Restart all Pods besides the cluster1-pxc-2 Pod (the recovery donor).

    $ for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do until [[ $(kubectl get pod cluster1-pxc-$i -o jsonpath='{.status.phase}') == 'Running' ]]; do sleep 10; done; kubectl exec cluster1-pxc-$i -- rm /var/lib/mysql/sst_in_progress; done
    $ kubectl delete pods --force --grace-period=0 cluster1-pxc-0 cluster1-pxc-1
    
  10. Wait for the successful startup of the Pods which were deleted during the previous step, and finally remove the cluster1-pxc-2 Pod:

    $ kubectl delete pods --force --grace-period=0 cluster1-pxc-2
    
  11. After the Pod startup, the cluster is fully recovered.

    Note

    If you have changed the update strategy on the 1st step, don’t forget to revert it back to SmartUpdate with the following command:

    $ kubectl patch pxc cluster1 --type=merge --patch '{"spec": {"updateStrategy": "SmartUpdate" }}'
    

Table Of Contents

Previous topic

Scale Percona XtraDB Cluster on Kubernetes and OpenShift

Next topic

Debug

Contact Us

For free technical help, visit the Percona Community Forum.
To report bugs or submit feature requests, open a JIRA ticket.
For paid support and managed or professional services, contact Percona Sales.