Upgrading Percona XtraDB Cluster

The following documents contain details about relevant changes in the 8.0 series of MySQL and Percona Server for MySQL. Make sure you deal with any incompatible features and variables mentioned in these documents when upgrading to Percona XtraDB Cluster 8.0.

Important changes in Percona XtraDB Cluster 8.0

Traffic encryption is enabled by default

The pxc_encrypt_cluster_traffic variable, which enables traffic encryption, is set to ON by default in Percona XtraDB Cluster 8.0.

Unless you configure a node accordingly (each node in your cluster must use the same SSL certificates) or try to join a cluster running Percona XtraDB Cluster 5.7 which unencrypted cluster traffic, the node will not be able to join resulting in an error.

... [ERROR] ... [Galera] handshake with remote endpoint ...
This error is often caused by SSL issues. ...

PXC Strict Mode is enabled by default

Percona XtraDB Cluster in 8.0 runs with PXC Strict Mode enabled by default. This will deny any unsupported operations and may halt the server if a strict mode validation fails. It is recommended to first start the node with the pxc_strict_mode variable set to PERMISSIVE in the MySQL configuration file (on Debian and Ubuntu /etc/mysql/mysql.conf.d/mysqld.cnf; on CentOS and Red Hat /etc/my.cnf).

After you check the log for any experimental or unsupported features and fix any encountered incompatibilities, you can set the variable back to ENFORCING at run time:

mysql> SET pxc_strict_mode=ENFORCING;

Also, switching back to ENFORCING may be done by restarting the node with the updated configuration file (on Debian and Ubuntu /etc/mysql/mysql.conf.d/mysqld.cnf; on CentOS and Red Hat /etc/my.cnf).

The configuration file layout has changed in Percona XtraDB Cluster 8.0

All configuration settings are stored in the default MySQL configuration file:

  • Path on Debian and Ubuntu: /etc/mysql/mysql.conf.d/mysqld.cnf
  • Path on Red Hat and CentOS: /etc/my.cnf

Before you start the upgrade, move your custom settings from /etc/mysql/percona-xtradb-cluster.conf.d/wsrep.cnf (on Debian and Ubuntu) or from /etc/percona-xtradb-cluster.conf.d/wsrep.cnf (on Red Hat and CentOS) to the new location accordingly.

caching_sha2_password is the default authentication plugin

In Percona XtraDB Cluster 8.0, the default authentication plugin is caching_sha2_password. The ProxySQL option –syncusers will not work if the Percona XtraDB Cluster user is created using caching_sha2_password. Use the mysql_native_password authentication plugin in these cases.

Be sure you are running on the latest 5.7 version before you upgrade to 8.0.

mysql_upgrade is part of SST

mysql_upgrade is now run automatically as part of SST. You do not have to run it manually when upgrading your system from an older version.

Major Upgrade Scenarios

Upgrading Percona XtraDB Cluster from 5.7 to 8.0 may have slightly different strategies depending on the configuration and workload on your Percona XtraDB Cluster cluster.

Note that the new default value of pxc-encrypt-cluster-traffic (set to ON versus OFF in Percona XtraDB Cluster 5.7) requires additional care. You cannot join a 5.7 node to a Percona XtraDB Cluster 8.0 cluster unless the node has traffic encryption enabled as the cluster may not have some nodes with traffic encryption enabled and some nodes with traffic encryption disabled. For more information, see Traffic encryption is enabled by default.

Scenario: No active parallel workload or with read-only workload

If there is no active parallel workload or the cluster has read-only workload while upgrading the nodes, complete the following procedure for each node in the cluster:

  1. Shutdown one of the node 5.7 cluster nodes.
  2. Remove 5.7 PXC packages without removing the data-directory.
  3. Install Percona XtraDB Cluster 8.0 packages.
  4. Restart the mysqld service.

Important

Before upgrading, make sure your application can work with a reduced cluster size. Also, during the upgrade, the cluster will operate with an even number of nodes - the cluster may run into the split-brain problem.

This upgrade flow auto-detects the presence of the 5.7 data directory and trigger the upgrade as part of the node bootup process. The data directory is upgraded to be compatible with Percona XtraDB Cluster 8.0. Then the node joins the cluster and enters synced state. The 3-node cluster is restored with 2 nodes running Percona XtraDB Cluster 5.7 and 1 node running Percona XtraDB Cluster 8.0.

Note

Since SST is not involved, SST based auto-upgrade flow is not started.

Percona XtraDB Cluster 8.0 uses Galera 4 while Percona XtraDB Cluster 5.7 uses Galera-3. The cluster will continue to use the protocol version 3 used in Galera 3 effectively limiting some of the functionality. With all nodes upgraded to version 8.0, protocol version 4 is applied.

Hint

The protocol version is stored in the protocol_version` column of the ``wsrep_cluster table.

mysql> USE mysql;
...
mysql> SELECT protocol_version from wsrep_cluster;
+------------------+
| protocol_version |
+------------------+
|                4 |
+------------------+
1 row in set (0.00 sec)

As soon as the last 5.7 node shuts down, the configuration of the remaining two nodes is updated to use protocol version 4. A new upgraded node will then join using protocol version 4 and the whole cluster will maintain protocol version 4 enabling the support for additional Galera 4 facilities.

It may take longer to join the last upgraded node since it will invite IST to obtain the configuration changes.

Note

Starting from Galera 4, the configuration changes are cached to gcache and the configuration changes are donated as part of IST or SST to help build the certification queue on the JOINING node. As other nodes (say n2 and n3), already using protocol version 4, donate the configuration changes when the JOINER node is booted.

The situation was different for the previous and penultimate nodes since the donation of the configuration changes is not supported by protocol version 3 that they used.

With IST involved on joining the last node, the smart IST flow is triggered to take care of the upgrade even before MySQL starts to look at the data directory.

Important

It is not recommended to restart the last node without upgrading it.

Scenario: Upgrading from Percona XtraDB Cluster 5.6 to Percona XtraDB Cluster 8.0

First, upgrade Percona XtraDB Cluster from 5.6 to the latest version of Percona XtraDB Cluster 5.7. Then proceed with the upgrade using the procedure described in Scenario: No active parallel workload or with read-only workload.

Minor upgrade

To upgrade the cluster, follow these steps for each node:

  1. Make sure that all nodes are synchronized.

  2. Stop the mysql service:

    $ sudo service mysql stop
    
  3. Upgrade Percona XtraDB Cluster and Percona XtraBackup packages. For more information, see Installing Percona XtraDB Cluster.

  4. Back up grastate.dat, so that you can restore it if it is corrupted or zeroed out due to network issue.

  5. Now, start the cluster node with 8.0 packages installed, Percona XtraDB Cluster will upgrade the data directory as needed - either as part of the startup process or a state transfer (IST/SST).

    In most cases, starting the mysql service should run the node with your previous configuration. For more information, see Adding Nodes to Cluster.

    $ sudo service mysql start
    

    Note

    On CentOS, the /etc/my.cnf configuration file is renamed to my.cnf.rpmsave. Make sure to rename it back before joining the upgraded node back to the cluster.

    PXC Strict Mode is enabled by default, which may result in denying any unsupported operations and may halt the server. For more information, see pxc-strict-mode is enabled by default.

    pxc-encrypt-cluster-traffic is enabled by default. You need to configure each node accordingly and avoid joining a cluster with unencrypted cluster traffic. For more information, see Traffic encryption is enabled by default.

  6. Repeat this procedure for the next node in the cluster until you upgrade all nodes.

Contact Us

For free technical help, visit the Percona Community Forum.
To report bugs or submit feature requests, open a JIRA ticket.
For paid support and managed or professional services, contact Percona Sales.