This guide describes the procedure for upgrading Percona XtraDB Cluster without downtime (rolling upgrade) to the latest 5.7 version. A “rolling upgrade” means there is no need to take down the complete cluster during the upgrade.
Both major upgrades (from 5.6 to 5.7 version) and minor ones (from 5.7.x to 5.7.y) can be done in this way. Rolling upgrades to 5.7 from versions older than 5.6 are not supported. Therefore if you are running Percona XtraDB Cluster version 5.5, it is recommended to shut down all nodes, then remove and re-create the cluster from scratch. Alternatively, you can perform a rolling upgrade from PXC 5.5 to 5.6, and then follow the current procedure to upgrade from 5.6 to 5.7.
The following documents contain details about relevant changes in the 5.7 series of MySQL and Percona Server. Make sure you deal with any incompatible features and variables mentioned in these documents when upgrading to Percona XtraDB Cluster 5.7.
To upgrade the cluster, follow these steps for each node:
Make sure that all nodes are synchronized.
Stop the mysql service:
$ sudo service mysql stop
Remove existing Percona XtraDB Cluster and Percona XtraBackup packages, then install Percona XtraDB Cluster version 5.7 packages. For more information, see Installing Percona XtraDB Cluster.
For example, if you have Percona software repositories configured, you might use the following commands:
On CentOS or RHEL:
$ sudo yum remove percona-xtrabackup* Percona-XtraDB-Cluster*
$ sudo yum install Percona-XtraDB-Cluster-57
On Debian or Ubuntu:
$ sudo apt-get remove percona-xtrabackup* percona-xtradb-cluster*
$ sudo apt-get install percona-xtradb-cluster-57
In case of Debian or Ubuntu,
the mysql service starts automatically after install.
Stop the service:
$ sudo service mysql stop
Back up grastate.dat, so that you can restore it
if it is corrupted or zeroed out due to network issue.
Start the node outside the cluster (in standalone mode)
by setting the wsrep_provider variable to none.
For example:
sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
Note
As of Percona XtraDB Cluster 5.7.6, the --skip-grant-tables option
is not required.
Note
To prevent any users from accessing this node while performing work on it, you may add –skip-networking to the startup options and use a local socket to connect, or alternatively you may want to divert any incoming traffic from your application to other operational nodes.
Open another session and run mysql_upgrade.
When the upgrade is done, stop the mysqld process.
You can either run sudo kill on the mysqld process ID,
or sudo mysqladmin shutdown with the MySQL root user credentials.
Note
On CentOS, the my.cnf configuration file
is renamed to my.cnf.rpmsave.
Make sure to rename it back
before joining the upgraded node back to the cluster.
Now you can join the upgraded node back to the cluster.
In most cases, starting the mysql service
should run the node with your previous configuration:
$ sudo service mysql start
For more information, see Adding Nodes to Cluster.
Note
As of version 5.7, Percona XtraDB Cluster runs with PXC Strict Mode enabled by default. This will deny any unsupported operations and may halt the server upon encountering a failed validation.
If you are not sure, it is recommended to first start the node
with the pxc_strict_mode variable set to PERMISSIVE
in the in the MySQL configuration file, my.cnf.
After you check the log for any experimental or unsupported features
and fix any encountered incompatibilities,
you can set the variable back to ENFORCING at run time:
mysql> SET pxc_strict_mode=ENFORCING;
Also switch back to ENFORCING may be done by restarting the node
with updated my.cnf.
Repeat this procedure for the next node in the cluster until you upgrade all nodes.
It is important that on rejoining, the node should synchronize using IST. For this, it is best not to leave the cluster node being upgraded offline for an extended period. More on this below.
When performing any upgrade (major or minor), SST could be initiated by the joiner node after the upgrade if the server was offline for some time. After SST completes, the data directory structure needs to be upgraded (using mysql_upgrade) once more time to ensure compatibility with the newer version of binaries.
To upgrade the cluster, follow these steps for each node:
Make sure that all nodes are synchronized.
Stop the mysql service:
$ sudo service mysql stop
Upgrade Percona XtraDB Cluster and Percona XtraBackup packages. For more information, see Installing Percona XtraDB Cluster.
For example, if you have Percona software repositories configured, you might use the following commands:
On CentOS or RHEL:
$ sudo yum update Percona-XtraDB-Cluster-57
On Debian or Ubuntu:
$ sudo apt-get install --only-upgrade percona-xtradb-cluster-57
In case of Debian or Ubuntu,
the mysql service starts automatically after install.
Stop the service:
$ sudo service mysql stop
Back up grastate.dat, so that you can restore it
if it is corrupted or zeroed out due to network issue.
Start the node outside the cluster (in standalone mode)
by setting the wsrep_provider variable to none.
For example:
sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
Note
As of Percona XtraDB Cluster 5.7.6, the --skip-grant-tables option
is not required.
Note
To prevent any users from accessing this node while performing work on it, you may add –skip-networking to the startup options and use a local socket to connect, or alternatively you may want to divert any incoming traffic from your application to other operational nodes.
Open another session and run mysql_upgrade.
When the upgrade is done, stop the mysqld process.
You can either run sudo kill on the mysqld process ID,
or sudo mysqladmin shutdown with the MySQL root user credentials.
Note
On CentOS, the my.cnf configuration file
is renamed to my.cnf.rpmsave.
Make sure to rename it back
before joining the upgraded node back to the cluster.
Now you can join the upgraded node back to the cluster.
In most cases, starting the mysql service
should run the node with your previous configuration:
$ sudo service mysql start
For more information, see Adding Nodes to Cluster.
Note
As of version 5.7, Percona XtraDB Cluster runs with PXC Strict Mode enabled by default. This will deny any unsupported operations and may halt the server upon encountering a failed validation.
If you are not sure, it is recommended to first start the node
with the pxc_strict_mode variable set to PERMISSIVE
in the in the MySQL configuration file, my.cnf.
After you check the log for any experimental or unsupported features
and fix any encountered incompatibilities,
you can set the variable back to ENFORCING at run time:
mysql> SET pxc_strict_mode=ENFORCING;
Also switch back to ENFORCING may be done by restarting the node
with updated my.cnf.
Repeat this procedure for the next node in the cluster until you upgrade all nodes.
It is important that on rejoining, the node should synchronize using IST. For this, it is best not to leave the cluster node being upgraded offline for an extended period. More on this below.
When performing any upgrade (major or minor), SST could be initiated by the joiner node after the upgrade if the server was offline for some time. After SST completes, the data directory structure needs to be upgraded (using mysql_upgrade) once more time to ensure compatibility with the newer version of binaries.
The following additional steps should be made to upgrade the data directory structure after SST (after the normal major or minor upgrade steps):
shutdown the node that rejoined the cluster using SST:
$ sudo service mysql stop
restart the node in standalone mode by setting the
wsrep_provider variable to none, e.g.:
sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
run mysql-upgrade
restart the node in cluster mode (e.g by executing sudo service mysql
start and make sure the cluster joins back using IST.
For general inquiries, please send us your question and someone will contact you.
