Percona XtraDB Cluster 8.0 Behavior Change for pxc-encrypt-cluster-traffic

Percona XtraDB Cluster 8.0 Behavior Change for pxc-encrypt-cluster-trafficPercona has enforced stronger security in Percona XtraDB Cluster (PXC) 8, but this requires some attention during the rollout of the new server version, so let see the why and what.

In PXC there are two different kinds of traffic: client-server exchange (ie: application traffic), and replication traffic. The latter refers to any SST/IST, write-set, and other service messages the nodes exchange.

In PXC 5.7 it is possible to activate SSL encryption by enabling the variable pxc-encrypt-cluster-traffic by following the instructions.

In PXC 8, we choose to enable encryption by default on all replication traffic, to have the highest out-of-box security enforcement.

While this is an obvious improvement, PXC 8 needs to behave differently from its predecessor PXC 5.7. This article is meant to help you in becoming more familiar with the new version of PXC.

Let’s Start With The Basics:

On a PXC cluster, in order to have the nodes connecting and to be able to communicate when SSL is enabled, the certificates must be the same on all of the nodes that are supposed to be part of the cluster, no matter if you are using PXC 8 or PXC 5.7.

PXC can generate the certificates by itself if you let it do so (by default). The files are generated in the data directory.

The problem here is that each node will generate a different set of certificates, which means each set of *.pem files on a node is different from the others. Given that, we need to replace the *.pem files by choosing a single set from one of the nodes and copy them over to each other node, otherwise, the nodes will not be able to communicate.

A best practice is to define a different location for your files, such that you can easily keep in sync between nodes, and that they will not be removed or overwrite during SST, something  like:

In order to generate the *.pem file at initialization, your my.cnf should not have any reference to the above, or PXC will expect to find the *.pem files at the given path.

An easy solution is to comment the settings, initialize the database, generate the *.pem, and copy them to the location you have decided.

Let us now consider the four different scenarios:

  1. Upgrade from 5.7 no pxc-encrypt-cluster-traffic
  2. Upgrade from 5.7 with pxc-encrypt-cluster-traffic
  3. New install PXC8 without pxc-encrypt-cluster-traffic
  4. New install with pxc-encrypt-cluster-traffic

Note: When upgrading a PXC 5.7 online via the addition of PXC 8 nodes, ensure you set PXC 8 to READ_ONLY mode.  Application writes cannot be sent to PXC 8 while there are PXC 5.7 nodes in the same cluster.  For example, include the following entry in your server’s /etc/my.cnf: PXC8 node(s) cannot receive writes until a PXC 5.7 node is part of the cluster.

Case 1 – Upgrade From PXC 5.7 No SSL

As mentioned previously, the difference between PXC 5.7 and 8 here is that the encryption is enabled by default.

If you try to have a PXC8 node join with defaults, you will get this error:

Indeed, an SSL issue because our 5.7 cluster IS NOT using SSL. We need to change the defaults for our PXC8, modifying the pxc-encrypt-cluster-traffic settings to:

Doing so, our new PXC8 node will join and perform SST:

It will also identify that the source is a 5.7 node and will perform an internal upgrade.

Case 2 – Upgrade From 5.7 With pxc-encrypt-cluster-traffic

So, you have SSL already active in your 5.7 cluster, and you want to upgrade to 8.

If your *.pem files are in the data directory, given you are migrating, this is the right moment to do the right thing and place them out of it in the new nodes.

In the new nodes, move them for best practice and define the path in the MySQL config file.

Steps:

  1. Create a safe place for the file like /etc/mysql/cert
  2. Copy file over from the donor
  3. Add the following to the my.cnf

  4. grant mysql user only read/write access
  5. Start your PXC8 node

At this point, the node will do SST, then the internal upgrade, and will join the 5.7 cluster.

Case 3 – New Install PXC8 Without pxc-encrypt-cluster-traffic

If you want to start a new PXC8 cluster without encryption, you must explicitly disable pxc_encrypt_cluster_traffic.

Given so, on all nodes add to your my.cnf:

And start the cluster as usual.

Case 4 – New Install With pxc-encrypt-cluster-traffic

As mentioned earlier, by default, PXC8 has pxc-encrypt-cluster-traffic enabled.

When booting the first node, you have two options:

If you do not have already the *.pem files and you like to have a new set automatically created, just:

  1. Comment the ssl entry (as above) in your my.cnf. PXC will create a new set for you.
  2. After the boot, you can stop the server
  3. Move the files in a safe place outside the data directory (like /etc/mysql/cert)
  4. Copy over them on any other node.
  5. Restart your first node
  6. Let the other nodes to join

Remember to grant permission to mysql user only:

Your new cluster will be up in a few seconds and all the replication/messaging traffic between nodes will be encrypted.

Conclusion

Enabling encrypted traffic between PXC 8 servers is expected to improve the security profile of your PXC environment but comes with additional complexity when upgrading or bootstrapping a new cluster. Operations, that if not done in the correct order, in case of an upgrade, may affect cluster stability.

Here my focus was to give simple instructions and understanding of this minor behavior change, which may cause a waste of time and frustration. PXC8 has many other additional features that we are going to illustrate in our blog, so follow us.

Please keep in mind that PXC8 is GA since April 2020 and PXC 8.0.19 is the first GA release, and as such we advise you to use it for heavy testing, but do not put it in production unless you have covered all the possible use cases and load scenarios.

Reference

Encrypting PXC Traffic

Upgrading Percona XtraDB Cluster

Share this post

Leave a Reply