Encrypting PXC Traffic¶
There are two kinds of traffic in Percona XtraDB Cluster:
- Client-Server traffic (the one between client applications and cluster nodes),
- Replication traffic, that includes SST, IST, write-set replication, and various service messages.
Percona XtraDB Cluster supports encryption for all types of traffic. Replication traffic encryption can be configured either automatically or manually.
Percona XtraDB Cluster uses the underlying MySQL encryption mechanism to secure communication between client applications and cluster nodes.
MySQL generates default key and certificate files and places them in the data directory. You can override auto-generated files with manually created ones, as described in the section Generating Keys and Certificates Manually.
The auto-generated files are suitable for automatic SSL configuration, but you should use the same key and certificate files on all nodes.
Specify the following settings in the
my.cnf configuration file
for each node:
[mysqld] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem [client] ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/client-cert.pem ssl-key=/etc/mysql/certs/client-key.pem
After it is restarted, the node uses these files to encrypt communication with clients. MySQL clients require only the second part of the configuration to communicate with cluster nodes.
MySQL generates the default key and certificate files and places them in the data directory. You can either use them or generate new certificates. For generation of new certificate please refer to Generating Keys and Certificates Manually section.
The traffic of each type is transferred via a different channel, and so it is important to configure secure channels for all 3 variants to completely secure the replication traffic.
Percona XtraDB Cluster supports a single configuration option which helps to secure the complete replication traffic, and is often referred to as SSL Automatic Configuration. You can also configure the security of each channel by specifying independent parameters.
The automatic configuration of the SSL encryption needs a key and certificate files. MySQL generates a default key and certificate files and places them in the data directory.
It is important that your cluster use the same SSL certificates on all nodes.
pxc-encrypt-cluster-traffic is enabled thereby using a secured
channel for replication. This variable is not dynamic and so it cannot be changed
pxc-encrypt-cluster-traffic has the effect of applying the following
pxc-encrypt-cluster-traffic=ONhas the effect of applying
- the following settings in the
[mysqld] wsrep_provider_options=”socket.ssl_key=server-key.pem;socket.ssl_cert=server-cert.pem;socket.ssl_ca=ca.pem” [sst] encrypt=4 ssl-key=server-key.pem ssl-ca=ca.pem ssl-cert=server-cert.pem
wsrep_provider_options, only the mentioned options
are affected (
socket.ssl_ca), the rest is not modified.
The default value of
pxc-encrypt-cluster-traffic helps improve the security
of your system.
pxc-encrypt-cluster-traffic is not enabled, anyone with the
access to your network can connect to any Percona XtraDB Cluster node either as a
client or as another node joining the cluster. This potentially
lets them query your data or get a complete copy of it.
If you must disable
pxc-encrypt-cluster-traffic, you need
to stop the cluster and update [mysqld] section of the configuration file:
pxc-encrypt-cluster-traffic=OFF of each node. Then, restart the cluster.
The automatic configuration of the SSL encryption needs key and certificate files. MySQL generates default key and certificate files and places them in data directory. These auto-generated files are suitable for automatic SSL configuration, but you should use the same key and certificate files on all nodes. Also you can override auto-generated files with manually created ones, as covered in Generating Keys and Certificates Manually.
The necessary key and certificate files are first searched at the
ssl-key options under
[mysqld]. If these options are
not set, the data directory is searched for
[sst] section is not searched.
If all three files are found, they are used to configure encryption. If any of the files is missing, a fatal error is generated.
If user wants to enable encryption for specific channel only or use different certificates or other mix-match, then user can opt for manual configuration. This helps to provide more flexibility to end-users.
To enable encryption manually, the location of the required key and certificate files shoud be specified in the Percona XtraDB Cluster configuration. If you do not have the necessary files, see Generating Keys and Certificates Manually.
Encryption settings are not dynamic. To enable it on a running cluster, you need to restart the entire cluster.
There are three aspects of Percona XtraDB Cluster operation, where you can enable encryption:
This refers to SST traffic during full data copy from one cluster node (donor) to the joining node (joiner).
This refers to all internal Percona XtraDB Cluster communication, such as, write-set replication, IST, and various service messages.
Encrypting SST Traffic¶
This refers to full data transfer that usually occurs when a new node (JOINER) joins the cluster and receives data from an existing node (DONOR).
For more information, see State Snapshot Transfer.
keyring_file plugin is used, then SST encryption is mandatory:
when copying encrypted data via SST, the keyring must be sent over
with the files for decryption. In this case following options are to
be set in
my.cnf on all nodes:
The cluster will not work if keyring configuration across nodes is different.
The only available SST method is
xtrabackup-v2 which uses Percona XtraBackup.
This is the only available SST method (the
wsrep_sst_method is always set
xtrabackup-v2), which uses Percona XtraBackup to perform non-blocking transfer
of files. For more information, see Percona XtraBackup SST Configuration.
Encryption mode for this method is selected using the
encrypt=0is the default value, meaning that encryption is disabled.
encrypt=4enables encryption based on key and certificate files generated with OpenSSL. For more information, see Generating Keys and Certificates Manually.
To enable encryption for SST using XtraBackup, specify the location of the keys and certificate files in the each node’s configuration under
[sst] encrypt=4 ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem
SSL clients require DH parameters to be at least 1024 bits,
due to the logjam vulnerability.
However, versions of
socat earlier than 1.7.3 use 512-bit parameters.
dhparams.pem file of required length
is not found during SST in the data directory,
it is generated with 2048 bits, which can take several minutes.
To avoid this delay, create the
dhparams.pem file manually
and place it in the data directory before joining the node to the cluster:
openssl dhparam -out /path/to/datadir/dhparams.pem 2048
For more information, see this blog post.
Encrypting Replication/IST Traffic¶
Replication traffic refers to the following:
- Write-set replication which is the main workload of Percona XtraDB Cluster (replicating transactions that execute on one node to all other nodes).
- Incremental State Transfer (IST) which is copying only missing transactions from DONOR to JOINER node.
- Service messages which ensure that all nodes are synchronized.
All this traffic is transferred via the same underlying communication channel
gcomm). Securing this channel will ensure that IST traffic,
write-set replication, and service messages are encrypted.
(For IST, a separate channel is configured using the same configuration
parameters, so 2 sections are described together).
To enable encryption for all these processes, define the paths to the key, certificate and certificate authority files using the following wsrep provider options:
To set these options, use the
in the configuration file:
You must use the same key and certificate files on all nodes, preferably those used for Encrypting Client-Server Communication.
Check :upgrade-certificate: section on how to upgrade existing certificates.
As mentioned above, MySQL generates default key and certificate files and places them in the data directory. If you want to override these certificates, the following new sets of files can be generated:
- Certificate Authority (CA) key and certificate to sign the server and client certificates.
- Server key and certificate to secure database server activity and write-set replication traffic.
- Client key and certificate to secure client communication traffic.
These files should be generated using OpenSSL.
Common Name value
used for the server and client keys and certificates
must differ from the value used for the CA certificate.
Generating CA Key and Certificate¶
The Certificate Authority is used to verify the signature on certificates.
Generate the CA key file:
$ openssl genrsa 2048 > ca-key.pem
Generate the CA certificate file:
$ openssl req -new -x509 -nodes -days 3600 -key ca-key.pem -out ca.pem
Generating Server Key and Certificate¶
Generate the server key file:
$ openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout server-key.pem -out server-req.pem
Remove the passphrase:
$ openssl rsa -in server-key.pem -out server-key.pem
Generate the server certificate file:
$ openssl x509 -req -in server-req.pem -days 3600 \ -CA ca.pem -CAkey ca-key.pem -set_serial 01 \ -out server-cert.pem
Generating Client Key and Certificate¶
Generate the client key file:
$ openssl req -newkey rsa:2048 -days 3600 \ -nodes -keyout client-key.pem -out client-req.pem
Remove the passphrase:
$ openssl rsa -in client-key.pem -out client-key.pem
Generate the client certificate file:
$ openssl x509 -req -in client-req.pem -days 3600 \ -CA ca.pem -CAkey ca-key.pem -set_serial 01 \ -out client-cert.pem
To verify that the server and client certificates are correctly signed by the CA certificate, run the following command:
$ openssl verify -CAfile ca.pem server-cert.pem client-cert.pem
If the verification is successful, you should see the following output:
server-cert.pem: OK client-cert.pem: OK
Failed validation caused by matching CN
Sometimes, an SSL configuration may fail if the certificate and the CA files contain the same CN.
To check if this is the case run
openssl command as follows and verify that the CN field differs for the Subject and Issuer lines.
$ openssl x509 -in server-cert.pem -text -noout
Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=www.percona.com, O=Database Performance., C=US ... Subject: CN=www.percona.com, O=Database Performance., C=AU ...
To obtain a more compact output run
openssl specifying -subject and -issuer parameters:
$ openssl x509 -in server-cert.pem -subject -issuer -noout
subject= /CN=www.percona.com/O=Database Performance./C=AU issuer= /CN=www.percona.com/O=Database Performance./C=US
Deploying Keys and Certificates¶
Use a secure method (for example,
to send the key and certificate files to each node.
Place them under the
or similar location where you can find them later.
Make sure that this directory is protected with proper permissions.
Most likely, you only want to give read permissions
to the user running
The following files are required:
Certificate Authority certificate file (
This file is used to verify signatures.
Server key and certificate files (
These files are used to secure database server activity and write-set replication traffic.
Client key and certificate files (
These files are required only if the node should act as a MySQL client. For example, if you are planning to perform SST using
Upgrading Certificates subsection covers the details on upgrading certificates, if necessary.
The following procedure shows how to upgrade certificates used for securing replication traffic when there are two nodes in the cluster.
Restart the first node with the
socket.ssl_caoption set to a combination of the the old and new certificates in a single file.
For example, you can merge contents of
cat old-ca.pem > upgrade-ca.pem && \ cat new-ca.pem >> upgrade-ca.pem
wsrep_provider_optionsvariable as follows:
Restart the first node with the new certificate files, as in the previous step.
You can remove the old certificate files.