Starting with Percona XtraDB Cluster (PXC) 8.0, replication traffic encryption is enabled by default. That said, it’s common to find clusters running without TLS that suddenly need it: a new compliance requirement, an audit finding, a network segment that is no longer considered trusted.
PXC has a variable for exactly that case, pxc-encrypt-cluster-traffic, which handles SSL encryption for inter-node traffic, including State Snapshot Transfer (SST), Incremental State Transfer (IST), and the group communication the nodes use for replication.
The variable is not dynamic, and turning it on normally costs a full cluster restart since the node that encrypts traffic listens on ssl:// while its peers are still on tcp://. If a node joins the cluster with TLS enabled while remaining nodes don’t, the restarting node fails to reach out to the other peers with the following error:
|
1 |
2026-09-02T02:16:02.359992Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: wrong version number |
This is a common OpenSSL error seen when a client sends encrypted TLS traffic to a server that expects unencrypted plaintext.
Starting with PXC 8.0.28 and in all PXC 8.4 versions, there is a way to work around the full cluster restart, which relies on the Galera socket.dynamic option. In this post I’ll go through how this option works and how it can be used to enable TLS without downtime.
This provider option lets a node establish communication with other peers using both encrypted and plaintext connections. The instance will initially open an encrypted connection, and if it fails since the other nodes are not using TLS, it will retry to establish an unencrypted connection.
The Joiner error log will show the following sequence:
|
1 2 3 |
2026-09-02T02:16:43.321209Z 0 [Note] [MY-000000] [Galera] Failed to establish connection: wrong version number 2026-09-02T02:16:43.321521Z 0 [Note] [MY-000000] [Galera] (565f0a92-8283, 'tcp://0.0.0.0:4567') connection established to 1c72fb64-ac57 tcp://172.18.0.3:4567 2026-09-02T02:16:43.321566Z 0 [Note] [MY-000000] [Galera] (565f0a92-8283, 'tcp://0.0.0.0:4567') connection established to 2001e9dd-a09a tcp://172.18.0.4:4567 |
The first line is the same error we saw previously: our node initiated a TLS handshake, but the peer responded with something that was not TLS. Without socket.dynamic, the node would stop at that point and never join, but with it, it retries (without encryption) and is able to establish a connection.
This Galera option can’t be changed at runtime, so you need to restart the node to enable it.
In this procedure, we consider a typical PXC architecture with 3 nodes. Enabling encryption requires two rolling restarts: the first turns TLS on while still allowing plaintext connections, and the second removes that option and forces only TLS connections.
Whether the certificates are auto-generated by MySQL or issued by us, every member of the cluster has to use the same key and certificate files. It’s recommended to keep the certificates outside the data directory for security and configuration clarity, and to keep them clear of SST since a joiner wipes its data directory before a full state transfer, and only files matching the [sst] cpat pattern survive. That default covers *.pem, so the MySQL-generated names are safe, but a .crt or .key left in the data directory is deleted.
– pxc-encrypt-cluster-traffic=ON, this switch adds TLS for group communication, IST, and SST.
– socket.dynamic=YES option inside the wsrep_provider_options variable, so each node can speak both protocols while the rollout is in progress.
– The ssl-* variables, in case we’re setting the certificates outside the default path, which is the data directory.
– An [sst] section with the encrypt=4 option and the ssl-* variables. Since pxc-encrypt-cluster-traffic enforces SST channel to encrypt=4, this is a safety measure for those that did not yet performed the change, so donor and joiner can communicate over TLS while the change has not been applied in all nodes. Even though nothing has been restarted, the SST script re-reads my.cnf on every transfer, so as soon as the file is in place the donors encrypt SST whether or not they have been restarted.
The my.cnf file shows the following values:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
[mysqld] pxc-encrypt-cluster-traffic=ON wsrep_provider_options="socket.dynamic=YES" ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem [sst] encrypt=4 ssl-ca=/etc/mysql/certs/ca.pem ssl-cert=/etc/mysql/certs/server-cert.pem ssl-key=/etc/mysql/certs/server-key.pem |
Wait for each node to rejoin and report Synced before moving to the next. A restarted node will try TLS first and then fall back.
You can verify the change was correctly applied by running the following query. Since the SSL settings are in the wsrep_provider_options variable, which is a long string of key/value pairs, we pull out only the ones we are interested in:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
mysql> SELECT -> REGEXP_SUBSTR(@@wsrep_provider_options, 'socket\\.dynamic = [^;]+') AS socket_dynamic, -> REGEXP_SUBSTR(@@wsrep_provider_options, 'socket\\.ssl = [^;]+') AS socket_ssl, -> REGEXP_SUBSTR(@@wsrep_provider_options, 'socket\\.ssl_ca = [^;]+') AS socket_ssl_ca, -> REGEXP_SUBSTR(@@wsrep_provider_options, 'socket\\.ssl_cert = [^;]+') AS socket_ssl_cert, -> REGEXP_SUBSTR(@@wsrep_provider_options, 'socket\\.ssl_key = [^;]+') AS socket_ssl_key\G *************************** 1. row *************************** socket_dynamic: socket.dynamic = YES socket_ssl: socket.ssl = YES socket_ssl_ca: socket.ssl_ca = /etc/mysql/certs/ca.pem socket_ssl_cert: socket.ssl_cert = /etc/mysql/certs/server-cert.pem socket_ssl_key: socket.ssl_key = /etc/mysql/certs/server-key.pem 1 row in set (0.00 sec) |
The provider option socket.ssl and ssl certificates are passed down from MySQL to the provider because pxc-encrypt-cluster-traffic is on.
That confirms the settings reached the provider. That said, the replication traffic is not encrypted yet.
We can check this by using tcpdump on node #1 to listen on the replication port (4567) packets coming from node #2:
|
1 |
$ tcpdump -i any -nn -A -s0 "tcp port 4567 and host node2" -c 2000 | grep -a table_test |
On node #2 we create the table_test that will be replicated to the other nodes:
|
1 2 |
mysql> create table test.table_test (c1 integer primary key); Query OK, 0 rows affected (0.10 sec) |
Since in PXC DDLs are replicated as a statement, tcpdump on node #1 shows the create table command in plaintext:
|
1 |
...x..x.j.....y............................ ..E......std............test.create table test.table_test (c1 integer primary key)....... |
Node #3 does not need the socket.dynamic option since the other two nodes are already capable of talking either TLS or plaintext.
We modify the my.cnf option file as below:
|
1 2 |
[mysqld] wsrep_provider_options="socket.dynamic=NO" |
And restart the node. This node will start and request only TLS from the other nodes, which they can respond to, since they already have all the changes in place.
The node will show reaching out to cluster peers over ssl://
|
1 2 |
2026-09-02T02:19:03.640009Z 0 [Note] [MY-000000] [Galera] (aac049fc-a401, 'ssl://0.0.0.0:4567') connection established to 8d5f826f-8dd1 ssl://172.18.0.3:4567 2026-09-02T02:19:03.640410Z 0 [Note] [MY-000000] [Galera] (aac049fc-a401, 'ssl://0.0.0.0:4567') connection established to 565f0a92-8283 ssl://172.18.0.2:4567 |
In order to disallow the non-encrypted option and to establish a TLS connection with the other nodes, nodes #1 and #2 should be restarted.
Similar to the previous step, when restarting, the nodes will establish communication on TLS.
Node #1
|
1 2 |
2026-09-02T02:20:12.565801Z 0 [Note] [MY-000000] [Galera] (d3d58399-9ea2, 'ssl://0.0.0.0:4567') connection established to aac049fc-a401 ssl://172.18.0.4:4567 2026-09-02T02:20:12.565855Z 0 [Note] [MY-000000] [Galera] (d3d58399-9ea2, 'ssl://0.0.0.0:4567') connection established to 8d5f826f-8dd1 ssl://172.18.0.3:4567 |
Node #2
|
1 2 |
2026-09-02T02:21:20.911318Z 0 [Note] [MY-000000] [Galera] (fc923848-a6d1, 'ssl://0.0.0.0:4567') connection established to d3d58399-9ea2 ssl://172.18.0.2:4567 2026-09-02T02:21:20.911339Z 0 [Note] [MY-000000] [Galera] (fc923848-a6d1, 'ssl://0.0.0.0:4567') connection established to aac049fc-a401 ssl://172.18.0.4:4567 |
To confirm the replication traffic is encrypted, we can perform the same check as in step 3, by running tcpdump on node #1 and listening on the replication port (4567) for packets coming from node #2:
|
1 |
$ tcpdump -i any -nn -A -s0 "tcp port 4567 and host node2" -c 2000 | grep -a table_test |
On node #2 we drop the table_test previously created:
|
1 2 |
mysql> drop table test.table_test; Query OK, 0 rows affected (0.04 sec) |
tcpdump does not show a matching pattern since the replication traffic is now encrypted.
We can use the same mechanism to disable TLS for whatever reason. As with enabling it, this takes two rolling restarts. Starting from a cluster where every node has pxc-encrypt-cluster-traffic=ON we perform the following steps:
– Restart nodes #1 and #2 with socket.dynamic=YES, keeping pxc-encrypt-cluster-traffic=ON, so they accept plaintext connections again.
– Restart node #3 with pxc-encrypt-cluster-traffic=OFF, it will come up in plaintext only, and the other two nodes will accept it because they are back to accepting both protocols.
– Restart nodes #1 and #2 with pxc-encrypt-cluster-traffic=OFF and without socket.dynamic, which leaves the whole cluster in plaintext.
– Finally, remove from the sst section the encrypt and SSL related variables.
Enabling TLS for inter-node traffic can be done without stopping the whole cluster. The socket.dynamic option lets a node accept both encrypted and plaintext connections during the transition, so the change can be rolled out one node at a time. Please note we’ll need at least two rolling restarts to achieve that the cluster fully communicates via TLS.
Resources
RELATED POSTS
