Using the keyring_vault Plugin with Percona Server for MySQL 5.7

September 17, 2018

Author

Jericho Rivera

MySQL

Percona Software

Security

Share this Post:

This is the first of a two-part series on using the keyring_vault plugin with Percona Server for MySQL 5.7. The second part, Backing up Percona Server for MySQL with keyring_vault plugin enabled, walks through how to use Percona Xtrabackup to backup from this instance and restore to another server and set it up as a slave with keyring_vault plugin.

What is the keyring_vault plugin?

The keyring_vault is a plugin that allows the database to interface with a Hashicorp Vault server to store and secure encryption keys. The Vault server then acts as a centralized encryption key management solution which is critical for security and for compliance with various security standards.

Configuring Vault

Create SSL certificates to be used by Vault. You can use the sample ssl.conf template below to generate the necessary files.

[root@vault1 ~]# cat /etc/sslkeys/ssl.conf<br>[req]<br>distinguished_name = req_distinguished_name<br>x509_extensions = v3_req<br>prompt = no<br><br>[req_distinguished_name]<br>C = US<br>ST = NC<br>L =  R<br>O = Percona<br>CN = *<br><br>[v3_req]<br>subjectKeyIdentifier = hash<br>authorityKeyIdentifier = keyid,issuer<br>basicConstraints = CA:TRUE<br>subjectAltName = @alt_names<br><br>[alt_names]<br>IP = 192.168.0.114

[root@vault1 ~]# cat /etc/sslkeys/ssl.conf [req] distinguished_name = req_distinguished_name x509_extensions = v3_req prompt = no [req_distinguished_name] C = US ST = NC L = R O = Percona CN = * [v3_req] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer basicConstraints = CA:TRUE subjectAltName = @alt_names [alt_names] IP = 192.168.0.114

Then run the two commands below to generate the cert and key files and the certificate chain:

$ openssl req -config ssl.conf -x509 -days 365 -batch -nodes -newkey rsa:2048 -keyout vault.key -out vault.crt<br>$ cat vault.key vault.crt &gt; vault.pem

1	$ openssl req -config ssl.conf -x509 -days 365 -batch -nodes -newkey rsa:2048 -keyout vault.key -out vault.crt<br>$ cat vault.key vault.crt > vault.pem

Once the SSL certificates are created start Vault with the sample configuration below. Take note that you should follow the suggested best practices when deploying Vault in production, this example is to get us by with a simple working setup.

[root@vault1 ~]# cat /etc/vault.hcl<br>listener "tcp" {<br>address = "192.168.0.114:8200"<br>tls_cert_file="/etc/sslkeys/vault.crt"<br>tls_key_file="/etc/sslkeys/vault.key"<br>}<br><br>storage "file" {<br>path = "/var/lib/vault"<br>}

1	[root@vault1 ~]# cat /etc/vault.hcl<br>listener "tcp" {<br>address = "192.168.0.114:8200"<br>tls_cert_file="/etc/sslkeys/vault.crt"<br>tls_key_file="/etc/sslkeys/vault.key"<br>}<br><br>storage "file" {<br>path = "/var/lib/vault"<br>}

Assuming Vault started up fine and you are able to unseal Vault, the next step is to create the policy file. For more details on initializing and unsealing Vault please read the manual here.

[root@vault1 ~]# cat /etc/vault/policy/dc1.hcl<br>path "secret/*" {<br>capabilities = ["list"]<br>}<br><br>path "secret/dc1/*" {<br>capabilities = ["create", "read", "delete", "update", "list"]<br>}

1	[root@vault1 ~]# cat /etc/vault/policy/dc1.hcl<br>path "secret/" {<br>capabilities = ["list"]<br>}<br><br>path "secret/dc1/" {<br>capabilities = ["create", "read", "delete", "update", "list"]<br>}

Create a Vault policy named dc1-secrets using the dc1.hcl file like this:

[root@vault1 ~]# vault policy write dc1-secrets /etc/vault/policy/dc1.hcl -ca-cert=/etc/sslkeys/vault.pem<br>Success! Uploaded policy: dc1-secrets

1	[root@vault1 ~]# vault policy write dc1-secrets /etc/vault/policy/dc1.hcl -ca-cert=/etc/sslkeys/vault.pem<br>Success! Uploaded policy: dc1-secrets

Next, create a token associated with the newly created policy:

[root@vault1 ~]# vault token create -policy=dc1-secrets -ca-cert=/etc/sslkeys/vault.pem &gt; dc1-token<br>[root@vault1 ~]# cat dc1-token<br>Key                  Value<br>---                  -----<br>token                be515093-b1a8-c799-b237-8e04ea90ad7a<br>token_accessor       4c1ba5c5-3fed-e9bb-d230-5bf1392e2d7e<br>token_duration       8760h<br>token_renewable      true<br>token_policies       ["dc1-secrets" "default"]<br>identity_policies    []<br>policies             ["dc1-secrets" "default"]

[root@vault1 ~]# vault token create -policy=dc1-secrets -ca-cert=/etc/sslkeys/vault.pem > dc1-token [root@vault1 ~]# cat dc1-token Key Value --- ----- token be515093-b1a8-c799-b237-8e04ea90ad7a token_accessor 4c1ba5c5-3fed-e9bb-d230-5bf1392e2d7e token_duration 8760h token_renewable true token_policies ["dc1-secrets" "default"] identity_policies [] policies ["dc1-secrets" "default"]

Setting up MySQL

The following instructions should work starting from Percona Server for MySQL 5.7.20-18 and through later versions.

Configure my.cnf with the following variables:

early-plugin-load="keyring_vault=keyring_vault.so"<br>loose-keyring_vault_config="/var/lib/mysql-keyring/keyring_vault.conf"<br><br>encrypt_binlog=ON<br>innodb_encrypt_online_alter_logs=ON<br>innodb_encrypt_tables=ON<br>innodb_temp_tablespace_encrypt=ON<br>master_verify_checksum=ON<br>binlog_checksum=CRC32<br><br>log_bin=mysqld-bin<br>binlog_format=ROW<br>server-id=1<br>log-slave-updates

early-plugin-load="keyring_vault=keyring_vault.so" loose-keyring_vault_config="/var/lib/mysql-keyring/keyring_vault.conf" encrypt_binlog=ON innodb_encrypt_online_alter_logs=ON innodb_encrypt_tables=ON innodb_temp_tablespace_encrypt=ON master_verify_checksum=ON binlog_checksum=CRC32 log_bin=mysqld-bin binlog_format=ROW server-id=1 log-slave-updates

Create the keyring_vault.conf file in the path above with the following contents:

[root@mysql1 ~]# cat /var/lib/mysql-keyring/keyring_vault.conf<br>vault_url = https://192.168.0.114:8200<br>secret_mount_point = secret/dc1/master<br>token = be515093-b1a8-c799-b237-8e04ea90ad7a<br>vault_ca = /etc/vault_ca/vault.pem

1	[root@mysql1 ~]# cat /var/lib/mysql-keyring/keyring_vault.conf<br>vault_url = https://192.168.0.114:8200<br>secret_mount_point = secret/dc1/master<br>token = be515093-b1a8-c799-b237-8e04ea90ad7a<br>vault_ca = /etc/vault_ca/vault.pem

Here we are using the vault.pem file generated by combining the vault.crt and vault.key files. Observe that our secret_mount_point is secret/dc1/master. We want to make sure that this mount point is unique across all servers, this is in fact advised in the manual here.

Ensure that the CA certificate is owned by mysql user:

[root@mysql1 ~]# ls -la /etc/vault_ca/<br>total 24<br>drwxr-xr-x  2 mysql mysql   41 Jul 14 11:39 .<br>drwxr-xr-x 63 root  root  4096 Jul 14 13:17 ..<br>-rw-------  1 mysql mysql 1139 Jul 14 11:39 vault.pem

1	[root@mysql1 ~]# ls -la /etc/vault_ca/<br>total 24<br>drwxr-xr-x 2 mysql mysql 41 Jul 14 11:39 .<br>drwxr-xr-x 63 root root 4096 Jul 14 13:17 ..<br>-rw------- 1 mysql mysql 1139 Jul 14 11:39 vault.pem

Initialize the MySQL data directory on the Master:

[root@mysql1 ~]# mysqld --initialize-insecure --datadir=/var/lib/mysql --user=mysql

1	[root@mysql1 ~]# mysqld --initialize-insecure --datadir=/var/lib/mysql --user=mysql

For production systems, we do not recommend using --initialize-insecure option, this is just to skip additional steps in this tutorial.

Finally, start mysqld instance and then test the setup by creating an encrypted table.

[root@mysql1 ~]# systemctl status mysqld<br>● mysqld.service - MySQL Server<br>Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled)<br>Active: active (running) since Sat 2018-07-14 23:53:16 UTC; 2s ago<br>Docs: man:mysqld(8)<br>http://dev.mysql.com/doc/refman/en/using-systemd.html<br>Process: 1401 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)<br>Process: 1383 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)<br>Main PID: 1403 (mysqld)<br>CGroup: /system.slice/mysqld.service<br>└─1403 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid<br><br>Jul 14 23:53:16 mysql1 systemd[1]: Starting MySQL Server...<br>Jul 14 23:53:16 mysql1 systemd[1]: Started MySQL Server.

[root@mysql1 ~]# systemctl status mysqld ● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled) Active: active (running) since Sat 2018-07-14 23:53:16 UTC; 2s ago Docs: man:mysqld(8) http://dev.mysql.com/doc/refman/en/using-systemd.html Process: 1401 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 1383 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 1403 (mysqld) CGroup: /system.slice/mysqld.service └─1403 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid Jul 14 23:53:16 mysql1 systemd[1]: Starting MySQL Server... Jul 14 23:53:16 mysql1 systemd[1]: Started MySQL Server.

At this point, you should have Percona Server for MySQL instance with tablespace encryption using Vault.