Using Vault to Store the Master Key for Data at Rest Encryption on Percona Server for MongoDB

Percona Server MongoDB EncryptionSince the release of Percona Server MongoDB 3.6.13 (PSMDB), you have been able to use Vault to store the encryption keys for data at rest encryption. Here’s how to set it up.

First, you need to have a Vault server up and running. My colleague, Jericho, has an article on setting up Vault for Percona Server titled Using the keyring_vault Plugin with Percona Server for MySQL 5.7. In this post, I will provide the same instructions for installing and setting up Hashicorp Vault for testing (Thank you Jericho!).

Vault Installation(run as root or use sudo):

1. Download, extract and install Vault:

Make sure to place it somewhere on your PATH

2. Place initial vault configuration in /etc/vault

You can use a port different than 8200. disable-mlock=true  is needed if you want to start the Vault server as a non-root user.

3. Generate SSL certificates. To be able to create SSL certificates without going through prompts, we will place those entries in a configuration file:

4. Run the commands below to generate the certificates and store them in /etc/vault.

5. Ensure that the keys are only accessible by the owner.

6. Set environment variables needed to access the vault.

7. Start Vault in the background.

8. Initialize Vault and store the unseal keys and the initial root token generated in this step:

9. Unseal the Vault by supplying the three unseal keys generated above. You need to unseal Vault every time the Vault server is started.

10. Now it’s time to log in to Vault and use the initial root token.

11. At this point, on Vault, you will need to enable the KV Secrets Engine – Version 2, the only engine supported by PSMDB. To do so, you will need to run the following:

12. Next, you will need to create a Vault Policy to access this datastore. As per https://www.vaultproject.io/docs/secrets/kv/kv-v2/, reading and writing versions need to be prefixed with the path data/.

13. To upload the policy to the Vault server, run:

14. Finally, you will need to create an access token to be used by MongoDB. Remember to create a token per MongoDB instance.

You will need to use the token above to connect MongoDB to Vault.

15. Done.

Incorporating Vault in MongoDB

In MongoDB, you’ll need to create a token based on the generated token and CA file generated in the Vault. Let’s place them in /etc/mongodb:

1. Create a mongodb config directory that will store both token and CA file.

2. Place the token value in the token file and copy the contents of vault.crt from the Vault server.

3. Ensure the files are readable by mongod only, as MongoDB will complain and fail during startup with lax permissions.

4. Place in /etc/mongod.conf the Vault configuration under the security section. As a sample naming convention, given that the hostname of this server is psmongodb1, the secret path will be secret/data/dc/psmongodb1 as well:

5. Then start MongoDB:

You can only enable encryption on an empty database. It’s not possible to re-encrypt an existing, previously unencrypted database. One way to do it is to create a backup, stop mongod, delete data files, enable encryption, start mongod, and restore the database from backup. If you have a replica set, you can apply changes in a rolling manner (one member at the time). Data will be synced automatically from other nodes. Each node has to be encrypted separately.

6. Check the logs if using the key was successful:

As per the logs, it was able to generate a master key and use the Percona encryption. At this point, data at rest encryption has been configured on this server.

7. Done.

Take note that if the Vault server is down, you will not be able to start up your server. Given this new dependency on obtaining the master key from Vault, be sure to design your architecture for fault tolerance.

Encryption Key Rotation

Each encryption key can be used only for a limited time. Rotating master key re-encrypts the keystore using a new master key. The newly generated master key is then stored in the Vault. The entire dataset isn’t re-encrypted.

1. Place in /etc/mongod.conf an extra line in the security section:

2. Then restart MongoDB:

You’ll get an error message (but that’s expected, as mongod process didn’t start successfully). You can check logs if the encryption key was successfully rotated:

3. Remove the extra line in /etc/mongod.conf configuration rotateMasterKey: true

4. Done. You can start mongod again, your key was rotated.

Please be advised that this is a simple guideline on how to set up a basic Vault server and should not be used as a template for production usage. We recommend using Percona Consulting Services to assist you in that matter.

Share this post

Comment (1)

  • sv Reply

    Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 1073 i am getting this one

    May 18, 2020 at 10:32 am

Leave a Reply