Configuring Percona Backup for MongoDB in a Multi-Instances Environment

Configuring and managing backups is one of the top priorities among the many tasks we deal with daily in our database instances.

In this context, it’s not rare to see deployments with more than one mongod process running per node, creating a multi-instance environment.

Although this approach is highly discouraged due to the potential risk of compromising your business’s availability, there are scenarios where such a configuration makes sense.

If you have a setup where a host runs more than one mongod process and you’re looking at how to properly configure Percona Backup for MongoDB (PBM) on that, this article will guide you through that scenario.

Before getting started, this article assumes two prior conditions:

1. Your mongod processes are already in place and configured.
2. You have Percona Server for MongoDB(PSMDB) initially installed on your machine.

   1. If not, please follow the installation instructions in the official documentation: PBM – Quickstart guide.

Those pre-conditions are crucial for the brevity and straightforwardness of the article.

Environment:

The environment and how you have configured your multi-instance node can vary heavily depending on your objectives.

For example, you might have a host with two mongod processes while a second one holds three more. Or if you have a Sharded Cluster,  you might also spread your instances in only three hosts.

To eliminate such an infinite possibility, we will be using a single host that holds the entire Replica Set of tree nodes as illustrated in the following diagram:

• Red Hat Linux 8
• Percona Server for MongoDB(PSMDB): 7.0.11-6
• Percona Backup for MongoDB: 2.5.0

Whether it’s a Replica Set with N nodes or a Sharded Cluster will not change the outcome of this article.

That’s because, and here is the first piece of advice.

• You must configure the pbm-agent on all mongod data-bearing nodes in your cluster. 

  • That includes replica set nodes that are currently PRIMARY or SECONDARIES and Config Server replica set nodes if the cluster is a Sharded Cluster.
  • Since they don’t have the data set, you don’t need to configure it on the arbiter or mongoSes.

• The configuration of pbm-agent and mongod will work as 1 to 1.

  • For each mongod, you must deploy and configure a pbm-agent accordingly.

Introduction

To configure PBM in this scenario of multi-instance environment, there are two possibilities:

1. Via a background process with the use of nohup and &:

2. Or use it via the service manager – systemd:

Although the first option looks more tempting due to its simplicity, it’s not the best option because it lacks further control of itself and other processes. PBM needs to have control of the mongod process over a few operations, such as the restoration where the database is restarted several times.

Losing control of a process mid-operation, which can lead to inconsistent results, is not what we are looking for.

In that matter, the recommended approach is to deploy individual PBM service files and let the systemd control them.

Process

1. Configuring the Service file.

• By default, after a clean installation of PBM, it already deploys the respective service file for the process:

We will use the default pbm-agent.service it to create the copy.

2. Create a template service file that you can be used for multiple instances:

• Edit the template service file to use instance-specific environment files and configurations:

The %i placeholder in the service file will be replaced with the service number that will pass later, making the service unit specific to that instance.

3. Now, let’s create environment files for each PBM Instance and deploy them to the system.

• Create environment files for each PBM instance to specify the connection details:

PBM Agents for the second and third MongoDB instances (mongod-2, mongod3 )

• Reload the systemd daemon to recognize the new service files:

4. Before starting the process, we need to create a user that PBM will use to manage PBM operations inside the Cluster.

Using the official documentation as a reference, let’s create the use as follows:

Note: Execute this step on a primary node of each replica set. In a sharded cluster, this means on every shard replica set and the config server replica set.

• Create a role that allows any action on any resource:

• Create the user and assign the role you created to it:

You can specify the username and password values and other options of the createUser command as long as the above roles are granted.

5. Configure the MongoDB connection URI for pbm-agent.

As mentioned initially in this article, each pbm-agent process connects to a mongod in a 1 to 1 configuration, in a standalone type of connection.

• First, let’s edit and configure /etc/sysconfig/pbm-agent:

• Now, let’s edit and configure /etc/sysconfig/pbm-agent2:

• Finally,  let’s edit and configure /etc/sysconfig/pbm-agent3:

• Ensure to set the right user we created for PBM, in this case – pbmuser:secretpwd .
• Ensure to set the respective port for the database.
• Ensure to set the right Replica Set name, in this case “replset”. 

When we initialize the pbm-agents, they will use their respective environment files to connect to the database. 

Note: Just a friendly reminder that we are using a Red Hat installation. For a Debian-based installation, please check the official documentation here for better reference on the paths and directory names.

Before we move forward with the following steps, this is how our configuration would be represented at the moment:

Your pbm-agents are configured but not running yet.

6. In the next step, configure the connection URI for PBM CLI.

At this phase, it’s important to highlight that PBM is composed of two components:

1. The pbm-agent – Which is the PBM process itself, and we already configured it.
2. The pbm-CLI – The PBM Client is a command-line utility that instructs pbm-agents to perform operations like backup, restore, status, and so on.

The PBM client connects to Replica Set via MongoDB URI connection string:

As a Replica Set connection string, you must pass all the nodes that are part of it. That’s in case a node becomes unavailable; the PBM client can detect and automatically route the connection to the available one.

7. Configuring the backup storage location.

Before we fire up the PBM for the first time, we need to configure the storage location. As the name says, this is where PBM will store your backups.

The current supported Storages are:

• S3-compatible storage.
• Filesystem type storage.
• Microsoft Azure Blob storage.

When configuring the storage location, it’s crucial that all mongod can connect to that same storage endpoint. In this demonstration, it’s simpler because the entire cluster is hosted on a single server. 

However, if your nodes are spread over different hosts, they must be able to reach the storage equally, regardless of whether it’s NFS or S3-compatible.

To configure:

1. Create a config file (e.g. pbm_config.yaml) in a path where PBM can read it:

       2.  Specify the storage information within:

• • Here, we are simulating an NFS mount point, thus using the storage type filesystem. For more information on the configuration reference for the supported storage, please check the documentation here:

       3. Load the file to PBM via the PBM client command:

• Sample output:

That’s the final result after configuring the PBM client:

All PBM components are configured but are not running yet.

8. Let’s start the pbm-agents:

• To watch their system logs, you can use journalctl and pass their service placeholder:

• You can also check the logs via the PBM client command:

• If everything goes smoothly, you should be able to see PBM successfully recognizing the Cluster and ready for work:

Congratulations, you have configured PBM in a multi-instance environment.

The final diagram of our setup would look as follows:

PBM and its components were successfully deployed and are connected and fully functional with the PSMDB cluster.

Conclusion

Deploying Percona Backup for MongoDB (PBM) in a multi-instance environment requires more steps and care than a regular installation. However, the article above aims to provide all the necessary steps, observations, and commands for a seamless installation and optimal tool functionality.

If you have any questions or encounter any problems, feel free to share them in the comments section or open a question in our Community Forum.