Deploying Percona Operator for MongoDB Across GKE Clusters with MCS

Ever needed a robust, highly available MongoDB setup that spans multiple Kubernetes clusters on GCP? This step-by-step guide shows you how to deploy the Percona Operator for MongoDB in two GKE clusters, linking them using Multi-Cluster Services (MCS) for seamless cross-cluster discovery and connectivity.

Step 1: Prepare your GKE clusters & enable MCS

1. Prepare your account for Multi-Cluster Services (MCS)

2. Create Two GKE Clusters

Use your preferred method (e.g., gcloud container clusters create) to set up main-cluster and replica-cluster. Workload Identity Federation is recommended by Google for MCS.

3. Enable Multi-Cluster Services (MCS)

In your fleet host project, run:

4. Register the clusters to the fleet

5. Grant the required Identity and Access Management (IAM) permissions for MCS Importer

6. Verify that MCS is enabled

Step 2: Install Percona Operator on both GKE clusters

Install the Operator in each cluster (main Cluster and replica Cluster):

1. Generate a kubeconfig file for each GKE cluster

2. Give permissions to your account to manage the GKE clusters

3. (Optional) Open one terminal to manage each cluster

Terminal 1:

Terminal 2:

4. Create the same namespace on both clusters and install the Percona Operator for MongoDB.

Terminal 1:

Terminal 2:

Step 3: Create the main PSMDB cluster

Make sure to create it in the psmdb namespace and use ClusterIP services (which are required for MCS).

Step 4: Export the secrets from the main cluster

Step 5: Modify secrets and apply to the GKE replica cluster

You need to remove the fields:

The following helper scripts can be used:

Now create the modified secrets on the GKE replica cluster:

Step 6: Create the replica PSMDB cluster

Make sure to create it in the psmdb namespace and remember to set “unmanaged: true” and updateStrategy to RollingUpdate or OnDelete.

Step 7: Get the names of the services

Run this step on both clusters:

If ServiceImports are missing, check the MCS controller logs and ensure ServiceExports were created by the operator. Replace the pod name below with your own operator pod:

Step 8: Add the replica nodes to the Main Cluster

Using the service names we got from the previous step, add each node from the replica side to the main cluster. Perform this for every shard and the config server replica set.

Step 9: Add the main nodes to the Replica Cluster

Similarly to the previous step, edit the yaml file on the replica side, and add the main nodes as external:

Step 10: Verify the configuration

Connect to a member of each replica set. I am using the config servers in the example:

Verify all members are present (I’ve removed some fields from the output for readability)

Step 11: Test the switchover process

Set the main cluster to unmanaged by editing the yaml file and applying the changes:

Now set the replica cluster to managed, so the operator assumes control of the nodes on the replica side:

Verify a new primary was elected in the replica side

Final thoughts

By combining the power of Percona Operator for MongoDB with GKE’s Multi-Cluster Services, you gain a resilient, scalable, and multi-region replica set architecture. Perfect for high-availability applications and disaster recovery use cases.