Ever needed a robust, highly available MongoDB setup that spans multiple Kubernetes clusters on GCP? This step-by-step guide shows you how to deploy the Percona Operator for MongoDB in two GKE clusters, linking them using Multi-Cluster Services (MCS) for seamless cross-cluster discovery and connectivity.
Step 1: Prepare your GKE clusters & enable MCS
1. Prepare your account for Multi-Cluster Services (MCS)
|
1 2 3 4 5 6 7 8 |
export PROJECT_ID=your_project_id gcloud services enable multiclusterservicediscovery.googleapis.com gkehub.googleapis.com cloudresourcemanager.googleapis.com trafficdirector.googleapis.com dns.googleapis.com |
2. Create Two GKE Clusters
Use your preferred method (e.g., gcloud container clusters create) to set up main-cluster and replica-cluster. Workload Identity Federation is recommended by Google for MCS.
|
1 2 3 4 5 6 |
gcloud container clusters create main-cluster --zone us-central1-a --cluster-version 1.32 --machine-type n1-standard-4 --num-nodes=3 --workload-pool=$PROJECT_ID.svc.id.goog |
|
1 2 3 4 5 6 |
gcloud container clusters create replica-cluster --zone us-central1-a --cluster-version 1.32 --machine-type n1-standard-4 --num-nodes=3 --workload-pool=$PROJECT_ID.svc.id.goog |
3. Enable Multi-Cluster Services (MCS)
In your fleet host project, run:
|
1 |
gcloud container fleet multi-cluster-services enable --project $PROJECT_ID |
4. Register the clusters to the fleet
|
1 2 3 |
gcloud container fleet memberships register main-cluster --gke-cluster us-central1-a/main-cluster --enable-workload-identity |
|
1 2 3 |
gcloud container fleet memberships register replica-cluster --gke-cluster us-central1-a/replica-cluster --enable-workload-identity |
5. Grant the required Identity and Access Management (IAM) permissions for MCS Importer
|
1 2 3 4 5 6 |
# get the fleet project number PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)") gcloud projects add-iam-policy-binding $PROJECT_ID --member "principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$PROJECT_ID.svc.id.goog/subject/ns/gke-mcs/sa/gke-mcs-importer" --role "roles/compute.networkViewer" |
6. Verify that MCS is enabled
|
1 |
gcloud container fleet multi-cluster-services describe --project $PROJECT_ID |
Step 2: Install Percona Operator on both GKE clusters
Install the Operator in each cluster (main Cluster and replica Cluster):
1. Generate a kubeconfig file for each GKE cluster
|
1 2 3 |
KUBECONFIG=./gcp-main_config gcloud container clusters get-credentials main-cluster --zone us-central1-a KUBECONFIG=./gcp-replica_config gcloud container clusters get-credentials replica-cluster --zone us-central1-a |
2. Give permissions to your account to manage the GKE clusters
|
1 2 3 |
kubectl --kubeconfig gcp-main_config create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account) kubectl --kubeconfig gcp-replica_config create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account) |
3. (Optional) Open one terminal to manage each cluster
Terminal 1:
|
1 |
kubectl --kubeconfig gcp-main_config config set-context $(kubectl config current-context) |
Terminal 2:
|
1 |
kubectl --kubeconfig gcp-replica_config config set-context $(kubectl config current-context) |
4. Create the same namespace on both clusters and install the Percona Operator for MongoDB.
Terminal 1:
|
1 2 3 |
kubectl create ns psmdb kubectl config set-context --current --namespace=psmdb kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.20.1/deploy/bundle.yaml |
Terminal 2:
|
1 2 3 |
kubectl create ns psmdb kubectl config set-context --current --namespace=psmdb kubectl apply --server-side -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.20.1/deploy/bundle.yaml |
Step 3: Create the main PSMDB cluster
Make sure to create it in the psmdb namespace and use ClusterIP services (which are required for MCS).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: main-cluster spec: crVersion: 1.20.1 image: percona/percona-server-mongodb:7.0.14-8-multi updateStrategy: SmartUpdate multiCluster: enabled: true DNSSuffix: svc.clusterset.local upgradeOptions: apply: disabled schedule: "0 2 * * *" secrets: users: my-cluster-name-secrets encryptionKey: my-cluster-name-mongodb-encryption-key replsets: - name: rs0 size: 3 expose: enabled: true type: ClusterIP volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 3 expose: enabled: true type: ClusterIP volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 3 expose: type: ClusterIP |
|
1 |
kubectl apply -f cr-main.yml |
Step 4: Export the secrets from the main cluster
|
1 2 3 4 |
kubectl get secret my-cluster-name-secrets -o yaml > my-cluster-secrets.yml kubectl get secret main-cluster-ssl -o yaml > main-cluster-ssl.yml kubectl get secret main-cluster-ssl-internal -o yaml > main-cluster-ssl-internal.yml kubectl get secret my-cluster-name-mongodb-encryption-key -o yaml > my-cluster-name-mongodb-encryption-key.yml |
Step 5: Modify secrets and apply to the GKE replica cluster
You need to remove the fields:
|
1 2 3 4 5 6 |
ownerReferences annotations creationTimestamp resourceVersion selfLink uid |
The following helper scripts can be used:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid)' my-cluster-secrets.yml > my-cluster-secrets-replica.yaml sed -i '' 's/main-cluster/replica-cluster/g' my-cluster-secrets-replica.yaml yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid)' main-cluster-ssl.yml > replica-cluster-ssl.yml sed -i '' 's/main-cluster/replica-cluster/g' replica-cluster-ssl.yml yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid)' main-cluster-ssl-internal.yml > replica-cluster-ssl-internal.yml sed -i '' 's/main-cluster/replica-cluster/g' replica-cluster-ssl-internal.yml yq eval 'del(.metadata.ownerReferences, .metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.selfLink, .metadata.uid)' my-cluster-name-mongodb-encryption-key.yml > my-cluster-name-mongodb-encryption-key2.yml sed -i '' 's/main-cluster/replica-cluster/g' my-cluster-name-mongodb-encryption-key2.yml |
Now create the modified secrets on the GKE replica cluster:
|
1 2 3 4 |
kubectl apply -f my-cluster-secrets-replica.yaml kubectl apply -f replica-cluster-ssl.yml kubectl apply -f replica-cluster-ssl-internal.yml kubectl apply -f my-cluster-name-mongodb-encryption-key2.yml |
Step 6: Create the replica PSMDB cluster
Make sure to create it in the psmdb namespace and remember to set “unmanaged: true” and updateStrategy to RollingUpdate or OnDelete.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: replica-cluster spec: unmanaged: true crVersion: 1.20.1 image: percona/percona-server-mongodb:7.0.14-8-multi multiCluster: enabled: true DNSSuffix: svc.clusterset.local updateStrategy: RollingUpdate upgradeOptions: apply: disabled schedule: "0 2 * * *" secrets: users: my-cluster-name-secrets encryptionKey: my-cluster-name-mongodb-encryption-key ssl: replica-cluster-ssl sslInternal: replica-cluster-ssl-internal replsets: - name: rs0 size: 3 expose: enabled: true type: ClusterIP volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 3 expose: enabled: true type: ClusterIP volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 1 expose: type: ClusterIP |
Step 7: Get the names of the services
Run this step on both clusters:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gke-mcs-51sqttasr2 ClusterIP 34.118.225.98 27017/TCP 8m1s gke-mcs-5ujp4u9efu ClusterIP 34.118.236.255 27017/TCP 2m39s gke-mcs-87ngrkrek2 ClusterIP 34.118.236.74 27017/TCP 10m gke-mcs-8gvu7dss6j ClusterIP 34.118.233.222 27017/TCP 2m41s gke-mcs-covcmpalsn ClusterIP 34.118.235.127 27017/TCP 10m gke-mcs-f9h00il386 ClusterIP 34.118.238.35 27017/TCP 8m7s gke-mcs-ksa245qkbn ClusterIP 34.118.231.205 27017/TCP 8m6s gke-mcs-mhng4qml79 ClusterIP 34.118.231.157 27017/TCP 8m3s gke-mcs-ukshfk963v ClusterIP 34.118.234.152 27017/TCP 10m gke-mcs-v2opr5f985 ClusterIP 34.118.233.131 27017/TCP 2m44s main-cluster-cfg ClusterIP None 27017/TCP 15m main-cluster-cfg-0 ClusterIP 34.118.230.116 27017/TCP 15m main-cluster-cfg-1 ClusterIP 34.118.239.70 27017/TCP 14m main-cluster-cfg-2 ClusterIP 34.118.233.221 27017/TCP 14m main-cluster-mongos ClusterIP 34.118.239.128 27017/TCP 15m main-cluster-rs0 ClusterIP None 27017/TCP 15m main-cluster-rs0-0 ClusterIP 34.118.229.138 27017/TCP 15m main-cluster-rs0-1 ClusterIP 34.118.225.147 27017/TCP 14m main-cluster-rs0-2 ClusterIP 34.118.237.74 27017/TCP 14m |
If ServiceImports are missing, check the MCS controller logs and ensure ServiceExports were created by the operator. Replace the pod name below with your own operator pod:
|
1 2 |
<span style="font-weight: 400;">kubectl get serviceimports</span> <span style="font-weight: 400;">kubectl logs pod/percona-server-mongodb-operator-64976cdb47-2zdqj</span> |
Step 8: Add the replica nodes to the Main Cluster
Using the service names we got from the previous step, add each node from the replica side to the main cluster. Perform this for every shard and the config server replica set.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: main-cluster spec: crVersion: 1.20.1 image: percona/percona-server-mongodb:7.0.14-8-multi updateStrategy: SmartUpdate multiCluster: enabled: true DNSSuffix: svc.clusterset.local upgradeOptions: apply: disabled schedule: "0 2 * * *" secrets: users: my-cluster-name-secrets encryptionKey: my-cluster-name-mongodb-encryption-key replsets: - name: rs0 size: 3 externalNodes: - host: replica-cluster-rs0-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: replica-cluster-rs0-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: replica-cluster-rs0-2.psmdb.svc.clusterset.local votes: 0 priority: 0 expose: enabled: true type: ClusterIP volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 3 externalNodes: - host: replica-cluster-cfg0-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: replica-cluster-cfg0-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: replica-cluster-cfg0-1.psmdb.svc.clusterset.local votes: 0 priority: 0 expose: enabled: true type: LoadBalancer volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 3 expose: type: ClusterIP |
|
1 |
kubectl apply -f cr-main-after.yml |
Step 9: Add the main nodes to the Replica Cluster
Similarly to the previous step, edit the yaml file on the replica side, and add the main nodes as external:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
replsets: - name: rs0 size: 3 externalNodes: - host: main-cluster-rs0-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: main-cluster-rs0-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: main-cluster-rs0-2.psmdb.svc.clusterset.local votes: 0 priority: 0 sharding: configsvrReplSet: externalNodes: - host: main-cluster-cfg-0.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: main-cluster-cfg-1.psmdb.svc.clusterset.local votes: 1 priority: 1 - host: main-cluster-cfg-2.psmdb.svc.clusterset.local votes: 0 priority: 0 |
Step 10: Verify the configuration
Connect to a member of each replica set. I am using the config servers in the example:
|
1 2 |
kubectl exec -it main-cluster-cfg-0 -- /bin/bash mongosh admin -u clusterAdmin -p **** |
Verify all members are present (I’ve removed some fields from the output for readability)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
rs.status().members [ { _id: 0, name: 'main-cluster-cfg-0.psmdb.svc.clusterset.local:27017', health: 1, state: 1, stateStr: 'PRIMARY', }, { _id: 1, name: 'main-cluster-cfg-1.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }, { _id: 2, name: 'main-cluster-cfg-2.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }, { _id: 3, name: 'replica-cluster-cfg-0.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }, { _id: 4, name: 'replica-cluster-cfg-1.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }, { _id: 5, name: 'replica-cluster-cfg-2.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }] |
Step 11: Test the switchover process
Set the main cluster to unmanaged by editing the yaml file and applying the changes:
|
1 |
vi cr-main.yml |
|
1 2 3 |
spec: unmanaged: true updateStrategy: RollingUpdate |
|
1 |
kubectl exec -f cr-main.yml |
Now set the replica cluster to managed, so the operator assumes control of the nodes on the replica side:
|
1 |
vi cr-replica.yml |
|
1 2 |
spec: unmanaged: false |
|
1 |
kubectl exec -f cr-replica.yml |
Verify a new primary was elected in the replica side
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
rs.status().members [ { _id: 0, name: 'main-cluster-cfg-0.psmdb.svc.clusterset.local:27017', health: 1, state: 2, stateStr: 'SECONDARY', }, ... }, { _id: 3, name: 'replica-cluster-cfg-0.psmdb.svc.clusterset.local:27017', health: 1, state: 1, stateStr: 'PRIMARY', }, ... ] |
Final thoughts
By combining the power of Percona Operator for MongoDB with GKE’s Multi-Cluster Services, you gain a resilient, scalable, and multi-region replica set architecture. Perfect for high-availability applications and disaster recovery use cases.
About the Author
Ivan GroenewoldPassionate about technology, Ivan Groenewold is a seasoned professional with extensive experience in database management, cloud infrastructure, and software development. With a focus on optimizing performance and scalability, Ivan excels in designing and implementing solutions for complex systems, particularly in MongoDB and cloud-native environments. Known for problem-solving and a results-driven mindset, Ivan combines technical expertise with a commitment to continuous learning, delivering high-quality solutions for modern enterprise applications.
Subscribe
0 Comments
Oldest
