As a DBA, it is important to monitor a database to help us troubleshoot or to understand the health of an instance. Percona Monitoring and Management (PMM v2) is open-source and does a great job in monitoring the databases like MongoDB, MySQL, PostgreSQL, etc.
In this blog post, we will see how to configure a sharded cluster in PMM2. This is a part two version of the previous one which was done with PMM v1, titled Configuring PMM Monitoring for MongoDB Cluster. I have listed the steps to configure the sharded cluster into PMM2 below:
Before configuring with PMM2, we will need to create a USER for monitoring from the database side. If you need to enable QAN (query analytics), then you will need to enable profiler and some more custom permission like “explainRole” to the user as well. Adding profiler adds up some more little load to the database, so it is better you do prior tests to analyze the load if you want to assess the extra load.
|
1 |
// Change role name / user / password as required<br><br>db.getSiblingDB("admin").createRole({<br> role: "explainRole",<br> privileges: [{<br> resource: {<br> db: "",<br> collection: ""<br> },<br> actions: [<br> "listIndexes",<br> "listCollections",<br> "dbStats",<br> "dbHash",<br> "collStats",<br> "find"<br> ]<br> }],<br> roles:[]<br>})<br><br><br>db.getSiblingDB("admin").createUser({<br> user: "pmm_mongodb",<br> pwd: "password",<br> roles: [<br> { role: "explainRole", db: "admin" },<br> { role: "clusterMonitor", db: "admin" },<br> { role: "read", db: "local" }<br> ]<br>}) |
This is optional. Run the instance with the profiler or add profiling at the database level to monitor queries in QAN (not applicable for mongos).
To start at the instance level (enables profiling for all databases):
|
1 |
mongod <other options> --profile 2 --slowms 200 --rateLimit 100 |
or in mongod.conf:
|
1 |
operationProfiling:<br> mode: all<br> slowOpThresholdMs: 200<br># (Below variable is available only with Percona Server for MongoDB.)<br> rateLimit: 100 |
To enable p[rofiling at DB level:
|
1 |
use dbname<br>db.setProfilingLevel(2) |
Here use the same –cluster option name for all members from the same cluster and provide service-name to identify it:
|
1 |
sudo pmm-admin add mongodb <br>--username=pmm_mongodb --password=password <br>--query-source=profiler <br>--cluster=mycluster <br>--service-name=myc_mongoc2 <br>--host=127.0.0.1 --port=37061 |
Then check whether the service was added successfully or not:
|
1 |
$ sudo pmm-admin list<br>Service type Service name Address and port Service ID<br>MongoDB myc_mongoc2 127.0.0.1:37061 /service_id/02e261a1-e8e0-4eb4-8043-8616424500de<br><br>Agent type Status Metrics Mode Agent ID Service ID<br>pmm_agent Connected /agent_id/281b4046-4f4b-4897-bd2e-b771d3e97922 <br>node_exporter Running push /agent_id/5e9b17a8-ecb9-47c3-8477-ce322047c4d9 <br>mongodb_exporter Running push /agent_id/0067dd85-9a0a-47dd-976e-ae779deb982b /service_id/5c92f132-3005-45ab-84df-7541c286c34a<br>mongodb_profiler_agent Running /agent_id/18d3d87a-9bb9-48c1-8e3e-d8bae3f043bb /service_id/02e261a1-e8e0-4eb4-8043-8616424500de |
I used localhost to deploy the sharded cluster for the testing purpose as below:
Members list:
|
1 |
1 mongos (37050), <br>3 shards consist of 3 member replicaSet each (37051-37059), <br>3 config members(37060-37062) |
Listing one mongod instance from the ps command:
|
1 |
balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ ps -ef | grep mongod -w | head -1<br>balaguru 41883 2846 1 13:01 ? 00:04:04 mongod --replSet configRepl --dbpath /home/balaguru/mongodb/testshard/data/configRepl/rs1/db --logpath /home/balaguru/mongodb/testshard/data/configRepl/rs1/mongod.log --port 37060 --fork --configsvr --wiredTigerCacheSizeGB 1 --profile 2 --slowms 200 --rateLimit 100 --logappend |
Adding mongodb services to pmm-admin:
|
1 |
balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s11 --host=127.0.0.1 --port=37051<br>MongoDB Service added.<br>Service ID : /service_id/cc6b3fed-ee16-494e-93f0-0d2e8f60a136<br>Service name: myc_s11--host=127.0.0.1<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s12 --host=127.0.0.1 --port=37052<br>MongoDB Service added.<br>Service ID : /service_id/235494d8-aaee-4ca0-bd3a-bf2259e87ecc<br>Service name: myc_s12<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s13 --host=127.0.0.1 --port=37053<br>MongoDB Service added.<br>Service ID : /service_id/55261675-41e7-40f1-95c9-08cac25c4f64<br>Service name: myc_s13<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s21 --host=127.0.0.1 --port=37054<br>MongoDB Service added.<br>Service ID : /service_id/5c92f132-3005-45ab-84df-7541c286c34a<br>Service name: myc_s21<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s22 --host=127.0.0.1 --port=37055<br>MongoDB Service added.<br>Service ID : /service_id/4de07a5b-5a47-4126-8824-80570bd72cef<br>Service name: myc_s22--host=127.0.0.1<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s23 --host=127.0.0.1 --port=37056<br>MongoDB Service added.<br>Service ID : /service_id/7bdaaa72-6e00-4f46-a2a9-5205d5f3fff5<br>Service name: myc_s23<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s31 --host=127.0.0.1 --port=37057<br>MongoDB Service added.<br>Service ID : /service_id/2028e075-bc65-4aae-bcdd-ec616b36e81b<br>Service name: myc_s31<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s32 --host=127.0.0.1 --port=37058<br>MongoDB Service added.<br>Service ID : /service_id/7659231c-f48f-4a65-b651-585ac1f058cd<br>Service name: myc_s32<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_s33 --host=127.0.0.1 --port=37059<br>MongoDB Service added.<br>Service ID : /service_id/2c224eaf-c0f1-482b-b23c-8ea4b914c8e5<br>Service name: myc_s33<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_mongoc1 --host=127.0.0.1 --port=37060<br>MongoDB Service added.<br>Service ID : /service_id/09e95cc5-40b7-4a53-9e35-2937ca23395f<br>Service name: myc_mongoc1<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_mongoc2 --host=127.0.0.1 --port=37061<br>MongoDB Service added.<br>Service ID : /service_id/02e261a1-e8e0-4eb4-8043-8616424500de<br>Service name: myc_mongoc2<br><br>balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin add mongodb --username=pmm_mongodb --password=password <br>--query-source=profiler --cluster=mycluster --service-name=myc_mongoc3 --host=127.0.0.1 --port=37062<br>MongoDB Service added.<br>Service ID : /service_id/421449d9-8ada-46dd-9c8a-84c0847a8742<br>Service name: myc_mongoc3 |
Listing the services added:
|
1 |
balaguru@vinodh-UbuntuPC:~/mongodb/testshard$ pmm-admin list<br>Service type Service name Address and port Service ID<br>MongoDB myc_mongoc2 127.0.0.1:37061 /service_id/02e261a1-e8e0-4eb4-8043-8616424500de<br>MongoDB myc_mongoc1 127.0.0.1:37060 /service_id/09e95cc5-40b7-4a53-9e35-2937ca23395f<br>MongoDB myc_s31 127.0.0.1:37057 /service_id/2028e075-bc65-4aae-bcdd-ec616b36e81b<br>MongoDB myc_s12 127.0.0.1:37052 /service_id/235494d8-aaee-4ca0-bd3a-bf2259e87ecc<br>MongoDB myc_s33 127.0.0.1:37059 /service_id/2c224eaf-c0f1-482b-b23c-8ea4b914c8e5<br>MongoDB myc_mongos 127.0.0.1:37050 /service_id/3f4f56be-6259-4579-88b7-bb4d0c29204b<br>MongoDB myc_mongoc3 127.0.0.1:37062 /service_id/421449d9-8ada-46dd-9c8a-84c0847a8742<br>MongoDB myc_s22 127.0.0.1:37055 /service_id/4de07a5b-5a47-4126-8824-80570bd72cef<br>MongoDB myc_s13 127.0.0.1:37053 /service_id/55261675-41e7-40f1-95c9-08cac25c4f64<br>MongoDB myc_s21 127.0.0.1:37054 /service_id/5c92f132-3005-45ab-84df-7541c286c34a<br>MongoDB myc_s32 127.0.0.1:37058 /service_id/7659231c-f48f-4a65-b651-585ac1f058cd<br>MongoDB myc_s23 127.0.0.1:37056 /service_id/7bdaaa72-6e00-4f46-a2a9-5205d5f3fff5<br>MongoDB myc_s11 127.0.0.1:37051 /service_id/cc6b3fed-ee16-494e-93f0-0d2e8f60a136<br><br>Agent type Status Metrics Mode Agent ID Service ID<br>pmm_agent Connected /agent_id/281b4046-4f4b-4897-bd2e-b771d3e97922 <br>node_exporter Running push /agent_id/5e9b17a8-ecb9-47c3-8477-ce322047c4d9 <br>mongodb_exporter Running push /agent_id/0067dd85-9a0a-47dd-976e-ae779deb982b /service_id/5c92f132-3005-45ab-84df-7541c286c34a <br>mongodb_exporter Running push /agent_id/071ec1ae-ff35-4fa1-a4c9-4d5bca705131 /service_id/09e95cc5-40b7-4a53-9e35-2937ca23395f <br>mongodb_exporter Running push /agent_id/5e045290-36c2-410b-86e9-b4945cd7ecfb /service_id/3f4f56be-6259-4579-88b7-bb4d0c29204b <br>mongodb_exporter Running push /agent_id/6331b519-da6e-47c0-be7e-92f2ac142fa5 /service_id/2c224eaf-c0f1-482b-b23c-8ea4b914c8e5 <br>mongodb_exporter Running push /agent_id/6ce78e1c-be6a-4ffd-844b-8afdc0ee5700 /service_id/235494d8-aaee-4ca0-bd3a-bf2259e87ecc <br>mongodb_exporter Running push /agent_id/6ed1bcc2-3561-4c65-95e1-11b3cc051194 /service_id/cc6b3fed-ee16-494e-93f0-0d2e8f60a136 <br>mongodb_exporter Running push /agent_id/7721bd24-7408-431d-abcb-3239459df75a /service_id/7659231c-f48f-4a65-b651-585ac1f058cd <br>mongodb_exporter Running push /agent_id/999c0152-656e-4941-a1fb-003df2dbfbf6 /service_id/2028e075-bc65-4aae-bcdd-ec616b36e81b <br>mongodb_exporter Running push /agent_id/9e63f2d9-7e75-45ee-927d-b1406d4797e0 /service_id/55261675-41e7-40f1-95c9-08cac25c4f64 <br>mongodb_exporter Running push /agent_id/ca3ab511-29eb-4c68-b037-23ab13fa92ff /service_id/4de07a5b-5a47-4126-8824-80570bd72cef <br>mongodb_exporter Running push /agent_id/cd1066eb-f917-4d7e-b284-8d8a8bc7c652 /service_id/7bdaaa72-6e00-4f46-a2a9-5205d5f3fff5 <br>mongodb_exporter Running push /agent_id/e2ef230a-d84b-428c-921b-b6da7c3180f3 /service_id/421449d9-8ada-46dd-9c8a-84c0847a8742 <br>mongodb_exporter Running push /agent_id/e3f7ba25-6592-4cb4-aae6-7431b3b6a6da /service_id/02e261a1-e8e0-4eb4-8043-8616424500de <br>mongodb_profiler_agent Running /agent_id/18d3d87a-9bb9-48c1-8e3e-d8bae3f043bb /service_id/02e261a1-e8e0-4eb4-8043-8616424500de <br>mongodb_profiler_agent Running /agent_id/1cf5ee8a-b5b5-4133-896c-fafccc164f54 /service_id/5c92f132-3005-45ab-84df-7541c286c34a <br>mongodb_profiler_agent Running /agent_id/4b13cc24-fbd2-47cc-955d-c2a65624d2be /service_id/55261675-41e7-40f1-95c9-08cac25c4f64 <br>mongodb_profiler_agent Running /agent_id/4de795cf-f047-49e6-a3bc-dc2ab1b2bc86 /service_id/cc6b3fed-ee16-494e-93f0-0d2e8f60a136 <br>mongodb_profiler_agent Running /agent_id/89ae83c7-e62c-48f6-9e8c-597ce978c8ce /service_id/4de07a5b-5a47-4126-8824-80570bd72cef <br>mongodb_profiler_agent Running /agent_id/98343388-a246-4767-8838-ded8f8de5191 /service_id/235494d8-aaee-4ca0-bd3a-bf2259e87ecc <br>mongodb_profiler_agent Running /agent_id/a5df9e6b-037e-486a-bc95-afe20095cf98 /service_id/7bdaaa72-6e00-4f46-a2a9-5205d5f3fff5 <br>mongodb_profiler_agent Running /agent_id/a6bda9b4-989a-427b-ae64-5deffc2b9ba2 /service_id/7659231c-f48f-4a65-b651-585ac1f058cd <br>mongodb_profiler_agent Running /agent_id/c59c40ca-63ee-4497-b297-403faa9d4ec0 /service_id/2c224eaf-c0f1-482b-b23c-8ea4b914c8e5 <br>mongodb_profiler_agent Running /agent_id/c7f84a08-4823-455b-93a3-168eee19329b /service_id/3f4f56be-6259-4579-88b7-bb4d0c29204b <br>mongodb_profiler_agent Running /agent_id/e85d0757-7542-4b38-bfed-81ded8bf309c /service_id/421449d9-8ada-46dd-9c8a-84c0847a8742 <br>mongodb_profiler_agent Running /agent_id/ed81849a-6fc9-46f3-a5dc-e6c288409009 /service_id/09e95cc5-40b7-4a53-9e35-2937ca23395f <br>mongodb_profiler_agent Running /agent_id/f9d26161-4827-4bed-a85f-cbe3ce9478ab /service_id/2028e075-bc65-4aae-bcdd-ec616b36e81b <br>vmagent Running push /agent_id/a662e1f6-31d3-4514-8f83-ea31e0165d61 |
From PMM Dashboards, you can then view the replSet summary as well as the sharded cluster summary.
This dashboard gives information about the sharded/unsharded databases, shards, chunks, cursor details, etc.

This dashboard tells about the replication information like replica lag, operations, heartbeat, ping time, etc.

This is the general dashboard for a MongoDB instance which provides generic information about the connections, memory usage, latency, etc

This is the main dashboard that you’ll need most to analyze the problems here as it shows the wiredTiger information. The main metrics that you need to monitor here are the WT cache utilization, evictions of modified or unmodified pages, write/read tickets utilization, index/objects scans, etc.

If you enable the profiling, then you could see the queries used in the database here. You can filter them easily as shown in the screenshot below. Also, you can get the explain plan to check whether they utilize the COLLSCAN (disk reads) or IXSCAN (uses index). Also, you can check the counts, load, etc.

As said, Percona Monitoring and Management 2 is very easy to configure to monitor the databases and it is recommended too. It’s better now rather than late to configure the monitoring. PMM2 is managed by Percona which is totally free and you can raise any bugs here – https://jira.percona.com/. If you have doubts, you can leave your questions here – https://forums.percona.com.
Complete the 2021 Percona Open Source Data Management Software Survey
Resources
RELATED POSTS