Before Percona Server for MongoDB 4.4 (PSMDB), the best practice to create an index was doing it in a rolling manner. Many folks used to create directly on Primary, resulting in the first index being created successfully on Primary and then replicated to Secondary nodes.
Starting from PSMDB 4.4, there was a new parameter commitQuorum introduced in the createIndex command. If you are not passing this parameter explicitly with the createIndex command, it will use the default settings on a replica set or sharded cluster and start building the index simultaneously across all data-bearing voting replica set members.
Below is the command used to create an index using commitQuorum as the majority:
|
1 |
db.getSiblingDB("acme").products.createIndex({ "airt" : 1 }, { }, "<strong>majority</strong>") |
The above command will run the index create command on the majority of data-bearing replica set members. There are other options available too when using commitQuorum:
Now we will see the scenarios of what happens when the index is created with the default and majority commitQuorum.
|
1 |
rs1:PRIMARY> db.products.createIndex({ "airt" : 1 }) |
|
1 |
{"t":{"$date":"2023-06-26T12:33:18.417+00:00"},"s":"I", "c":"INDEX", "id":20384, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Index build: starting","attr":{"namespace":"acme.products","buildUUID":{"uuid":{"$uuid":"58f4e7bf-7b8f-4eb6-8de0-0ad774c4b51f"}},"properties":{"v":2,"key":{"airt":1.0},"name":"airt_1"},"method":"Hybrid","maxTemporaryMemoryUsageMB":200}} |
|
1 |
{"t":{"$date":"2023-06-26T12:33:18.417+00:00"},"s":"I", "c":"INDEX", "id":20384, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Index build: starting","attr":{"namespace":"acme.products","buildUUID":{"uuid":{"$uuid":"58f4e7bf-7b8f-4eb6-8de0-0ad774c4b51f"}},"properties":{"v":2,"key":{"airt":1.0},"name":"airt_1"},"method":"Hybrid","maxTemporaryMemoryUsageMB":200}}} |
|
1 |
{"t":{"$date":"2023-06-26T12:33:28.445+00:00"},"s":"I", "c":"INDEX", "id":20384, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Index build: starting","attr":{"namespace":"acme.products","buildUUID":{"uuid":{"$uuid":"58f4e7bf-7b8f-4eb6-8de0-0ad774c4b51f"}},"properties":{"v":2,"key":{"airt":1.0},"name":"airt_1"},"method":"Hybrid","maxTemporaryMemoryUsageMB":200}} |
|
1 |
rs1:PRIMARY> rs.status().members.forEach(function (d) {print(d.name) + " " + print(d.stateStr)});<br>127.0.0.1:27017<br>PRIMARY<br>localhost:27018<br>SECONDARY<br>localhost:27019<br>(not reachable/healthy)<br>rs1:PRIMARY> |
|
1 |
rs1:PRIMARY> db.products.createIndex({ "airt" : 1 }) |
|
1 |
rs1:PRIMARY> db.printSecondaryReplicationInfo()<br>source: localhost:27018<br> syncedTo: Mon Jun 26 2023 17:56:30 GMT+0000 (UTC)<br> 0 secs (0 hrs) behind the primary<br>source: localhost:27019<br> syncedTo: Thu Jan 01 1970 00:00:00 GMT+0000 (UTC)<br> 1687802190 secs (468833.94 hrs) behind the primary<br>rs1:PRIMARY> |
|
1 |
rs1:PRIMARY> db.currentOp(true).inprog.forEach(function(op){ if(op.msg!==undefined) print(op.msg) })<br>Index Build: draining writes received during build<br>rs1:PRIMARY> Date()<br>Mon Jun 26 2023 18:07:26 GMT+0000 (UTC)<br>rs1:PRIMARY> |
|
1 |
"active" : true,<br> "currentOpTime" :"2023-06-26T19:04:33.175+00:00",<br> "opid" : 329147,<br> "lsid" : {<br> "id" :UUID("dd9672f8-4f56-47ce-8ceb-31caf5e8baf8"),<br> "uid": BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=")<br> },<br> "secs_running" : NumberLong(4214),<br> "microsecs_running" : NumberLong("4214151233"),<br> "op" : "command",<br> "ns" : "acme.products",<br> "command" : {<br> "createIndexes" : "products",<br> "indexes" : [<br> {<br> "key" : {<br> "airt" : 1<br> },<br> "name" : "airt_1"<br> }<br> ],<br> "lsid" : {<br> "id" :UUID("dd9672f8-4f56-47ce-8ceb-31caf5e8baf8")<br> },<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687801980, 1),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> },<br> "$db" : "acme"<br> } |
|
1 |
{"t":{"$date":"2023-06-26T17:54:21.419+00:00"},"s":"I", "c":"STORAGE", "id":3856203, "ctx":"IndexBuildsCoordinatorMongod-1","msg":"Index build: waiting for next action before completing final phase","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}}}} |
|
1 |
{"t":{"$date":"2023-06-26T17:54:21.424+00:00"},"s":"I", "c":"STORAGE", "id":3856203, "ctx":"IndexBuildsCoordinatorMongod-1","msg":"Index build: waiting for next action before completing final phase","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}}}} |
|
1 |
rs1:PRIMARY> db.products.getIndexes()<br>[<br> {<br> "v" : 2,<br> "key" : {<br> "_id" : 1<br> },<br> "name" : "_id_"<br> },<br> {<br> "v" : 2,<br> "key" : {<br> "airt" : 1<br> },<br> "name" : "airt_1"<br> }<br>]<br>rs1:PRIMARY> |
Below is the explain plan of a query, where we can see the query is doing COLLSCAN instead of IXSCAN:
|
1 |
rs1:PRIMARY> db.products.find({"airt" : 1.9869362536440427}).explain()<br>{<br> "queryPlanner" : {<br> "plannerVersion" : 1,<br> "namespace" : "acme.products",<br> "indexFilterSet" : false,<br> "parsedQuery" : {<br> "airt" : {<br> "$eq" : 1.9869362536440427<br> }<br> },<br> "queryHash" : "65E2F79D",<br> "planCacheKey" : "AA490985",<br> "winningPlan" : {<br> "stage" : "COLLSCAN",<br> "filter" : {<br> "airt" : {<br> "$eq" : 1.9869362536440427<br> }<br> },<br> "direction" : "forward"<br> },<br> "rejectedPlans" : [ ]<br> },<br> "serverInfo" : {<br> "host" : "ip-172-31-82-235.ec2.internal",<br> "port" : 27017,<br> "version" : "4.4.22-21",<br> "gitVersion" : "be7a5f4a1000bed8cf1d1feb80a20664d51503ce"<br>} |
Now I will bring up the third node, and we will see that index op will complete.
Index status:
|
1 |
rs1:PRIMARY> db.products.createIndex({ "airt" : 1 })<br>{<br> "createdCollectionAutomatically" : false,<br> "numIndexesBefore" : 1,<br> "numIndexesAfter" : 2,<br> "commitQuorum" : "votingMembers",<br> "ok" : 1,<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687806737, 3),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> },<br> "operationTime" : Timestamp(1687806737, 3)<br>}<br>rs1:PRIMARY> |
Now will run the same query, and we can see index (IXSCAN) is getting used as the index was created successfully above:
|
1 |
rs1:PRIMARY> db.products.find({"airt" : 1.9869362536440427}).explain()<br>{<br> "queryPlanner" : {<br> "plannerVersion" : 1,<br> "namespace" : "acme.products",<br> "indexFilterSet" : false,<br> "parsedQuery" : {<br> "airt" : {<br> "$eq" : 1.9869362536440427<br> }<br> },<br> "queryHash" : "65E2F79D",<br> "planCacheKey" : "AA490985",<br> "winningPlan" : {<br> "stage" : "FETCH",<br> "inputStage" : {<br> "stage" : "IXSCAN",<br> "keyPattern" : {<br> "airt" : 1<br> },<br> "indexName" : "airt_1",<br> "isMultiKey" : false,<br> "multiKeyPaths" : {<br> "airt" : [ ]<br> },<br> "isUnique" : false,<br> "isSparse" : false,<br> "isPartial" : false,<br> "indexVersion" : 2,<br> "direction" : "forward",<br> "indexBounds" : {<br> "airt" : [<br> "[1.986936253644043, 1.986936253644043]"<br> ]<br> }<br> }<br> },<br> "rejectedPlans" : [ ]<br> },<br> "serverInfo" : {<br> "host" : "ip-172-31-82-235.ec2.internal",<br> "port" : 27017,<br> "version" : "4.4.22-21",<br> "gitVersion" : "be7a5f4a1000bed8cf1d1feb80a20664d51503ce"<br> } |
Primary logs once the third node came up and the index was created successfully:
|
1 |
{"t":{"$date":"2023-06-26T19:12:17.450+00:00"},"s":"I", "c":"STORAGE", "id":3856201, "ctx":"conn40","msg":"Index build: commit quorum satisfied","attr":{"indexBuildEntry":{"_id":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"},"collectionUUID":{"$uuid":"a963b7e7-1054-4a5f-a935-a5be8995cff0"},<strong>"commitQuorum":"votingMembers"</strong>,"indexNames":["airt_1"],"commitReadyMembers":["127.0.0.1:27017","localhost:27018","localhost:27019"]}}}<br><br>{"t":{"$date":"2023-06-26T19:12:17.450+00:00"},"s":"I", "c":"STORAGE", "id":3856204, "ctx":"IndexBuildsCoordinatorMongod-1","msg":"Index build: received signal","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}},<strong>"action":"Commit quorum Satisfied"</strong>}}<br><br>{"t":{"$date":"2023-06-26T19:12:17.451+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"IndexBuildsCoordinatorMongod-1","msg":"Index build: done building","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}},"namespace":"acme.products","index":"airt_1","commitTimestamp":{"$timestamp":{"t":1687806737,"i":2}}}}<br><br>{"t":{"$date":"2023-06-26T19:12:17.452+00:00"},"s":"I", "c":"STORAGE", "id":20663, "ctx":"IndexBuildsCoordinatorMongod-1","msg":"Index build: completed successfully","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}},"namespace":"acme.products","uuid":{"uuid":{"$uuid":"a963b7e7-1054-4a5f-a935-a5be8995cff0"}},"indexesBuilt":1,"numIndexesBefore":1,"numIndexesAfter":2}}<br><br>{"t":{"$date":"2023-06-26T19:12:17.554+00:00"},"s":"I", "c":"INDEX", "id":20447, "ctx":"conn34","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"46451b37-141f-4312-a219-4b504736ab5b"}}}}<br><br>{"t":{"$date":"2023-06-26T19:12:17.554+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn34","msg":"Slow query","attr":{"type":"command","ns":"acme.products","appName":"MongoDB Shell","command":{"createIndexes":"products","indexes":[{"key":{"airt":1.0},"name":"airt_1"}],"lsid":{"id":{"$uuid":"dd9672f8-4f56-47ce-8ceb-31caf5e8baf8"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1687801980,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"$db":"acme"},"numYields":0,"reslen":271,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":3}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":4}},"ReplicationStateTransition":{"acquireCount":{"w":5}},"Global":{"acquireCount":{"r":1,"w":4}},"Database":{"acquireCount":{"w":3}},"Collection":{"acquireCount":{"r":1,"w":1,"W":1}},"Mutex":{"acquireCount":{"r":3}}},"flowControl":{"acquireCount":3,"timeAcquiringMicros":7},"storage":{"data":{"bytesRead":98257,"timeReadingMicros":3489}},"protocol":"op_msg",<strong>"durationMillis":4678530</strong>}} |
Above, you can see how much time it took to complete the index build; the op was running till the third node was down.
|
1 |
rs1:PRIMARY> rs.status().members.forEach(function (d) {print(d.name) + " " + print(d.stateStr)});<br>127.0.0.1:27017<br>PRIMARY<br>localhost:27018<br>SECONDARY<br>localhost:27019<br>(not reachable/healthy)<br>rs1:PRIMARY> |
|
1 |
rs1:PRIMARY> db.products.createIndex({ "airt" : 1 }, { }, "majority")<br>{<br> "createdCollectionAutomatically" : false,<br> "numIndexesBefore" : 1,<br> "numIndexesAfter" : 2,<br> "commitQuorum" : "majority",<br> "ok" : 1,<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687808148, 4),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> },<br> "operationTime" : Timestamp(1687808148, 4)<br>}<br>rs1:PRIMARY> |
|
1 |
{"t":{"$date":"2023-06-26T19:35:48.821+00:00"},"s":"I", "c":"STORAGE", "id":3856201, "ctx":"conn7","msg":"Index build: commit quorum satisfied","attr":{"indexBuildEntry":{"_id":{"$uuid":"5f8f75ee-aa46-42a6-b4c2-59a68fea47a7"},"collectionUUID":{"$uuid":"a963b7e7-1054-4a5f-a935-a5be8995cff0"},<strong>"commitQuorum":"majority"</strong>,"indexNames":["airt_1"],"commitReadyMembers":["127.0.0.1:27017","localhost:27018"]}}}<br><br>{"t":{"$date":"2023-06-26T19:35:48.821+00:00"},"s":"I", "c":"STORAGE", "id":3856204, "ctx":"IndexBuildsCoordinatorMongod-3","msg":"Index build: received signal","attr":{"buildUUID":{"uuid":{"$uuid":"5f8f75ee-aa46-42a6-b4c2-59a68fea47a7"}},<strong>"action":"Commit quorum Satisfied"</strong>}}<br><br>{"t":{"$date":"2023-06-26T19:35:48.822+00:00"},"s":"I", "c":"INDEX", "id":20345, "ctx":"IndexBuildsCoordinatorMongod-3","msg":"Index build: done building","attr":{"buildUUID":{"uuid":{"$uuid":"5f8f75ee-aa46-42a6-b4c2-59a68fea47a7"}},"namespace":"acme.products","index":"airt_1","commitTimestamp":{"$timestamp":{"t":1687808148,"i":3}}}}<br><br>{"t":{"$date":"2023-06-26T19:35:48.824+00:00"},"s":"I", "c":"STORAGE", "id":20663, "ctx":"IndexBuildsCoordinatorMongod-3","msg":"Index build: completed successfully","attr":{"buildUUID":{"uuid":{"$uuid":"5f8f75ee-aa46-42a6-b4c2-59a68fea47a7"}},"namespace":"acme.products","uuid":{"uuid":{"$uuid":"a963b7e7-1054-4a5f-a935-a5be8995cff0"}},"indexesBuilt":1,"numIndexesBefore":1,"numIndexesAfter":2}}<br><br>{"t":{"$date":"2023-06-26T19:35:48.923+00:00"},"s":"I", "c":"INDEX", "id":20447, "ctx":"conn34","msg":"Index build: completed","attr":{"buildUUID":{"uuid":{"$uuid":"5f8f75ee-aa46-42a6-b4c2-59a68fea47a7"}}}}<br><br>{"t":{"$date":"2023-06-26T19:35:48.923+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn34","msg":"Slow query","attr":{"type":"command","ns":"acme.products","appName":"MongoDB Shell","command":{"createIndexes":"products","indexes":[{"key":{"airt":1.0},"name":"airt_1"}],"commitQuorum":"majority","lsid":{"id":{"$uuid":"dd9672f8-4f56-47ce-8ceb-31caf5e8baf8"}},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1687808123,"i":1}},"signature":{"hash":{"$binary":{"base64":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","subType":"0"}},"keyId":0}},"$db":"acme"},"numYields":0,"reslen":266,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":3}},"FeatureCompatibilityVersion":{"acquireCount":{"r":1,"w":4}},"ReplicationStateTransition":{"acquireCount":{"w":5}},"Global":{"acquireCount":{"r":1,"w":4}},"Database":{"acquireCount":{"w":3}},"Collection":{"acquireCount":{"r":1,"w":1,"W":1}},"Mutex":{"acquireCount":{"r":3}}},"flowControl":{"acquireCount":3,"timeAcquiringMicros":7},"storage":{},"protocol":"op_msg",<strong>"durationMillis":2469</strong>}} |
So far, we have discussed how to use commitQuorum and when to use it. Now we will see a scenario when one node (voting) is down for any reason, and someone created an index with default commitQuorum. The op will keep running, and you want to kill the op.
I created the index with the default commitQuorum when one node is down.
Status of nodes:
|
1 |
rs1:PRIMARY> rs.status().members.forEach(function (d) {print(d.name) + " " + print(d.stateStr)});<br>127.0.0.1:27017<br>PRIMARY<br>localhost:27018<br>SECONDARY<br>localhost:27019<br>(not reachable/healthy)<br>rs1:PRIMARY> |
CurrentOp:
|
1 |
"active" : true,<br>"currentOpTime" : "2023-06-26T21:27:41.304+00:00",<br>"opid" : 536535,<br>"lsid" : {<br> "id" : UUID("dd9672f8-4f56-47ce-8ceb-31caf5e8baf8"),<br> "uid" : BinData(0,"47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=")<br> },<br> "secs_running" : NumberLong(264),<br> "microsecs_running" : NumberLong(264345444),<br> "op" : "command",<br> "ns" : "acme.products",<br> "command" : {<br> "createIndexes" : "products",<br> "indexes" : [<br> {<br> "key" : {<br> "airt" : 1<br> },<br> "name" : "airt_1"<br> }<br> ],<br> "lsid" : {<br> "id" : UUID("dd9672f8-4f56-47ce-8ceb-31caf5e8baf8")<br> },<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687814589, 2),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> },<br> "$db" : "acme"<br> } |
Now you need to kill the above opid to release the above op:
|
1 |
rs1:PRIMARY> db.killOp(536535)<br>{<br> "info" : "attempting to kill op",<br> "ok" : 1,<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687815189, 2),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> },<br> "operationTime" : Timestamp(1687815189, 2)<br>}<br>rs1:PRIMARY><br>rs1:PRIMARY> db.products.createIndex({ "airt" : 1 })<br>{<br> "operationTime" : Timestamp(1687815192, 2),<br> "ok" : 0,<br> "errmsg" : "operation was interrupted",<br> "code" : 11601,<br> "codeName" : "Interrupted",<br> "$clusterTime" : {<br> "clusterTime" : Timestamp(1687815192, 2),<br> "signature" : {<br> "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),<br> "keyId" : NumberLong(0)<br> }<br> }<br>}<br>rs1:PRIMARY><br>rs1:PRIMARY> db.products.getIndexes()<br>[ { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_" } ]<br>rs1:PRIMARY> |
Above, we can see when we killed the op, and the index creation op got killed.
We have seen how commitQuorum works while creating indexes from PSMDB 4.4. Still, the best practice is to create indexes in a rolling manner.
We recommend checking out our products for Percona Server for MongoDB, Percona Backup for MongoDB, and Percona Operator for MongoDB. We also recommend checking out our blog MongoDB: Why Pay for Enterprise When Open Source Has You Covered?
Resources
RELATED POSTS