MongoDB Replica set Scenarios and Internals – Part 1

MongoDB replica sets replication internals rThe MongoDB® replica set is a group of nodes with one set as the primary node, and all other nodes set as secondary nodes. Only the primary node accepts “write” operations, while other nodes can only serve “read” operations according to the read preferences defined. In this blog post, we’ll focus on some MongoDB replica set scenarios, and take a look at the internals.

In a subsequent post, I talk about MongoDB Elections.

Example configuration

We will refer to a three node replica set that includes one primary node and two secondary nodes running as:

Here, the primary is running on port 25001, and the two secondaries are running on ports 25002 and 25003 on the same host.

Secondary nodes can only sync from Primary?

No, it’s not mandatory. Each secondary can replicate data from the primary or any other secondary to the node that is syncing. This term is also known as chaining, and by default, this is enabled.

In the above replica set, you can see that secondary node "_id":2   is syncing from another secondary node "_id":1   as "syncingTo" : "" 

This can also be found in the logs as here the parameter chainingAllowed :true   is the default setting.


That means that a secondary member node is able to replicate from another secondary member node instead of from the primary node. This helps to reduce the load from the primary. If the replication lag is not tolerable, then chaining could be disabled.

For more details about chaining and the steps to disable it please refer to my earlier blog post here.

Ok, then how does the secondary node select the source to sync from?

If Chaining is False

When chaining is explicitly set to be false, then the secondary node will sync from the primary node only or could be overridden temporarily.

If Chaining is True

Example for “unable to find a member to sync from” then, in the next attempt, finding a candidate to sync from

This can be found in the log like this. On receiving the message from rsBackgroundSync thread could not find member to sync from, the whole internal process restarts and finds a member to sync from i.e. sync source candidate:, which means it is now syncing from node running on port 25001.

  • Once the sync source node is selected, SyncSourceResolver probes the sync source to confirm that it is able to fetch the oplogs.
  • RollbackID is also fetched i.e. rbid  after the first batch is returned by oplogfetcher.
  • If all eligible sync sources are too fresh, such as during initial sync, then the syncSourceStatus Oplog start is missing and earliestOpTimeSeen will set a new minValid.
  • This minValid is also set in the case of rollback and abrupt shutdown.
  • If the node has a minValid entry then this is checked for the eligible sync source node.

Example showing the selection of a new sync source when the existing source is found to be invalid

Here, as the logs show, during sync the node chooses a new sync source. This is because it found the original sync source is not ahead, so not does not contain recent oplogs from which to sync.

  • If the secondary node is too far behind the eligible sync source node, then the node will enter maintenance node and then resync needs to be call manually.
  • Once the sync source is chosen, BackgroundSync starts oplogFetcher.

Example for oplogFetcher

Here is an example of fetching oplog from the “” collection, and checking for the greater than required timestamp.

When and what details replica set nodes communicate with each other?

At a regular interval, all the nodes communicate with each other to check the status of the primary node, check the status of the sync source, to get the oplogs and so on.

ReplicationCoordinator has ReplicaSetConfig that has a list of all the replica set nodes, and each node has a copy of it. This makes nodes aware of other nodes under same replica set.

This is how nodes communicate in more detail:

Heartbeats: This checks the status of other nodes i.e. alive or die

heartbeatInterval: Every node, at an interval of two seconds, sends the other nodes a heartbeat to make them aware that “yes I am alive!”

heartbeatTimeoutSecs: This is a timeout, and means that if the heartbeat is not returned in 10 seconds then that node is marked as inaccessible or simply die.

Every heartbeat is identified by these replica set details:

  • replica set config version
  • replica set name
  • Sender host address
  • id from the replicasetconfig

The source code could be referred to from here.

When the remote node receives the heartbeat, it processes this data and validates if the details are correct. It then prepares a ReplSetHeartbeatResponse, that includes:

  • Name of the replica set, config version, and optime details
  • Details about primary node as per the receiving node.
  • Sync source details and state of receiving node

This heartbeat data is processed, and if primary details are found then the election gets postponed.

TopologyCoordinator checks for the heartbeat data and confirms if the node is OK or NOT. If the node is OK then no action is taken. Otherwise it needs to be reconfigured or else initiate a priority takeover based on the config.

Response from oplog fetcher

To get the oplogs from the sync source, nodes communicate with each other. This oplog fetcher fetches oplogs through “find” and “getMore”. This will only affect the downstream node that gets metadata from its sync source to update its view from the replica set.

OplogQueryMetadata only comes with OplogFetcher responses

OplogQueryMetadata comes with OplogFetcher response and ReplSetMetadata comes with all the replica set details including configversion and replication commands.

Communicate to update Position commands:

This is to get an update for replication progress. ReplicationCoordinatorExternalState creates SyncSourceFeedback sends replSetUpdatePosition commands.

It includes Oplog details, Replicaset config version, and replica set metadata.

If a new node is added to the existing replica set, how will that node get the data?

If a new node is added to the existing replica set then the “initial sync” process takes place. This initial sync can be done in two ways:

  1. Just add the new node to the replicaset and let initial sync threads restore the data. Then it syncs from the oplogs until it reaches the secondary state.
  2. Copy the data from the recent data directory to the node, and restart this new node. Then it will also sync from the oplogs until it reaches the secondary state.

This is how it works internally

When “initial sync” or “rsync” is called by ReplicationCoordinator  then the node goes to “STARTUP2” state, and this initial sync is done in DataReplicator

  • A sync source is selected to get the data from, then it drops all the databases except the local database, and oplogs are recreated.
  • DatabasesCloner asks syncsource for a list of the databases, and for each database it creates DatabaseCloner.
  • For each DatabaseCloner it creates CollectionCloner to clone the collections
  • This CollectionCloner calls ListIndexes on the syncsource and creates a CollectionBulkLoader for parallel index creation while data cloning
  • The node also checks for the sync source rollback id. If rollback occurred, then it restarts the initial sync. Otherwise, datareplicator is done with its work and then replicationCoordinator assumes the role for ongoing replication.

Example for the “initial sync” :

Here node enters   "STARTUP2"- "transition to STARTUP2"

Then sync source gets selected and drops all the databases except the local database.  Next, replication oplog is created and CollectionCloner is called.

Local database not dropped: because every node has its own “local” database with its own and other nodes’ information, based on itself, this database is not replicated to other nodes.

Finished fetching all the oplogs, and finishing up initial sync.

What are oplogs and where do these reside?

oplogs stands for “operation logs”. We have used this term so many times in this blog post as these are the mandatory logs for the replica set. These operations are in the capped collection called “”  that resides in “local” database.

Below, this is how oplogs are stored in the collection “” that includes details for timestamp, operations, namespace, output.

It consists of rolling update operations coming to the database. Then these oplogs replicate to the secondary node(s) to maintain the high availability of the data in case of failover.

When the replica MongoDB instance starts, it creates an oplog ocdefault size. For Wired tiger, the default size is 5% of disk space, with a lower bound size of 990MB. So here in the example it creates 990MB of data. If you’d like to learn more about oplog size then please refer here

What if the same oplog is applied multiple times, will that not lead to inconsistent data?

Fortunately, oplogs are Idempotent that means the value will remain unchanged, or will provide the same output, even when applied multiple times.

Let’s check an example:

For the $inc operator that will increment the value by 1 for the filed “item”, if this oplog is applied multiple times then the result might lead to an inconsistent record if this is not Idempotent. However, rather than increasing the item value multiple times, it is actually applied only once.

This is how these operations are stored in oplog, here this $inc value is stored in oplog as $set

That means that however many  times it is applied, it will generate the same results, so no inconsistent data!

I hope this blog post helps you to understand multiple scenarios for MongoDB replica sets, and how data replicates to the nodes.

Share this post

Comments (4)

  • alex penazzi Reply

    Interesting article. Thanks for this Aayushi.

    October 10, 2018 at 7:20 am
  • Pooja Rathore Reply

    It was a great article..I will read your other blogs as well.

    October 17, 2018 at 8:28 am
  • Deepak Reply

    Quite informative blogs

    January 23, 2019 at 8:30 am
  • max Reply

    Hello,I use PSMDB v3.4.16-2.14 replica set , I met a problem,The following log appears every 30 minutes in the primary node log files . Thank you for your reply。。

    2019-09-25T11:06:45.329+0800 I COMMAND [conn3720107] command command: find { find: “”, filter: { ts: { $gte: Timestamp 1569378708000|3, $lte: Timestamp 1569380585000|17 } }, skip: 0 } planSummary: COLLSCAN cursorid:35876048945 keysExamined:0 docsExamined:93489807 numYields:731510 nreturned:101 reslen:51525 locks:{ Global: { acquireCount: { r: 1463022 } }, Database: { acquireCount: { r: 731511 } }, oplog: { acquireCount: { r: 731511 } } } protocol:op_query 219136ms
    2019-09-25T11:36:13.034+0800 I COMMAND [conn3721466] command command: find { find: “”, filter: { ts: { $lte: Timestamp 1569382307000|9, $gte: Timestamp 1569380585000|17 } }, skip: 0 } planSummary: COLLSCAN cursorid:37832025936 keysExamined:0 docsExamined:93521292 numYields:732104 nreturned:101 reslen:48012 locks:{ Global: { acquireCount: { r: 1464210 } }, Database: { acquireCount: { r: 732105 } }, oplog: { acquireCount: { r: 732105 } } } protocol:op_query 264646ms
    2019-09-25T12:05:16.439+0800 I COMMAND [conn3722523] command command: find { find: “”, filter: { ts: { $gte: Timestamp 1569382307000|9, $lte: Timestamp 1569384094000|3 } }, skip: 0 } planSummary: COLLSCAN cursorid:38346597871 keysExamined:0 docsExamined:92372199 numYields:722827 nreturned:101 reslen:51087 locks:{ Global: { acquireCount: { r: 1445656 } }, Database: { acquireCount: { r: 722828 } }, oplog: { acquireCount: { r: 722828 } } } protocol:op_query 221660ms
    2019-09-25T12:34:20.203+0800 I COMMAND [conn3723089] command command: find { find: “”, filter: { ts: { $gte: Timestamp 1569384094000|3, $lte: Timestamp 1569385892000|11 } }, skip: 0 } planSummary: COLLSCAN cursorid:37141151636 keysExamined:0 docsExamined:92395424 numYields:722587 nreturned:101 reslen:52316 locks:{ Global: { acquireCount: { r: 1445176 } }, Database: { acquireCount: { r: 722588 } }, oplog: { acquireCount: { r: 722588 } } } protocol:op_query 167030ms
    2019-09-25T13:05:15.034+0800 I COMMAND [conn3723649] command command: find { find: “”, filter: { ts: { $gte: Timestamp 1569385892000|11, $lte: Timestamp 1569387712000|6 } }, skip: 0 } planSummary: COLLSCAN cursorid:38511743876 keysExamined:0 docsExamined:92412676 numYields:722892 nreturned:101 reslen:39843 locks:{ Global: { acquireCount: { r: 1445786 } }, Database: { acquireCount: { r: 722893 } }, oplog: { acquireCount: { r: 722893 } } } protocol:op_query 202054ms
    2019-09-25T13:35:36.457+0800 I COMMAND [conn3724549] command command: find { find: “”, filter: { ts: { $lte: Timestamp 1569389527000|1, $gte: Timestamp 1569387712000|6 } }, skip: 0 } planSummary: COLLSCAN cursorid:36755678812 keysExamined:0 docsExamined:92425916 numYields:723060 nreturned:101 reslen:35475 locks:{ Global: { acquireCount: { r: 1446122 } }, Database: { acquireCount: { r: 723061 } }, oplog: { acquireCount: { r: 723061 } } } protocol:op_query 207805ms
    2019-09-25T14:06:02.128+0800 I COMMAND [conn3725685] command command: find { find: “”, filter: { ts: { $gte: Timestamp 1569389527000|1, $lte: Timestamp 1569391333000|8 } }, skip: 0 } planSummary: COLLSCAN cursorid:36351666103 keysExamined:0 docsExamined:92440017 numYields:723399 nreturned:101 reslen:50981 locks:{ Global: { acquireCount: { r: 1446800 } }, Database: { acquireCount: { r: 723400 } }, oplog: { acquireCount: { r: 723400 } } } protocol:op_query 228515ms

    September 25, 2019 at 2:39 am

Leave a Reply