Last Resort: How to Use a Backup to Start a Secondary Instance for MongoDB

Backup to Start a SecondaryIn this blog post, I’ll look at how you can use a backup to start a secondary instance for MongoDB.

Although the documentation says it is not possible to use a backup to start a secondary, sometimes this is the only possible way to start a new instance. In this blog post, we will explain how to bypass this limitation and use a backup to start a secondary instance.

The initial sync/rsync or snapshot works fine when the instances are in the same data center, but it can fail or be really slow. Much slower than moving a compressed backup between data centers.

Not every backup can be used as a source for starting a replica set. The backup must have the oplog file. This means the backup must be done in a previously existent replica set using the --oplog flag point in time backup when dumping the collections. The time spent to move and restore the backup file must be less than the oplog window.

Please follow the next steps to create a secondary from a backup:

  1. Create a backup using the --oplog command.
  2. Backup the replica set collection from the local database.

  3. After backup finishes please confirm the file is in the backup folder.
  4. Use bsondump to convert the oplog to JSON. We will use the last oplog entry as a starting point for the replica set.
  5. Initiate the new instance without the replica set parameter configured. At this point, the instance will act as a single instance.
  6. Restore the database normally using --oplogreplay to apply all the oplogs that have been recorded while the backup was running.
  7. Connect to this server and use the local database to create the collection. Please use the same value as the other members (e.g., 20 GB).
  8. From the generated in step 4, get the last line and copy the fields ts and h values to a MongoDB document.

  9. Insert the JSON value to the collection that was created before.
  10. Restore the replset collection to the local database.
  11. Stop the service and edit the parameter replica set name to match the existing replica set.
  12. Connect to the primary and add this new host. The new host must start catching up the oplog and get in sync after a few hours/minutes, depending on the number of operations the replica set handles. It is important to consider adding this new secondary as a hidden secondary, without votes if possible, to avoid triggering an election. When the secondary is added to the replica set drivers, it will start using this host to perform reads. If you don’t add the server with hidden: true, the application will read inconsistent data (old data).

  13. Please check the replication lag, and once the seconds behind master is near to zero, change the host parameters in the replica set to hidden: false and priority or votes.
  14. We are considering a replica set with three members, where the new secondary has the ID 2 in the member’s array. Use the following command to unhide the secondary and make it available for reads. The priority and votes depend on your environment. Please notice you might need to change the member ID.

I hope this tutorial helps in an emergency situation. Please consider using initial sync, disk snapshots and hot backups before using this method.

Feel free to reach out me on twitter @AdamoTonete or @percona.

Share this post