EmergencyEMERGENCY? Get 24/7 Help Now!

MongoDB point-in-time backups made easy

 | September 20, 2016 |  Posted In: MongoDB

PREVIOUS POST
NEXT POST

MongoDB point-in-time backupsIn this blog post we’ll look at MongoDB point-in-time backups, and work with them.

Mongodump is the base logical backup tool included with MongoDB. It takes a full BSON copy of database/collections, and optionally includes a log of changes during the backup used to make it consistent to a point in time. Mongorestore is the tool used to restore logical backups created by Mongodump. I’ll use these tools in the steps in this article to restore backed-up data. This article assumes a mongodump-based backup that was taken consistently with oplog changes (by using the command flag “–oplog”), and the backup is being restored to a MongoDB instance.

In this example, a mongodump backup is gathered and restored for the base collection data, and separately the oplogs/changes necessary to restore the data to a particular point-in-time are collected and applied to this data.

Note: Percona developed a backup tool named mongodb_consistent_backup, which is a wrapper for ‘mongodump’ with added cluster-wide backup consistency. The backups created by mongodb_consistent_backup (in Dump/Mongodump mode) can be restored using the same steps as a regular “mongodump” backup.

Stages

Stage 1: Get a Mongodump Backup

Mongodump Command Flags
–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplog

Required for any replset member! Causes “mongodump” to capture the oplog change log during the backup for consistent to one point in time.

–gzip

Optional. For mongodump >= 3.2, enables inline compression on the backup files.

Steps
  1. Get a mongodump backup via (pick one):
    • Running “mongodump” with the correct flags/options to take a backup (w/oplog) of the data:
    • Use the latest daily automatic backup, if it exists.

Stage 2: Restore the Backup Data

Steps
  1. Locate the shard PRIMARY member.
  2. Triple check you’re restoring the right backup to the right shard/host!
  3. Restore a mongodump-based backup to the PRIMARY node using the steps in this article: Restore a Mongodump Backup.
  4. Check for errors.
  5. Check that all SECONDARY members are in sync with the PRIMARY.

Stage 3: Get Oplogs for Point-In-Time-Recovery

In this stage, we will gather the changes needed to roll the data forward from the time of backup to the time/oplog-position to which we would like to restore.

In this example below, let’s pretend someone accidentally deleted an entire collection at oplog timestamp: “Timestamp(1470923942, 3)” and we want to fix it. If we decrement the Timestamp increment (2nd number) of “Timestamp(1470923942, 3)” we will have the last change before the accidental command, which in this case is: “Timestamp(1470923942, 2)“. Using the timestamp, we can capture and replay the oplogs from when the backup occurred to just before the issue/error.

A start and end timestamp are required to get the oplog data. In all cases, this will need to be gathered manually, case-by-case.

Helper Script

 

Script Usage:

 

Steps
  1. Find the PRIMARY member that contains the oplogs needed for the PITR restore.
  2. Determine the “end” Timestamp() needed to restore to. This oplog time should be before the problem occurred.
  3. Determine the “start” Timestamp() from right before the backup was taken.
    1. This timestamp doesn’t need to be exact, so something like a Timestamp() object equal-to “a few min before the backup started” is fine, but the more accurate you are, the fewer changes you’ll need to re-apply (which saves on restore time).
  4. Use the MongoToolsAndSnippets script: “get_oplog_range.sh (above in “Helper Script”) to dump the oplog time-ranges you need to restore to your chosen point-in-time. In this example I am gathering the oplog between two point-in-times (also passing in –username/–password flags in quotes the 3rd parameter):
    1. The starting timestamp: the BSON timestamp from before the mongodump backup in “Stage 2: Restore Collection Data” was taken, in this example. “Timestamp(1470923918, 0)” is a time a few seconds before my mongodump was taken (does not need to be exact).
    2. The end timestamp: the end BSON Timestamp to restore to, in this example. “Timestamp(1470923942, 2)” is the last oplog-change BEFORE the problem occurred.

    Example:

    Note: all additional mongodump flags (optional 3rd field) must be in quotes!

  5. Double check it worked by looking for the ‘oplog.bson‘ file and checking that the file has some data in it (168mb in the below example):

     

Stage 4: Apply Oplogs for Point in Time Recovery (PITR)

In this stage, we apply the time-range-based oplogs gathered in Stage 3 to the restored data set to bring it from the time of the backup to a particular point in time before a problem occurred.

Mongorestore Command Flags
–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017).  If authorization is enabled, add –user/–password flags also.

–oplogReplay

Required. This is needed to replay the oplogs in this step.

–dir

Required. The path to the mongodump data.

Steps
  1. Copy the “dump” directory containing only the “oplog.bson”. file (captured in Stage 3) to the host that needs the oplog changes applied (the restore host).
  2. Run “mongorestore” on the “dump” directory to replay the oplogs into the instance. Make sure the “dump” dir contains only “oplog.bson”!
  3. Validate the data was restored with the customer or using any means possible (examples: .count() queries, some random .find() queries, etc.).
PREVIOUS POST
NEXT POST
David Murphy

David is the Practice Manager for MongoDB @ Percona.

He joined Percona in Oct 2015, before that he has been deep in both the MySQL and MongoDB database communities for some time. Other passions include DevOps , tool building, and security.

5 Comments

  • David. Thanks for sharing the steps.

    Things get a little dicey with Rollbacks though. Maybe multiple replica’s need to be queried and only dump the oplog entries that have been ‘majority’ propagated?

    Also on sharded clusters we need to avoid the rebalancing oplog entries.

    • You are absolutely correct here, there is another blog post on consistent backups that deal with sharding and its backup complexities to the point we published a tool on the labs part of our github.

      Regarding rollbacks you can only safely restore to the point the rollbacks happened then apply the rollback data. After this point reality diverges and you would have to pick a side. It’s not pretty but it’s true.

      • Makes sense. Whats a good way to detect rollbacks in this process ?

        1. You would need to take oplog dumps on each replica
        2. Compare the entries from each replica at restore time to find a rollback?
        3. Find the latest common oplog entry among the two divergent entries and restore up to it
        4. Then maybe pick the oplog entries from the latest master (as a fork in the road after the rollback).

        Seems fairly complicated though.

  • The mongodump query needs a $and wrapping the upper and lower bounds because it is querying on the same field, not using one returns everything earlier than TS_END including earlier than TS_START

    MONGODUMP_QUERY='{ "$and": [ {"ts" : { "$gte" : '$TS_START' }}, {"ts" : { "$lte" : '$TS_END' }} ] }'

  • Hi David,

    Can you tell how can we extract the timestamp that you have considered to get the dumps in an interval of time ?
    How to get the start value and from which file.

    Regards

Leave a Reply