Downloads

Blog

Contact Us

MongoDB Point-in-Time Backups Made Easy

September 20, 2016

Author

David Murphy

MongoDB

Share this Post:

In this blog post, we’ll look at MongoDB point-in-time backups, and work with them.

Mongodump

Mongodump is the base logical backup tool included with MongoDB. It takes a full BSON copy of database/collections, and optionally includes a log of changes during the backup used to make it consistent to a point in time. Mongorestore is the tool used to restore logical backups created by Mongodump. I’ll use these tools in the steps in this article to restore backed-up data. This article assumes a mongodump-based backup that was taken consistently with oplog changes (by using the command flag “–oplog”), and the backup is being restored to a MongoDB instance.

In this example, a mongodump backup is gathered and restored for the base collection data, and separately the oplogs/changes necessary to restore the data to a particular point-in-time are collected and applied to this data.

Note: Percona developed a backup tool named mongodb_consistent_backup, which is a wrapper for ‘mongodump’ with added cluster-wide backup consistency. The backups created by mongodb_consistent_backup (in Dump/Mongodump mode) can be restored using the same steps as a regular “mongodump” backup.

Stages

Stage 1: Get a Mongodump Backup

Mongodump Command Flags

–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017). If authorization is enabled, add –user/–password flags also.

–oplog

Required for any replset member! Causes “mongodump” to capture the oplog change log during the backup for consistent to one point in time.

–gzip

Optional. For mongodump >= 3.2, enables inline compression on the backup files.

Steps

Get a mongodump backup via (pick one):

Running “mongodump” with the correct flags/options to take a backup (w/oplog) of the data:

				
				1

						$ mongodump --host localhost --port 27017 --oplog --gzip<br><br>2016-08-15T12:32:28.930+0200    writing wikipedia.pages to<br>2016-08-15T12:32:31.932+0200    [#########...............]  wikipedia.pages  674/1700   (39.6%)<br>2016-08-15T12:32:34.931+0200    [####################....]  wikipedia.pages  1436/1700  (84.5%)<br>2016-08-15T12:32:37.509+0200    [########################]  wikipedia.pages  2119/1700  (124.6%)<br>2016-08-15T12:32:37.510+0200    done dumping wikipedia.pages (2119 documents)<br>2016-08-15T12:32:37.521+0200    writing captured oplog to<br>2016-08-15T12:32:37.931+0200    [##......................]  .oplog  44/492   (8.9%)<br>2016-08-15T12:32:39.648+0200    [########################]  .oplog  504/492  (102.4%)<br>2016-08-15T12:32:39.648+0200    dumped 504 oplog entries

Use the latest daily automatic backup, if it exists.

Stage 2: Restore the Backup Data

Steps

Locate the shard PRIMARY member.

Triple check you’re restoring the right backup to the right shard/host!

Restore a mongodump-based backup to the PRIMARY node using the steps in this article: Restore a Mongodump Backup.

Check for errors.

Check that all SECONDARY members are in sync with the PRIMARY.

Stage 3: Get Oplogs for Point-In-Time-Recovery

In this stage, we will gather the changes needed to roll the data forward from the time of backup to the time/oplog-position to which we would like to restore.

In this example below, let’s pretend someone accidentally deleted an entire collection at oplog timestamp: “Timestamp(1470923942, 3)” and we want to fix it. If we decrement the Timestamp increment (2nd number) of “Timestamp(1470923942, 3)” we will have the last change before the accidental command, which in this case is: “Timestamp(1470923942, 2)“. Using the timestamp, we can capture and replay the oplogs from when the backup occurred to just before the issue/error.

A start and end timestamp are required to get the oplog data. In all cases, this will need to be gathered manually, case-by-case.

Helper Script

#!/bin/bash<br>#<br># This tool will dump out a BSON file of MongoDB oplog changes based on a range of Timestamp() objects.<br># The captured oplog changes can be applied to a host using 'mongorestore --oplogReplay --dir /path/to/dump'.<br><br>set -e<br><br>TS_START=$1<br>TS_END=$2<br>MONGODUMP_EXTRA=$3<br><br>function usage_exit() {<br>  echo "Usage $0: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]"<br>  exit 1<br>}<br><br>function check_bson_timestamp() {<br>  local TS=$1<br>  echo "$TS" | grep -qP "^Timestamp(d+,sd+)$"<br>  if [ $? -gt 0 ]; then<br>    echo "ERROR: Both timestamp fields must be in BSON Timestamp format, eg: 'Timestamp(########, #)'!"<br>    usage_exit<br>  fi<br>}<br><br>if [ -z "$TS_START" ] || [ -z "$TS_END" ]; then<br>  usage_exit<br>else<br>  check_bson_timestamp "$TS_START"<br>  check_bson_timestamp "$TS_END"<br>fi<br><br>MONGODUMP_QUERY='{ "ts" : { "$gte" : '$TS_START' }, "ts" : { "$lte" : '$TS_END' } }'<br>MONGODUMP_FLAGS='--db=local --collection=oplog.rs'<br>[ ! -z "$MONGODUMP_EXTRA" ] && MONGODUMP_FLAGS="$MONGODUMP_FLAGS $MONGODUMP_EXTRA"<br><br>if [ -d dump ]; then<br>  echo "'dump' subdirectory already exists! Exiting!"<br>  exit 1<br>fi<br><br>echo "# Dumping oplogs from '$TS_START' to '$TS_END'..."<br><br>mkdir dump<br>mongodump $MONGODUMP_FLAGS --query "$MONGODUMP_QUERY" --out - >dump/oplog.bson<br><br>if [ -f dump/oplog.bson ]; then<br>  echo "# Done!"<br>else<br>  echo "ERROR: Cannot find oplog.bson file! Exiting!"<br>  exit 1<br>fi<br>

1

#!/bin/bash # # This tool will dump out a BSON file of MongoDB oplog changes based on a range of Timestamp() objects. # The captured oplog changes can be applied to a host using 'mongorestore --oplogReplay --dir /path/to/dump'. set -e TS_START=$1 TS_END=$2 MONGODUMP_EXTRA=$3 function usage_exit() { echo "Usage $0: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]" exit 1 } function check_bson_timestamp() { local TS=$1 echo "$TS" | grep -qP "^Timestamp(d+,sd+)$" if [ $? -gt 0 ]; then echo "ERROR: Both timestamp fields must be in BSON Timestamp format, eg: 'Timestamp(########, #)'!" usage_exit fi } if [ -z "$TS_START" ] || [ -z "$TS_END" ]; then usage_exit else check_bson_timestamp "$TS_START" check_bson_timestamp "$TS_END" fi MONGODUMP_QUERY='{ "ts" : { "$gte" : '$TS_START' }, "ts" : { "$lte" : '$TS_END' } }' MONGODUMP_FLAGS='--db=local --collection=oplog.rs' [ ! -z "$MONGODUMP_EXTRA" ] && MONGODUMP_FLAGS="$MONGODUMP_FLAGS $MONGODUMP_EXTRA" if [ -d dump ]; then echo "'dump' subdirectory already exists! Exiting!" exit 1 fi echo "# Dumping oplogs from '$TS_START' to '$TS_END'..." mkdir dump mongodump $MONGODUMP_FLAGS --query "$MONGODUMP_QUERY" --out - >dump/oplog.bson if [ -f dump/oplog.bson ]; then echo "# Done!" else echo "ERROR: Cannot find oplog.bson file! Exiting!" exit 1 fi

Script Usage:

1	$ ./dump_oplog_range.sh<br>Usage ./dump_oplog_range.sh: [Start-BSON-Timestamp] [End-BSON-Timestamp] [Extra-Mongodump-Flags (in quotes for multiple)]

Steps

Find the PRIMARY member that contains the oplogs needed for the PITR restore.

Determine the “end” Timestamp() needed to restore to. This oplog time should be before the problem occurred.

Determine the “start” Timestamp() from right before the backup was taken.
1. This timestamp doesn’t need to be exact, so something like a Timestamp() object equal-to “a few min before the backup started” is fine, but the more accurate you are, the fewer changes you’ll need to re-apply (which saves on restore time).

Use the MongoToolsAndSnippets script: “get_oplog_range.sh” (above in “Helper Script”) to dump the oplog time-ranges you need to restore to your chosen point-in-time. In this example I am gathering the oplog between two point-in-times (also passing in –username/–password flags in quotes the 3rd parameter):

The starting timestamp: the BSON timestamp from before the mongodump backup in “Stage 2: Restore Collection Data” was taken, in this example. “Timestamp(1470923918, 0)” is a time a few seconds before my mongodump was taken (does not need to be exact).

The end timestamp: the end BSON Timestamp to restore to, in this example. “Timestamp(1470923942, 2)” is the last oplog-change BEFORE the problem occurred.

Example:

1

$ wget -q https://raw.githubusercontent.com/percona/MongoToolsAndSnippets/master/rdba/dump_oplog_range.sh $ bash ./dump_oplog_range.sh 'Timestamp(1470923918, 0)' 'Timestamp(1470923942, 2)' '--username=secret --password=secret --host=mongo01.example.com --port=27024' # Dumping oplogs from 'Timestamp(1470923918, 0)' to 'Timestamp(1470923942, 2)'... 2016-08-12T13:11:17.676+0200    writing local.oplog.rs to stdout 2016-08-12T13:11:18.120+0200    dumped 22 documents # Done!

Note: all additional mongodump flags (optional 3rd field) must be in quotes!

Double check it worked by looking for the ‘oplog.bson‘ file and checking that the file has some data in it (168mb in the below example):

1

$ ls -alh dump/oplog.bson -rw-rw-r--. 1 tim tim 168M Aug 12 13:11 dump/oplog.bson

Stage 4: Apply Oplogs for Point in Time Recovery (PITR)

In this stage, we apply the time-range-based oplogs gathered in Stage 3 to the restored data set to bring it from the time of the backup to a particular point in time before a problem occurred.

Mongorestore Command Flags

–host/–port (and –user/–password)

Required, even if you’re using the default host/port (localhost:27017). If authorization is enabled, add –user/–password flags also.

–oplogReplay

Required. This is needed to replay the oplogs in this step.

–dir

Required. The path to the mongodump data.

Steps

Copy the “dump” directory containing only the “oplog.bson”. file (captured in Stage 3) to the host that needs the oplog changes applied (the restore host).

Run “mongorestore” on the “dump” directory to replay the oplogs into the instance. Make sure the “dump” dir contains only “oplog.bson”!

1

$ mongorestore --host localhost --port 27017 --oplogReplay --dir ./dump 2016-08-12T13:12:28.105+0200    building a list of dbs and collections to restore from dump dir 2016-08-12T13:12:28.106+0200    replaying oplog 2016-08-12T13:12:31.109+0200    oplog   80.0 MB 2016-08-12T13:12:34.109+0200    oplog   143.8 MB 2016-08-12T13:12:35.501+0200    oplog   167.8 MB 2016-08-12T13:12:35.501+0200    done