Recovery Times – Part Deux

In a follow-up experiment to an earlier post on TokuDB recovery times, I tried to create a better apples-to-apples comparison to InnoDB recovery time. If I measure recovery times when both DBs are doing the same amount of work, TokuDB requires only 2s to recover from a crash, compared to 1020s for InnoDB.

Background

In the first experiment, I compared recovery times when both storage engines (TokuDB and InnoDB) were inserting at maximum rates. In that experiment, following a power cord pull and server restart, TokuDB recovered in 501s, InnoDB in 18505s. In response, it was suggested that I run a patched version of InnoDB that vastly improves InnoDB recovery times. The patched version recovered in 1020s. (Quite an improvement!)

The first experiment ignored the fact that TokuDB was inserting >16,000 rows/s into the table, while InnoDB was inserting less than 200 rows/s. Given that TokuDB was doing so much more work, you would expect a longer recovery time. This post discusses an experiment where recovery times were measured when both DBs were doing the same amount of work.

The experiment

As before, I ran on the following server:

The disks and RAID controller were configured with the write-caches OFF.

I modified iiBench to generate rows at a rate of ~200 rows per second for insertion into a MySQL database so that both storage engines would perform insertions at the same rate. After inserting 250M rows, I pulled the power cords, then measured the time to restart mysqld after the machine was rebooted. I repeated this experiment inserting ~2000 rows per second. The results:

The reduced workload meant there was far less log data to scan and recover. These results also match our findings when we recover from power plug pulls on typical applications, which don’t necessarily use all of TokuDB’s insertion bandwidth at all times.

A final note: it takes mysqld 1s to start when no recovery is required.

Looks like the upper hand, is on the other foot!