“Percona has performed very well in production,” said Barth. “We’ve had no issues with the software, which gives us a high degree of confidence to move forward with our multi-master strategy.”

Percona Technologies

Percona XtraDB Cluster provides PagerDuty with a variety of critical benefits. For example, in the event of failover, instead of the significant downtime and the performance hit that occurred with DRDB and MySQL Community Edition, Barth can simply mark the current preferred master as down, so new connections go to a different node. Percona XtraDB Cluster can then be restarted on the master with no downtime or performance hit. This is then repeated until all three nodes are restarted. “Our Percona XtraDB Cluster remains up the entire time, and there’s no impact on customers,” said Barth.

Higher Performance Storage – The DRDB pair used Amazon Elastic Block Store (EBS) to store data. While EBS is reliable for data storage, companies can experience extended downtime due to network issues. Because Percona XtraDB Cluster makes it easy to spin up a new cluster node and grab data from another existing node, PagerDuty had the confidence to move to Amazon EC2 Instance Store instead of EBS. EC2 Instance Store is local disk storage for the Percona Server virtual machine—and therefore eliminates network issues—but it is “ephemeral” storage, that is, it is available only during the lifetime of the host virtual machine. “This is a huge benefit to our Percona cluster,” said Barth. “We are no longer dependent on EBS to be up and reliable in order for our database to be up and reliable. With Instance Store, our data is on a device that’s physically attached to the computer running our virtual machine, yet we have the confidence that with three active Percona XtraDB Cluster nodes, our data is safe.”

Asynchronous Slave Replication – Percona XtraDB Cluster allows async slaves to be moved between any of the cluster nodes with no impact on performance. Because of the way the cluster annotates the binary logs, replication can begin immediately from a known coordinate point on any node. “Our database is now large enough that a restore from a MySQL dump would take 15 hours and from a binary backup, it would take 2 to 3 hours. The great benefit for us is that now, by simply calculating the coordinates on another node, we can be up and running in a few minutes,” said Barth.

As a secondary backup strategy, PagerDuty uses Percona XtraBackup to create full and incremental binary backups of its database on a dedicated backup host. Once again, the key benefit is recovery time. If necessary, this binary restore would take just a couple of hours instead of the 15 hours required for a MySQL dump.

PagerDuty uses a variety of tools from the Percona Toolkit. For example, a pt-query-digest is run every night to identify slow queries; pt-heartbeat monitors async slaves; and pt-slave-delay manages a delayed slave setup for “fat finger errors.”