High availability for MySQL on Amazon EC2 – Part 1 – Intro

Like many, I have been seduced by the power and flexibility of Amazon EC2. Being able to launch new instances at will depending on the load, is almost too good to be true. Amazon has also some drawbacks, availability is not guaranteed and discovery protocols relying on Ethernet broadcast or multicast cannot be used. That means, it is not easy to have a truly highly available solution for MySQL on EC2. If a MySQL instance fails, here are some challenges that we face:

1. Detect the failure
2. Kill the faulty instance
3. Free the shared resources (ex: EBS volumes)
4. Launch a new instances
5. Reconfigure the shared resources
6. Reconfigure instance monitoring
7. Reconfigure the database clients

Facing these challenges, a customer recently asked me if I could build a viable HA solution for MySQL on EC2. The goal is to have a cheap small instance monitor the availability of a large (any size) and taking measures to keep MySQL available. A few weeks later, I ended up with a solution that work and is decently elegant using Pacemaker and Heartbeat. The setup is fairly complex, being in the Amazon EC2 virtual world is not a simplification, far from. Because of the complexity, the story will be broken into multiple posts:

1. Part 1 – Intro (this post)
2. Part 2 – Setting up the initial instances
3. Part 3 – Configuring the HA resources
4. Part 4 – The instance restart script
5. Part 5 – The instance monitoring script
6. Part 6 – Publishing the MySQL server location
7. Part 7 – Pitfalls to avoid

Hopefully, I should be able to write those posts quickly but since consulting is my primary duty, I don’t have much control over my workload.

Stay tuned!