Using keepalived for HA on top of Percona XtraDB Cluster

Percona XtraDB Cluster (PXC) itself manages quorum and node failure.  Minorities of nodes in a network partition situation will move themselves into a Non-primary state and not allow any DB activity.  Nodes in such a state will be easily detectable via SHOW GLOBAL STATUS variables.

It’s common to use HAproxy with PXC for load balancing purposes, but what if you are planning to just send traffic to a single node?  We would standardly use keepalived to HA HAproxy, and keepalived supports track_scripts that can monitor whatever we want, so why not just monitor PXC directly?

If we have clustercheck working on all hosts:

Then we can just install keepalived and the this config on all nodes:

And start the keepalived service. The virtual IP above will be brought up on an active node in the cluster and moved around if clustercheck fails.

If I shutdown PXC on node2:

We can see node2 moves to a FAULT state and the VIP moves to node1 instead.  This provides us with a very simple way to do Application to PXC high availability.

A few additional notes:

  • You can disqualify donors (i.e., make clustercheck fail on Donor/Desynced nodes) by setting the 3rd argument to clustercheck to 0.  Setting this to 1 means Donors can retain the VIP.
  • Each keepalived instance monitors its own state only, hence the @localhost GRANT.  This is much cleaner than exposing clustercheck as a web port via xinetd.
  • It’s possible to more complex things with keepalived like multiple vips, node weighting, etc.
  • Keepalived can track over multiple network interfaces (in this example, just eth1) for better reliability.

Share this post

Comments (6)

  • Ashok Kumar Reply

    Team,

    Please help me how can sync multiple extraDB cluster between two location.

    Thanks
    Ashok

    October 27, 2013 at 12:46 am
  • Tom Diederich Reply

    Hi again, Ashok,

    Once again our forums are the place to ask questions like this – however please try to include as much detail as possible. Here’s the url to the Percona XtraDB Cluster forums category: https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster

    October 27, 2013 at 7:59 pm
  • Marco Londero Reply

    Hi guys,

    the conf is slightly bugged. If you don’t start slaves forcing “state BACKUP”, the IP will be configured on all interfaces.

    Marco

    November 18, 2013 at 11:13 am
  • CaptTofu Reply

    Hi Jay!

    Trying to get this working on a Docker container – is this anything you’ve done?

    February 3, 2014 at 4:47 pm
    • Peter H Reply

      What’s are the benefits running ha-proxy/percona inside the docker container ? I see quite a lot of people neding a lot of applications for what reason ? There is always a nat and linux bridge between the container and the underlying os which is additional layer causing unnecessary complexity and slowness. I see people use docker under wery inappropriate circumstances.

      September 24, 2017 at 5:08 pm
  • Jay Janssen Reply

    @CaptTofu — can’t say that I have.

    February 4, 2014 at 8:21 am

Leave a Reply