Announcement

Announcement Module
Collapse
No announcement yet.

Cluster members out of sync

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Cluster members out of sync

    We just moved a large Invision Power Board forum to a 2 node Percona Cluster (behind HAProxy) from a single Percona server. As soon as the load on the forum goes up, servers quickly become desynchronized and depending on which node is hit, different data is displayed.

    Nodes were configured according to: http://www.percona.com/doc/percona-x...l#centos-howto

    Both are showing they are in sync when running the following command:
    Code:
    show status like 'wsrep%';
    Error logs are empty on both nodes.

    Is XtraDB Cluster not fast enough to keep up with this type of usage? CPU load and memory usage are very low on both servers so I'm not sure what could be the problem.

    As soon as HAProxy configuration is adjusted to send all requests to a single server, all problems disappear.

    Also, it seems that data was not just delayed between the hosts, but some of it was never duplicated. Shouldn't sync errors show up in log files?

    I can verify that replication works by adding data on any of the nodes and accessing it from any other node, but forum content is still completely out of sync. What could cause this?

    This may be a stupid question, but does DB need to be offline during the initial SST copy? It seems that only DB content from the initial copy at node start gets replicated, but nothing after that.

    Any help is appreciated.
    Last edited by daq; 07-10-2013, 06:40 PM. Reason: Placing all questions in one post

  • #2
    A minimum of 3 cluster nodes is recommended or a garbd arbitrator.
    You cannot properly achieve quorum with only 2 nodes.

    Also, this may help explain more:
    http://www.mysqlperformanceblog.com/...-only-2-nodes/

    Comment


    • #3
      I saw a similar article and added another node, which did not resolve the problem, but is a good idea anyway. I noticed that some of the tables are still using MyISAM engine which I suspect is what's causing the problem. I will convert them to InnoDB and hope that replication works as expected.

      Thanks for the suggestion.

      Comment


      • #4
        MyISAM tables were the problem. Cluster is now syncing correctly! Thanks.

        Comment

        Working...
        X