Group Replication: Shipped Too Early

Group Replication: Shipped Too Early

PREVIOUS POST
NEXT POST

Group ReplicationThis blog post is my overview of Group Replication technology.

With Oracle clearly entering the “open source high availability solutions” arena with the release of their brand new Group Replication solution, I believe it is time to review the quality of the first GA (production ready) release.

TL;DR: Having examined the technology, it is my conclusion that Oracle seems to have released the GA version of Group Replication too early. While the product is definitely “working prototype” quality, the release seems rushed and unfinished. I found a significant number of issues, and I would personally not recommend it for production use.

It is obvious that Oracle is trying hard to ship technology to compete with Percona XtraDB Cluster, which is probably why they rushed to claim Group Replication GA quality.

If you’re all set to follow along and test Group Replication yourself, simplify the initial setup by using this Docker image. We can review some of the issues you might face together.

For the record, I tested the version based on MySQL 5.7.17 release.

No automatic provisioning

First off, the first thing you’ll find is there is NO way to automatically setup of a new node.

If you need to setup new node or recover an existing node from a fatal failure, you’ll need to manually provision the slave.

Of course, you can clone a slave using Percona XtraBackup or LVM by employing some self-developed scripts. But given the high availability nature of the product, one would expect Group Replication to automatically re-provision any failed node.

Bug: stale reads on nodes

Please see this bug:

One line summary: while any secondary nodes are “catching up” to whatever happened on a first node (it takes time to apply changes on secondary nodes), reads on a secondary node could return stale data (as shown in the bug report).

This behavior brings us back to the traditional asynchronous replication slave behavior (i.e., Group Replication’s predecessor).

It also contradicts the Group Replication documentation, which states: “There is a built-in group membership service that keeps the view of the group consistent and available for all servers at any given point in time.” (See https://dev.mysql.com/doc/refman/5.7/en/group-replication.html.)

I might also mention here that Percona XtraDB Cluster prevents stale reads (see https://www.percona.com/doc/percona-xtradb-cluster/5.7/wsrep-system-index.html#wsrep_sync_wait).

Bug: nodes become unusable after a big transaction, refusing to execute further transactions

There are two related bugs:

One line summary: after running a big transaction, any secondary nodes become unusable and refuse to perform any further transactions.

Obscure error messages

It is not uncommon to see cryptic error messages while testing Group Replication. For example:

This is fairly useless and provides little help until I check the mysqld error log. The log provides a little bit more information:

Discussion:

The items highlighted above might not seem too bad at first, and you could assume that your workload won’t be affected. However, stale reads and node dysfunctions basically prevent me from running a more comprehensive evaluation.

My recommendation:

If you care about your data, then I recommend not using Group Replication in production. Currently, it looks like it might cause plenty of headaches, and it is easy to get inconsistent results.

For the moment, Group Replication appears an advanced – but broken – traditional MySQL asynchronous replication.

I understand Oracle’s dilemma. Usually people are hesitant to test a product that is not GA. So in order to get feedback from users, Oracle needs to push the product to GA. Oracle must absolutely solve the issues above during future QA cycles.

PREVIOUS POST
NEXT POST

Share this post

Comments (8)

  • Jean-François Gagné Reply

    Hi Vadim, please allow me to leave this link here, everything has already been written somewhere else. Regards, JFG
    https://www.facebook.com/jfg956/posts/10155145654669274

    February 16, 2017 at 2:49 am
    • Vadim Tkachenko Reply

      You are welcome to post comments here, not just links to Facebook account.
      On the topic note – I would appreciate a practical feedback or disagreement, not just the fact that you feel “annoyed”

      February 16, 2017 at 11:02 am
  • Rick Pizzi Reply

    Actually, I mostly agree with Vadim. Group replication is rather new technology; interesting for sure, but should not be advertised as production ready IMHO. On the other hand, with Galera Cluster being a way more mature and proven solution, I can see why Oracle is in a rush with this…

    February 16, 2017 at 4:06 am
  • tuparov1 Reply

    Hi,
    add this https://bugs.mysql.com/bug.php?id=84774 performance issue.

    February 16, 2017 at 6:48 am
    • Vadim Tkachenko Reply

      Thanks for adding, but in this case frankly I do not see the reported issue serious enough to be a showstopper. Small performance variances happen all the time.

      February 16, 2017 at 10:54 am
  • Mark Callaghan Reply

    Docs for wsrep_sync_wait leave a lot to be desired. Maybe the feature is better than what GR offers today, but that is hard to tell from such sparse docs — https://www.percona.com/doc/percona-xtradb-cluster/5.7/wsrep-system-index.html#wsrep_sync_wait

    February 17, 2017 at 10:45 am

Leave a Reply