Group Replication: Shipped Too Early
This blog post is my overview of Group Replication technology.
With Oracle clearly entering the “open source high availability solutions” arena with the release of their brand new Group Replication solution, I believe it is time to review the quality of the first GA (production ready) release.
TL;DR: Having examined the technology, it is my conclusion that Oracle seems to have released the GA version of Group Replication too early. While the product is definitely “working prototype” quality, the release seems rushed and unfinished. I found a significant number of issues, and I would personally not recommend it for production use.
It is obvious that Oracle is trying hard to ship technology to compete with Percona XtraDB Cluster, which is probably why they rushed to claim Group Replication GA quality.
If you’re all set to follow along and test Group Replication yourself, simplify the initial setup by using this Docker image. We can review some of the issues you might face together.
For the record, I tested the version based on MySQL 5.7.17 release.
No automatic provisioning
First off, the first thing you’ll find is there is NO way to automatically setup of a new node.
If you need to setup new node or recover an existing node from a fatal failure, you’ll need to manually provision the slave.
Of course, you can clone a slave using Percona XtraBackup or LVM by employing some self-developed scripts. But given the high availability nature of the product, one would expect Group Replication to automatically re-provision any failed node.
Bug: stale reads on nodes
Please see this bug:
- https://bugs.mysql.com/bug.php?id=84900: getting inconsistent results on different nodes
One line summary: while any secondary nodes are “catching up” to whatever happened on a first node (it takes time to apply changes on secondary nodes), reads on a secondary node could return stale data (as shown in the bug report).
This behavior brings us back to the traditional asynchronous replication slave behavior (i.e., Group Replication’s predecessor).
It also contradicts the Group Replication documentation, which states: “There is a built-in group membership service that keeps the view of the group consistent and available for all servers at any given point in time.” (See https://dev.mysql.com/doc/refman/5.7/en/group-replication.html.)
I might also mention here that Percona XtraDB Cluster prevents stale reads (see https://www.percona.com/doc/percona-xtradb-cluster/5.7/wsrep-system-index.html#wsrep_sync_wait).
Bug: nodes become unusable after a big transaction, refusing to execute further transactions
There are two related bugs:
- https://bugs.mysql.com/bug.php?id=83218: DML operations in multiple sessions fail
- https://bugs.mysql.com/bug.php?id=84901: can’t execute transaction on the second node
One line summary: after running a big transaction, any secondary nodes become unusable and refuse to perform any further transactions.
Obscure error messages
It is not uncommon to see cryptic error messages while testing Group Replication. For example:
|
1 2 |
mysql> commit; ERROR 3100 (HY000): Error on observer while running replication hook 'before_commit'. |
This is fairly useless and provides little help until I check the mysqld error log. The log provides a little bit more information:
|
1 |
2017-02-09T02:05:36.996776Z 18 [ERROR] Plugin group_replication reported: '[GCS] Gcs_packet's payload is too big. Only the packets smaller than 2113929216 bytes can be compressed.' |
Discussion:
The items highlighted above might not seem too bad at first, and you could assume that your workload won’t be affected. However, stale reads and node dysfunctions basically prevent me from running a more comprehensive evaluation.
My recommendation:
If you care about your data, then I recommend not using Group Replication in production. Currently, it looks like it might cause plenty of headaches, and it is easy to get inconsistent results.
For the moment, Group Replication appears an advanced – but broken – traditional MySQL asynchronous replication.
I understand Oracle’s dilemma. Usually people are hesitant to test a product that is not GA. So in order to get feedback from users, Oracle needs to push the product to GA. Oracle must absolutely solve the issues above during future QA cycles.
Comments (8)
Hi Vadim, please allow me to leave this link here, everything has already been written somewhere else. Regards, JFG
https://www.facebook.com/jfg956/posts/10155145654669274
You are welcome to post comments here, not just links to Facebook account.
On the topic note – I would appreciate a practical feedback or disagreement, not just the fact that you feel “annoyed”
Hi Vadim, the facebook post is public, it links to this my previously published post in the 2nd comment. I realized it was not obvious as more comments were added, so I edited the facebook post and added the original link below.
http://jfg-mysql.blogspot.nl/2015/11/disappointment-at-mysql-ecosystem.html
Actually, I mostly agree with Vadim. Group replication is rather new technology; interesting for sure, but should not be advertised as production ready IMHO. On the other hand, with Galera Cluster being a way more mature and proven solution, I can see why Oracle is in a rush with this…
Hi,
add this https://bugs.mysql.com/bug.php?id=84774 performance issue.
Thanks for adding, but in this case frankly I do not see the reported issue serious enough to be a showstopper. Small performance variances happen all the time.
Docs for wsrep_sync_wait leave a lot to be desired. Maybe the feature is better than what GR offers today, but that is hard to tell from such sparse docs — https://www.percona.com/doc/percona-xtradb-cluster/5.7/wsrep-system-index.html#wsrep_sync_wait
Mark,
This is fair, we will improve documentation.