September 1, 2014

Row-based replication, MySQL 5.6 upgrades and temporal data types

Whither your rollback plan?

MySQL 5.6 upgrades are in full swing these days and knowing how to safely upgrade from MySQL 5.5 to 5.6 is important. When upgrading a replication environment, it’s important that you can build a migration plan that safely allows for your upgrade with minimal risk — rollback is often a very important component to this.

For many people this means upgrading slaves first and then the master.  The strategy of an older master replicating to a newer slave is well known and has been supported in MySQL replication for a very long time.  To be specific:  you can have a MySQL 5.6 slave of a 5.5 master and this should work fine until you upgrade your master and/or promote one of the slaves to be the master.

However, there are those of us who like to live on the edge and do unsupported things.  Suppose that when you cut over to that MySQL 5.6 master your application completely breaks.  What would your rollback plan be?   In such a case, leaving a 5.5 slave of the new 5.6 master (or perhaps a dual-master setup with 5.5 and 5.6) would be useful to allow you to rollback to but still have the data written on the 5.6 master.

What might break?

With Statement-based replication (SBR), you are generally ok with this type of setup, provided you aren’t doing any MySQL 5.6 syntax-specific things until you don’t have any more 5.5 slaves.  However, with Row-based replication (RBR), things are a bit trickier, particularly when column formats change.

Now, one nice new feature of MySQL 5.6 is the improvement of the storage requirements for DATETIME fields as well as the addition of fractional second support for TIME, DATETIME, and TIMESTAMP.   This is great, but unfortunately this is a new column format that 5.5 clearly would not understand.  Does this put our 5.6 to 5.5 replication in jeopardy?    The answer is, if we’re careful, NO.

Quite simply, MySQL 5.6 supports both old and new types and mysql_upgrade does not make such a conversion on existing tables.  Only NEW tables or REBUILT tables in 5.6 will use the new format.  Any tables from 5.5 with a simple mysql_upgrade to 5.6 will still be using the old types.  For more information on how to find columns in 5.6 that are using the old format, see Ike Walker’s excellent blog post on the topic.  (Thanks Ike!)

An imperfect test

To test this out, I created a simple experiment.  I have a master and slave using RBR, both on 5.5, and I setup pt-heartbeat to update the master.  I realized that pt-heartbeat actually uses a varchar for the timestamp field — I suspect this makes multiple database support easier.  However, since pt-heartbeat’s update uses a NOW() to populate that field, I can convert it to a DATETIME:

So my heartbeat table now has a 5.5 DATETIME, pt-heartbeat is working properly, and the heartbeat is replicating to the slave.  Now I will upgrade my master to MySQL 5.6:

I can now verify that Ike’s INFORMATION_SCHEMA queries correctly detect the ‘heartbeat.ts’ column as the old format:

To make replication work from MySQL 5.6 to 5.5, I also had to add a few backwards compatibility options on the master:

Once I fixed that up, I can verify my slave is still working after this and receiving heartbeats. Clearly the new formats are not a show-stopper for backwards replication compatibility.

But, if I’m not careful on MySQL 5.6, and rebuild the table, the new format does clearly bite me:

TL;DR

What does all this teach us?

While the MySQL version is important, for RBR what matters most is the actual current format for each column.  Your master and slave(s) MUST have the same column formats for RBR to work right.

So, the new temporal formats do not necessarily break RBR replication back to 5.5, provided:

  • All base MySQL 5.6 enhancements to replication are disabled (GTIDs, binlog checksums and the RBR v2 format)
  • Tables with temporal formats are preserved in their 5.5 formats until all 5.5 nodes are retired.
  • You can avoid creating any new tables on the MySQL 5.6 master with temporal formats

However, I want  to make it clear that MySQL 5.6 to 5.5 replication is technically unsupported.  I have not exhausted all possibilities for problems with 5.6 to 5.5 RBR replication, just this specific one. If you choose to make an upgrade strategy that relies on backwards replication in this way, be prepared for it to not work and test it thoroughly in advance.  The purpose of this post is to simply point out that data type formats, in and of themselves, do not necessarily break RBR backwards compatibility.

About Jay Janssen

Jay joined Percona in 2011 after 7 years at Yahoo working in a variety of fields including High Availability architectures, MySQL training, tool building, global server load balancing, multi-datacenter environments, operationalization, and monitoring. He holds a B.S. of Computer Science from Rochester Institute of Technology.

Comments

  1. Some related bugs:
    http://bugs.mysql.com/bug.php?id=72522 Replication compatibility of 5.5/5.6 should mention checksums etc.
    http://bugs.mysql.com/bug.php?id=72521 Upgrade of timestamp columns
    http://bugs.mysql.com/bug.php?id=70124 Timestamp and datetimes self-incompatible during replication
    http://bugs.mysql.com/bug.php?id=71111 strange replication broken on slave

    If you enable checksums, gtid or binlog-rows-query-log-events then replication from 5.6 to 5.5 will also fail.
    http://code.openark.org/blog/mysql/when-you-cannot-replicate-from-master-m-to-slave-s

  2. Robert says:

    Slightly OT, but: what about MySQL 5.6 -> MySQL 5.6 replication? will this work with different temporal types?

    E.g. when I add a column to a table on the slave, all existing datetime fields in this table will be converted to the new format in recent versions of MySQL. The format on the master will of course stay the same. With previous versions of MySQL this led to “Column … cannot be converted from type ‘datetime’ to ‘datetime’” errors.

    The reason why I’d want to alter a table on the slave in the first place: the use case here are tables that are too huge to do an ALTER on the master – even with online DDL. The strategy here would be to the ALTER on the slave first, do a master switch, then ALTER the former master.

    Do you know if this will break replication? Or is there datetime to datetime conversion in recent MySQL versions?

  3. @Daniel: thanks for the info. GTID is another one of course and I’ve updated the post to include it.

    @Robert: Good question — I didn’t test it directly, but I’m pretty sure it works, otherwise 5.5 -> 5.6 replication compatibility (which should be supported irregardless of column formats) is violated.

  4. Robert says:

    To follow up I ran some tests and found that:

    - replication of old type -> new type works
    - replication of new type -> old type produces the above error

    regardless of MySQL versions involved. It makes sense when one thinks about it as replication of new to old temporal type might loose microsecond precision.

    For my scenario that means that replication will break as soon as the master switch is done (the new master cannot replicate anymore to the former master). Running a hotcopy to set up a fresh slave will solve this of course.

Speak Your Mind

*