How to Deal with XA Transactions Recovery

XA TransactionsFor most people (including me until recently) database XA transactions are a fuzzy concept. In over eight years with Percona, I have never had to deal with XA transactions. Then a few weeks ago I got two customers having issues with XA transactions. That deserves a post.

XA 101

What are XA transactions? XA transactions are useful when you need to coordinate a transaction between different systems. The simplest example could be simply two storage engines within MySQL. Basically, it follows this sequence:

  2. Some SQL statements
  3. XA END

Once prepared, the XA transaction survives a MySQL crash. Upon restart, you’ll see something like this in the MySQL error log:

The command xa recover shows you an output like:

There are some binary data that can’t be shown in HTML. The XA Xid is made of three fields: gtrid (global trx id), bqual (branch qualifier) and formatId. Java applications use all three fields. For my example above, I used “X’01020304′,’bqual’,1234”. You can trust Java application servers to be creative with Xid values. With MySQL 5.7, you can output the data part in hex with convert xid :

The Problem

If you do nothing, the prepared transaction stays there forever and holds locks and a read view open. As a consequence, the history list grows without bound along with your ibdata1 file, where the undo entries are kept. If you have slaves, they all have the prepared transaction too (at least with 5.7). No fun.

As a consequence, if you are using XA transactions, you MUST check if there are prepared transactions pending after the server or mysqld restarted. If you find such transactions, you need to commit or roll them back, depending on what is involved.

But how do you commit these XA transactions? The problem here is the output of xa recover. As it is, the output is unusable if there is a bqual field or non-default formatID field:

The Fix

Looking back at the xa recover convert xid output above, the gtrid_length and bqual_length are provided. With the use of these values, you can extract the parts of the data field which gives us:

  • gtrid = 0x01020304
  • bqual = 0x627175616C

And, of course, the formatID is 1234. Altogether, we have:

Which finally works! On 5.6 the convert xid option is not available. You have to be a bit more creative:

But there is a limitation in 5.6: you can only XA commit/rollback transactions that belong to your session. That means after a crash you are out of luck. To get rid of these you need to promote a slave or perform a logical dump and restore. The best plan is to avoid the use of XA transactions with 5.6.

I submitted this bug to Percona Server for MySQL in order to get a usable output out of xa recover convert xid. If you think this is important, vote for it!

Share this post

Comments (6)

  • sbester September 22, 2017 at 3:23 pm
  • Wagner Bianchi

    Good article for someone that suffered with XA before, thanks for writing it, Yves! Cheers!

    September 26, 2017 at 6:26 pm
  • Bingi

    I am getting error even after splitting based on the length

    mysql> XA COMMIT 0x7023010095EDCC9A6369736861322C7365727665722C503130,0x306369736861322C7365727665722C503130302C00,4871251;
    ERROR 1397 (XAE04): XAER_NOTA: Unknown XID

    April 10, 2018 at 7:22 pm
    • Paul Cooper

      Looks like you’re missing the 3rd parameter: formatId (get it from “xa recover convert xid”)

      February 20, 2020 at 12:54 pm
      • Paul Cooper

        Nope, sorry, I lied… you have the 3 parameters, just much longer than the ones I’ve seen. Not sure what the problem is.

        February 20, 2020 at 12:56 pm
  • Paul Cooper

    Thanks for this article… helped a lot.

    February 20, 2020 at 12:55 pm

Comments are closed.

Use Percona's Technical Forum to ask any follow-up questions on this blog topic.