I had a customer recently who a few strange errors in their mysqld.err log:

This customer was running Percona Server 5.1 and they got this error on two tables during a maintenance window when they were adding indexes to the same tables.  We had a suspicion that it had something to do with Fast index creation in Innodb, and that it had been corrected when the ALTER TABLE completed because the errors had not recurred.

Reproducing the error on a test system is simple:

  1. create an Innodb table
  2. make a copy of the .frm file
  3. do an ALTER TABLE to add an index
  4. then copy the old .frm file back into place
  5. re-open the table  (Might need a FLUSH TABLES or mysqld restart here)

From my testing, I saw that the error only happened when the table was opened and not on every table access.  So, it was a possibility that the indexes were out of sync and we weren’t seeing new errors in the log simply because the table hadn’t been re-opened.

But, before getting too crazy, how can we verify the problem still exists?  We need a way to compare the output of SHOW CREATE TABLE to what Innodb thinks.  What Innodb thinks is in the Innodb Data dictionary.

  • The first recommendation I got was to simply use the INFORMATION_SCHEMA.INNODB_SYS_INDEXES table, which exists in Percona Server 5.1, but doesn’t appear in MySQL until 5.6 (if the manual is to be trusted).  I’d probably consider this on a newer version of Percona Server or MysqL 5.6.
  • Another person (I’m looking at you, Baron) was adverse to trusting INNODB_SYS_INDEXES from some bad experiences with it, and suggested the Innodb Table monitor instead, see my next post for how that turned out, but this basically will regurgitate the entire Innodb Data dictionary to the mysqld error log file.
  • If I had to do it over again, I think I’d simply try doing:  FLUSH TABLES table1, table2; to force the tables to close and be reopened and simply see if the error message comes back.  That might something of a performance impact, but it seems to be the most stable.
In this case, it turned out that the indexes were not out of sync, so I didn’t have to do anything to fix it.
However if I did have to fix it, I found on my test table that the extra index in Innodb could be removed by doing:

This, of course, rebuilds the whole table based on the .frm table definition and removes the existing index in Innodb, which might not be desirable, but at least you can re-add it later.  However, it’s not the greatest thing to do on a live production database master if you can help it.

Another solution might be to figure out what index was missing via the Innodb data dictionary (more on that in a minute), create a separate table identical to the existing .frm, add that index to it, and copy the new .frm back over the original.  Kind of scary.

My advice is to ensure the error still exists before trying to fix it.

6 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Peter Zaitsev

Jay,

I wonder what happens if old .frm file will also contain smaller number of columns. Will the recovery work in this case or
would alter table fail too ?

Peter Zaitsev

Jay,

What if you try to insert in the table when frm is in mismatch with actual table structure ?

Also what I read here is different – if for some reason you get old .frm you will have a silent data loss on ALTER TABLE – think about the case when “foo” was some important data added a month ago and when for some reason old .frm file was restored.

Might be Innodb needs more checks here.

Raghavendra

Regarding INNODB_SYS_INDEXES from IS, when reading about them, I saw a note which I remember now, that they are in sync with the disk, in other words, the dictionary on disk is the source of truth which is followed, not the memory.

PS: Got the article — http://blogs.innodb.com/wp/?p=636

“There is another benefit of Information Schema SYSTEM TABLES in that, by design, the data is read from system table directly, rather than fetching data from their in memory representations (like innodb_table_monitor does). So it is a true representation of what we have on disk. If there is an mismatch (unlikely) between the in memory information and on disk data, these system tables will display the real image on disk, and help DBA and even developer to debug or better understand the system metadata behavior.”