Innodb vs MySQL index counts
I had a customer recently who a few strange errors in their mysqld.err log:
[ERROR] Table database_name/table_name contains 8 indexes inside InnoDB, which is different from the number of indexes 7 defined in the MySQL
This customer was running Percona Server 5.1 and they got this error on two tables during a maintenance window when they were adding indexes to the same tables. We had a suspicion that it had something to do with Fast index creation in Innodb, and that it had been corrected when the ALTER TABLE completed because the errors had not recurred.
Reproducing the error on a test system is simple:
- create an Innodb table
- make a copy of the .frm file
- do an ALTER TABLE to add an index
- then copy the old .frm file back into place
- re-open the table (Might need a FLUSH TABLES or mysqld restart here)
From my testing, I saw that the error only happened when the table was opened and not on every table access. So, it was a possibility that the indexes were out of sync and we weren’t seeing new errors in the log simply because the table hadn’t been re-opened.
But, before getting too crazy, how can we verify the problem still exists? We need a way to compare the output of SHOW CREATE TABLE to what Innodb thinks. What Innodb thinks is in the Innodb Data dictionary.
- The first recommendation I got was to simply use the INFORMATION_SCHEMA.INNODB_SYS_INDEXES table, which exists in Percona Server 5.1, but doesn’t appear in MySQL until 5.6 (if the manual is to be trusted). I’d probably consider this on a newer version of Percona Server or MysqL 5.6.
- Another person (I’m looking at you, Baron) was adverse to trusting INNODB_SYS_INDEXES from some bad experiences with it, and suggested the Innodb Table monitor instead, see my next post for how that turned out, but this basically will regurgitate the entire Innodb Data dictionary to the mysqld error log file.
- If I had to do it over again, I think I’d simply try doing: FLUSH TABLES table1, table2; to force the tables to close and be reopened and simply see if the error message comes back. That might something of a performance impact, but it seems to be the most stable.
In this case, it turned out that the indexes were not out of sync, so I didn’t have to do anything to fix it.
However if I did have to fix it, I found on my test table that the extra index in Innodb could be removed by doing:
ALTER TABLE table_name ENGINE=Innodb;
This, of course, rebuilds the whole table based on the .frm table definition and removes the existing index in Innodb, which might not be desirable, but at least you can re-add it later. However, it’s not the greatest thing to do on a live production database master if you can help it.
Another solution might be to figure out what index was missing via the Innodb data dictionary (more on that in a minute), create a separate table identical to the existing .frm, add that index to it, and copy the new .frm back over the original. Kind of scary.
My advice is to ensure the error still exists before trying to fix it.