While working on a recent support issue as a Percona Support Engineer, I got one question from a customer asking how to monitor ALTER TABLE progress. Actually, for MySQL 5.5 and prior versions, it’s quite difficult to ALTER the table in a running production environment especially for large tables (with millions of records). Because it will rebuild and lock the table affecting the performance as well as our users. Therefore even if we start ALTER it’s really important to know when it will finish. Even while creating the index, ALTER TABLE will not rebuild the table if fast_index_creation is ON but still it might lock the table.
fast_index_creation feature was introduced in MySQL 5.5 and higher versions. Also available in MySQL 5.1 with the InnoDB Plugin
From MySQL 5.6 and later, “Online DDL” feature has been introduced, which is enhancing many other types of ALTER TABLE operations to avoid “copying the table” and “locking.” It also allows SELECT, INSERT, UPDATE, and DELETE statements to run while the table is being altered. So in the latest version, we can control the copy of file and locking by using ALGORITHM and LOCK options. But even in MySQL 5.6, there are some ALTER operations which still required table rebuild i.e ADD/DROP column, change the data type, ADD/DROP primary key, etc. You can check the table here for more details.
“Summary of Online Status for DDL Operations” http://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-overview.html
So in any case, if required, we can check the ALTER TABLE progress with below solutions.
One of the solutions is to use the Percona Toolkit utility, pt-online-schema-change which will ALTER the table without locking it for a long time and show the progress. i.e
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
nilnandan@Dell-XPS:~$ pt-online-schema-change --alter "ADD COLUMN phone INT" u=root,p=root,D=nil,t=nil_test --execute No slaves found. See --recursion-method if host Dell-XPS has slaves. Not checking slave lag because no slaves were found and --check-slave-lag was not specified. Operation, tries, wait: copy_rows, 10, 0.25 create_triggers, 10, 1 drop_triggers, 10, 1 swap_tables, 10, 1 update_foreign_keys, 10, 1 Altering `nil`.`nil_test`... Creating new table... Created new table nil._nil_test_new OK. Altering new table... Altered `nil`.`_nil_test_new` OK. 2014-02-07T12:20:54 Creating triggers... 2014-02-07T12:20:54 Created triggers OK. 2014-02-07T12:20:54 Copying approximately 12583349 rows... Copying `nil`.`nil_test`: 29% 01:12 remain Copying `nil`.`nil_test`: 60% 00:38 remain Copying `nil`.`nil_test`: 91% 00:08 remain 2014-02-07T12:22:33 Copied rows OK. 2014-02-07T12:22:33 Swapping tables... 2014-02-07T12:22:33 Swapped original and new tables OK. 2014-02-07T12:22:33 Dropping old table... 2014-02-07T12:22:33 Dropped old table `nil`.`_nil_test_old` OK. 2014-02-07T12:22:33 Dropping triggers... 2014-02-07T12:22:33 Dropped triggers OK. Successfully altered `nil`.`nil_test`. nilnandan@Dell-XPS:~$ |
While using pt-online-schema-change, many times I get questions such as: “What will happen to those data changes (INSERT/UPDATE/DELETE) which are performing during the ALTER TABLE?”
Here, I would like to explain briefly about it. While running pt-online-schema-change, if we will check data dir,
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
root@Dell-XPS:/var/lib/mysql/nil# ll total 830524 drwx------ 2 mysql mysql 4096 Feb 7 12:20 ./ drwxr-xr-x 5 mysql mysql 4096 Feb 7 12:05 ../ -rw-rw---- 1 mysql mysql 65 Jan 31 12:12 db.opt -rw-rw---- 1 mysql mysql 8616 Feb 7 12:06 nil_test.frm -rw-rw---- 1 mysql mysql 822083584 Feb 7 12:18 nil_test.ibd -rw-rw---- 1 mysql mysql 8648 Feb 7 12:20 _nil_test_new.frm -rw-rw---- 1 mysql mysql 28311552 Feb 7 12:20 _nil_test_new.ibd -rw-rw---- 1 mysql mysql 944 Feb 7 12:20 nil_test.TRG -rw-rw---- 1 mysql mysql 40 Feb 7 12:20 pt_osc_nil_nil_test_del.TRN -rw-rw---- 1 mysql mysql 40 Feb 7 12:20 pt_osc_nil_nil_test_ins.TRN -rw-rw---- 1 mysql mysql 40 Feb 7 12:20 pt_osc_nil_nil_test_upd.TRN |
We can see that it will create triggers (separate trigger for INSERT, UPDATE and DELETE) on the original table to update corresponding rows to the new table( _nil_test_new). So any modifications happened to the data in original tables during the copy, will be reflected in the new table.
NOTE: This tool will not work if any triggers are already defined on the table.
But what if we don’t want to use pt-online-schema-change and run regular ALTER TABLE on MySQL prompt? After some research, I found many ways to calculate the progress of ALTER TABLE especially with innodb_file_per_table is ON. (innodb_file_per_table is ON by default in MySQL 5.6.)
- Calculate the progress by checking the size of temporary tablespace.
With innodb_file_per_table=1,when we are running ALTER TABLE, InnoDB creates a temporary tablespace within the same data directory with random name starting from #sql ending with .ibd like #sql-1c80_27.ibd. i.e
|
1 2 3 4 5 6 |
root@Dell-XPS:/var/lib/mysql/nil# ll ... -rw-rw---- 1 mysql mysql 8682 Feb 7 13:33 nil_test.frm -rw-rw---- 1 mysql mysql 335544320 Feb 7 13:34 nil_test.ibd -rw-rw---- 1 mysql mysql 8716 Feb 7 13:35 #sql-1c80_27.frm -rw-rw---- 1 mysql mysql 23068672 Feb 7 13:35 #sql-1c80_27.ibd |
While altering table, InnoDB reads original ibd file like nil_test.ibd and writes new pages to #sql-1c80_27.ibd. So with file size of nil_test.ibd and temporary #sql-1c80_27.ibd, we can check the ALTER TABLE progress with something like,
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
#!/bin/bash while true do A=`du -m '#sql'*'.ibd' 2>/dev/null|cut -f 1`; # if $A -lt 0 ; if [[ -z "$A" ]] ; then echo "Done"; exit 0 ; fi TABLENAME='nil_test'; TT=$TABLENAME.ibd; B=`du -m $TT |cut -f 1`; echo "ALTER TABLE $TABLENAME...$A MB written to tmp tablespace out of $B MB"; sleep 10 done |
When we run ALTER on mysql, we can simply run this script in data dir and check the progress like,
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
mysql> ALTER TABLE nil_test ADD COLUMN phone int; Query OK, 7582912 rows affected (58.54 sec) Records: 7582912 Duplicates: 0 Warnings: 0 root@Dell-XPS:/var/lib/mysql/nil# ./alter_table_monitor.sh ALTER TABLE nil_test...23 MB written in tmp file out of 485 MB ALTER TABLE nil_test...73 MB written in tmp file out of 485 MB ALTER TABLE nil_test...121 MB written in tmp file out of 485 MB ALTER TABLE nil_test...173 MB written in tmp file out of 485 MB ALTER TABLE nil_test...225 MB |