In this blog post, we’ll look at how the command line tool autotokubackup can make TokuDB hot backups easier.
I would like to share an experimental tool named autotokubackup, for TokuBackup. This tool is aimed at helping simplify the life of TokuDB system administrators. autotokubackup is written in the Python language.
General information:
So why would you need this tool? Let’s clarify a bit what you might face while using tokubackup. You have a backup solution that you can use from the MySQL shell:
|
1 |
mysql > set tokudb_backup_dir='/var/lib/tokubackupdir';<br> |
Now you want to automate this process. The first problem is that the second backup will fail, because it’s required that the backup directory is empty before starting a backup process. One solution is to create time-stamped directories and for the backups.
Further, you have a backup policy that requires some other necessary files are copied as part of your backup process. You need to write a script to put those files into a separate folder under backup directory.
Another issue you will face is the lack of any clear output on backup progress. The shell just pauses until the backup completes. The one possible way to obtain information about the backup process is displaying the MySQL processlist in a separate MySQL shell. But it isn’t the best way, and there are some issues, as reported here: Unclear status information of backup state while taking backups using TokuBackup.
Generally, we need to know which files are backed up during the backup process. There should also be a clear message indicating the end of the backup process.
To make your life easier, the autotokubackup tool:
To start, we only need two things:
To install the tool you can use following methods:
* From source:
|
1 |
cd /home<br>git clone https://github.com/Percona-Lab/autotokubackup.git<br>cd autotokubackup<br>python3 setup.py install<br> |
* or via pip3:
|
1 |
pip3 install autotokubackup<br> |
The result will be something like:
|
1 |
Collecting autotokubackup<br> Downloading autotokubackup-1.1-py3-none-any.whl<br>Collecting watchdog>=0.8.3 (from autotokubackup)<br> Downloading watchdog-0.8.3.tar.gz (83kB)<br> 100% |████████████████████████████████| 92kB 8.2MB/s <br>Collecting click>=3.3 (from autotokubackup)<br> Downloading click-6.7-py2.py3-none-any.whl (71kB)<br> 100% |████████████████████████████████| 71kB 10.6MB/s <br>Collecting mysql-connector>=2.0.2 (from autotokubackup)<br> Downloading mysql-connector-2.1.4.zip (355kB)<br> 100% |████████████████████████████████| 358kB 4.7MB/s <br>Collecting PyYAML>=3.10 (from watchdog>=0.8.3->autotokubackup)<br> Downloading PyYAML-3.12.tar.gz (253kB)<br> 100% |████████████████████████████████| 256kB 6.5MB/s <br>Collecting argh>=0.24.1 (from watchdog>=0.8.3->autotokubackup)<br> Downloading argh-0.26.2-py2.py3-none-any.whl<br>Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->autotokubackup)<br> Downloading pathtools-0.1.2.tar.gz<br><br> |
After that there should be a configuration file for this tool located -> /etc/tokubackup.conf.
The structure of the config file is:
|
1 |
[MySQL]<br>mysql=/usr/bin/mysql<br>user=root<br>password=12345<br>port=3306<br>socket=/var/run/mysqld/mysqld.sock<br>host=localhost<br>datadir=/var/lib/mysql<br><br><br>[Backup]<br>backupdir=/var/lib/tokubackupdir<br><br>[Copy]<br># The following copy_file_x options allow you to copy various files together with your backup<br># Highly recommended; a copy of your my.cnf file (usually /etc/my.cnf) and any cnf files referenced from it (i.e. includedir etc.)<br># You can also include other files you would like to take a copy of, like for example a text report or the mysqld error log<br># copy_file_1=<br># copy_file_2=<br># copy_file_...=<br># copy_file_10=<br><br>#copy_file_1=/etc/my.cnf<br>#copy_file_2=/var/log/messages<br>#copy_file_3=<br>#copy_file_4=<br>#copy_file_5=<br>#copy_file_6=<br>#copy_file_7=<br>#copy_file_8=<br>#copy_file_9=<br>#copy_file_10=<br> |
You can change options to reflect your environment and start to use. Available command line options for the tool can be displayed using --help
|
1 |
tokubackup --help<br>Usage: tokubackup [OPTIONS]<br><br>Options:<br>--backup Take full backup using TokuBackup.<br>--version Version information.<br>--defaults_file TEXT Read options from the given file<br>--help Show this message and exit.<br> |
You can prepare different config files. For example, one for the slave. Specify using the –defaults_file option, and the overall result of the run should be something like the below:
|
1 |
tokubackup --backup --defaults_file=/etc/tokubackup_node2.conf <br>Backup will be stored in /var/lib/tokubackupdir/2017-02-09_20-25-40<br>Running backup command => /home/sh/percona-server/5.7.17/bin/mysql -uroot --password=msandbox --host=localhost --socket=/tmp/mysql_sandbox20194.sock -e set tokudb_backup_dir='/var/lib/tokubackupdir/2017-02-09_20-25-40'<br>mysql: [Warning] Using a password on the command line interface can be insecure.<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_data<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_logs<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_temp<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/log000000000006.tokulog29<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.rollback<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.environment<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.directory<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tc.log<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/client-key.pem<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-cert.pem<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-key.pem<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca.pem<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca-key.pem<br>Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/auto.cnf<br>Completed - OK<br><br> |
The backup directory will store the following:
|
1 |
ls -l 2017-02-09_20-25-40/<br><br> copied_files - Directory for copied files.<br> global_variables - File for MySQL global variables. <br> mysql_data_dir - Directory for copied MySQL datadir.<br> session_variables - File for MySQL session variables. <br> tokubackup_binlog_info - File for storing binary log position.(The new feature for TokuBackup) [Not released yet]<br> tokubackup_slave_info - File for storing slave info.(The new feature for TokuBackup) [Not released yet]<br> |
That’s it. If you test it and find bugs, send a feature request to further improve our “helper.” Thanks! 🙂
Resources
RELATED POSTS
