I would like to share an experimental tool named autotokubackup, for TokuBackup. This tool is aimed at helping simplify the life of TokuDB system administrators. autotokubackup is written in the Python language.
- If you need more information about TokuBackup, read here: TokuBackup Doc
- The project is in Github under Percona-Lab: autotokubackup
- The PyPi link for package: PyPi – autotokubackup
So why would you need this tool? Let’s clarify a bit what you might face while using tokubackup. You have a backup solution that you can use from the MySQL shell:
mysql > set tokudb_backup_dir='/var/lib/tokubackupdir';
Now you want to automate this process. The first problem is that the second backup will fail, because it’s required that the backup directory is empty before starting a backup process. One solution is to create time-stamped directories and for the backups.
Further, you have a backup policy that requires some other necessary files are copied as part of your backup process. You need to write a script to put those files into a separate folder under backup directory.
Another issue you will face is the lack of any clear output on backup progress. The shell just pauses until the backup completes. The one possible way to obtain information about the backup process is displaying the MySQL processlist in a separate MySQL shell. But it isn’t the best way, and there are some issues, as reported here: Unclear status information of backup state while taking backups using TokuBackup.
Generally, we need to know which files are backed up during the backup process. There should also be a clear message indicating the end of the backup process.
To make your life easier, the autotokubackup tool:
- Automates the TokuDB database backup procedures
- Creates timestamped backups inside the backup directory, overcoming the need for copy/remove old backups to empty the backup directory
- Copies all necessary files for your backup policy (you can specify up to ten supplementary files to be in the backup directory as part of backup process)
- Clearly describes what is going to be in the backup director, by showing newly created files inside backup directory
- Clearly shows the end of backup process
To start, we only need two things:
- Installed Percona Server with TokuDB engine + TokuBackup plugin
- Installed Python3
To install the tool you can use following methods:
* From source:
git clone https://github.com/Percona-Lab/autotokubackup.git
python3 setup.py install
* or via pip3:
pip3 install autotokubackup
The result will be something like:
Collecting watchdog>=0.8.3 (from autotokubackup)
Downloading watchdog-0.8.3.tar.gz (83kB)
100% |████████████████████████████████| 92kB 8.2MB/s
Collecting click>=3.3 (from autotokubackup)
Downloading click-6.7-py2.py3-none-any.whl (71kB)
100% |████████████████████████████████| 71kB 10.6MB/s
Collecting mysql-connector>=2.0.2 (from autotokubackup)
Downloading mysql-connector-2.1.4.zip (355kB)
100% |████████████████████████████████| 358kB 4.7MB/s
Collecting PyYAML>=3.10 (from watchdog>=0.8.3->autotokubackup)
Downloading PyYAML-3.12.tar.gz (253kB)
100% |████████████████████████████████| 256kB 6.5MB/s
Collecting argh>=0.24.1 (from watchdog>=0.8.3->autotokubackup)
Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->autotokubackup)
After that there should be a configuration file for this tool located -> /etc/tokubackup.conf.
The structure of the config file is:
# The following copy_file_x options allow you to copy various files together with your backup
# Highly recommended; a copy of your my.cnf file (usually /etc/my.cnf) and any cnf files referenced from it (i.e. includedir etc.)
# You can also include other files you would like to take a copy of, like for example a text report or the mysqld error log
You can change options to reflect your environment and start to use. Available command line options for the tool can be displayed using
Usage: tokubackup [OPTIONS]
--backup Take full backup using TokuBackup.
--version Version information.
--defaults_file TEXT Read options from the given file
--help Show this message and exit.
You can prepare different config files. For example, one for the slave. Specify using the –defaults_file option, and the overall result of the run should be something like the below:
tokubackup --backup --defaults_file=/etc/tokubackup_node2.conf
Backup will be stored in /var/lib/tokubackupdir/2017-02-09_20-25-40
mysql: [Warning] Using a password on the command line interface can be insecure.
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_data
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_logs
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_temp
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/log000000000006.tokulog29
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.rollback
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.environment
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.directory
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tc.log
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/client-key.pem
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-cert.pem
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-key.pem
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca.pem
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca-key.pem
Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/auto.cnf
Completed - OK
The backup directory will store the following:
ls -l 2017-02-09_20-25-40/
global_variables - File for MySQL global variables.
mysql_data_dir - Directory for copied MySQL datadir.
session_variables - File for MySQL session variables.
tokubackup_binlog_info - File for storing binary log position.(The new feature for TokuBackup) [Not released yet]
tokubackup_slave_info - File for storing slave info.(The new feature for TokuBackup) [Not released yet]
That’s it. If you test it and find bugs, send a feature request to further improve our “helper.” Thanks! 🙂