Making backups over the network can be done in two ways: either save on disk and transfer or just transfer without saving. Both ways have their strong and weak points. The second way, particularly, is highly dependent on the upload speed, which would either reduce or increase the backup time. Other factors that influence it are chunk size and the number of upload threads.
Percona XtraBackup 2.4.14 has gained S3 streaming, which is the capability to upload backups directly to s3-compatible storage without saving locally first. This feature was developed because we wanted to improve the upload speeds of backups in Percona Operator for XtraDB Cluster.
There are many implementations of S3 Compatible Storage: AWS S3, Google Cloud Storage, Digital Ocean Spaces, Alibaba Cloud OSS, MinIO, and Wasabi.
We’ve measured the speed of AWS CLI, gsutil, MinIO client, rclone, gof3r and the xbcloud tool (part of Percona XtraBackup) on AWS (in single and multi-region setups) and on Google Cloud. XtraBackup was compared in two variants: a default configuration and one with tuned chunk size and amount of uploading threads.
Here are the results.
The backup data was streamed from the AWS EC2 instance to the AWS S3, both in the us-east-1 region.
| tool | settings | CPU | max mem | speed | speed comparison |
|---|---|---|---|---|---|
| AWS CLI | default settings | 66% | 149Mb | 130MiB/s | baseline |
| AWS CLI | 10Mb block, 16 threads | 68% | 169Mb | 141MiB/s | +8% |
| MinIO client | not changeable | 10% | 679Mb | 59MiB/s | -55% |
| rclone rcat | not changeable | 102% | 7138Mb | 139MiB/s | +7% |
| gof3r | default settings | 69% | 252Mb | 97MiB/s | -25% |
| gof3r | 10Mb block, 16 threads | 77% | 520Mb | 108MiB/s | -17% |
| xbcloud | default settings | 10% | 96Mb | 25MiB/s | -81% |
| xbcloud | 10Mb block, 16 threads | 60% | 185Mb | 134MiB/s | +3% |
Tip: If you run MySQL on an EC2 instance to make backups inside one region, do snapshots instead.
The backup data was streamed from AWS EC2 in us-east-1 to AWS S3 in eu-central-1.
| tool | settings | CPU | max mem | speed | speed comparison |
|---|---|---|---|---|---|
| AWS CLI | default settings | 31% | 149Mb | 61MiB/s | baseline |
| AWS CLI | 10Mb block, 16 threads | 33% | 169Mb | 66MiB/s | +8% |
| MinIO client | not changeable | 3% | 679Mb | 20MiB/s | -67% |
| rclone rcat | not changeable | 55% | 9307Mb | 77MiB/s | +26% |
| gof3r | default settings | 69% | 252Mb | 97MiB/s | +59% |
| gof3r | 10Mb block, 16 threads | 77% | 520Mb | 108MiB/s | +77% |
| xbcloud | default settings | 4% | 96Mb | 10MiB/s | -84% |
| xbcloud | 10Mb block, 16 threads | 59% | 417Mb | 123MiB/s | +101% |
Tip: Think about disaster recovery, and what will you do when the whole region is not available. It makes no sense to back up to the same region; always transfer backups to another region.
The backup data were streamed from Compute Engine instance in us-east1 to Cloud Storage europe-west3. Interestingly, Google Cloud Storage supports both native protocol and S3(interoperability) API. So, Percona XtraBackup can transfer data to Google Cloud Storage directly via S3(interoperability) API.
| tool | settings | CPU | max mem | speed | speed comparison |
|---|---|---|---|---|---|
| gsutil | not changeable, native protocol | 8% | 246Mb | 23MiB/s | etalon |
| rclone rcat | not changeable, native protocol | 6% | 61Mb | 16MiB/s | -30% |
| xbcloud | default settings, s3 protocol | 3% | 97Mb | 9MiB/s | -61% |
| xbcloud | 10Mb block, 16 threads, s3 protocol | 50% | 417Mb | 133MiB/s | +478% |
Tip: A cloud provider can block your account due to many reasons, such as human or robot mistakes, inappropriate content abuse after hacking, credit card expire, sanctions, etc. Think about disaster recovery and what will you do when a cloud provider blocks your account, it may make sense to back up to another cloud provider or on-premise.
xbcloud tool (part of Percona XtraBackup) is 2-5 times faster with tuned settings on long-distance with native cloud vendor tools, and 14% faster and requires 20% less memory than analogs with the same settings. Also, xbcloud is the most reliable tool for transferring backups to S3-compatible storage because of two reasons:
.md5/filename.md5 file and verifies sums on the download (gof3r does the same).
PS: Please find instructions on GitHub if you would like to reproduce this article’s results.
Resources
RELATED POSTS
