Faster Streaming Backups – Introducing Percona XtraBackup FIFO Parallel Stream

When it comes to backups, there are several options for saving backup files. You can choose to save them locally on the same server, stream them to different servers, or store them in object storage. Percona XtraBackup facilitates streaming through the use of an auxiliary tool called xbcloud.

STDOUT Datasink

This diagram displays the process of streaming a backup to object storage utilizing the current STDOUT datasink:

• XtraBackup spawns multiple copy threads. Each one will be reading a chunk of data from a specific file.
• Each copy thread will write the chunk of data to a pipe (aka STDOUT).
• Xbcloud will have a red thread that will be responsible for reading each chunk of data from STDIN. This chunk will be uploaded to object storage utilizing an async request, and a callback will be added to an event handler list.
• The event handler thread of xbcloud will be responsible for checking when the TCP socket has received the response from the object storage and executing the callback depending on the return (success or failure).

When testing the speed of XtraBackup streaming, we can see a limitation of around 1.8 Gbps. This architecture works fine when uploading data via WAN (we mostly will not have this speed over WAN). However, needing to stream backups over the LAN has become more popular, and 10Gbps or more is the standard setup.

After careful consideration of the above architecture, it has become clear that even though xbcloud can act super fast reading from STDIN, doing an asynchronous operation to Object Storage, and delegating the process to wait for the response to the event handler worker thread, there is only one thread on the XtraBackup side that can be writing at the same time to STDOUT. This is done via a mutex to ensure a chunk of data is written from start to finish. Otherwise, we would have the interference of other copy thread data being appended to STDOUT while the other thread has not completed its initial chunk.

FIFO Datasink

Percona XtraBackup 8.0.33-28 introduces a new datasink for true parallel streaming. This new datasink utilizes FIFO (named pipes) to achieve true parallelism to stream data from XtraBackup to xbcloud. Extending the previous diagram, the new data stream will look like below:

• Multiple FIFO files will be created, which will be responsible for storing the data from XtraBackup to xbcloud.
• Each XtraBackup copy thread will be bound to a FIFO file in a round-robin fashion.
•  Xbcloud will have multiple read threads to read from the FIFO streams.

Usage

We introduced three new parameters:

• fifo-streams=# Number of FIFO files to use for parallel datafiles stream. Setting this parameter to 1 disables FIFO, sending the stream to STDOUT. The default is kept as 1 (STDOUT) for backward compatibility.
• fifo-dir=path Directory to write Named Pipe.
• fifo-timeout=# How many seconds to wait for the other end to open the stream for reading. Default is 60 seconds

The current behavior and XtraBackup and cloud are not changed. The default value of –fifo-streams is 1, so we still have a single-line command:

Performance

To test the new FIFO datasink, we have created 1TB of data into multiple tables. The link speed between the source server and the destination server ( utilizing MinIO ) is ~9.2Gbps:

The test was performed without any load (no new redo log during the backup) to ensure both tests are both pushing 1TB of data from two servers.

The results are shown in the below graph:

• STDOUT – Utilized 239 MBps (1.8 Gbps) and took  01:25:24 to push 1TB of data.
• FIFO ( 8 pipes) – Utilized 1.15 GBps (9.2 Gbps) and took only  00:16:01 to push the same 1TB of data.

Summary

If you utilize streaming inside your local network, you can definitely benefit from faster backups by streaming data in parallel using the new FIFO datasink.

Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server for MySQL and MySQL.

Learn more about Percona XtraBackup