Increasing slow MySQL query performance with the parallel query execution

MySQL and Scaling-up (using more powerful hardware) was always a hot topic. Originally MySQL did not scale well with multiple CPUs; there were times when InnoDB performed poorer with more CPU cores than with fewer CPU cores. MySQL 5.6 can scale significantly better; however, there is still 1 big limitation: 1 SQL query will eventually use only 1 CPU core (no parallelism). Here is what I mean by that: let’s say we have a complex query which will need to scan millions of rows and may need to create a temporary table; in this case MySQL will not be able to scan the table in multiple threads (even with partitioning) so the single query will not be faster on the more powerful server. On the contrary, a server with more slower CPUs will show worse performance than the server with less (but faster) CPUs.

MySQL parallel query execution

To address this issue we can use a parallel query execution. Vadim wrote about the PHP asynchronous calls for MySQL. Another way to increase the parallelism will be to use “sharding” approach, for example with Shard Query. I’ve decided to test out the parallel (asynchronous) query execution with a relatively large table: I’ve used the US Flights Ontime performance database, which was originally used by Vadim in the old post Analyzing air traffic performance. Let’s see how this can help us increase the performance of the complex query reports.

Parallel Query Example

To illustrate the parallel query execution with MySQL I’ve created the following table:

And loaded 26 years of data into it. The table is 56G with ~152M rows.

Software: Percona 5.6.15-63.0. Hardware: Supermicro; X8DTG-D; 48G of RAM; 24xIntel(R) Xeon(R) CPU L5639 @ 2.13GHz, 1xSSD drive (250G)

So we have 24 relatively slow CPUs

Simple query

Now we can run some queries. The first query is very simple: find all flights per year (in the US):

As we have the index on YearD, the query will use the index:

The query is simple, however, it will have to scan 150M rows. Here is the results of the query (cached):

The query took 54 seconds and utilized only 1 CPU core. However, this query is perfect for running in parallel.  We can run 26 parallel queries, each will count its own year. I’ve used the following shell script to run the queries in the background:

Here are the results:

So the total execution time is ~5 (10x faster) seconds. Each individual results are here: