October 1, 2014

Linux IO Schedulers and MySQL

Found a great article about Linux IO Schedulers today which is quite interesting. It goes in details about schedulers and explains in which of workloads which of schedulers is best.

The interesting thing this article points out is – there are multiple versions of each of the schedulers, while name remains the same. This means unless you really know mapping between kernel versions and scheduler versions it is very hard to evaluate benchmark results.

This could be noticed by benchmarks we’ve done over years. Long time ago “AS” scheduler could be several times slower than deadline for MySQL workloads such as SysBench or DBT2 when it went down to 30% difference and in the last runs we’ve done difference was not really significant.

This article also points out benchmarking IO schedulers you should look at more numbers than aggregate bandwidth – you also better to measure per client bandwidth as well as max latency as this is what can be the problem. Take a look at these old results for example. It also means you’d better to perform IO scheduler benchmarks on mixed load with different of task, for example mixing OLTP with some reporting queries if you really want to see the difference.

From the article it looks like CFQ should be good choice for databases and it is also found to work pretty well by some benchmarks we’ve done. The only question if it is doing as good as it could – In the docs it is mentioned it uses “per process” scheduling while MySQL is single process but single thread – does each thread gets its own queue in reality or is it shared ?

We should look into this when we’ll run more benchmarks for IO Schedulers.

About Peter Zaitsev

Peter managed the High Performance Group within MySQL until 2006, when he founded Percona. Peter has a Master's Degree in Computer Science and is an expert in database kernels, computer hardware, and application scaling.

Comments

  1. n says:

    Each thread gets its own queue. You can also do ionice per thread.

    However it doesn’t quite work for AIO which gets submitted from other threads.

    I think it’s pretty important to always measure latencies too, not just aggregate throughput

  2. peter says:

    That is good to hear each threads gets its own queue. I remember in 2.4 times there were compatibility issues with threads and read-ahead which was tracked per process not per thread.

    Regarding ionice it is tough to use with database as there can be priority inversion due to lock held by the thread etc.

    What would be great to see implemented is per request priorities so log flushes for example would get high priority while background dirty pages flush low priority.

  3. I did some brief research on the four available schedulers and came across this page :

    http://www.redhat.com/magazine/008jun05/features/schedulers/

    Redhat also claims that the CFQ scheduler is the best overall for database applications. But it also looks like deadline may be an option as well. Am I correct in stating that the deadline scheduler offers better long-term performance compared to CFQ? Or am I misreading that graph?

    At any rate, the scheduler can be selected on the fly for many distributions by changing the value of /sys/block//queue/scheduler to one of the four scheduler names. In Fedora, if you cat this file it will display the four available schedulers and the current one is listed in brackets.

    Redhat Enterprise, and thus CentOS, only allow scheduler changes on bootup. This stems from an older 2.6 kernel being used.

    Fun stuff. Worth running some benchmarks on…

  4. The comment system happily ate part of the filename I posted above.. it should be :

    /sys/block/DEVICENAME/queue/scheduler

    And example would be something like :

    /sys/block/hda/queue/scheduler

  5. peter says:

    Yes, I’ve seen that paper. The problem is it is rather old… the IO schedulers are changing quickly as you can read in original article.

    Yeah it is a pity CentOS/RHEL only allow scheduler to be changed on boot. I think it is just because it is based on older kernel. version 5 hopefully will not have this problem.

    I guess CFQ vs Deadline depends on applications I’ve seen mixed results.

  6. veridicus says:

    Very interesting article. Thanks for pointing us to it!

Speak Your Mind

*