As you might have seen MySQL QA Team has published their benchmarks for MySQL 5.0.72 and 5.1.30.
It is interesting to compare with results I posted previously
The quote from the results mentioned above:
“Maybe youâ€™ve seen some claims by others in the MySQL community that MySQL 5.1 runs slower than MySQL 5.0. Maybe youâ€™ve also seen some claims by others in the MySQL community that MySQL 5.1 runs faster than MySQL 5.0.
Guess what? Theyâ€™re both right. “
But is it really what results are telling us ?
I do not think so. When you’re doing benchmarks you should be comparing best performance settings for given application and conditions. For example it is unfair to compare results with different innodb_buffer_pool_size or innodb_flush_log_at_trx_commit but for innodb_thread_concurrency – you should be picking the value which makes your workload to run fastest.
Lets look at the graphs provided in these benchmarks and see what value is best for MySQL 5.1 and for MySQL 5.0 respectively ?
Both versions do their best with innodb_thread_concurrency=0 and 5.0 is slightly but consistently faster. Same as in my results.
So I would interpret these results differently.
MySQL 5.0 is faster if you configure it right. If you configure it wrong the regression is going to be worse than for MySQL 5.1.
You can’t really use this results to tell MySQL 5.1 will be winner in cases when small values of innodb_thread_concurrency get best performance. The things can be completely different in this case. They may or may not, there is simply no data at all.
It is also very interesting to see benchmark run with innodb_thread_concurrency=1000 – This is exactly the value which you should never use. The limit of 1000 threads inside the kernel is by far too large (so it is same as 0 – unlimited) but it will add another mutex to deal with for the queue implementation.
What would be really interesting to learn though is why MySQL 5.1 gets so better when threads get queued up, what kind of changes in MySQL 5.0 result in this behavior. It also would be good to run profiling for these results to see where these few percent are lost for MySQL 5.0 to see if they are possible to be reclaimed.
Finally – if I would be doing QA, and benchmarks as part of them I would try to use options which are close to what people would use in production. Or at least explain why they are set so. Is it because 5.1 does not show too good results with standard settings or is it just omission ? Again quite possibly nothing will change but may be not.
In particular: innodb_support_xa=0, innodb_doublewrite=0 are not normally used in production and they do add some overhead.
When looking for results more relevant for production I also would keep binary log enabled – most big installations of MySQL use replication or at least binary log to get point in time recovery from backup. Also Innodb log files of 2*650M are larger than practical for most applications because of too large recovery time.
Anyways. It is great to see MySQL QA Team has published some benchmarks now and I can’t wait to see more. If we have data and the good disclosure (settings, versions, hardware) as we have in this case we can make our own mind of results and draw our own conclusions.
Also indeed it is a good time to try MySQL 5.1 for your environment. If you spot any regressions from MySQL 5.0 it will likely take a time to get them fixed.