October 23, 2014

PBXT in tpcc-like benchmark

Finally I was able to run PBXT 1.0.11 pre-GA in tpcc-like workload, apparently there was bug with did not allow me to get the result earlier, and I am happy to see that PBXT team managed it.

For initial runs I took tpcc 100 warehouses ( about 10GB of data) which fully fits into memory (32 GB on server),
and compared 1 and 16 users in MySQL-5.1.46/PBXT and Percona Server / XtraDB – 5.1.45-rel10.2. As workload is totally memory based it will show how PBXT scales in CPU-bond cases on 16 cores systems.

As storage system it was Intel SSD X25-M card.

While full results and config are on Wiki:
http://www.percona.com/docs/wiki/benchmark:pbxt:tpcc:start

there are graphs for 1 user:

and 16 users:

Interesting to see that in case with 1 user the maximal throughput in PBXT is about 1.5x better XtraDB, but
there periodical drops which are very similar to periodical drops in InnoDB without adaptive checkpointing, and I guess it is also related to checkpoint activity.
The final results are also better for PBXT: 5785.567 TpmC vs 4905.967 TpmC ( XtraDB)

For 16 threads final result is: 26129.350 TpmC for PBXT and 29485.518 TpmC for XtraDB , and from the graph you can see that the maximal throughput is about identical, while PBXT spends more time in drops area. Again it looks like PBXT are not fully keeping up with checkpoint activity and I am looking PBXT addresses this problem also. Beside this issue PBXT looks pretty good and in next round I am going to run IO intensive workloads.

About Vadim Tkachenko

Vadim leads Percona's development group, which produces Percona Clould Tools, the Percona Server, Percona XraDB Cluster and Percona XtraBackup. He is an expert in solid-state storage, and has helped many hardware and software providers succeed in the MySQL market.

Comments

  1. peter says:

    Vadim,

    What is about XtraDB outliers 16 threads ? I see there seems to be some dips with multiple dots at regular intervals and there is also bursts of higher transaction rates after that. Do you have any explanation for it ?

    It would be also nice to see what is 99 percentile response time or something in both cases. The PBXT results for 16 threads seems to have a lot higher variance with some dots being on 0 all together.

  2. I’m very happy we’ve been able to benchmark PBXT. Ditto what Peter says — the
    XtraDB outliers look like something newly revealed with this benchmark? PBXT’s
    performance seems rather unstable over time in these graphs. I looked at the
    raw stats and got the 95th percentile and 99th percentile R and stddev(R)
    (assuming R = 10 / X, which I know is not really true, but…) Here’s what I
    got for the single-threaded case:

    And for the 16-thread case,

  3. Andy says:

    Vadim,

    Thanks for the benchmark.

    How do PBXT’s table size compared to XtraDB’s — are they generally bigger or smaller for the same data?

    For your IO bound tests, I’d be interested in seeing the impact of XtraDB’s data compression. For example comparing PBXT, uncoompressed XtraDB with compressed XtraDB to see if compression brings any performance improvements.

  4. Sorry, I forgot to sort the columns in my spreadsheet before doing the 16-thread numbers… let me fix that and post new ones.

  5. Here are revised stats for the 16-thread case.

  6. peter says:

    Thanks Baron,

    I Start to like 99th percentile a lot better than 95 these days. As long as you have fair number of events 1000+ it is a lot close to what you need. Being slow every 20th request which 95th allows is not OK these days.

  7. Dimitri says:

    Vadim,

    Thank you to share it!

    I’m curious also about your observations: was I/O load higher on PBXT vs XtraDB? if yes, how much?

    Did you try other PBXT settings/tuning? did Paul work with you here? may we consider the current PBXT configuration setting as optimal? any other observation you may to share?..

    I was also surprised to see some TPS drops on XtraDB – why it happens? – if I remember well there was no TPS drops before during TPCC-like tests..

    Thanks again!

    Rgds,
    -Dimitri

  8. Dimitri says:

    Vadim,

    Thank you to share it!

    I’m curious also about your observations: was I/O load higher on PBXT vs XtraDB? if yes, how much?

    Did you try other PBXT settings/tuning? did Paul work with you here? may we consider the current PBXT configuration setting as optimal? any other observation you may to share?..

    I was also surprised to see some TPS drops on XtraDB on 16 users – why it happens? – if I remember well there was no TPS drops before during TPCC-like tests..

    Thanks again!

    Rgds,
    -Dimitri

  9. Vadim says:

    Dimitri,

    PBXT settings were recommended by Paul, but I am ready to make another run with different tunings.

    I did not gather io stats this time, I need to figure out how to make dim_STAT publish graphs automatically :), may be we should talk and you share some wisdom, I have couple questions.

    XtraDB has drops at the warmup stage, and it is regular case ( maybe we will fix it sometime). After warmup the
    results are stable.. beside periodically outliers, which I am not sure how to explain yet.

  10. Dimitri says:

    Hi Vadim,

    Will be great to see with PBXT a sort of guide of configuration settings advices :-)
    As it’ll probably vary depending on workload – will be nice to avoid any know issues for those who will try to test it too..

    No problem with any help with dim_STAT – and I’m happy you’re trying to adapt it :-)
    It may probably also clarify TPS drops on XtraDB..

    BTW, seems there are some issues with comments on the site – mine was posted twice, and I did not receive any email notifications about your answer..

    Rgds,
    -Dimitri

Speak Your Mind

*