Percona Server for MongoDB 4.2 vs 4.4 in Python TPCC Benchmark

September 15, 2020

Author

Vadim Tkachenko

Benchmarks

Insight for DBAs

MongoDB

Share this Post:

Percona Server for MongoDB 4.2 vs. 4.4 Following my previous blogs on py-tpcc benchmark for MongoDB, Evaluating the Python TPCC MongoDB Benchmark and Evaluating MongoDB Under Python TPCC 1000W Workload, and the recent release of Percona Server for MongoDB 4.4, I wanted to evaluate 4.2 vs 4.4 in similar scenarios.

Hardware Specs

For the client and server, I will use identical bare metal servers, connected via a 10Gb network.

The node specification:

# Percona Toolkit System Summary Report ######################<br>        Date | 2020-09-14 16:52:46 UTC (local TZ: EDT -0400)<br>    Hostname | node3<br>      System | Supermicro; SYS-2028TP-HC0TR; v0123456789 (Other)<br>    Platform | Linux<br>     Release | Ubuntu 20.04.1 LTS (focal)<br>      Kernel | 5.4.0-42-generic<br>Architecture | CPU = 64-bit, OS = 64-bit<br># Processor ##################################################<br>  Processors | physical = 2, cores = 28, virtual = 56, hyperthreading = yes<br>      Models | 56xIntel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz<br>      Caches | 56x35840 KB<br># Memory #####################################################<br>       Total | 251.8G<br>  Swappiness | 0<br> DirtyPolicy | 80, 5<br> DirtyStatus | 0, 0<br>

1

The drive I used for the storage in this benchmark is a Samsung SM863 SATA SSD.

MongoDB Topology

For MongoDB I used:

Single node instance without limiting cache size. As the bare metal server has 250GB of RAM, MongoDB should allocate 125GB of memory for WiredTiger cache and the rest will be used for OS cache. This should produce more CPU bound workload.

Single node instance with limited cache size. For WiredTiger cache I will set a limit 25GB, and to limit OS cache I will limit the memory available to a mongodb instance to 50GB, as described in Using Cgroups to Limit MySQL and MongoDB memory usage.

However I did not use cgroups in this case, but I rather used Docker to run different versions and set limits.

The script to start Percona Server for MongoDB in docker with memory limits:

> bash startserver.sh 4.4 1<br>=== script startserver.sh ===<br>docker run -d --name db$2 -m 50g  <br>          -v /mnt/data/psmdb$2-$1:/data/db <br>          --net=host <br>          percona/percona-server-mongodb:$1 --replSet "rs$2" --port $(( 27016 + $2 )) <br>          --logpath /data/db/server1.log --slowms=10000 --wiredTigerCacheSizeGB=25 <br><br>sleep 10<br><br>mongo mongodb://127.0.0.1:$(( 27016 + $2 )) --eval "rs.initiate( { _id : 'rs$2',  members: [      { _id: 0, host: '172.16.0.3:$(( 27016 + $2 ))' }   ] })"<br>

1

> bash startserver.sh 4.4 1 === script startserver.sh === docker run -d --name db$2 -m 50g -v /mnt/data/psmdb$2-$1:/data/db --net=host percona/percona-server-mongodb:$1 --replSet "rs$2" --port $(( 27016 + $2 )) --logpath /data/db/server1.log --slowms=10000 --wiredTigerCacheSizeGB=25 sleep 10 mongo mongodb://127.0.0.1:$(( 27016 + $2 )) --eval "rs.initiate( { _id : 'rs$2', members: [ { _id: 0, host: '172.16.0.3:$(( 27016 + $2 ))' } ] })"

MongoDB Versions:

Percona Server for MongoDB 4.2.9-9

Percona Server for MongoDB 4.4.0-1

Benchmark Results

Unlimited Memory

The results are in New Order Transactions per Minute (NOTPM), and more is better:

Clients	4.2	4.4
10	541.31	691.89
30	999.89	1105.88
50	1048.50	1171.35
70	1095.72	1335.90
90	1184.38	1433.09
110	1210.18	1521.56
130	1231.38	1575.23
150	1245.31	1680.81
170	1224.13	1668.33
190	1300.11	1641.45
210	1240.86	1619.58
230	1220.89	1575.57
250	1237.86	1545.01

MongoDB 4.4 Unlimited Memory

Limited Memory, 50GB in Total and 25GB for Cache

The results are in New Order Transactions per Minute (NOTPM), and more is better:

Clients	4.2	4.4
10	351.45	377.29
30	483.88	447.22
50	535.34	522.59
70	576.30	574.14
90	604.49	582.10
110	618.59	542.11
130	593.31	386.33
150	386.67	301.75
170	265.91	298.80
190	259.56	301.38
210	254.57	301.88
230	249.47	299.15
250	251.03	300.00

MongoDB 4.2 Limited Memory

Observation

Actually I wanted to perform more benchmarks on 4.4 vs 4.2, but some interesting behavior in 4.4 made me reconsider my plans and I’ve gotten distracted trying to understand the issue, and I share this in the post MongoDB 4.4 Performance Regression: Overwhelmed by Memory.

Besides that, in my tests, 4.4 outperformed 4.2 in case of unlimited memory, but I want to consider a variation of throughput during the benchmark so we are working on a py-tpcc version that would report data with 1-sec resolution. Also, I want to re-evaluate how 4.4 would perform in a long-running benchmark, as the current length of the benchmark is 900 sec.

In the case with limited memory, 4.4 did identically or worse than 4.2 with concurrency over 100 clients.

Both versions did not handle the increased number of clients well, showing worse results with 150 clients compared to 10 clients.

Learn more about the history of Oracle, the growth of MongoDB, and what really qualifies software as open source. If you are a DBA, or an executive looking to adopt or renew with MongoDB, this is a must-read!

Download “Is MongoDB the New Oracle?”