In a previous blog post: “Tuning Linux for MongoDB,” I covered several tunings for an efficient MongoDB deployment on Linux in Production. This post expands on that one.
While I felt the tuning Linux for MongoDB was a very useful blog post that results in a great baseline tuning, something bugged me about how much effort and touch-points were required to achieve an efficient Linux installation for MongoDB. More importantly, I noticed some cases where the tunings (example: changes to disk I/O scheduler in /etc/udev.d) were ignored on some recent RedHat and CentOS versions. With these issues in mind, I started to investigate better solutions for achieving the tuned baseline.
In RedHat (and thus CentOS) 7.0, a daemon called “tuned” was introduced as a unified system for applying tunings to Linux. tuned operates with simple, file-based tuning “profiles” and provides an admin command-line interface named “tuned-adm” for applying, listing and even recommending tuned profiles.
Some operational benefits of tuned:
- File-based configuration – Profile tunings are contained in a simple, consolidated files
- Swappable profiles – Profiles are easily changed back/forth
- Standards compliance – Using tuned profiles ensures tunings are not overridden or ignored
Note: If you use configuration management systems like Puppet, Chef, Salt, Ansible, etc., I suggest you configure those systems to deploy tunings via tuned profiles instead of applying tunings directly, as tuned will likely start to fight this automation, overriding the changes.
The default available tuned profiles (as of RedHat 7.2.1511) are:
The profiles that are generally interesting for database usage are:
“A server profile for typical latency performance tuning. This profile disables dynamic tuning mechanisms and transparent hugepages. It uses the performance governer for p-states through cpuspeed, and sets the I/O scheduler to deadline.”
“A server profile for typical throughput performance tuning. It disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. CPU governor is set to performance.”
- network-latency – Includes “latency-performance,” disables transparent_hugepages, disables NUMA balancing and enables some latency-based network tunings.
- network-throughput – Includes “throughput-performance” and increases network stack buffer sizes.
I find “network-latency” is the closest match to our recommended tunings, but some additional changes are still required.
The good news is tuned was designed to be flexible, so I decided to make a MongoDB-specific profile: enter “tuned-percona-mongodb”.
“tuned-percona-mongodb” is a performance-focused tuned profile for MongoDB on Linux, and is currently considered experimental (no gurantees/warranties). It’s hosted in our Percona-Lab Github repo.
tuned-percona-mongodb applies the following tunings (from the previous tuning article) on a Redhat/CentOS 7+ host:
- Disabling of transparent huge pages
- Kernel network tunings (sysctls)
- Virtual memory dirty ratio changes (sysctls)
- Virtual memory “swappiness” (sysctls)
- Block-device readahead settings (on all disks except /dev/sda by default)
- Block-device I/O scheduler (on all disks except /dev/sda by default)
The following tunings that our previous tuning article didn’t cover are also applied:
- CPU C-States and Energy-Bias tunings for high performance
- Disk Aggressive Link Power Management (ALPM) tuning, if available/supported
After a successful deployment of this profile, only these recommendations are outstanding:
- Filesystem type and mount options:
Tuned does not handle filesystem mount options, this needs to be done manually in /etc/fstab. To quickly summarize: we recommend the XFS or EXT4 filesystem type for MongoDB data when using MMAPv1 or RocksDB storage engines, and XFS ONLY when using WiredTiger. For all filesystems, using the mount options “rw,noatime” will reduce some activity.
- NUMA disabling or interleaving:
Tuned does not handle NUMA settings and these still need to be handled via the MongoDB init script or the BIOS on/off switch.
- Linux ulimits:
Tuned does not set Linux ulimit settings. However, Percona Server for MongoDB RPM packages do this for you at startup! See “LimitNOFILE” and “LimitNPROC” in “/usr/lib/systemd/system/mongod.service” for more information.
Tuned does not handle installation of RPM packages or enabling of services. You will need to install the “ntp” package and enable/start the “ntpd” service manually:Shell123sudo yum install ntpsudo systemctl enable ntpdsudo systemctl start ntpd
The installation of this profile is as simple as checking-out the repository with a “git” command and then running “sudo make enable”, full output here:
$ git clone https://github.com/Percona-Lab/tuned-percona-mongodb
$ cd tuned-percona-mongodb
$ sudo make enable
if [ -d /etc/tuned ]; then
cp -dpR percona-mongodb /etc/tuned/percona-mongodb;
echo "### 'tuned-percona-mongodb' is installed. Enable with 'make enable'.";
echo "### ERROR: cannot find tuned config dir at /etc/tuned!";
### 'tuned-percona-mongodb' is installed. Enable with 'make enable'.
tuned-adm profile percona-mongodb
Current active profile: percona-mongodb
In the example above you can see “percona-mongodb” is now the active tuned profile on the system (mentioned on the last output line).
The tuned profile files are installed to “/etc/tuned/percona-mongodb”, as seen here:
$ ls -alh /etc/tuned/percona-mongodb/*.*
-rwxrwxr-x. 1 root root 677 Nov 22 20:00 percona-mongodb.sh
-rw-rw-r--. 1 root root 1.4K Nov 22 20:00 tuned.conf
Let’s check that the “deadline” i/o scheduler is now the current scheduler on any disk that isn’t /dev/sda (“sdb” used below):
$ cat /sys/block/sdb/queue/scheduler
noop [deadline] cfq
Transparent huge pages should be disabled (it is!):
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
$ cat /sys/kernel/mm/transparent_hugepage/defrag
always madvise [never]
Block-device readahead should be 32 (16kb) on /dev/sdb (looks good!):
$ blockdev --getra /dev/sdb
That was easy!
To uninstall the profile, run “sudo make uninstall” in the github checkout directory:
if [ -d /etc/tuned/percona-mongodb ]; then
echo "### Disabling tuned profile 'tuned-percona-mongodb'";
echo "### Changing tuned profile to 'latency-performance', adjust if necessary after!";
tuned-adm profile latency-performance;
echo "tuned-percona-mongodb profile not installed!";
### Disabling tuned profile 'tuned-percona-mongodb'
### Changing tuned profile to 'latency-performance', adjust if necessary after!
Current active profile: latency-performance
if [ -d /etc/tuned/percona-mongodb ]; then rm -rf /etc/tuned/percona-mongodb; fi
Note: the uninstallation will enable the “latency-performance” tuned profile, change this after the uninstall if needed
To confirm the uninstallation, let’s check if the block-device readahead is set back to default (256/128kb):
$ sudo blockdev --getra /dev/sdb
So far tuned shows a lot of promise for tuning Linux for MongoDB, providing a single, consistent interface for tuning the Linux operating system. In the future, I would like to see the documentation for tuned improve. However, its simplicity makes the need for documentation rarely necessary.
As mentioned, after applying “tuned-percona-mongodb” you still need to configure an NTP server, NUMA (in some cases) and the filesystem type+tunings manually. The majority of the time, effort and room for mistakes is greatly reduced using this method.
If you have any issues with this profile for tuning Linux for MongoDB, or have any questions, please create a Github issue at this URL: https://github.com/Percona-Lab/tuned-percona-mongodb/issues/new.