I have recently been involved in diagnosing the reasons behind OOM invocation that would kill the MySQL server process. Of course these servers were primarily running MySQL. As such the MySQL server process was the one with the largest amount of memory allocated.
But the strange thing was that in all the cases, there was no swapping activity seen and there were enough pages in the page cache. Ironically all of these servers were CentOS 6.4 running kernel version 2.6.32-358. Another commonality was the fact that vm.swappiness was set to 0. This is a pretty much standard practice and one that is applied on nearly every server that runs MySQL.
Looking into this further I realized that there was a change introduced in kernel 3.5-rc1 that altered the swapping behavior when “vm.swappiness=0”.
Below is the description of the commit that changed “vm.swappiness=0” behavior, together with the diff:
|
1 |
$ git show fe35004fbf9eaf67482b074a2e032abb9c89b1dd<br>commit fe35004fbf9eaf67482b074a2e032abb9c89b1dd<br>Author: Satoru Moriya <satoru.moriya@hds.com><br>Date: Tue May 29 15:06:47 2012 -0700<br><br>mm: avoid swapping out with swappiness==0<br><br>Sometimes we'd like to avoid swapping out anonymous memory. In<br>particular, avoid swapping out pages of important process or process<br>groups while there is a reasonable amount of pagecache on RAM so that we<br>can satisfy our customers' requirements.<br><br>OTOH, we can control how aggressive the kernel will swap memory pages with<br>/proc/sys/vm/swappiness for global and<br>/sys/fs/cgroup/memory/memory.swappiness for each memcg.<br><br>But with current reclaim implementation, the kernel may swap out even if<br>we set swappiness=0 and there is pagecache in RAM.<br><br>This patch changes the behavior with swappiness==0. If we set<br>swappiness==0, the kernel does not swap out completely (for global reclaim<br>until the amount of free pages and filebacked pages in a zone has been<br>reduced to something very very small (nr_free + nr_filebacked < high<br>watermark)).<br><br>Signed-off-by: Satoru Moriya <satoru.moriya@hds.com><br>Acked-by: Minchan Kim <minchan@kernel.org><br>Reviewed-by: Rik van Riel <riel@redhat.com><br>Acked-by: Jerome Marchand <jmarchan@redhat.com><br>Signed-off-by: Andrew Morton <akpm@linux-foundation.org><br>Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org><br><br>diff --git a/mm/vmscan.c b/mm/vmscan.c<br>index 67a4fd4..ee97530 100644<br>--- a/mm/vmscan.c<br>+++ b/mm/vmscan.c<br>@@ -1761,10 +1761,10 @@ static void get_scan_count(struct mem_cgroup_zone *mz, struct scan_control *sc,<br>* proportional to the fraction of recently scanned pages on<br>* each list that were recently referenced and in active use.<br>*/<br>- ap = (anon_prio + 1) * (reclaim_stat->recent_scanned[0] + 1);<br>+ ap = anon_prio * (reclaim_stat->recent_scanned[0] + 1);<br>ap /= reclaim_stat->recent_rotated[0] + 1;<br><br>- fp = (file_prio + 1) * (reclaim_stat->recent_scanned[1] + 1);<br>+ fp = file_prio * (reclaim_stat->recent_scanned[1] + 1);<br>fp /= reclaim_stat->recent_rotated[1] + 1;<br>spin_unlock_irq(&mz->zone->lru_lock);<br><br>@@ -1777,7 +1777,7 @@ out:<br>unsigned long scan;<br><br><code> scan = zone_nr_lru_pages(mz, lru);<br>- if (priority || noswap) {<br>+ if (priority || noswap || !vmscan_swappiness(mz, sc)) {<br>scan >>= priority;<br>if (!scan && force_scan)<br>scan = SWAP_CLUSTER_MAX; |
This change was merged into the RHEL kernel 2.6.32-303:
|
1 |
* Mon Aug 27 2012 Jarod Wilson <jarod@redhat.com> [2.6.32-303.el6]<br>...<br>- [mm] avoid swapping out with swappiness==0 (Satoru Moriya) [787885] |
This obviously changed the way we think about “vm.swappiness=0”. Previously, setting this to 0 was thought to reduce the tendency to swap userland processes but not disable that completely. As such it was expected to see little swapping instead of OOM.
This applies to all RHEL/CentOS kernels > 2.6.32-303 and to other distributions that provide newer kernels such as Debian and Ubuntu. Or any other distribution where this change has been backported as in RHEL.
Let me share with you memory zones related statistics that were logged to the system log from one of the OOM event.
|
1 |
Mar 11 11:01:45 db01 kernel: Node 0 DMA: 4*4kB 2*8kB 2*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15680kB<br>Mar 11 11:01:45 db01 kernel: Node 0 DMA32: 6*4kB 22*8kB 444*16kB 374*32kB 129*64kB 26*128kB 15*256kB 17*512kB 2*1024kB 0*2048kB 0*4096kB = 45448kB<br>Mar 11 11:01:45 db01 kernel: Node 0 Normal: 825*4kB 1012*8kB 382*16kB 169*32kB 69*64kB 74*128kB 14*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 42436kB<br>Mar 11 11:01:45 db01 kernel: 452844 total pagecache pages<br>Mar 11 11:01:45 db01 kernel: 0 pages in swap cache<br>Mar 11 11:01:45 db01 kernel: Swap cache stats: add 0, delete 0, find 0/0<br>Mar 11 11:01:45 db01 kernel: Free swap = 4128760kB<br>Mar 11 11:01:45 db01 kernel: Total swap = 4128760kB<br>...<br>Mar 11 11:01:45 db01 kernel: Node 0 DMA free:15680kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15284kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes<br>Mar 11 11:01:45 db01 kernel: Node 0 DMA32 free:45448kB min:25140kB low:31424kB high:37708kB active_anon:1741812kB inactive_anon:520348kB active_file:4792kB inactive_file:462576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:386328kB writeback:76268kB mapped:936kB shmem:0kB slab_reclaimable:20420kB slab_unreclaimable:6964kB kernel_stack:0kB pagetables:572kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:142592 all_unreclaimable? no<br>Mar 11 11:01:45 db01 kernel: Node 0 Normal free:42436kB min:42316kB low:52892kB high:63472kB active_anon:3041852kB inactive_anon:643624kB active_file:340156kB inactive_file:1003512kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:979444kB writeback:22040kB mapped:15616kB shmem:180kB slab_reclaimable:41052kB slab_unreclaimable:35996kB kernel_stack:2720kB pagetables:19912kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:31552 all_unreclaimable? no |
As can be seen the amount of free memory and the amount of memory in the page cache was greater than the high watermark, which prevented any swapping activity. Yet unnecessary memory pressure caused OOM to be invoked which killed the MySQL server process.
MySQL getting OOM’ed is bad for many reasons and can have an undesirable impact such as causing loss of uncommitted transactions or transactions not yet flushed to the log because of innodb_flush_log_at_trx_commit=0, or a much more heavy impact because of cold caches upon restart.
I prefer the old behavior of vm.swappiness and as such I now set it to a value of “1”. Setting vm.swappiness=0 would mean that you will now have to be much more accurate in how you configure the size of various global and session buffers.
