Announcement

Announcement Module
Collapse
No announcement yet.

Cannot Account for memory in use. 5.1.60

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Cannot Account for memory in use. 5.1.60

    Version:5.1.60-12.5

    Currently have a system that is using up almost 50G of memory usage, we are more or less expecting it to take up closer to 40G.

    I have calculated what I can from SHOW ENGINE INNODB STATUS and the global variables and status output, but cant seem to find where the memory went.

    It seemed to rise up gradually over at least a week (from the output of SAR)

    I have gathered pt-summary,pt-mysql-summary, meminfo, and innodb output for the purposes of diagnostics.

    Here is the output of meminfo

    $ cat /proc/meminfo
    MemTotal: 49447676 kB
    MemFree: 646504 kB
    Buffers: 271340 kB
    Cached: 742204 kB
    SwapCached: 2923228 kB
    Active: 47088424 kB
    Inactive: 1432660 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 49447676 kB
    LowFree: 646504 kB
    SwapTotal: 8388600 kB
    SwapFree: 343988 kB
    Dirty: 108 kB
    Writeback: 0 kB
    AnonPages: 45193080 kB
    Mapped: 23292 kB
    Slab: 116260 kB
    PageTables: 113000 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    CommitLimit: 33112436 kB
    Committed_AS: 59970984 kB
    VmallocTotal: 34359738367 kB
    VmallocUsed: 268736 kB
    VmallocChunk: 34359469315 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0
    Hugepagesize: 2048 kB


    Here is the ps output:

    mysql 5346 8.9 91.3 53300120 45152588 ? Sl Jan27 7987:51 /usr/local/mysql-5.1.60/sbin/mysqld --defaults-extra-file=/etc/my-3306.cnf --basedir=/usr/local/mysql --datadir=/var/lib/mysql_data/3306 --user=mysql --log-error=/var/lib/mysql_logs/3306/error.log --open-files-limit=8192 --pid-file=/var/run/mysql/3306.pid --socket=/var/run/mysql/3306.sock --port=3306

    Shows us as using almost 50G. However Innodb is only using

    Total memory allocated 39655047168; in additional pool allocated 0
    Internal hash tables (constant factor + variable factor)
    Adaptive hash index 617273808 (611993944 + 5279864)
    Page hash 38250584
    Dictionary cache 420153435 (152999984 + 267153451)
    File system 12351904 (82672 + 12269232)
    Lock system 95636480 (95624824 + 11656)
    Recovery system 0 (0 + 0)
    Threads 411640 (406936 + 4704)
    Dictionary memory allocated 267153451
    Buffer pool size 2359295
    Buffer pool size, bytes 38654689280
    Free buffers 1
    Database pages 3922659
    Old database pages 1447992
    Modified db pages 49
    Pending reads 0
    Pending writes: LRU 0, flush list 0, single page 0
    Pages made young 45206245, not young 0
    0.00 youngs/s, 0.00 non-youngs/s
    Pages read 43531033, created 27926938, written 129329407
    0.00 reads/s, 0.00 creates/s, 11.45 writes/s
    Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
    Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
    LRU len: 3922659, unzip_LRU len: 392265
    I/O sum[1485]:cur[0], unzip sum[38]:cur[0]



    There seems to be a wide gap there in whats currently being used and what mysql thinks its using.

    We have no MyISAM except for the mysql.* of course.

    ODD SIDE NOTE: Most of our innodb tables are using compression@8k.

    I have gathered up plenty of other diagnostic information but not sure what else to post.

    Can someone point me in the right direction of where to look ?
Working...
X