Announcement

Announcement Module
Collapse
No announcement yet.

Percona XtraDB Cluster Memory Leak or config problem ?

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Percona XtraDB Cluster Memory Leak or config problem ?

    Hi all,

    We are using Percona XtraDB Cluster in production for now a few months and along our migration planning, we saw a memory usage problem that is during for now a few weeks, it can be seen on the following attached image link.

    We discovered the problem after one of our node (sothis) crashed on week 42 (we cannot see it on the month view of the graph attached but all memory and then swap was totally filled, after this the mysqld process litterally crashed). So after doing some upgrade tests on a test cluster, we finally decided to upgrade the cluster to next version 5.5.31 on week45, the memory increase was still present but growing more slowly than 5.5.30 so we upgraded to v5.5.33 on the beginning of this week but this does not seem to solve the problem ... 5.5.34 will be our next try but I'm now wondering if the problem is not elsewhere, bad settings in config file or other ?
    For information, sarapis and sothis node are "really in production" and serving requests but sopdet node is only getting replication traffic, not serving any requests for the moment, also memory usage is different on each node because we direct traffic for some databases directly on node sarapis and traffic for some other databases pool on node sothis.

    Could anyone advise on which indicators, settings, metrics ... I can check to investigate and find the cause of this problem please ?

    Best regards,

    Laurent
    Dropbox is a free service that lets you bring your photos, docs, and videos anywhere and share them easily. Never email yourself a file again!

  • #2
    Note that in general Galera based cluster members require more memory then standard MySQL. That especially matters when large transactions come into equation.
    Check if your servers have to deal with large transactions and split them into smaller ones if possible.
    You can also attach your my.cnf here, to check potentially important variables.

    Comment


    • #3
      Hi przemek,

      Thanks for your reply and sorry for late reply !
      You can find my configuration for these nodes by clicking on this link : https://groups.google.com/group/code...huser=0&view=1

      Any advises/clues would be appreciated as it seems swap is still increasing after memory being fully filed by now on one of our node
      Sample below :


      20131205-160001 - sarapis - mysqld (used) RSS : 26525776 / VMSIZE : 33301664 - SysMEM (used) Mem : 26164764 / Swap : 499724
      20131205-170002 - sarapis - mysqld (used) RSS : 26575160 / VMSIZE : 33301664 - SysMEM (used) Mem : 26189804 / Swap : 499680
      20131205-180002 - sarapis - mysqld (used) RSS : 26624768 / VMSIZE : 33301468 - SysMEM (used) Mem : 26214152 / Swap : 499604
      20131205-190002 - sarapis - mysqld (used) RSS : 26658748 / VMSIZE : 33301468 - SysMEM (used) Mem : 26232364 / Swap : 499052
      20131205-200002 - sarapis - mysqld (used) RSS : 26676988 / VMSIZE : 33301468 - SysMEM (used) Mem : 26235600 / Swap : 498972
      20131205-210002 - sarapis - mysqld (used) RSS : 26684976 / VMSIZE : 33301468 - SysMEM (used) Mem : 26236536 / Swap : 498936
      20131205-220002 - sarapis - mysqld (used) RSS : 26577644 / VMSIZE : 33301468 - SysMEM (used) Mem : 26238076 / Swap : 508864
      20131205-230002 - sarapis - mysqld (used) RSS : 26586440 / VMSIZE : 33301468 - SysMEM (used) Mem : 26242256 / Swap : 508832
      20131206-000002 - sarapis - mysqld (used) RSS : 26594888 / VMSIZE : 33301468 - SysMEM (used) Mem : 26244532 / Swap : 508760
      20131206-010001 - sarapis - mysqld (used) RSS : 26388076 / VMSIZE : 33301468 - SysMEM (used) Mem : 26222812 / Swap : 528620
      20131206-020001 - sarapis - mysqld (used) RSS : 26382248 / VMSIZE : 33301468 - SysMEM (used) Mem : 26218464 / Swap : 542908
      20131206-030001 - sarapis - mysqld (used) RSS : 26390612 / VMSIZE : 33301468 - SysMEM (used) Mem : 26217884 / Swap : 542860
      20131206-040001 - sarapis - mysqld (used) RSS : 26456368 / VMSIZE : 33301468 - SysMEM (used) Mem : 26229232 / Swap : 542824
      20131206-050001 - sarapis - mysqld (used) RSS : 26465328 / VMSIZE : 33301468 - SysMEM (used) Mem : 26230684 / Swap : 542780
      20131206-060001 - sarapis - mysqld (used) RSS : 26485588 / VMSIZE : 33301468 - SysMEM (used) Mem : 26241716 / Swap : 542724
      20131206-070001 - sarapis - mysqld (used) RSS : 26492912 / VMSIZE : 33301468 - SysMEM (used) Mem : 26242800 / Swap : 542700
      20131206-080001 - sarapis - mysqld (used) RSS : 26540080 / VMSIZE : 33301468 - SysMEM (used) Mem : 26295568 / Swap : 532364
      20131206-090001 - sarapis - mysqld (used) RSS : 26621024 / VMSIZE : 33301468 - SysMEM (used) Mem : 26437716 / Swap : 488996
      20131206-100002 - sarapis - mysqld (used) RSS : 27185156 / VMSIZE : 33301468 - SysMEM (used) Mem : 27025748 / Swap : 498500
      20131206-110001 - sarapis - mysqld (used) RSS : 26824156 / VMSIZE : 33301468 - SysMEM (used) Mem : 27375088 / Swap : 594192
      20131206-120001 - sarapis - mysqld (used) RSS : 27247632 / VMSIZE : 33301468 - SysMEM (used) Mem : 27574268 / Swap : 593168
      20131206-130001 - sarapis - mysqld (used) RSS : 27486916 / VMSIZE : 33301468 - SysMEM (used) Mem : 27672732 / Swap : 592484
      20131206-140001 - sarapis - mysqld (used) RSS : 27526692 / VMSIZE : 33301468 - SysMEM (used) Mem : 27690748 / Swap : 592360
      20131206-150001 - sarapis - mysqld (used) RSS : 27589388 / VMSIZE : 33301468 - SysMEM (used) Mem : 27719524 / Swap : 592216
      20131206-160001 - sarapis - mysqld (used) RSS : 27070148 / VMSIZE : 33301468 - SysMEM (used) Mem : 27020940 / Swap : 1064568

      Regards,

      Laurent

      Comment


      • #4
        Laurent,

        Are you sure you really need following variables different then default values?
        Code:
        read_rnd_buffer_size = 16M
        sort_buffer_size = 16M
        join_buffer_size = 4M
        See this for reference: http://ronaldbradford.com/blog/dont-...rs-2010-03-08/

        And what was the reason to have this so huge:
        innodb_log_buffer_size = 340M

        If you indeed have that huge transactions, you should know galera based clusters won't like them.

        Comment

        Working...
        X