Tweak Innodb parameters

  • Filter
  • Time
  • Show
Clear All
new posts

  • Tweak Innodb parameters


    The machine has 12 gb ram. It only houses one huge cache table (flushed twice per day). The table is BLOB. Currently about 10 GB is cached, so there's room for using more ram.

    I could really need some tips how to tweak it.

    This is my.cnf:

    # Uncomment the following if you are using InnoDB tables
    innodb_data_home_dir = /var/lib/mysql/
    innodb_data_file_path = ibdata1:200M:autoextend
    innodb_log_group_home_dir = /var/lib/mysql/
    innodb_log_arch_dir = /var/lib/mysql/
    # You can set .._buffer_pool_size up to 50 - 80 %
    # of RAM but beware of setting memory usage too high
    innodb_buffer_pool_size = 384M
    innodb_additional_mem_pool_size = 20M
    # Set .._log_file_size to 25 % of buffer pool size
    innodb_log_file_size = 100M
    innodb_log_buffer_size = 8M
    innodb_flush_log_at_trx_commit = 1
    innodb_lock_wait_timeout = 50

    Here's output of status for innodb params.

    Com_show_innodb_status 0
    Innodb_buffer_pool_pages_data 24475
    Innodb_buffer_pool_pages_dirty 10365
    Innodb_buffer_pool_pages_flushed 26657589
    Innodb_buffer_pool_pages_free 71
    Innodb_buffer_pool_pages_latched 52
    Innodb_buffer_pool_pages_misc 30
    Innodb_buffer_pool_pages_total 24576
    Innodb_buffer_pool_read_ahead_rnd 566045
    Innodb_buffer_pool_read_ahead_seq 162349
    Innodb_buffer_pool_read_requests 3165607053
    Innodb_buffer_pool_reads 12897128
    Innodb_buffer_pool_wait_free 0
    Innodb_buffer_pool_write_requests 1052043302
    Innodb_data_fsyncs 86361799
    Innodb_data_pending_fsyncs 0
    Innodb_data_pending_reads 1
    Innodb_data_pending_writes 0
    Innodb_data_read 4257353728
    Innodb_data_reads 15859383
    Innodb_data_writes 92817184
    Innodb_data_written 1746445824
    Innodb_dblwr_pages_written 26657589
    Innodb_dblwr_writes 367841
    Innodb_log_waits 0
    Innodb_log_write_requests 644676246
    Innodb_log_writes 85553632
    Innodb_os_log_fsyncs 85632317
    Innodb_os_log_pending_fsyncs 0
    Innodb_os_log_pending_writes 0
    Innodb_os_log_written 70796288
    Innodb_page_size 16384
    Innodb_pages_created 18758388
    Innodb_pages_read 50853538
    Innodb_pages_written 26657589
    Innodb_row_lock_current_waits 0
    Innodb_row_lock_time 261
    Innodb_row_lock_time_avg 0
    Innodb_row_lock_time_max 25
    Innodb_row_lock_waits 1301
    Innodb_rows_deleted 1906012
    Innodb_rows_inserted 84545781
    Innodb_rows_read 213763389
    Innodb_rows_updated 0

    Any tips?

  • #2
    Do you have any specific problems with performance?
    Because the only thing I can say generic is that these two:
    innodb_buffer_pool_size = 384M
    innodb_additional_mem_pool_size = 20M
    could be increased a lot, up to about 80% of RAM which could mean something like this:
    innodb_buffer_pool_size = 9G
    innodb_additional_mem_pool_size = 200M
    But that depends on how big you database actually is, because if your DB is smaller than that this setting is worthless and you should maybe focus on allowing more memory to be used for sorting or join buffers etc.
    That is why it is important to say if you experience any specific problems.

    When you say "flushed twice per day", what do you actually mean?
    That it is deleted and rebuilt or what?

    If you are experiencing slow insert/updates but the rest is going fast then you should change this:
    innodb_flush_log_at_trx_commit = 1
    to a value of 2 instead.
    But if you do you should also be aware that if the server crashes then you could loose the last changes that you have performed in the database.


    • #3
      Do you know how to calculate innodb_additional_mem_pool_size?


      • #4
        1) no specific problem. The database can hold up to 40-50G of data.

        2) It's cached data, so the whole table (theres really only one table on it) gets dropped and rebuilt twice a day.

        3) No, I dont!

        Thanks for your reply


        • #5
          If it's only one table then you can keep the setting:
          innodb_additional_mem_pool_size = 20M
          There is no use to allow more since it will never be used.

          And actually the value of 200M is very large and I want to retract that statement from my previous post. Sorry about that!

          No unfortunately I don't have a good way to calculate it.
          But it doesn't really matter since the only thing that will happen if this buffer is too small is that MySQL will allocate the memory anyway but it will write error messages about it to the MySQL error log.
          So if you see these errors turn up then you can just increase the setting.


          • #6

            Did not work out very well.

            The only thing I can think of would be the key_buffer already so high so innodb buffer pool could not allocate its 3G I chose.
            (And having a key buffer above say 16M makes no sense in this case as I only have one table, an innodb so I will of course lower it)
            What do you think?

            080414 8:22:47 InnoDB: Error: cannot allocate 3221241856 bytes of
            InnoDB: memory with malloc! Total allocated memory
            InnoDB: by InnoDB 39688400 bytes. Operating system errno: 12
            InnoDB: Check if you should increase the swap file or
            InnoDB: ulimits of your operating system.
            InnoDB: On FreeBSD check you have compiled the OS with
            InnoDB: a big enough maximum process size.
            InnoDB: Note that in most 32-bit computers the process
            InnoDB: memory space is limited to 2 GB or 4 GB.
            InnoDB: We keep retrying the allocation for 60 seconds...
            InnoDB: Fatal error: cannot allocate the memory for the buffer pool
            ^G/usr/sbin/mysqld: Out of memory (Needed 3814433792 bytes)
            ^G/usr/sbin/mysqld: Out of memory (Needed 2860824576 bytes)
            080414 8:23:47 [Note] /usr/sbin/mysqld: ready for connections.
            Version: '5.0.37-standard-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL)
            080414 8:24:02 [ERROR] /usr/sbin/mysqld: Incorrect information in file: './something/somethingelse.frm'

            Linux something.net 2.6.9-55.0.12.ELsmp #1 SMP Fri Nov 2 11:19:08 EDT 2007 i686 i686 i386 GNU/Linux

            # cat /proc/meminfo
            MemTotal: 12469000 kB

            # Example MySQL config file for very large systems.
            # This is for a large system with memory of 1G-2G where the system runs mainly
            # MySQL.
            # You can copy this file to
            # /etc/my.cnf to set global options,
            # mysql-data-dir/my.cnf to set server-specific options (in this
            # installation this directory is /usr/local/var) or
            # ~/.my.cnf to set user-specific options.
            # In this file, you can use all long options that a program supports.
            # If you want to know which options a program supports, run the program
            # with the "--help" option.

            # The following options will be passed to all MySQL clients
            #password = your_password
            port = 3306
            socket = /var/lib/mysql/mysql.sock

            # Here follows entries for some specific programs

            # The MySQL server
            port = 3306
            socket = /var/lib/mysql/mysql.sock
            key_buffer = 4096M
            #Edited by PM default is 1M
            max_allowed_packet = 40M
            table_cache = 1024
            sort_buffer_size = 2M
            read_buffer_size = 2M
            read_rnd_buffer_size = 8M
            myisam_sort_buffer_size = 64M
            thread_cache_size = 8
            query_cache_size = 32M
            # Try number of CPU's*2 for thread_concurrency
            thread_concurrency = 8

            # log-long-format

            # Don't listen on a TCP/IP port at all. This can be a security enhancement,
            # if all processes that need to connect to mysqld run on the same host.
            # All interaction with mysqld must be made via Unix sockets or named pipes.
            # Note that using this option without enabling named pipes on Windows
            # (via the "enable-named-pipe" option) will render mysqld useless!

            # Replication Master Server (default)
            # binary logging is required for replication

            # required unique id between 1 and 2^32 - 1
            # defaults to 1 if master-host is not set
            # but will not function as a master if omitted
            server-id = 1

            # Replication Slave (comment out master section to use this)
            # To configure this host as a replication slave, you can choose between
            # two methods :
            # 1) Use the CHANGE MASTER TO command (fully described in our manual) -
            # the syntax is:
            # where you replace , , by quoted strings and
            # by the master's port number (3306 by default).
            # Example:
            # CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
            # MASTER_USER='joe', MASTER_PASSWORD='secret';
            # OR
            # 2) Set the variables below. However, in case you choose this method, then
            # start replication for the first time (even unsuccessfully, for example
            # if you mistyped the password in master-password and the slave fails to
            # connect), the slave will create a master.info file, and any later
            # change in this file to the variables' values below will be ignored and
            # overridden by the content of the master.info file, unless you shutdown
            # the slave server, delete master.info and restart the slaver server.
            # For that reason, you may want to leave the lines below untouched
            # (commented) and instead use CHANGE MASTER TO (see above)
            # required unique id between 2 and 2^32 - 1
            # (and different from the master)
            # defaults to 2 if master-host is set
            # but will not function as a slave if omitted
            #server-id = 2
            # The replication master for this slave - required
            #master-host =
            # The username the slave will use for authentication when connecting
            # to the master - required
            #master-user =
            # The password the slave will authenticate with when connecting to
            # the master - required
            #master-password =
            # The port the master is listening on.
            # optional - defaults to 3306
            #master-port =
            # binary logging - not required for slaves, but recommended

            # Point the following paths to different dedicated disks
            #tmpdir = /tmp/
            #log-update = /path-to-dedicated-directory/hostname

            # Uncomment the following if you are using BDB tables
            #bdb_cache_size = 384M
            #bdb_max_lock = 100000

            # Uncomment the following if you are using InnoDB tables
            innodb_data_home_dir = /var/lib/mysql/
            innodb_data_file_path = ibdata1:200M:autoextend
            innodb_log_group_home_dir = /var/lib/mysql/
            innodb_log_arch_dir = /var/lib/mysql/
            # You can set .._buffer_pool_size up to 50 - 80 %
            # of RAM but beware of setting memory usage too high
            innodb_buffer_pool_size = 3G
            innodb_additional_mem_pool_size = 20M
            # Set .._log_file_size to 25 % of buffer pool size
            innodb_log_file_size = 256M
            innodb_log_buffer_size = 8M
            innodb_flush_log_at_trx_commit = 1
            innodb_lock_wait_timeout = 50

            max_allowed_packet = 16M

            # Remove the next comment character if you are not familiar with SQL

            key_buffer = 256M
            sort_buffer_size = 256M
            read_buffer = 2M
            write_buffer = 2M

            key_buffer = 256M
            sort_buffer_size = 256M
            read_buffer = 2M
            write_buffer = 2M



            • #7
              I guess you're using 32bit OS.

              Using 64bit OS to overcome this limitation.


              • #8
                What exactly is the limitation?


                • #9
                  You should of course lower it but I don't think that is the problem because this buffer will not be allocated unless there are any indexes to use ti.

                  No, your problem is probably OS related instead.

                  Which OS are you running?
                  Not a 64bit version i assume?

                  Either that or you have a ulimit set globally for how much memory each process are allowed to use.

                  On 32bit OS'es one process in itself can very rarely be allowed to allocate more than about 2-3GB due to limitations with using 32bit memory address space.

                  And on some Linux/BSD distributions they have a global ulimit that doesn't allow user processes to be larger than a certain size.
                  First check the ulimit limit with:

                  ulimit -a

                  If it says unlimited then it is not a ulimit problem and then unfortunately you need to run a 64bit version of your OS to be able to use more memory (which I think you should anyway since you have a pretty hefty server with a lot of RAM available).

                  And when you upgrade your OS to a 64 bit version you should also make sure that you are running a 64 bit version of MySQL


                  • #10

                    I guess I will have to reinstall servers etc.
                    I will revoke this thread once all is done, probably a few weeks.

                    Thanks for your answers guys!