Too many connections? No problem!

PREVIOUS POST
NEXT POST

Did that ever happen to you in production?

Just happened to one of our customers. Want to know what we did?

For demo purposes I’ll use sandbox here (so the ./use is actually executing mysql cli). Oh and mind it is not a general-purpose best-practice, but rather a break-and-enter hack when the server is flooded. So, when this happens in production, the problem is – how do you quickly regain access to mysql server to see what are all the sessions doing and how do you do that without restarting the application? Here’s the trick:

And here’s the result:

Credit for the gdb magic goes to Domas.

Few notes:

  • You would usually have one connection reserved for SUPER user, but that does not help if your application is connecting as a SUPER user (which is a bad idea anyway).
  • This worked for me on 5.0.87-percona-highperf, but use it at your own risk and better test it before you actually have to do it in production.
  • This example assumes you had less than 5000 max_connections configured ;)
PREVIOUS POST
NEXT POST

Comments

  1. says

    Aurimas,

    One question one may have is how does GDB able to execute MySQL command “set max_connections=5000″ – the thing is it is not MySQL but GDB command in this case which just happen to look the same which sets global variable with max_connections name to value of 5000.

  2. says

    Great tip!

    The only caveat is that if you have a runaway app that it is trying to create as many connections as it can, you have to be *very* quick to get your connection before the run away application does.

    Does this method work for other global variables? It’d be great if you can set / reset ‘wait_timeout’ to automatically clean sleeping connections, in my experience, a small wait_timeout can close ‘sleeping’ connections before too many are created.

    My $.02
    G

  3. says

    We always configure with settings like:

    set-variable = max_connections=1500
    set-variable = max_user_connections=300

    Where there are < 10 users that will connect to the server. This leaves connections open for super users. We typically have 2-3 application users for our mysql servers. So, with 3, only 900 connections could be used up by our users. You can additionally restrict per user connections if you want to tweak it even further.

  4. Istvan Podor says

    @Brian:

    Yes,when I talked to one of my friend he just mentioned the same.

    @Aurimas:

    Great great article. Thanks for sharing! :)

  5. says

    Brian,

    thanks – indeed, there are many things you can do upfront to avoid running into this, unfortunately most of the systems we end up working with are systems that were not configured by us just yet.

  6. says

    Gerry,

    yeah, that’s so true. An alternative to being quick is setting max_connections to a much higher value than you expect your app to reach.

    Regarding other global variables – it should work but symbols may not match the names in mysql server so you have to check the source to figure out what’s what. To change global wait_timeout and interactive_timeout one would set global_system_variables.net_wait_timeout and global_system_variables.net_interactive_timeout however each thread has a local variable set which I’m not sure how to modify.

  7. says

    Brian,

    Right this is generally good configuration advice. Also Aurimas mentioned there is connection reserved for SUPER user… why that would not be used ? Because there are many cases when we have to deal with badly configured systems :)

  8. Mark Callaghan says

    In sql/set_var.cc a function is called when max_connections is changed. I think it is resize_thr_alarm. You have not done that here.

  9. mikeh says

    Doesn’t this imply you’re running a build of MySQL with debugging symbols? Surely most people aren’t.

  10. Mark Callaghan says

    It frequently works for me without calling that function. It occasionally doesn’t and I get obscure error messages that go away after I call it.

  11. says

    Aurimas,

    What I’d do is to change connections to higher value so you can connect and when change it again using MySQL way so it is resized as needed.
    I would be careful calling functions in GDB unless you’re sure it is safe – mutexes etc may cause unwanted side effects.

  12. says

    i was impress this thing additional to this i work an application that uses vb.net and i use a control backgroundworker as my asyncronous mysql connection to database to retrieve data every second after a long running application suddenly ived enctour a too many connection in my mysql server 5+.

  13. Romeo says

    I have tried this command in our redhat server w/ mysql having “too many connection” problem & it doesn’t work.

    gdb -p $(cat data/mysql_sandbox5087.pid) -ex “set max_connections=5000″ -batch

    > gdb -p $17391 -ex “set max_connections=5000″ -batch
    GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
    Copyright 2004 Free Software Foundation, Inc.
    GDB is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    Type “show copying” to see the conditions.
    There is absolutely no warranty for GDB. Type “show warranty” for details.
    This GDB was configured as “i386-redhat-linux-gnu”…set max_connections=5000: No such file or directory.

    /mysql/data/-batch: No such file or directory.
    (gdb)

    I have tried “–batch” and putting a complete command syntax for all still returns the above error.

    We can’t restart the DB or the server. What should I do next?

  14. Romeo says

    I have tried that but it is giving this error:
    > gdb -pid 17391 -ex “set max_connections=5000″ -batch
    set max_connections=5000: No such file or directory.
    (no debugging symbols found)
    Using host libthread_db library “/lib/tls/libthread_db.so.1″.
    0x082e674f in ?? ()

  15. Andrews says

    Hello, every one.

    Aurimas excellent article,

    I have read almost all the comments because a few days ago we had the same problem with our new web site, but I have a question about this change and I will share with you to see if you can answer it.

    When doing this kind of change? No need to change some other side of apache? Considering that increasing the number of connections is possible that the workload increase on the web server or not?

  16. says

    Romeo, could it be the symbols (debug package) is not available for the version of MySQL you are using? Also, this may be gdb dependent, but are you sure you should be using -pid instead of –pid or -p ?

    Andrews – this article is only about changing max_connections when MySQL server is using all of the available slots and you need to log into it to fix it (say kill some mysql queries) without restarting mysql server. Apache httpd settings is entirely different (and actually more important) topic, but there’s no simple answer here i.e. it depends on why you are hitting the max connections limit in the first place.

    Aurimas

  17. Andrews says

    Aurimas, thanks a lot for your reply.

    Could you refer to me some article to check it out about this topic, the idea is make a balance between apache conections and MySQL max_conections.

    Thanks in advance.

  18. JoseM says

    Hello everyone;

    I have my website develop in PHP with MYSQL, these are MYSQL values:

    datadir=/data/mysql
    socket=/data/mysql/mysql.sock
    max_connections = 600
    wait_timeout = 60
    interactive_timeout=1800
    skip-name-resolve
    skip-bdb
    default-time-zone=America/Caracas
    log-slow-queries=/var/log/mysql-slow-queries.log
    long_query_time=15
    log-queries-not-using-indexes
    query_cache_type=1
    query_cache_size=32M
    thread_cache_size=30
    table_cache=4096
    join_buffer_size=6M
    key_buffer_size=12M
    server-id = 1
    log-bin=/data/mysql/drbd
    expire_logs_days = 5
    max_binlog_size = 100M
    [client]
    socket=/data/mysql/mysql.sock

    When I start the database service that starts smoothly with a low number of connections and so remains for a few minutes, but then so sopresiva connections to the database increases reaching the maximum of connections and I have to re-restart the server.
    I’ve checked the security aspect and not an attack, will have any suggestions or ideas of what might be causing thereby increase the number of connections?
    Thanks a lot

Leave a Reply

Your email address will not be published. Required fields are marked *