Recently, I encountered a situation where MySQL gave error messages that I had never seen before:
2018-12-12T14:36:45.571440Z 0 [ERROR] Error log throttle: 150 'Can't create thread to handle new connection' error(s) suppressed
2018-12-12T14:36:45.571456Z 0 [ERROR] Can't create thread to handle new connection(errno= 11)
2018-12-12T14:37:47.748575Z 0 [ERROR] Error log throttle: 940 'Can't create thread to handle new connection' error(s) suppressed
2018-12-12T14:37:47.748595Z 0 [ERROR] Can't create thread to handle new connection(errno= 11)
I was thinking maybe we hit some ulimit limitations or similar, but all the usual suspects were set high enough, and we were not even close to them.
After googling and discussing with the customer, I found they had had similar issues in the past, and I learned something new. Actually it is relatively new, as it has been around for a few years but is not that well known. It is called
Specify the maximum number of tasks that may be created in the unit. This ensures that the number of tasks accounted for the unit (see above) stays below a specific limit. This either takes an absolute number of tasks or a percentage value that is taken relative to the configured maximum number of tasks on the system. If assigned the special value “
infinity“, no tasks limit is applied. This controls the “
pids.max” control group attribute. For details about this control group attribute, see pids.txt.
It was introduced to systemd in 2015:
I’d like to introduce DefaultTasksMax= that controls the default
value of the per-unit TasksMax= by default, and would like it to
set to some value such 1024 out-of-the-box. This will mean that any
service or scope created will by default be limited to 1024
tasks. This of course is a change from before that has the
potential to break some daemons that maintain an excessive number
of processes or threads. However, I think it’s a much better choice
to raise the limit for them, rather than stay unlimited for all
services by default. I think 1024 is not particularly low, but also
not particularly high. Note that the kernel by default limits the
number of processes to 32K in total anyway.
In the end, we can see in this commit they chose 512 to be the default settings for
TasksMax, which means services that are not explicitly configured otherwise will only be able to create at most 512 processes or threads.
Why 512? I have read through the email list and there was some discussion about what should be the default. Eventually, I found this comment from one of the developers:
Anyway, for now I settled for the default TasksMax= setting of 512 for
all units, plus 4096 for the per-user slices and 8192 for each nspawn
instance. Let’s see how this will work out.
So this is how 512 become the default and no one has touched it since. MySQL is able to reach that limit and can cause error messages like those we see above.
You can increase this limit by creating a file called /etc/systemd/system/mysqld.service :
You can use a specific number like 4096 (or any other number based on your workload), or
infinity which means MySQL can start as many processes as it wants.
Not everyone will reach this limit, but if MySQL is giving error messages like this you should also check
TasksMax as well as the other usual suspects. The easiest way to verify the current setting is:
#> systemctl show -p TasksMax mysql