TasksMax: Another Setting That Can Cause MySQL Error Messages

TasksMax: Another Setting That Can Cause MySQL Error Messages

PREVIOUS POST
NEXT POST

TasksMax setting causing errors MySQLRecently, I encountered a situation where MySQL gave error messages that I had never seen before:

I was thinking maybe we hit some ulimit limitations or similar, but all the usual suspects were set high enough, and we were not even close to them.

After googling and discussing with the customer, I found they had had similar issues in the past, and I learned something new. Actually it is relatively new, as it has been around for a few years but is not that well known. It is called TasksMax:

Specify the maximum number of tasks that may be created in the unit. This ensures that the number of tasks accounted for the unit (see above) stays below a specific limit. This either takes an absolute number of tasks or a percentage value that is taken relative to the configured maximum number of tasks on the system. If assigned the special value “infinity“, no tasks limit is applied. This controls the “pids.max” control group attribute. For details about this control group attribute, see pids.txt.

Source Manual.

It was introduced to systemd in 2015:

I’d like to introduce DefaultTasksMax= that controls the default
value of the per-unit TasksMax= by default, and would like it to
set to some value such 1024 out-of-the-box. This will mean that any
service or scope created will by default be limited to 1024
tasks. This of course is a change from before that has the
potential to break some daemons that maintain an excessive number
of processes or threads. However, I think it’s a much better choice
to raise the limit for them, rather than stay unlimited for all
services by default. I think 1024 is not particularly low, but also
not particularly high. Note that the kernel by default limits the
number of processes to 32K in total anyway.

In the end, we can see in this commit they chose 512 to be the default settings for TasksMax, which means services that are not explicitly configured otherwise will only be able to create at most 512 processes or threads.

Why 512? I have read through the email list and there was some discussion about what should be the default. Eventually, I found this comment from one of the developers:

Anyway, for now I settled for the default TasksMax= setting of 512 for
all units, plus 4096 for the per-user slices and 8192 for each nspawn
instance. Let’s see how this will work out.

So this is how 512 become the default and no one has touched it since. MySQL is able to reach that limit and can cause error messages like those we see above.

You can increase this limit by creating a file called /etc/systemd/system/mysqld.service  :

You can use a specific number like 4096 (or any other number based on your workload), or infinity which means MySQL can start as many processes as it wants.

Conclusion

Not everyone will reach this limit, but if MySQL is giving error messages like this you should also check TasksMax as well as the other usual suspects. The easiest way to verify the current setting is:


Photo by Vlad Tchompalov on Unsplash

PREVIOUS POST
NEXT POST

Share this post

Comments (2)

  • Nils Reply

    In this case, ulimit is a very classic misdirect, with people often changing settings for PAM even though PAM is usually not involved when launching services. The reasoning behind this setting is to avoid fork bombs or other ways of exhausting the pid space. By the way, there is also a system wide maximum for pids which you can set via sysctl (kernel.pid_max), which is often the second suspect when troubleshooting issues like this 😉

    January 2, 2019 at 11:25 am
  • Sherry Reply

    May I know the kernel version?

    January 3, 2019 at 1:51 pm

Leave a Reply