Returning to InnoDB scalability

I’m again returning to InnoDB scalability and related bug #15815 as it hurts many users and customers using multi-cpu servers.
Short intro into problem:
On 4-CPU box 1 thread executes full-table scan select query for 8 sec,
but with 4 threads – each thread executes query for 240 sec.
It is very strange as threads use only SELECT queries and ideally there should be no
any problem in concurrent enviroment, especially for CPU-bound workload.

I did the profiling which show the problem is with “buffer pool” mutex, which protects innodb_buffer_pool.
In details, for each scanned row InnoDB calls block_get / block_release functions
which aquire/release the block related to the current row. And the problem functions
block_get / block_release use mutex_lock(buffer_pool_mutex) / mutex_unlock(buffer_pool_mutex)
calls. So global mutex is accessed for each scanned rows, what in multi-CPU/multi-threading
enviroment results in “mutex ping-pong” problem.

Looking in source code of block_get / block_release function I don’t see obviosly reasons
to use global lock which can not be weaken to block level. I tried to replace buffer_pool_mutex in
these place to block_mutex and I got impressive results: now
each of 4 threads executes query for 11 sec.
So performance is increased by 240/11 ~= 21 times, and if before we had negative scalability, currently
scalabilty factor is 2.9 (the result with 4 threads = 2.9 * result with 1 thread).

The current patch can not be considered as stable as it touches many InnoDB subsystems,
but it looks like right direction to solve problem.

Share this post

Comments (43)

  • Zemo

    If possible could you post this patch ?

    Thanks,

    July 28, 2006 at 3:53 am
  • Kristopher Tate

    Wow, this patch would help me greatly — the whole problem explains a lot! haha, and here I thought it was on my side of things…

    Just to let you know, I read / check this blog every day, and it has been a great resource to me and my efforts. Thanks.

    Best,

    Kristopher Tate
    cto & founder — bluebridge tech / zooomr

    July 28, 2006 at 8:50 am
  • Vadim

    Kristopher,

    Thank you for your words!
    It is very valuable to know my work is interesting.

    Thanks,
    Vadim.

    July 28, 2006 at 9:27 am
  • Vadim

    Zemo,

    I could post it, but I don’t want.
    I know this patch crashes server on INSERT queries, so I’d want to fix obviously problems
    before make it public.

    Let me one-two week, maybe I will find a solution.

    July 28, 2006 at 9:29 am
  • Zemo

    Thanks, this problem is really plagueing us at the momment , that is why I asked.

    We currently run an 8way Opteron system, with MySQL/Innodb. MySQL just cant seem to scale very well. Our performance is 5x worse then our regular Dual Opteron server that our database was on. I find it mind blowing that MySQL AB allowed this problem to go unresolved for so long.

    July 29, 2006 at 7:12 am
  • Vadim

    Zemo,

    Yes, it is known problem with InnoDB, and no solution is available.
    Well… you can use innodb_thread_concurrency=2 or 4, but in this case your 8way Opteron box will work as single Dual Core 🙂

    There might be several sources of bad scalability.
    Can you post SHOW MUTEX STATUS output ?
    It would be interesting to look if you have contention on buffer_pool mutex or something else.

    August 1, 2006 at 9:40 am
  • Vadim

    Zemo,

    I modified patch and it does not crash InnoDB in my tests, so
    I want to propose it for you.
    http://www.mysqlperformanceblog.com/files/patches/patch.bufferpool1.diff
    (Place it in 5.0.22 source tree and execute patch -p1 < patch.bufferpool1.diff). Don't consider this patch as stable! Please let me know how it works and report about crashes, thank you!

    August 3, 2006 at 12:37 pm
  • Zemo

    Valdim,

    Thanks!.. I’ll try this out..
    As for the information you requested.. The only workaround to make our server usable was to put innodb_thread_concurrency=4 in our my.conf .

    For the SHOW MUTEX STATUS output , here you go ::

    mysql> SHOW MUTEX STATUS ;
    +—————————+————-+————–+————+————-+———-+———–+—————+
    | Mutex | Module | Count | Spin_waits | Spin_rounds | OS_waits | OS_yields | OS_waits_time |
    +—————————+————-+————–+————+————-+———-+———–+—————+
    | &(trx->undo_mutex) | trx0trx.c | 1891 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1927 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1863 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1951 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1998 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2070 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1977 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1953 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2061 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2167 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1911 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2027 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2035 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2030 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2170 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2106 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2037 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2121 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2176 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2094 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2100 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2098 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 1961 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2085 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2082 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2180 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2020 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2177 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2180 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2231 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2267 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2119 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2162 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2115 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2135 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2134 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2183 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2150 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2248 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2326 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2252 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2160 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2213 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2262 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2259 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2295 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2219 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2219 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2402 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2283 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2342 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2224 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2263 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2162 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2243 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2171 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2362 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2210 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2265 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2242 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2237 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2164 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2274 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2302 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2265 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2155 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2235 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2270 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2336 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2195 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2253 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2167 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2149 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2227 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2202 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2337 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2222 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2266 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2291 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2314 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2290 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2199 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2368 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2204 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2231 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2340 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2301 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2342 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2269 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2206 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2337 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2274 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2239 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2324 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2269 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2225 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2429 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2107 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2205 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2265 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2312 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2280 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2258 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2301 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2365 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2430 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2294 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2203 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2331 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2214 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2221 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2343 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2285 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2181 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2263 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2270 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2328 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2254 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2423 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2219 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2399 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2341 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2318 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2252 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2322 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2286 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2314 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2399 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2266 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2217 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2421 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2226 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2313 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2399 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2377 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2419 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2317 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2360 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2154 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2232 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2199 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2361 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2397 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2268 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2321 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2364 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2408 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2275 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2318 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2246 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2457 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2387 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2370 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2232 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2250 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2401 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2289 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2471 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2324 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2403 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2327 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2345 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2246 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2312 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2429 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2394 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2362 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2378 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2399 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2341 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2330 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2392 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2294 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2392 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2404 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2291 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2426 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2392 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2260 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2319 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2442 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2365 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2417 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2307 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2456 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2325 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2388 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2428 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2277 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2440 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2444 | 0 | 0 | 0 | 0 | 0 |
    | &(trx->undo_mutex) | trx0trx.c | 2375 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 58209 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 196943 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 23 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1269 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 3386 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 50828 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 215 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 218 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 7182 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 92 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 92 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 32 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 41671 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 140242 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1512980 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 174 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 205 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 51 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 23 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 23 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 639 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 174 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 291 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 178 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 23 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 6034 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 8 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 178 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 174 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 272 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 572 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 192331 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 174 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 195 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 7238 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 300 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1218397 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 34 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 92 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 36 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 92 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 92 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 50 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 44 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1044 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 30 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 32 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 32 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 28 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 85 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 87 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 19 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 85 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 91 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 126 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 12821 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 19708 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 202 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 205 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 224 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 53 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 53 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 25 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 25 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1135 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 716 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 207 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 214 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 206 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 853 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 39 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 39 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 39 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 78 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 39 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 36 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 33 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 33 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 33 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 41 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 36 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 88 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 34 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 39 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 1830 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 95 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 579 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 102 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 395 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 150 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 146 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 468 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 217 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 16509 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 170 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 863234 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 19897 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 27028410 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 158 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 522 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 554 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 70211676 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 932554 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 144 | 0 | 0 | 0 | 0 | 0 |
    | &(table->autoinc_mutex) | dict0mem.c | 3494 | 0 | 0 | 0 | 0 | 0 |
    | &(purge_sys->mutex) | trx0purge.c | 229886976 | 0 | 0 | 0 | 0 | 0 |
    | &(rseg->mutex) | trx0rseg.c | 589392072 | 582932 | 5876949 | 88884 | 197014 | 0 |
    | &ibuf_bitmap_mutex | ibuf0ibuf.c | 371437 | 0 | 0 | 0 | 0 | 0 |
    | &ibuf_mutex | ibuf0ibuf.c | 3849561 | 0 | 0 | 0 | 0 | 0 |
    | &dict_foreign_err_mutex | dict0dict.c | 372930 | 0 | 0 | 0 | 0 | 0 |
    | &(dict_sys->mutex) | dict0dict.c | 259527067 | 51378 | 274779 | 1632 | 3479 | 0 |
    | &(recv_sys->mutex) | log0recv.c | 64 | 12 | 121 | 6 | 6 | 0 |
    | &ios_mutex | srv0start.c | 13112284 | 4 | 0 | 0 | 0 | 0 |
    | &(log_sys->mutex) | log0log.c | 773885913 | 466730 | 4249691 | 136192 | 180760 | 0 |
    | &(buf_pool->mutex) | buf0buf.c | 341813408575 | 5131618035 | 5198457973 | 6753293 | 3944667 | 0 |
    | &(system->mutex) | fil0fil.c | 81742208 | 66697 | 53105 | 35 | 40 | 0 |
    | &srv_misc_tmpfile_mutex | srv0start.c | 372182 | 0 | 0 | 0 | 0 | 0 |
    | &srv_dict_tmpfile_mutex | srv0start.c | 55459 | 0 | 0 | 0 | 0 | 0 |
    | &srv_monitor_file_mutex | srv0start.c | 1 | 0 | 0 | 0 | 0 | 0 |
    | &srv_innodb_monitor_mutex | srv0srv.c | 57877 | 0 | 0 | 0 | 0 | 0 |
    | &kernel_mutex | srv0srv.c | 8326330740 | 17826281 | 130886065 | 1071867 | 2746135 | 0 |
    | &thr_local_mutex | thr0loc.c | 811915 | 0 | 0 | 0 | 0 | 0 |
    | &(pool->mutex) | mem0pool.c | 150512213895 | 1329851104 | 2160576205 | 459450 | 389068 | 0 |
    | &rw_lock_list_mutex | sync0sync.c | 138309 | 0 | 0 | 0 | 0 | 0 |
    | rw_lock_mutexes | | 395971608584 | 222851003 | 305839693 | 411527 | 114906 | 0 |
    +—————————+————-+————–+————+————-+———-+———–+—————+
    388 rows in set (0.03 sec)

    August 4, 2006 at 6:27 am
  • Vadim

    Zemo,

    Thank you,
    As you see
    buf_pool->mutex is most expensive one:
    | &(buf_pool->mutex) | buf0buf.c | 341813408575 | 5131618035 | 5198457973 | 6753293 | 3944667 | 0 |

    So I think you hit the described problem and my patch should relax load on the mutex.
    Please inform me about results!

    August 4, 2006 at 8:50 am
  • Zemo

    I will be testing this out on our quad opteron 875 server this weekend, I’ll keep you posted on the results.

    Thanks!,

    August 4, 2006 at 3:57 pm