GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Percona / Innodb crashed three times (found possible reason and fix)

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Percona / Innodb crashed three times (found possible reason and fix)

    What could be wrong here. I switched from Mysql 5.5.23 to Percona 5.5.23 and converted most of the tables from MyISAM to Innodb.

    Since then Percona has stalled three times and every time I have had to do 'kill -9 mysql' so that I could start it again ('mysql stop' just keeps doing ......... ........).

    Here are the logs:

    1)

    Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652914944 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
    InnoDB: Pending preads 0, pwrites 0

    =====================================
    120523 21:00:18 INNODB MONITOR OUTPUT
    =====================================
    Per second averages calculated from the last 36 seconds
    -----------------
    BACKGROUND THREAD
    -----------------
    srv_master_thread loops: 368042 1_second, 367727 sleeps, 36613 10_second, 2156 background, 2156 flush
    srv_master_thread log flush and writes: 425253
    ----------
    SEMAPHORES

    ----------
    OS WAIT ARRAY INFO: reservation count 77805, signal count 105351
    --Thread 139679652914944 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679655577344 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679656908544 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679656642304 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679654778624 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139656811095808 has waited at buf0flu.c line 1481 for 256.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679652382464 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679661434624 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679663032064 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679665428224 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656747890432 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679657441024 has waited at buf0buf.c line 2527 for 253.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139656381863680 has waited at buf0buf.c line 2527 for 252.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679663830784 has waited at dict0dict.c line 742 for 252.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679664895744 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679655843584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679662499584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679661967104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    --Thread 139679655311104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679664363264 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679652116224 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679663564544 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679653447424 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679664629504 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679657707264 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679661168384 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679736440576 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679657973504 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679656376064 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679659570944 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679654512384 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679652648704 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679660369664 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679651849984 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679658505984 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679735641856 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679657174784 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679664097024 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679658239744 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

    --Thread 139679736174336 has waited at dict0dict.c line 742 for 240.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679735375616 has waited at dict0dict.c line 742 for 239.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679654246144 has waited at dict0dict.c line 742 for 239.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679734843136 has waited at dict0dict.c line 742 for 237.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679653181184 has waited at dict0dict.c line 742 for 235.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679663298304 has waited at dict0dict.c line 742 for 235.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139679661700864 has waited at dict0dict.c line 742 for 234.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656758646528 has waited at buf0buf.c line 2527 for 232.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 139679660103424 has waited at dict0dict.c line 742 for 230.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656747624192 has waited at dict0dict.c line 742 for 228.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656747357952 has waited at dict0dict.c line 742 for 223.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656747091712 has waited at dict0dict.c line 742 for 222.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656746825472 has waited at dict0dict.c line 742 for 221.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656746559232 has waited at dict0dict.c line 742 for 219.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656746292992 has waited at dict0dict.c line 742 for 214.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656746026752 has waited at dict0dict.c line 742 for 207.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656745760512 has waited at dict0dict.c line 742 for 203.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656745494272 has waited at dict0dict.c line 742 for 202.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656745228032 has waited at dict0dict.c line 742 for 194.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656744961792 has waited at dict0dict.c line 742 for 194.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656744695552 has waited at dict0dict.c line 742 for 139.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656744429312 has waited at dict0dict.c line 742 for 139.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656744163072 has waited at dict0dict.c line 742 for 98.000 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656743896832 has waited at dict0dict.c line 742 for 39.000 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 139656743630592 has waited at dict0dict.c line 742 for 17.000 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    Mutex spin waits 174645, rounds 933148, OS waits 8825
    RW-shared spins 65088, rounds 1443040, OS waits 45489

    RW-excl spins 15637, rounds 777875, OS waits 22758
    Spin rounds per wait: 5.34 mutex, 22.17 RW-shared, 49.75 RW-excl
    --------
    FILE I/O
    --------
    I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
    I/O thread 1 state: waiting for completed aio requests (log thread)
    I/O thread 2 state: waiting for completed aio requests (read thread)
    I/O thread 3 state: waiting for completed aio requests (read thread)
    I/O thread 4 state: waiting for completed aio requests (read thread)
    I/O thread 5 state: waiting for completed aio requests (read thread)
    I/O thread 6 state: waiting for completed aio requests (write thread)
    I/O thread 7 state: waiting for completed aio requests (write thread)
    I/O thread 8 state: waiting for completed aio requests (write thread)
    I/O thread 9 state: waiting for completed aio requests (write thread)
    Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
    ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
    Pending flushes (fsync) log: 0; buffer pool: 0
    175995 OS file reads, 5893732 OS file writes, 793642 OS fsyncs
    0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
    -------------------------------------
    INSERT BUFFER AND ADAPTIVE HASH INDEX
    -------------------------------------
    Ibuf: size 1, free list len 8868, seg size 8870, 995 merges
    merged operations:
    insert 12034, delete mark 756, delete 157
    discarded operations:
    insert 0, delete mark 0, delete 0
    Hash table size 42499631, node heap has 8001 buffer(s)
    0.00 hash searches/s, 0.00 non-hash searches/s
    ---
    LOG
    ---
    Log sequence number 73408964340
    Log flushed up to 73408964340
    Last checkpoint at 73408964310
    Max checkpoint age 433446544
    Checkpoint age target 419901340
    Modified age 30
    Checkpoint age 30
    0 pending log writes, 0 pending chkp writes
    2419317 log i/o's done, 0.00 log i/o's/second
    ----------------------
    BUFFER POOL AND MEMORY
    ----------------------
    Total memory allocated 22062039040; in additional pool allocated 0
    Internal hash tables (constant factor + variable factor)
    Adaptive hash index 471089648 (339997048 + 131092600)
    Page hash 21250552 (buffer pool 0 only)
    Dictionary cache 90295032 (85000912 + 5294120)
    File system 400352 (82672 + 317680)
    Lock system 53141048 (53125256 + 15792)
    Recovery system 0 (0 + 0)
    Dictionary memory allocated 5294120

    InnoDB: ###### Diagnostic info printed to the standard error stream
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652914944 has waited at dict0dict.c line 742 for 272.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655577344 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656908544 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656642304 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654778624 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656811095808 has waited at buf0flu.c line 1481 for 270.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652382464 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661434624 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663032064 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679665428224 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747890432 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657441024 has waited at buf0buf.c line 2527 for 267.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656381863680 has waited at buf0buf.c line 2527 for 266.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663830784 has waited at dict0dict.c line 742 for 266.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664895744 has waited at dict0dict.c line 742 for 265.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655843584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679662499584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661967104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655311104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664363264 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652116224 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663564544 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679653447424 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664629504 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657707264 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661168384 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679736440576 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657973504 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656376064 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679659570944 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654512384 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652648704 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679660369664 has waited at buf0buf.c line 2527 for 260.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679651849984 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679658505984 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679735641856 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657174784 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664097024 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679658239744 has waited at dict0dict.c line 742 for 256.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679736174336 has waited at dict0dict.c line 742 for 254.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679735375616 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654246144 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679734843136 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679653181184 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663298304 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661700864 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    InnoDB: Warning: a long semaphore wait:
    --Thread 139656758646528 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679660103424 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747624192 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
    InnoDB: Pending preads 0, pwrites 0
    InnoDB: ###### Diagnostic info printed to the standard error stream
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652914944 has waited at dict0dict.c line 742 for 303.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655577344 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656908544 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656642304 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654778624 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656811095808 has waited at buf0flu.c line 1481 for 301.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652382464 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661434624 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663032064 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679665428224 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747890432 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657441024 has waited at buf0buf.c line 2527 for 298.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656381863680 has waited at buf0buf.c line 2527 for 297.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663830784 has waited at dict0dict.c line 742 for 297.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664895744 has waited at dict0dict.c line 742 for 296.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655843584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679662499584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661967104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679655311104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664363264 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652116224 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663564544 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679653447424 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664629504 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657707264 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661168384 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679736440576 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657973504 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679656376064 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

    InnoDB: Warning: a long semaphore wait:
    --Thread 139679659570944 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654512384 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679652648704 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679660369664 has waited at buf0buf.c line 2527 for 291.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679651849984 has waited at dict0dict.c line 742 for 289.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679658505984 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679735641856 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679657174784 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679664097024 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679658239744 has waited at dict0dict.c line 742 for 287.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679736174336 has waited at dict0dict.c line 742 for 285.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679735375616 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679654246144 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679734843136 has waited at dict0dict.c line 742 for 282.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679653181184 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679663298304 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139679661700864 has waited at dict0dict.c line 742 for 279.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656758646528 has waited at buf0buf.c line 2527 for 277.00 seconds the semaphore:
    S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
    a writer (thread id 139679655577344) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

    InnoDB: Warning: a long semaphore wait:
    --Thread 139679660103424 has waited at dict0dict.c line 742 for 275.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747624192 has waited at dict0dict.c line 742 for 273.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747357952 has waited at dict0dict.c line 742 for 268.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656747091712 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656746825472 has waited at dict0dict.c line 742 for 266.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656746559232 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656746292992 has waited at dict0dict.c line 742 for 259.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656746026752 has waited at dict0dict.c line 742 for 252.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656745760512 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: Warning: a long semaphore wait:
    --Thread 139656745494272 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
    Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
    InnoDB: Pending preads 0, pwrites 0
    InnoDB: ###### Diagnostic info printed to the standard error stream
    120523 21:26:09 [Warning] Too many connections
    120523 21:26:09 [Warning] Too many connections
    .....

    120523 21:35:49 mysqld_safe Number of processes running now: 0
    120523 21:35:49 mysqld_safe mysqld restarted
    120523 21:35:49 [Note] Flashcache bypass: disabled
    120523 21:35:49 [Note] Flashcache setup error is : ioctl failed

    120523 21:35:49 [Note] Plugin 'FEDERATED' is disabled.
    120523 21:35:49 InnoDB: The InnoDB memory heap is disabled
    120523 21:35:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    120523 21:35:49 InnoDB: Compressed tables use zlib 1.2.3
    120523 21:35:49 InnoDB: Using Linux native AIO
    120523 21:35:49 InnoDB: Initializing buffer pool, size = 20.0G
    120523 21:35:50 InnoDB: Completed initialization of buffer pool
    120523 21:35:50 InnoDB: highest supported file format is Barracuda.
    InnoDB: Log scan progressed past the checkpoint lsn 73408964310
    120523 21:35:50 InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 73408964340
    120523 21:35:52 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
    InnoDB: Apply batch completed
    120523 21:35:53 InnoDB: Waiting for the background threads to start
    120523 21:35:54 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 73408964340
    120523 21:35:54 [Note] Event Scheduler: Loaded 0 events
    120523 21:35:54 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

  • #2
    2.


    Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479082170112 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479079507712 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140502064256768 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479088293632 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479081903872 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479080572672 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479075780352 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479111403264 has waited at buf0flu.c line 1481 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
    InnoDB: Pending preads 0, pwrites 0

    =====================================
    120603 21:00:07 INNODB MONITOR OUTPUT
    =====================================
    Per second averages calculated from the last 30 seconds
    -----------------
    BACKGROUND THREAD
    -----------------
    srv_master_thread loops: 519884 1_second, 519864 sleeps, 51784 10_second, 2305 background, 2305 flush
    srv_master_thread log flush and writes: 551646
    ----------
    SEMAPHORES
    ----------
    OS WAIT ARRAY INFO: reservation count 83453, signal count 157821
    --Thread 140479082170112 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479079507712 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140502064256768 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479088293632 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479081903872 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479080572672 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479075780352 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479111403264 has waited at buf0flu.c line 1481 for 247.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479100913408 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479079241472 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    --Thread 140479077111552 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479078708992 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479088027392 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140502064789248 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479086163712 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479081371392 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140479089624832 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140502065055488 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140479090157312 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140479086429952 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140502004061952 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    --Thread 140479074449152 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    --Thread 140479084300032 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1

    --Thread 140479071254272 has waited at dict0dict.c line 742 for 58.000 seconds the semaphore:
    Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
    waiters flag 1
    Mutex spin waits 218766, rounds 734044, OS waits 8318
    RW-shared spins 96974, rounds 1807098, OS waits 56729
    RW-excl spins 22955, rounds 698339, OS waits 18061
    Spin rounds per wait: 3.36 mutex, 18.63 RW-shared, 30.42 RW-excl
    --------
    FILE I/O
    --------
    I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
    I/O thread 1 state: waiting for completed aio requests (log thread)
    I/O thread 2 state: waiting for completed aio requests (read thread)
    I/O thread 3 state: waiting for completed aio requests (read thread)
    I/O thread 4 state: waiting for completed aio requests (read thread)
    I/O thread 5 state: waiting for completed aio requests (read thread)
    I/O thread 6 state: waiting for completed aio requests (write thread)
    I/O thread 7 state: waiting for completed aio requests (write thread)
    I/O thread 8 state: waiting for completed aio requests (write thread)
    I/O thread 9 state: waiting for completed aio requests (write thread)
    Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
    ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
    Pending flushes (fsync) log: 0; buffer pool: 0
    210627 OS file reads, 9171450 OS file writes, 1138010 OS fsyncs
    0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
    -------------------------------------
    INSERT BUFFER AND ADAPTIVE HASH INDEX
    -------------------------------------
    Ibuf: size 1, free list len 8868, seg size 8870, 2975 merges
    merged operations:
    insert 12697, delete mark 15496, delete 493
    discarded operations:
    insert 0, delete mark 0, delete 0
    Hash table size 42499631, node heap has 8787 buffer(s)
    0.00 hash searches/s, 0.00 non-hash searches/s
    ---
    LOG
    ---
    Log sequence number 82147830513
    Log flushed up to 82147830513
    Last checkpoint at 82147805141
    Max checkpoint age 433446544
    Checkpoint age target 419901340
    Modified age 483
    Checkpoint age 25372
    0 pending log writes, 0 pending chkp writes
    3583346 log i/o's done, 0.00 log i/o's/second
    ----------------------
    BUFFER POOL AND MEMORY
    ----------------------
    Total memory allocated 22062039040; in additional pool allocated 0
    Internal hash tables (constant factor + variable factor)
    Adaptive hash index 483967472 (339997048 + 143970424)
    Page hash 21250552 (buffer pool 0 only)
    Dictionary cache 89768970 (85000912 + 4768058)
    File system 402992 (82672 + 320320)
    Lock system 53138416 (53125256 + 13160)
    Recovery system 0 (0 + 0)
    Dictionary memory allocated 4768058

    InnoDB: ###### Diagnostic info printed to the standard error stream
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479079507712 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140502064256768 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479088293632 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479081903872 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479080572672 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
    InnoDB: Warning: a long semaphore wait:
    --Thread 140479075780352 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
    S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
    a writer (thread id 140479082170112) has reserved it in mode exclusive
    number of readers 0, waiters flag 1, lock_word: 0
    Last time read locked in file buf0buf.c line 2527
    Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

    120604 00:28:08 mysqld_safe Starting mysqld daemon with databases from /sdd/mysql
    120604 0:28:08 [Note] Flashcache bypass: disabled
    120604 0:28:08 [Note] Flashcache setup error is : ioctl failed

    120604 0:28:08 [Note] Plugin 'FEDERATED' is disabled.
    120604 0:28:08 InnoDB: The InnoDB memory heap is disabled
    120604 0:28:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
    120604 0:28:08 InnoDB: Compressed tables use zlib 1.2.3
    120604 0:28:08 InnoDB: Using Linux native AIO
    120604 0:28:08 InnoDB: Initializing buffer pool, size = 20.0G
    120604 0:28:09 InnoDB: Completed initialization of buffer pool
    120604 0:28:09 InnoDB: highest supported file format is Barracuda.
    InnoDB: Log scan progressed past the checkpoint lsn 82147805141
    120604 0:28:09 InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 82147830513
    120604 0:28:11 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
    InnoDB: Apply batch completed
    120604 0:28:13 InnoDB: Waiting for the background threads to start
    120604 0:28:14 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 82147830513
    120604 0:28:14 [Note] Event Scheduler: Loaded 0 events
    120604 0:28:14 [Note] /usr/local/mysql/bin/mysqld: ready for connections.


    (I made this a little bit shorter because of those similar lines there.... Thread numer has waited... and so on...

    Comment


    • #3
      3.


      Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
      InnoDB: Warning: a long semaphore wait:
      --Thread 139789856839424 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      InnoDB: Warning: a long semaphore wait:
      --Thread 139789782632192 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      InnoDB: Warning: a long semaphore wait:
      --Thread 139766884292352 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
      InnoDB: Pending preads 0, pwrites 0

      =====================================
      120611 21:00:09 INNODB MONITOR OUTPUT
      =====================================
      Per second averages calculated from the last 23 seconds
      -----------------
      BACKGROUND THREAD
      -----------------
      srv_master_thread loops: 655361 1_second, 655337 sleeps, 65197 10_second, 3784 background, 3784 flush
      srv_master_thread log flush and writes: 698575
      ----------
      SEMAPHORES
      ----------
      OS WAIT ARRAY INFO: reservation count 119896, signal count 213227
      --Thread 139789856839424 has waited at buf0buf.c line 2527 for 249.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      --Thread 139789782632192 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139766884292352 has waited at buf0buf.c line 2527 for 249.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      --Thread 139789780236032 has waited at buf0buf.c line 2527 for 248.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      --Thread 139789781567232 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0
      Last time read locked in file buf0buf.c line 2527
      Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
      --Thread 139789856573184 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139789785827072 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139789779437312 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1

      --Thread 139766866614016 has waited at dict0dict.c line 742 for 226.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139766865815296 has waited at dict0dict.c line 742 for 163.00 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139766865549056 has waited at dict0dict.c line 742 for 49.000 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      --Thread 139766866347776 has waited at dict0dict.c line 742 for 38.000 seconds the semaphore:
      Mutex at 0x2440b738 '&dict_sys->mutex', lock var 1
      waiters flag 1
      Mutex spin waits 2114760, rounds 1000421, OS waits 8770
      RW-shared spins 148421, rounds 2962902, OS waits 89565
      RW-excl spins 32041, rounds 834483, OS waits 21034
      Spin rounds per wait: 0.47 mutex, 19.96 RW-shared, 26.04 RW-excl
      --------
      FILE I/O
      --------
      I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
      I/O thread 1 state: waiting for completed aio requests (log thread)
      I/O thread 2 state: waiting for completed aio requests (read thread)
      I/O thread 3 state: waiting for completed aio requests (read thread)
      I/O thread 4 state: waiting for completed aio requests (read thread)
      I/O thread 5 state: waiting for completed aio requests (read thread)
      I/O thread 6 state: waiting for completed aio requests (write thread)
      I/O thread 7 state: waiting for completed aio requests (write thread)
      I/O thread 8 state: waiting for completed aio requests (write thread)
      I/O thread 9 state: waiting for completed aio requests (write thread)
      Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
      ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
      Pending flushes (fsync) log: 0; buffer pool: 0
      213895 OS file reads, 11478013 OS file writes, 1496154 OS fsyncs
      0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
      -------------------------------------
      INSERT BUFFER AND ADAPTIVE HASH INDEX
      -------------------------------------
      Ibuf: size 1, free list len 8868, seg size 8870, 2347 merges
      merged operations:
      insert 14321, delete mark 5444, delete 512
      discarded operations:
      insert 0, delete mark 0, delete 0
      Hash table size 42499631, node heap has 13984 buffer(s)
      0.00 hash searches/s, 0.00 non-hash searches/s
      ---
      LOG
      ---
      Log sequence number 88320533337
      Log flushed up to 88320533337
      Last checkpoint at 88320533337
      Max checkpoint age 433446544
      Checkpoint age target 419901340
      Modified age 0
      Checkpoint age 0
      0 pending log writes, 0 pending chkp writes
      4361377 log i/o's done, 0.00 log i/o's/second
      ----------------------
      BUFFER POOL AND MEMORY
      ----------------------
      Total memory allocated 22062039040; in additional pool allocated 0
      Internal hash tables (constant factor + variable factor)
      Adaptive hash index 569115120 (339997048 + 229118072)
      Page hash 21250552 (buffer pool 0 only)
      Dictionary cache 87971625 (85000912 + 2970713)
      File system 402992 (82672 + 320320)
      Lock system 53140672 (53125256 + 15416)
      Recovery system 0 (0 + 0)
      Dictionary memory allocated 2970713
      InnoDB: ###### Diagnostic info printed to the standard error stream
      InnoDB: Warning: a long semaphore wait:
      --Thread 139789856839424 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
      S-lock on RW-latch at 0x27dd7f8 '&buf_pool->page_hash_latch'
      a writer (thread id 139789856839424) has reserved it in mode exclusive
      number of readers 0, waiters flag 1, lock_word: 0

      120611 21:52:29 mysqld_safe Starting mysqld daemon with databases from /sdd/mysql
      120611 21:52:29 [Note] Flashcache bypass: disabled
      120611 21:52:29 [Note] Flashcache setup error is : ioctl failed

      120611 21:52:29 [Note] Plugin 'FEDERATED' is disabled.
      120611 21:52:29 InnoDB: The InnoDB memory heap is disabled
      120611 21:52:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      120611 21:52:29 InnoDB: Compressed tables use zlib 1.2.3
      120611 21:52:29 InnoDB: Using Linux native AIO
      120611 21:52:29 InnoDB: Initializing buffer pool, size = 20.0G
      120611 21:52:30 InnoDB: Completed initialization of buffer pool
      120611 21:52:30 InnoDB: highest supported file format is Barracuda.
      InnoDB: The log sequence number in ibdata files does not match
      InnoDB: the log sequence number in the ib_logfiles!
      120611 21:52:30 InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      120611 21:52:33 InnoDB: Waiting for the background threads to start
      120611 21:52:34 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 88320533337
      120611 21:52:34 [Note] Event Scheduler: Loaded 0 events
      120611 21:52:34 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

      Comment


      • #4
        It seems that it happens every time at 20.56. The only thing that happens exactly at that minute is vBulletin's Thread Views cron task. However that runs every hour (at 56 minute point) so any ideas?

        Comment


        • #5
          How are your backups performed?
          Could you have a SAN that takes snapshots of the disk?
          Or a remote backupserver that locks files during backup?

          I do not believe in coincidences and although there isn't anything in the local crontab. Things like backups (which are very often executed by a remote server) and sometimes some applications contain grooming or monitoring/software upgrade/patch solutions that are network wide run regularly.

          Good luck!

          Comment


          • #6
            Hello!

            Thanks for you suggestion - unfortunately this can't be a backup issue - I only take backups at 04.00 am using innobackupex.

            That specific cron task performs only vBulletin (discussion forum software) thread views counting. "This script updates thread views once per hour. This option is suggested for forums with large numbers of threads or users.".

            Of course this whole issue could be something else but it oddly coincides with that 56 minute mark (because that script runs every hour and these stalls have happened only at 8 pm 56min.

            Hopefully somebody is able to go through those logs and explain what could cause this (I know that error is partly of because Innodb waits for something and then spits out that "--Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:" when it's 'tired' of waiting. BUT of course the big thing is that why it stalls the whole mysql until I kill -9 it.

            Comment


            • #7
              It happened again (fourth time) This time at 15.00 and if I count backwards that waiting period of 240 seconds or so it happened again at 56 minute point.

              Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124260845312 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124300289792 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
              Mutex at 0x259b6ed8 '&dict_sys->mutex', lock var 1
              waiters flag 1
              InnoDB: Warning: a long semaphore wait:
              --Thread 140101320640256 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124263507712 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124300556032 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
              Mutex at 0x259b6ed8 '&dict_sys->mutex', lock var 1
              waiters flag 1
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124262176512 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
              InnoDB: Pending preads 0, pwrites 0

              =====================================
              120614 15:00:03 INNODB MONITOR OUTPUT
              =====================================
              Per second averages calculated from the last 25 seconds
              -----------------
              BACKGROUND THREAD
              -----------------
              srv_master_thread loops: 226990 1_second, 226980 sleeps, 22579 10_second, 1376 background, 1376 flush
              srv_master_thread log flush and writes: 241183

              OS WAIT ARRAY INFO: reservation count 46612, signal count 76434
              --Thread 140124260845312 has waited at buf0buf.c line 2527 for 242.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              --Thread 140124300289792 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
              Mutex at 0x259b6ed8 '&dict_sys->mutex', lock var 1
              waiters flag 1
              --Thread 140101320640256 has waited at buf0buf.c line 2527 for 242.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              --Thread 140124263507712 has waited at buf0buf.c line 2527 for 242.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              --Thread 140124300556032 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
              Mutex at 0x259b6ed8 '&dict_sys->mutex', lock var 1
              waiters flag 1
              ......
              Mutex spin waits 188592, rounds 255805, OS waits 2270
              RW-shared spins 67580, rounds 1287291, OS waits 40076
              RW-excl spins 8431, rounds 176374, OS waits 4151
              Spin rounds per wait: 1.36 mutex, 19.05 RW-shared, 20.92 RW-excl
              --------
              FILE I/O
              --------
              I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
              I/O thread 1 state: waiting for completed aio requests (log thread)
              I/O thread 2 state: waiting for completed aio requests (read thread)
              I/O thread 3 state: waiting for completed aio requests (read thread)
              I/O thread 4 state: waiting for completed aio requests (read thread)
              I/O thread 5 state: waiting for completed aio requests (read thread)
              I/O thread 6 state: waiting for completed aio requests (write thread)
              I/O thread 7 state: waiting for completed aio requests (write thread)
              I/O thread 8 state: waiting for completed aio requests (write thread)
              I/O thread 9 state: waiting for completed aio requests (write thread)
              Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
              ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
              Pending flushes (fsync) log: 0; buffer pool: 0
              162297 OS file reads, 3148308 OS file writes, 503682 OS fsyncs
              0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
              -------------------------------------
              INSERT BUFFER AND ADAPTIVE HASH INDEX
              -------------------------------------
              Ibuf: size 1, free list len 8868, seg size 8870, 2611 merges
              merged operations:
              insert 7492, delete mark 14839, delete 337
              discarded operations:
              insert 0, delete mark 0, delete 0
              Hash table size 42499631, node heap has 4455 buffer(s)
              0.08 hash searches/s, 0.00 non-hash searches/s
              ---
              LOG
              ---
              Log sequence number 90419530995
              Log flushed up to 90419530995
              Last checkpoint at 90419530995
              Max checkpoint age 433446544
              Checkpoint age target 419901340
              Modified age 0
              Checkpoint age 0
              0 pending log writes, 0 pending chkp writes
              1176001 log i/o's done, 0.00 log i/o's/second
              ----------------------
              BUFFER POOL AND MEMORY
              ----------------------
              Total memory allocated 22062039040; in additional pool allocated 0
              Internal hash tables (constant factor + variable factor)
              Adaptive hash index 412991984 (339997048 + 72994936)
              Page hash 21250552 (buffer pool 0 only)
              Dictionary cache 87839493 (85000912 + 2838581)
              File system 405192 (82672 + 322520)
              Lock system 53139544 (53125256 + 14288)
              Recovery system 0 (0 + 0)
              Dictionary memory allocated 2838581
              InnoDB: ###### Diagnostic info printed to the standard error stream
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124260845312 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
              S-lock on RW-latch at 0x3d8c7f8 '&buf_pool->page_hash_latch'
              a writer (thread id 140124260845312) has reserved it in mode exclusive
              number of readers 0, waiters flag 1, lock_word: 0
              Last time read locked in file buf0buf.c line 2527
              Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
              InnoDB: Warning: a long semaphore wait:
              --Thread 140124300289792 has waited at dict0dict.c line 742 for 272.00 seconds the semaphore:
              Mutex at 0x259b6ed8 '&dict_sys->mutex', lock var 1
              ...

              and then kill -9 and restart:

              120614 15:03:18 [Note] Flashcache bypass: disabled
              120614 15:03:18 [Note] Flashcache setup error is : ioctl failed

              120614 15:03:18 [Note] Plugin 'FEDERATED' is disabled.
              120614 15:03:18 InnoDB: The InnoDB memory heap is disabled
              120614 15:03:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
              120614 15:03:18 InnoDB: Compressed tables use zlib 1.2.3
              120614 15:03:18 InnoDB: Using Linux native AIO
              120614 15:03:18 InnoDB: Initializing buffer pool, size = 20.0G
              120614 15:03:19 InnoDB: Completed initialization of buffer pool
              120614 15:03:19 InnoDB: highest supported file format is Barracuda.
              InnoDB: The log sequence number in ibdata files does not match
              InnoDB: the log sequence number in the ib_logfiles!
              120614 15:03:19 InnoDB: Database was not shut down normally!
              InnoDB: Starting crash recovery.
              InnoDB: Reading tablespace information from the .ibd files...
              InnoDB: Restoring possible half-written data pages from the doublewrite
              InnoDB: buffer...
              120614 15:03:21 InnoDB: Waiting for the background threads to start
              120614 15:03:22 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 90419530995
              120614 15:03:22 [Note] Event Scheduler: Loaded 0 events
              120614 15:03:22 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
              Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240

              Comment


              • #8
                Great.. now it happened fifth time (twice today) - this time at 18.00.

                Comment


                • #9
                  Ok - again.

                  I think I found the reason.

                  That vBulletin's Thread View update script contains both "truncate" table and "drop" table.
                  $vbulletin->db->query_write("TRUNCATE TABLE " . TABLE_PREFIX . "threadviews");
                  and
                  $vbulletin->db->query_write("DROP TABLE IF EXISTS " . TABLE_PREFIX . $aggtable);

                  I think I encountered that issue with InnoDB's slow drop table function as described here:
                  http://www.mysqlperformanceblog.com/2011/02/03/performance-p roblem-with-innodb-and-drop-table/

                  and

                  http://www.mysqlperformanceblog.com/2011/04/20/drop-table-pe rformance/

                  I think my two options are 1) either to use that "innodb_lazy_drop_table" option or 2) instead of updating thread views on a hourly basis, update them immediately.

                  Can someone read the restarting logs and give an opinion if according to those logs during every restart tables were either ok, or repaired.
                  I'm just curious because the restart log is very different in first one compared to the rest "InnoDB: Progress in percents: 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed"
                  as the rest didn't have that. Is it because the transaction log was probably flushed during or just before that stall except in the first and second one?

                  "Modified age 30
                  Checkpoint age 30"
                  versus
                  "Modified age 0
                  Checkpoint age 0"

                  Comment

                  Working...
                  X