GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

--apply-log sporadic failures

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • --apply-log sporadic failures

    Hi all!
    Trying to fight sporadic restore failures. I'm backuping about 200 mysql instances from slave replicas, average of 6 slave instances per server. Script is simply runs innobackupex over instances one by one.
    Backup string: timeout -k 30s 3h ionice -c 3 -n 5 innobackupex --safe-slave-backup --slave-info --socket=/var/run/mysqld/mysqld-$ID.sock --defaults-file=/etc/mysql/mysqld-$ID.cnf --stream=xbstream \
    /tmp | pbzip2 --fast | ssh -c arcfour $backupserver "dd of=/backup/mysql/$masterserver-`date +%Y-%M-%d`.xbz2"

    After all backups created, on backup server I starting backup consistency checks by simply extracting backup and applying logs:
    pbzip2 -vdc $file | xbstream -v -x -C /backup/checkdir/ && cd /backup/checkdir/

    innobackupex --ibbackup=xtrabackup_55 --apply-log /backup/checkdir/

    And sometime I got the following errors applying log:

    InnoDB: Doing recovery: scanned up to log sequence number 4925695334912 (35 %)
    InnoDB: Doing recovery: scanned up to log sequence number 4925700577792 (36 %)
    140510 12:30:17 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 0 1 2 3 InnoDB: Probable data corruption on page 2284374
    InnoDB: Original record (compact record)
    InnoDB: on that page.
    InnoDB: Cannot find the dir slot for record (compact record)
    InnoDB: on that page!
    140510 12:30:17 InnoDB: Page dump in ascii and hex (16384 bytes):
    len 16384; hex 000000000022db560022db540022db570000047ad99e7cf945 bf00000000000000000000009300203aad807f000000000000 0002007c007d00000000000000000000000000000000018100 00000000000000000000000000000000000000010002002069 6e66696d756d0006000b0$
    InnoDB: End of page dump
    140510 12:30:17 InnoDB: Page checksum 545978757 (32bit_calc: 3531001921), prior-to-4.0.14-form checksum 1476022376
    InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 1146
    InnoDB: Page lsn 1146 3651042553, low 4 bytes of lsn at page end 3651042553
    InnoDB: Page number (if stored to page already) 2284374,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 147
    InnoDB: Page may be an index page where index id is 385
    140510 12:30:17 InnoDB: Assertion failure in thread 140664688744192 in file page0page.c line 153
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
    InnoDB: about forcing recovery.
    innobackupex: Error:
    innobackupex: ibbackup failed at /usr/local/bin/innobackupex line 2560.

    OR

    InnoDB: Doing recovery: scanned up to log sequence number 4988629964288 (61 %)
    InnoDB: Doing recovery: scanned up to log sequence number 4988635207168 (62 %)
    140510 13:20:18 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 140510 13:20:19 InnoDB: Assertion failure in thread 140548991547136 in file rem0rec.c line 569
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
    InnoDB: about forcing recovery.
    innobackupex: Error:
    innobackupex: ibbackup failed at /usr/local/bin/innobackupex line 2560.

    OR

    InnoDB: Doing recovery: scanned up to log sequence number 4827916240896 (30 %)
    InnoDB: Doing recovery: scanned up to log sequence number 4827921483776 (37 %)
    140509 11:23:17 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 InnoDB: Probable data corruption on page
    2307494
    InnoDB: Original record (compact record)
    InnoDB: on that page.
    InnoDB: Cannot find the dir slot for record (compact record)
    InnoDB: on that page!
    140509 11:23:22 InnoDB: Page dump in ascii and hex (16384 bytes):
    len 16384; hex 000000 .... ---CUT---
    ...
    InnoDB: End of page dump
    140509 11:23:22 InnoDB: Page checksum 1244108629 (32bit_calc: 3907307010), prior-to-4.0.14-form checksum 2889030552
    InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 1124
    InnoDB: Page lsn 1124 377661183, low 4 bytes of lsn at page end 377661183
    InnoDB: Page number (if stored to page already) 2307494,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 204
    InnoDB: Page may be an index page where index id is 523
    140509 11:23:22 InnoDB: Assertion failure in thread 139719660447488 in file page0page.c line 153
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
    InnoDB: about forcing recovery.
    innobackupex: Error:
    innobackupex: ibbackup failed at /usr/local/bin/innobackupex line 2560.


    There are no any regularity which backup fails: it could be 8 file broken or 0 from day to day, on different instances on different slave servers.
    Using:
    Percona mysqld Ver 5.5.16 for Linux on x86_64 (Source distribution)
    xtrabackup 2.1.8 (also tested on many previous versions)
    on Linux 2.6.39.2 x86_64

  • #2
    Updated xtrabackup to version 2.1.9. Still same problems.

    Comment


    • #3
      No chance to get any support on this important meter, when I have data corruption, using xtrabackup?

      Comment

      Working...
      X