GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

Xtrabackup is failing once log is applied

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Xtrabackup is failing once log is applied

    MariaDB 5.5.29, Xtrabackup 2.0.5 on el5 - all from repo. Works fine on several servers with smaller datasets, but fails on the big one (350G):


    InnoDB: Apply batch completed
    InnoDB: In a MySQL replication slave the last master binlog file
    InnoDB: position 0 995545447, file name mysql-bin.001309
    InnoDB: and relay log file
    InnoDB: position 0 995545731, file name ./mysql-relay-bin.002034
    InnoDB: Last MySQL binlog file position 0 276207520, file name ./mysql-bin.000005
    130417 10:18:35 InnoDB: Waiting for the background threads to start
    130417 10:18:36 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 13758730250258
    130417 10:18:38 InnoDB: Assertion failure in thread 1191156032 in file xtrabackup.c line 1632
    InnoDB: Failing assertion: cset == 0
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
    InnoDB: about forcing recovery.
    innobackupex: Error:
    innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.



    Any suggestions what to check? Workarounds may be?
    Thanks!


  • #2
    Do you use special features of MariaDB that are incompatible with standard InnoDB?

    Comment


    • #3
      Not really, only default ones. Not using area tables ether.

      Comment


      • #4
        I also have a problem.


        Packages info:

        Percona-Server-server-55-5.5.29-rel29.4.401.rhel6.x86_64
        percona-xtrabackup-2.0.5-499.rhel6.x86_64


        Commands:

        # cat /data/backup/mysql/2013-04-19_11-04-34/xtrabackup_checkpoints
        backup_type = full-prepared
        from_lsn = 0
        to_lsn = 308184144593
        last_lsn = 308882154381

        # innobackupex --apply-log --redo-only --use-memory=16G /data/backup/mysql/2013-04-19_11-04-34/ --incremental-dir=/data/backup/mysql/2013-04-19_13-42-17

        Partial error log:
        ......
        ......
        ......
        InnoDB: Doing recovery: scanned up to log sequence number 309111833600 (88 %)
        InnoDB: Doing recovery: scanned up to log sequence number 309114524155 (88 %)
        130419 17:33:34 InnoDB: Starting an apply batch of log records to the database...
        InnoDB: Progress in percents: 0 1 InnoDB: Probable data corruption on page 1662
        InnoDB: Original record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 32
        0: len 4; hex 00000043; asc C;;
        1: len 1; hex 00; asc ;;
        2: len 4; hex 000047a0; asc G ;;
        3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
        4: len 4; hex 81757165; asc uqe;;
        5: len 8; hex 8000013e1e903b68; asc > ;h;;
        6: len 8; hex 800000000042e94f; asc B O;;

        InnoDB: on that page.
        InnoDB: Cannot find the dir slot for record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
        0: len 4; hex 00000043; asc C;;
        1: len 1; hex 00; asc ;;
        2: len 4; hex 000047ce; asc G ;;
        3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
        4: len 4; hex 81828972; asc r;;
        5: len 8; hex 8000013e1ea8e7d0; asc > ;;
        6: len 8; hex 800000000042e9e9; asc B ;;

        InnoDB: on that page!
        130419 17:33:36 InnoDB: Page dump in ascii and hex (16384 bytes):
        ......
        ......
        ......
        InnoDB: End of page dump
        130419 17:33:36 InnoDB: Page checksum 2139845666 (32bit_calc: 3413430305), prior-to-4.0.14-form checksum 705155133
        InnoDB: stored checksum 700663176, prior-to-4.0.14-form stored checksum 71
        InnoDB: Page lsn 71 3951049913, low 4 bytes of lsn at page end 3951049913
        InnoDB: Page number (if stored to page already) 1662,
        InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
        InnoDB: Page may be an index page where index id is 18446744069414584320
        InnoDB: (index "CLUST_IND" of table "SYS_IBUF_TABLE")
        130419 17:33:36 InnoDB: Assertion failure in thread 140549583533824 in file page0page.c line 153
        InnoDB: We intentionally generate a memory trap.
        InnoDB: Submit a detailed bug report to [URL="http://bugs.mysql.com"]http://bugs.mysql.com[/URL="http://bugs.mysql.com"].
        InnoDB: If you get repeated assertion failures or crashes, even
        InnoDB: immediately after the mysqld startup, there may be
        InnoDB: corruption in the InnoDB tablespace. Please refer to
        InnoDB: [URL="http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html"]http://dev.mysql.com/doc/refman/5.5/...-recovery.html[/URL="http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html"]
        InnoDB: about forcing recovery.
        innobackupex: Error:
        innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.

        Last edited by harveyzh; 04-19-2013, 06:05 AM.

        Comment


        • #5
          I do encounter the same problem, like DmitryK - however, I do encounter it on MySQL instead of mariaDB.

          First of all, my versions:
          MySQL 5.5.27
          Percona XtraBackup 2.0.6

          What's strange in my case, is the fact that the backups for two of the three instances on my backup node can be restored without any hassle - but the third, and largest one ( about 1.2TB data-dir / 437GB zipped backup ) ultimately fails with the above mentioned assertion failure:

          Code:
          130426  9:04:02  InnoDB: Assertion failure in thread 139909354481408 in file xtrabackup.c line 1601
          InnoDB: Failing assertion: cset == 0
          I am using the following statements to create and restore the backup

          Creation:
          Code:
          innobackupex --slave-info --safe-slave-backup --stream=tar ./ | pigz -p 5 | split -d -b 100G - $FILENAME
          Restore:
          Code:
          cat 2013-04-23--11-51-59.xtrabackup.tar.gz* | pigz -dc | tar xif - -C /export/backup-tmp/
          Any advice would be appreciated.

          Greetings,

          Jan

          Comment


          • #6
            I replaced MariaDB with Percona 5.5.30-30.2 and installed xtrabackup 2.0.6 - same problem with normal (full, no streaming) backup!

            Comment

            Working...
            X