Announcement

Announcement Module
Collapse
No announcement yet.

Database page corruption detected

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Database page corruption detected

    xtrabackup fails after attempting to write backup of large datafile. The ibd file in the data directory is 123gigs. Can anyone give me some advice or direction here?

    Header of backup log:


    InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oyand Percona Inc 2009-2012. All Rights Reserved.This software is published underthe GNU GENERAL PUBLIC LICENSE Version 2, June 1991.120905 00:00:01 innobackupex: Starting mysql with options: --unbuffered --120905 00:00:01 innobackupex: Connected to database with mysql child process (pid=9788)120905 00:00:08 innobackupex: Connection to database server closedIMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!".innobackupex: Using mysql Ver 14.14 Distrib 5.5.27, for Linux (x86_64) using readline 5.1innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.innobackupex: Created backup directory /home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08120905 00:00:08 innobackupex: Starting mysql with options: --unbuffered --120905 00:00:08 innobackupex: Connected to database with mysql child process (pid=9815)120905 00:00:10 innobackupex: Connection to database server closed120905 00:00:10 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08 --parallel=4innobackupex: Waiting for ibbackup (pid=9822) to suspendinnobackupex: Suspend file '/home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08/xtrabackup_suspended'xtrabackup_55 version 2.0.2 for Percona Server 5.5.16 Linux (x86_64) (revision id: 461)xtrabackup: uses posix_fadvise().xtrabackup: cd to /home/db/dataxtrabackup: Target instance is assumed as followings.xtrabackup: innodb_data_home_dir = /home/db/data/innodb/xtrabackup: innodb_data_file_path = ibdata1:10M:autoextendxtrabackup: innodb_log_group_home_dir = /home/db/data/innodb/xtrabackup: innodb_log_files_in_group = 2xtrabackup: innodb_log_file_size = 268435456120905 0:00:10 InnoDB: Using Linux native AIOxtrabackup: using O_DIRECT


    Error Output:

    [01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Database page corruption detected at page 3999937. retrying...[01] xtrabackup: Error: 10 retries resulted in fail. File ./dbname/tablename.ibd seems to be corrupted.[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.[01] xtrabackup: Error: failed to copy datafile.innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 374.

  • #2
    Does it fail w/o parallel copy (remove --parallel)? I also have tables that are 100GB-200GB+ and would fail. This was on an pre 2.0 version though. I also had an issue where tables were being truncated while the copy was running, but believe this has been fixed for quite some time, just never got around to trying it again. I have been living with LVM snapshots for the time being.

    Comment


    • #3
      Are you using "barracuda" InnoDB compression by chance? Try checking the value of innodb_file_format (mysql> show variables like 'innodb_file_format'

      If so:

      https://groups.google.com/forum/?fromgroups=#!topic/percona- discussion/GR8GP9gLCBI

      https://bugs.launchpad.net/percona-xtrabackup/+bug/498660/

      Is it possible your tables are actually corrupted? What does the error log of your actual MySQL instance say? Here's an article on recovering from InnoDB corruption:

      http://www.mysqlperformanceblog.com/2008/07/04/recovering-in nodb-table-corruption/

      Comment


      • #4
        @swerner, I removed parallel option last night and the failure remains.

        @dannycolumbus, innodb_file_format = Antelope. Its likely I do have data corruption. Tonight, I attempted an optimize table and it failed when it got to that page, and dumped a hex and ascii representation of the data.


        InnoDB: Database page corruption on disk or a failedInnoDB: file read of page 3999937.InnoDB: You may have to recover from a backup.120909 20:05:56 InnoDB: Page dump in ascii and hex (16384 bytes):InnoDB: End of page dump120909 20:05:56 InnoDB: Page checksum 539377103 (32bit_calc: 986786231), prior-to-4.0.14-form checksum 3002691923InnoDB: stored checksum 4148786784, prior-to-4.0.14-form stored checksum 785653984InnoDB: Page lsn 132 4266642038, low 4 bytes of lsn at page end 471143840InnoDB: Page number (if stored to page already) 3999937,InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 544InnoDB: Page may be an index page where index id is 1220InnoDB: (index "PRIMARY" of table "dbname"."tablename")InnoDB: Database page corruption on disk or a failedInnoDB: file read of page 3999937.InnoDB: You may have to recover from a backup.InnoDB: It is also possible that your operatingInnoDB: system has corrupted its own file cacheInnoDB: and rebooting your computer removes theInnoDB: error.InnoDB: If the corrupt page is an index pageInnoDB: you can also try to fix the corruptionInnoDB: by dumping, dropping, and reimportingInnoDB: the corrupt table. You can use CHECKInnoDB: TABLE to scan your table for corruption.InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.


        I am hoping its a simple case, and am currently rebuilding the table using innodb_force_recovery = 1 and inserting into a myisam table.

        Thanks, I will post update once I know more.

        Comment

        Working...
        X