Announcement

Announcement Module
Collapse
No announcement yet.

innobackupex sporadic failures

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • innobackupex sporadic failures

    I'm running into a problem where innobackup is failing, sometimes. My backup script is running innobackupex like this:

    innobackupex --stream=tar --user=bkp_user --password=xxxxx ./ 2>> /mnt/backup/logs/backup_2013-04-20.log | pigz -c -p 20 -6 - > /mnt/backup/backup_2013-04-20.tar.gz

    This works most of the time, but sometimes it fails on me. When it fails, it reports one of the following messages:

    innobackupex: Created backup directory /root
    tar: -: Cannot write: Broken pipe
    tar: Error is not recoverable: exiting now
    innobackupex: Error: Failed to stream 'backup-my.cnf': Inappropriate ioctl for device at /usr/bin/innobackupex line 381.

    [01] Streaming /data/db/schemas/ibdata
    innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 381.

    How can I troubleshoot this, or gather additional information about the failures?

  • #2
    Originally posted by tgoodaire View Post
    I'm running into a problem where innobackup is failing, sometimes. My backup script is running innobackupex like this: innobackupex --stream=tar --user=bkp_user --password=xxxxx ./ 2>> /mnt/backup/logs/backup_2013-04-20.log | pigz -c -p 20 -6 - > /mnt/backup/backup_2013-04-20.tar.gz This works most of the time, but sometimes it fails on me. When it fails, it reports one of the following messages: innobackupex: Created backup directory /root tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now innobackupex: Error: Failed to stream 'backup-my.cnf': Inappropriate ioctl for device at /usr/bin/innobackupex line 381. [01] Streaming /data/db/schemas/ibdata innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 381. How can I troubleshoot this, or gather additional information about the failures?
    Hello,

    have you checked that you are not running out of disk space around this time? Not only on the destination backup directory, check also you tmpdir setting because this is where xtrabackup_logfile is saved before being streamed. How long does it take to complete the backup? If you have Cacti or something else, maybe you can get a quick glimpse of disk usage.
    Our documentation has a lot of answers about common questions on Percona software, have you checked there before posting that question here? http://www.percona.com/forums/core/i...lies/smile.png

    Join us at the annual Percona Live MySQL Users Conference - http://www.percona.com/live/mysql-conference-2014/

    Comment


    • tgoodaire
      tgoodaire commented
      Editing a comment
      I don't think that I'm running out of disk space. Nagios hasn't sent out any alerts and the disk usage looks ok. We haven't set tmpdir, so it should be the system default of /tmp. /tmp is 30GB with only 2 MB used and the backup job fails in 7 seconds. It very strange that this is just failing sometimes.

      Full log is here:

      Running backup at Thu Apr 25 16:30:01 UTC 2013


      InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
      and Percona Ireland Ltd 2009-2012. All Rights Reserved.

      This software is published under
      the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

      130425 16:30:01 innobackupex: Starting mysql with options: --defaults-file='/etc/mysql/my.cnf' --password=xxxxxxxx --user='bkpuser' --host='localhost' --unbuffered --
      130425 16:30:01 innobackupex: Connected to database with mysql child process (pid=14098)
      130425 16:30:07 innobackupex: Connection to database server closed
      IMPORTANT: Please check that the backup run completes successfully.
      At the end of a successful backup run innobackupex
      prints "completed OK!".

      innobackupex: Using mysql Ver 14.14 Distrib 5.1.63, for debian-linux-gnu (x86_64) using readline 6.1
      innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

      innobackupex: Created backup directory /root
      tar: -: Cannot write: Broken pipe
      tar: Error is not recoverable: exiting now
      innobackupex: Error: Failed to stream 'backup-my.cnf': Inappropriate ioctl for device at /usr/bin/innobackupex line 381.
      innobackupex command failed at Thu Apr 25 16:30:07 UTC 2013

  • #3
    I've the same problem and wondering how to fix that. It happens on the donor node and i've even tried to move the hardcoded /tmp path to a different path, but have the same problem there. The new path /var/lib/mysql got all permissions for the mysql user ready.

    Comment


    • #4
      @tgoodaire,

      I would suggest to simplify the command first, to see if any parts are failing. As you can see here I have removed the pigz and pipe to stderr i.e.

      innobackupex --stream=tar --user=bkp_user --password=xxxxx ./ | cat - > /mnt/backup/backup_2013-04-20.tar.gz


      Our documentation has a lot of answers about common questions on Percona software, have you checked there before posting that question here? http://www.percona.com/forums/core/i...lies/smile.png

      Join us at the annual Percona Live MySQL Users Conference - http://www.percona.com/live/mysql-conference-2014/

      Comment


      • tgoodaire
        tgoodaire commented
        Editing a comment
        For now, I've worked around the problem by doing a simple bash loop to retry the backup three times if it fails. So far it's been working ok. If it fails once, the second attempt works. It's still a mystery to me why it's failing sometimes.
    Working...
    X