Emergency

Receiving error:select() error: Bad address during backup.

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Receiving error:select() error: Bad address during backup.

    We have 6 databases in Amazon's US East region. Our backup script connects over SSH to the 6 database servers and executes the following command:

    sudo innobackupex --defaults-extra-file=/home/dbcontrol/.my.cnf.us-prod.backup --socket=/tmp/mysql.sock --slave-info --safe-slave-backup --safe-slave-backup-timeout=1200 --use-memory=4G --stream=xbstream --parallel=6 --compress --compress-threads=6 --encrypt=AES256 --encrypt-key=blah --encrypt-threads=3 /home/dbcontrol/backup

    The xbstream file is getting streamed to the central server we use to back up and finally gets copied over to an S3 bucket for long-term storage.

    If I run each backup serially I'm not seeing the problem but when we run all six in parallel I'm getting the error:

    select() error: Bad address

    Which gets written to the log file millions of times generating log files well over 16GB in size. Actually it doesn't stop writing this until I kill the process.

    Since it doesn't happen when I run it serially (at least so far) I thought maybe I was hitting some kind of open file handle limit or something but I've checked all of the servers and everything is configured per the documentation's recommendations so I'm at a loss. I'm not even sure if the error is coming from XtraBackup or SSH.

    Has anyone else run into this problem or have any knowledge of it?

    Thanks!

    Joe

  • #2
    Hi joeramsey ,

    Have you tried running this with additional verbosity in ssh via the -vvv option syntax? Since the error doesn't occur on serial application I suspect you are seeing a network contention / ssh based error here, not PXB.

    The other concept worth exploring is to use socat / netcat streaming rather than SSH. https://www.percona.com/doc/percona-...kx_stream.html

    Comment


    • #3
      Michael,

      I had thought about that but wasn't sure if it was a PXB error so thought I'd post in the forums to check. I'll do that and post the results.

      I'll also check out the link you mention.

      Thanks for the quick response and suggestions!!!

      Joe

      Comment

      Working...
      X