Announcement

Announcement Module
Collapse
No announcement yet.

Unable to bring up second node in my cluster

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Unable to bring up second node in my cluster

    I have 3 VMs configured in RHEL 6.5 … latest percona release. The first node builds fine and starts up as expected. I create the database and tablespaces on the second node … however when attempting to start I get:
    2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 28675028)
    2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0
    2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete.
    2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group.
    WSREP_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185)
    WSREP_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693)
    2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028
    2014-07-02 12:35:58 29662 [Note] Plugin 'FEDERATED' is disabled.
    2014-07-02 12:35:58 29662 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-07-02 12:35:58 29662 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-07-02 12:35:58 29662 [Note] InnoDB: Compressed tables use zlib 1.2.3
    2014-07-02 12:35:58 29662 [Note] InnoDB: Using Linux native AIO
    2014-07-02 12:35:58 29662 [Note] InnoDB: Using CPU crc32 instructions
    2014-07-02 12:35:58 29662 [Note] InnoDB: Initializing buffer pool, size = 3.0G
    2014-07-02 12:35:58 29662 [Note] InnoDB: Completed initialization of buffer pool
    2014-07-02 12:35:58 29662 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-07-02 12:35:58 29662 [Note] InnoDB: Log scan progressed past the checkpoint lsn 39408212788
    2014-07-02 12:35:58 29662 [Note] InnoDB: Database was not shutdown normally!
    2014-07-02 12:35:58 29662 [Note] InnoDB: Starting crash recovery.
    2014-07-02 12:35:58 29662 [Note] InnoDB: Reading tablespace information from the .ibd files...
    2014-07-02 12:35:58 29662 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/innodb_table_stats uses space ID: 1 at filepath: ./mysql/innodb_table_stats.ibd. Cannot open tablespace edw_target/png_edw_guid which uses space ID: 1 at filepath: /usr/local/sg/mysqldata/services/edw_target/png_edw_guid.ibd
    2014-07-02 12:35:58 7f58aef617e0 InnoDB: Operating system error number 2 in a file operation.
    InnoDB: The error means the system cannot find the path specified.
    InnoDB: If you are installing InnoDB, remember that you must create
    InnoDB: directories yourself, InnoDB does not create them.
    InnoDB: Error: could not open single-table tablespace file ./edw_target/png_edw_guid.ibd
    InnoDB: We do not continue the crash recovery, because the table may become
    InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
    InnoDB: To fix the problem and start mysqld:
    InnoDB: 1) If there is a permission problem in the file and mysqld cannot
    InnoDB: open the file, you should modify the permissions.
    InnoDB: 2) If the table is not needed, or you can restore it from a backup,
    InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
    InnoDB: crash recovery and ignore that table.
    InnoDB: 3) If the file system or the disk is broken, and you cannot remove
    InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
    InnoDB: and force InnoDB to continue crash recovery here.
    140702 12:35:58 mysqld_safe mysqld from pid file /usr/local/sg/chaldron/mysql/data/mysql/usphlliweb89.pid ended

    Permissions are set to same as on the first server. It must be getting the db log files from the first node since there is no data in this node.
    Any ideas?

  • #2
    Hi,


    2014-07-02 12:35:39 29662 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 28675028)
    2014-07-02 12:35:39 29662 [Note] WSREP: Requesting state transfer: success, donor: 0
    2014-07-02 12:35:57 29662 [Note] WSREP: 0 (uatchaldrondb1): State transfer to 1 (uatchaldrondb3) complete.
    2014-07-02 12:35:57 29662 [Note] WSREP: Member 0 (uatchaldrondb1) synced with group.
    WSREP_SST: [INFO] Joiner cleanup. (20140702 12:35:58.185)
    WSREP_SST: [INFO] Joiner cleanup done. (20140702 12:35:58.693)
    2014-07-02 12:35:58 29662 [Note] WSREP: SST complete, seqno: 28675028

    It seems state transfer didn't finish normally because

    * writeset position(28675028) doesn't seems to be changed
    * no file cleanup log ( normally it seems we can see the log of the deleted files )

    clear 2nd node data directory and restart might work well.

    The original problem(InnoDB: Attempted to open a previously opened tablespace) might hit windows Installer related bug as follows.
    http://bugs.mysql.com/bug.php?id=67595
    Last edited by taka-h; 07-23-2014, 07:31 AM.

    Comment


    • #3
      This is the error I am getting from innodbbackup.log:
      A tcp dump shows the joiner hanging up the tcp request immediately ... I have turned off the firewalls for these tests

      InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
      and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

      This software is published under
      the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

      Get the latest version of Percona XtraBackup, documentation, and help resources:
      http://www.percona.com/xb/p

      140908 11:56:46 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/var/lib/mysql/mysql.sock' as 'admin' (using password: YES).
      140908 11:56:46 innobackupex: Connected to MySQL server
      IMPORTANT: Please check that the backup run completes successfully.
      At the end of a successful backup run innobackupex
      prints "completed OK!".

      innobackupex: Using mysql server version 5.6.15-56-log

      innobackupex: Created backup directory /tmp/tmp.iotsptp2sG
      ^Gxbstream: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      innobackupex: 'xbstream -c' returned with exit code 1.
      innobackupex: Error: Failed to stream '/tmp/tmp.FmtmkMcRUl/backup-my.cnf': 1 at /usr//bin/innobackupex line 4745.

      Comment

      Working...
      X