Emergency

Wresp breakdown WSREP: fork() failed: 12 (Cannot allocate memory)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Wresp breakdown WSREP: fork() failed: 12 (Cannot allocate memory)

    We use percona cluster on debian wheezy. last cluster was build two weeks ago and worked properly.
    Today we tried startuing a new one and get memory errors from wresp.


    dpkg -l|grep galera
    ii percona-xtradb-cluster-galera-3 3.9.3494.wheezy amd64 Metapackage for latest version of galera3.
    ii percona-xtradb-cluster-galera-3.x 3.9.3494.wheezy amd64 Galera components of Percona XtraDB Cluster


    150310 17:48:52 mysqld_safe Starting mysqld_using_numa daemon with databases from /data/mysql
    150310 17:48:52 mysqld_safe WSREP: Running position recovery with --log_error='/data/mysql/wsrep_recovery.35GNao' --pid-file='/data/mysql/galeranode-02-recover.pid'
    2015-03-10 17:48:55 0 [Warning] Using unique option prefix sort_buffer instead of sort_buffer_size is deprecated and will be removed in a future release. Please use the full
    name instead.
    2015-03-10 17:48:55 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the fu
    ll name instead.
    2015-03-10 17:48:55 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
    2015-03-10 17:48:55 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more d
    etails).
    150310 17:48:58 mysqld_safe WSREP: Recovered position fafe51f0-c743-11e4-9db3-765772f3dde5:15
    Log of wsrep recovery (--wsrep-recover):
    2015-03-10 17:48:55 15254 [Warning] Using unique option prefix myisam_recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please
    use the full name instead.
    2015-03-10 17:48:55 15254 [Warning] option 'innodb-log-files-in-group': unsigned value 1 adjusted to 2
    InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
    InnoDB: The log block size is set to 4096.
    2015-03-10 17:48:55 7f563d5f5720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the optio
    n innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
    2015-03-10 17:48:55 7f563d5f5720 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTE
    D transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/...ansaction.html.
    2015-03-10 17:48:55 15254 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2015-03-10 17:48:55 15254 [Note] InnoDB: The InnoDB memory heap is disabled
    2015-03-10 17:48:55 15254 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2015-03-10 17:48:55 15254 [Note] InnoDB: Memory barrier is not used
    2015-03-10 17:48:55 15254 [Note] InnoDB: Compressed tables use zlib 1.2.7
    2015-03-10 17:48:55 15254 [Note] InnoDB: Using Linux native AIO
    2015-03-10 17:48:55 15254 [Note] InnoDB: Not using CPU crc32 instructions
    2015-03-10 17:48:55 15254 [Note] InnoDB: Initializing buffer pool, size = 5.5G
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    InnoDB: HugeTLB: Warning: Failed to allocate 601882624 bytes. errno 22
    InnoDB HugeTLB: Warning: Using conventional memory pool
    2015-03-10 17:48:55 15254 [Note] InnoDB: Completed initialization of buffer pool
    2015-03-10 17:48:55 15254 [Note] InnoDB: Highest supported file format is Barracuda.
    2015-03-10 17:48:55 15254 [Note] InnoDB: 128 rollback segment(s) are active.
    2015-03-10 17:48:55 15254 [Note] InnoDB: Waiting for purge to start
    2015-03-10 17:48:55 15254 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 1647917
    2015-03-10 17:48:55 15254 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
    2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_port=9998'
    2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_port_wr=9999'
    2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_threads=16'
    2015-03-10 17:48:55 15254 [Warning] /usr/sbin/mysqld: unknown variable 'loose_handlersocket_threads_wr=1'
    2015-03-10 17:48:56 15254 [Note] RSA private key file not found: /data/mysql//private_key.pem. Some authentication plugins will not work.
    2015-03-10 17:48:56 15254 [Note] RSA public key file not found: /data/mysql//public_key.pem. Some authentication plugins will not work.
    2015-03-10 17:48:56 15254 [Note] Server hostname (bind-address): '*'; port: 33306
    2015-03-10 17:48:56 15254 [Note] IPv6 is available.
    2015-03-10 17:48:56 15254 [Note] - '::' resolves to '::';
    2015-03-10 17:48:56 15254 [Note] Server socket created on IP: '::'.
    2015-03-10 17:48:56 15254 [Note] WSREP: Recovered position: fafe51f0-c743-11e4-9db3-765772f3dde5:15
    2015-03-10 17:48:56 15254 [Note] Binlog end
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'partition'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'BLACKHOLE'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_DELETED'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_METRICS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMPMEM'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP_RESET'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_CMP'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_LOCKS'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'INNODB_TRX'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_RSEG'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
    2015-03-10 17:48:56 15254 [Note] Shutting down plugin 'InnoDB'
    2015-03-10 17:48:56 15254 [Note] InnoDB: FTS optimize thread exiting.
    2015-03-10 17:48:56 15254 [Note] InnoDB: Starting shutdown...
    2015-03-10 17:48:58 15254 [Note] InnoDB: Shutdown completed; log sequence number 1647927
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'ARCHIVE'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MEMORY'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MyISAM'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'CSV'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'MRG_MYISAM'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'sha256_password'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'mysql_old_password'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'mysql_native_password'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'wsrep'
    2015-03-10 17:48:58 15254 [Note] Shutting down plugin 'binlog'
    2015-03-10 17:48:58 15254 [Note] /usr/sbin/mysqld: Shutdown complete


  • #2
    2015-03-10 17:49:03 0 [Warning] Using unique option prefix sort_buffer instead of sort_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
    2015-03-10 17:49:03 0 [Warning] Using unique option prefix thread_cache instead of thread_cache_size is deprecated and will be removed in a future release. Please use the full name instead.
    2015-03-10 17:49:03 0 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
    2015-03-10 17:49:03 0 [Note] WSREP: wsrep_start_position var submitted: 'fafe51f0-c743-11e4-9db3-765772f3dde5:15'
    2015-03-10 17:49:03 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2015-03-10 17:49:03 15327 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_load(): Galera 3.9(r93aca2d) by Codership Oy <info@codership.com> loaded successfully.
    2015-03-10 17:49:03 15327 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
    2015-03-10 17:49:03 15327 [Note] WSREP: Found saved state: fafe51f0-c743-11e4-9db3-765772f3dde5:-1
    2015-03-10 17:49:03 15327 [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/; base_host = 172.16.123.205; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT10S; evs.inactive_timeout = PT1M; evs.install_timeout = PT1M; evs.join_retrans_period = PT1S; evs.keepalive_period = PT3S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT30S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 4G; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false;
    2015-03-10 17:49:03 15327 [Note] WSREP: Service thread queue flushed.
    2015-03-10 17:49:03 15327 [Note] WSREP: Assign initial position for certification: 15, protocol version: -1
    2015-03-10 17:49:03 15327 [Note] WSREP: wsrep_sst_grab()
    2015-03-10 17:49:03 15327 [Note] WSREP: Start replication
    2015-03-10 17:49:03 15327 [Note] WSREP: Setting initial position to fafe51f0-c743-11e4-9db3-765772f3dde5:15
    2015-03-10 17:49:03 15327 [Note] WSREP: protonet asio version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: Using CRC-32C for message checksums.
    2015-03-10 17:49:03 15327 [Note] WSREP: backend: asio
    2015-03-10 17:49:03 15327 [Warning] WSREP: access file(/data/mysql//gvwstate.dat) failed(No such file or directory)
    2015-03-10 17:49:03 15327 [Note] WSREP: restore pc from disk failed
    2015-03-10 17:49:03 15327 [Note] WSREP: GMCast version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
    2015-03-10 17:49:03 15327 [Note] WSREP: EVS version 0
    2015-03-10 17:49:03 15327 [Note] WSREP: gcomm: connecting to group 'sso-uat', peer '172.16.123.204:,172.16.123.205:,172.16.123.206:,1 72.16.123.207:'
    2015-03-10 17:49:03 15327 [Warning] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') address 'tcp://172.16.123.205:4567' points to own listening address, blacklisting
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') address 'tcp://172.16.123.205:4567' pointing to uuid 5bcb2aad is blacklisted, skipping
    2015-03-10 17:49:03 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
    2015-03-10 17:49:03 15327 [Note] WSREP: declaring f844ae82 at tcp://172.16.123.204:4567 stable
    2015-03-10 17:49:03 15327 [Note] WSREP: Node f844ae82 state prim
    2015-03-10 17:49:03 15327 [Note] WSREP: view(view_id(PRIM,5bcb2aad,6) memb {
    5bcb2aad,0
    f844ae82,0
    } joined {
    } left {
    } partitioned {
    })
    2015-03-10 17:49:03 15327 [Note] WSREP: save pc into disk
    2015-03-10 17:49:03 15327 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.123.206:4567
    2015-03-10 17:49:03 15327 [Note] WSREP: discarding pending addr without UUID: tcp://172.16.123.207:4567
    2015-03-10 17:49:04 15327 [Note] WSREP: gcomm: connected
    2015-03-10 17:49:04 15327 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
    2015-03-10 17:49:04 15327 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
    2015-03-10 17:49:04 15327 [Note] WSREP: Opened channel 'sso-uat'
    2015-03-10 17:49:04 15327 [Note] WSREP: Waiting for SST to complete.
    2015-03-10 17:49:04 15327 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: sent state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: got state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2 from 0 (galeranode-02)
    2015-03-10 17:49:04 15327 [Note] WSREP: STATE EXCHANGE: got state msg: 5c64cc5b-c745-11e4-8b3f-1e0b689180a2 from 1 (galeranode-01)
    2015-03-10 17:49:04 15327 [Note] WSREP: Quorum results:
    version = 3,
    component = PRIMARY,
    conf_id = 5,
    members = 1/2 (joined/total),
    act_id = 17,
    last_appl. = -1,
    protocols = 0/7/3 (gcs/repl/appl),
    group UUID = 377d1d1f-c723-11e4-be0f-f205bd80e309
    2015-03-10 17:49:04 15327 [Note] WSREP: Flow-control interval: [23, 23]
    2015-03-10 17:49:04 15327 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 17)
    2015-03-10 17:49:04 15327 [Note] WSREP: State transfer required:
    Group state: 377d1d1f-c723-11e4-be0f-f205bd80e309:17
    Local state: fafe51f0-c743-11e4-9db3-765772f3dde5:15
    2015-03-10 17:49:04 15327 [Note] WSREP: New cluster view: global state: 377d1d1f-c723-11e4-be0f-f205bd80e309:17, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 3
    2015-03-10 17:49:04 15327 [Warning] WSREP: Gap in state sequence. Need state transfer.
    2015-03-10 17:49:04 15327 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' '
    2015-03-10 17:49:04 15327 [ERROR] WSREP: fork() failed: 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to execute: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' : 12 (Cannot allocate memory)
    2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
    2015-03-10 17:49:04 15327 [ERROR] Aborting
    2015-03-10 17:49:06 15327 [Note] WSREP: Closing send monitor...
    2015-03-10 17:49:06 15327 [Note] WSREP: Closed send monitor.
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: terminating thread
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: joining thread
    2015-03-10 17:49:06 15327 [Note] WSREP: gcomm: closing backend
    2015-03-10 17:49:06 15327 [Note] WSREP: (5bcb2aad, 'tcp://0.0.0.0:4567') turning message relay requesting off
    2015-03-10 17:49:07 15327 [Note] WSREP: view(view_id(NON_PRIM,5bcb2aad,6) memb {
    5bcb2aad,0
    } joined {
    } left {
    } partitioned {
    f844ae82,0
    })
    2015-03-10 17:49:07 15327 [Note] WSREP: view((empty))
    2015-03-10 17:49:07 15327 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
    2015-03-10 17:49:07 15327 [Note] WSREP: gcomm: closed
    2015-03-10 17:49:07 15327 [Note] WSREP: Flow-control interval: [16, 16]
    2015-03-10 17:49:07 15327 [Note] WSREP: Received NON-PRIMARY.
    2015-03-10 17:49:07 15327 [Note] WSREP: Shifting PRIMARY -> OPEN (TO: 17)
    2015-03-10 17:49:07 15327 [Note] WSREP: Received self-leave message.
    2015-03-10 17:49:07 15327 [Note] WSREP: Flow-control interval: [0, 0]
    2015-03-10 17:49:07 15327 [Note] WSREP: Received SELF-LEAVE. Closing connection.
    2015-03-10 17:49:07 15327 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 17)
    2015-03-10 17:49:07 15327 [Note] WSREP: RECV thread exiting 0: Success
    2015-03-10 17:49:07 15327 [Note] WSREP: recv_thread() joined.
    2015-03-10 17:49:07 15327 [Note] WSREP: Closing replication queue.
    2015-03-10 17:49:07 15327 [Note] WSREP: Closing slave action queue.
    2015-03-10 17:49:07 15327 [Note] WSREP: Service disconnected.
    2015-03-10 17:49:07 15327 [Note] WSREP: rollbacker thread exiting
    2015-03-10 17:49:08 15327 [Note] WSREP: Some threads may fail to exit.
    2015-03-10 17:49:08 15327 [Note] Binlog end
    2015-03-10 17:49:08 15327 [Note] /usr/sbin/mysqld: Shutdown complete

    Error in my_thread_global_end(): 1 threads didn't exit
    150310 17:49:13 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended


    Anybody some hints?

    Thx in advance

    Simon

    Comment


    • #3
      Can you also post error log from the DONOR node and post relevant wsrep_* variables for further analysis

      Comment


      • #4
        2015-03-10 17:49:04 15327 [ERROR] WSREP: fork() failed: 12 (Cannot allocate memory)
        2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to execute: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172.16.123.205' --auth 'root:Uz6oebu6' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '15327' '' : 12 (Cannot allocate memory)
        2015-03-10 17:49:04 15327 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
        2015-03-10 17:49:04 15327 [ERROR] Aborting
        It seems that the server where this is running has a small space in RAM and when xtrabackup comes in to prepare a backup, it use to allocate 100MB RAM by default and consequently, failing by a lack of RAM space. It's going to consider the size of dataset being copied to the backup set. Just to confirm, is this a small VM or similar thing?

        Additionally, the fork() called out my attention: http://en.wikipedia.org/wiki/Fork_%28system_call%29

        Cheers,
        WB

        Comment


        • #5
          I had the same problem as you, I've solved enabling SWAP in the DONOR and other nodes.

          Comment

          Working...
          X