Emergency

Percona cluster on ubuntu 16.04

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Percona cluster on ubuntu 16.04

    I'm trying to test a percona cluster with 3 nodes to verify if it will work for our production environment.
    I have followed the instructions on https://www.percona.com/doc/percona-...ntu_howto.html for all 3 nodes.
    The problem seem to stem from the fact that during the apt-get install process the package bootstraps each node, so when I have updated the config and try to join it to node 1 which I manually bootstrapped the other nodes does not join but instead create their own clusters by resuming the original bootstrap from the apt process.

    What files do I need to clear out to make it start joining the new cluster instead of creating its own?

    I have tried to completely clear out /var/lib/mysql of all files but then it just gives "WSREP: Failed to recover position:" errors instead.
    Any indication on where to go would be much helpful, I can paste the config etc if you need them.

  • #2
    There is difference in the command to start bootstrap node
    [root@pxc1 ~]# /etc/init.d/mysql bootstrap-pxc and then followup nodes [root@pxc2 ~]# /etc/init.d/mysql start ​​​​​​​Are you missing anything ?

    Comment


    • #3
      Here is the steps I am taking to get it setup, let me know if i miss anything.
      This is ran on a brand new Ubuntu 16.04 installation, nothing installed during the initial install besides SSH server.

      On all 3 nodes:

      wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb
      sudo dpkg -i percona-release_0.1-4.$(lsb_release -sc)_all.deb

      sudo apt-get update
      sudo apt-get install percona-xtradb-cluster-57

      (Setting a strong root password during install)

      sudo service mysql stop

      sudo nano /etc/mysql/my.cnf

      (Not actually including the start and stop lines)
      ---- Start My.cnf ---
      # Generated by Percona Configuration Wizard (http://tools.percona.com/) version REL5-20120208
      # Configuration name NODE1 generated for hlagnevall at 2017-12-15 18:54:55

      [mysql]

      # CLIENT #
      port = 3306
      socket = /var/lib/mysql/mysql.sock

      [mysqld]

      # GENERAL #
      user = mysql
      default-storage-engine = InnoDB
      socket = /var/lib/mysql/mysql.sock
      pid-file = /var/lib/mysql/mysql.pid

      # MyISAM #
      key-buffer-size = 32M
      myisam-recover = FORCE,BACKUP

      # SAFETY #
      max-allowed-packet = 16M
      max-connect-errors = 1000000
      innodb = FORCE

      # DATA STORAGE #
      datadir = /var/lib/mysql/
      default_storage_engine = InnoDB

      # BINARY LOGGING #
      log-bin = /var/lib/mysql/mysql-bin
      expire-logs-days = 14
      sync-binlog = 1
      binlog_format = ROW

      # CACHES AND LIMITS #
      tmp-table-size = 32M
      max-heap-table-size = 32M
      query-cache-type = 0
      query-cache-size = 0
      max-connections = 500
      thread-cache-size = 50
      open-files-limit = 65535
      table-definition-cache = 4096
      table-open-cache = 4096

      # INNODB #
      innodb-flush-method = O_DIRECT
      innodb-log-files-in-group = 2
      innodb-log-file-size = 256M
      innodb-flush-log-at-trx-commit = 1
      innodb-file-per-table = 1
      innodb-buffer-pool-size = 6G
      innodb_autoinc_lock_mode = 2
      innodb_buffer_pool_instances = 8

      # LOGGING #
      log-error = /var/log/mysql/mysql-error.log
      log-queries-not-using-indexes = 1
      slow-query-log = 1
      slow-query-log-file = /var/log/mysql/mysql-slow.log

      # CLUSTER #
      wsrep_provider = /usr/lib/libgalera_smm.so
      wsrep_cluster_address = gcomm://10.2.20.60,10.2.20.61,10.2.20.62
      wsrep_node_address = 10.2.20.60 (Updating this line to match the IP of the node)
      wsrep_sst_method = xtrabackup-v2
      wsrep_cluster_name = pxc-cluster
      wsrep_sst_auth = sstuser:Str0ngpa$$w0rdsaregr3at
      pxc_strict_mode = ENFORCING

      !includedir /etc/mysql/conf.d/
      !includedir /etc/mysql/percona-xtradb-cluster.conf.d/

      ----END MY.CNF-----------

      On Node 1:

      sudo /etc/init.d/mysql bootstrap-pxc

      mysql -u root -p

      CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'Str0ngpa$$w0rdsaregr3at';
      GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
      FLUSH PRIVILEGES;

      show status like 'wsrep%';
      +----------------------------------+--------------------------------------+
      | Variable_name | Value |
      +----------------------------------+--------------------------------------+
      | wsrep_local_state_uuid | ed51d9e0-e048-11e7-8b82-3e3d248f928a |
      | wsrep_protocol_version | 7 |
      | wsrep_last_committed | 9 |
      | wsrep_replicated | 9 |
      | wsrep_replicated_bytes | 2058 |
      | wsrep_repl_keys | 9 |
      | wsrep_repl_keys_bytes | 279 |
      | wsrep_repl_data_bytes | 1203 |
      | wsrep_repl_other_bytes | 0 |
      | wsrep_received | 2 |
      | wsrep_received_bytes | 152 |
      | wsrep_local_commits | 0 |
      | wsrep_local_cert_failures | 0 |
      | wsrep_local_replays | 0 |
      | wsrep_local_send_queue | 0 |
      | wsrep_local_send_queue_max | 1 |
      | wsrep_local_send_queue_min | 0 |
      | wsrep_local_send_queue_avg | 0.000000 |
      | wsrep_local_recv_queue | 0 |
      | wsrep_local_recv_queue_max | 2 |
      | wsrep_local_recv_queue_min | 0 |
      | wsrep_local_recv_queue_avg | 0.500000 |
      | wsrep_local_cached_downto | 1 |
      | wsrep_flow_control_paused_ns | 0 |
      | wsrep_flow_control_paused | 0.000000 |
      | wsrep_flow_control_sent | 0 |
      | wsrep_flow_control_recv | 0 |
      | wsrep_flow_control_interval | [ 100, 100 ] |
      | wsrep_flow_control_interval_low | 100 |
      | wsrep_flow_control_interval_high | 100 |
      | wsrep_flow_control_status | OFF |
      | wsrep_cert_deps_distance | 1.000000 |
      | wsrep_apply_oooe | 0.000000 |
      | wsrep_apply_oool | 0.000000 |
      | wsrep_apply_window | 1.000000 |
      | wsrep_commit_oooe | 0.000000 |
      | wsrep_commit_oool | 0.000000 |
      | wsrep_commit_window | 1.000000 |
      | wsrep_local_state | 4 |
      | wsrep_local_state_comment | Synced |
      | wsrep_cert_index_size | 1 |
      | wsrep_cert_bucket_count | 22 |
      | wsrep_gcache_pool_size | 3738 |
      | wsrep_causal_reads | 0 |
      | wsrep_cert_interval | 0.000000 |
      | wsrep_ist_receive_status | |
      | wsrep_ist_receive_seqno_start | 0 |
      | wsrep_ist_receive_seqno_current | 0 |
      | wsrep_ist_receive_seqno_end | 0 |
      | wsrep_incoming_addresses | 10.2.20.60:3306 |
      | wsrep_desync_count | 0 |
      | wsrep_evs_delayed | |
      | wsrep_evs_evict_list | |
      | wsrep_evs_repl_latency | 0/0/0/0/0 |
      | wsrep_evs_state | OPERATIONAL |
      | wsrep_gcomm_uuid | d0f49173-e049-11e7-87e1-9e6186249fb6 |
      | wsrep_cluster_conf_id | 1 |
      | wsrep_cluster_size | 1 |
      | wsrep_cluster_state_uuid | ed51d9e0-e048-11e7-8b82-3e3d248f928a |
      | wsrep_cluster_status | Primary |
      | wsrep_connected | ON |
      | wsrep_local_bf_aborts | 0 |
      | wsrep_local_index | 0 |
      | wsrep_provider_name | Galera |
      | wsrep_provider_vendor | Codership Oy <info@codership.com> |
      | wsrep_provider_version | 3.22(r8678538) |
      | wsrep_ready | ON |
      +----------------------------------+--------------------------------------+
      67 rows in set (0.01 sec)

      On nodes 2 & 3:

      sudo /etc/init.d/mysql start
      [....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
      failed!

      The /var/log/mysql/error.log file is empty.

      And this happens every time I try to do it again over and over, I have blown away each server and redeployed following the steps multiple times.
      I even went as far as sign up for the percona config generator (hence why I have the larger config file now) to make sure I wasn't missing anything basic from the my.cnf file.

      Comment


      • #4
        Here is the /var/log/mysqld.log file from node3.
        Attached Files

        Comment


        • #5
          hlagnevall Are you still having problems running a 3-node PXC Cluster?

          Comment


          • #6
            Yes still having issues, I have since deleted the VM's and recreated them again and have reran all the configurations multiple time. I get the same results, each node attempts to create it's own cluster instead of joining even though only the first one is bootstrapped.

            Comment


            • #7
              Looks like it has something to do with version 5.7, I just reinstalled with 5.6 following the exact same instructions and everything is now working fine.

              Comment


              • #8
                I was looking at the installation code this weekend.

                I wonder if some of the cnf files in /etc/mysql/percona-xtradb-cluster.conf.d/ were overriding your values (specifically the wsrep_cluster_address in wsrep.cnf,, since they would be included after your settings). Maybe it would be different if the

                !includedir /etc/mysql/conf.d/
                !includedir /etc/mysql/percona-xtradb-cluster.conf.d/

                were placed at the top of the config file instead of the bottom.
                Last edited by Kenn Takara; 01-24-2018, 08:22 PM.

                Comment

                Working...
                X