Deploying OpenEverest on a 3-Node K3s Cluster

Part 1 of 2: From bare Ubuntu servers to a running MySQL cluster

OpenEverest is an open source platform for running databases such as MySQL, PostgreSQL and MongoDB on Kubernetes. You get a web console and an API, and Kubernetes operators handle provisioning, scaling, backups and failover behind the scenes.

This post builds a small but realistic lab from scratch: a 3-node K3s Kubernetes cluster on Ubuntu 26.04, OpenEverest on top of it, and a 3-node Percona XtraDB Cluster (PXC) for MySQL that you can connect to from your laptop. Every step explains what the command does and why it is needed.

Part 2 of this series looks inside the MySQL cluster: how to read Galera’s health indicators, and a live failover test where we kill the primary database pod while writes keep running.

What You Will Build

• K3s: a lightweight, CNCF-certified Kubernetes distribution. A single binary includes containerd (the container runtime), Flannel (pod networking), CoreDNS (service discovery) and the local-path storage provisioner.
• Helm: the Kubernetes package manager, used to install OpenEverest.
• OpenEverest: the control plane (everest-server for the UI/API, everest-operator for reconciling database clusters).
• Percona XtraDB Cluster: a 3-node, synchronously replicated MySQL cluster with HAProxy in front of it, created from the OpenEverest UI.

Server Inventory

Each node needs a unique hostname and a unique IP address. Kubernetes identifies nodes by hostname, so two VMs cloned from the same image with the same hostname will conflict.

The nodes must be able to reach each other on these ports:

Phase 1: Prepare All Three Nodes

Where to run: all three servers. Open three terminal windows and SSH into each one.

Step 1.1: Connect to Each Server

Open one SSH session per server so you can run the preparation commands side by side.

Step 1.2: Install Packages and Disable Swap

This refreshes the package index and installs the tools the rest of the guide needs:

• curl and wget download the K3s and Helm installers.
• open-iscsi and nfs-common are not needed by local-path storage, but they are prerequisites for replicated storage such as Longhorn or NFS volumes. Installing them now makes it easy to switch storage later.

Swap is turned off because the kubelet expects it to be disabled by default. With swap on, memory limits on database pods become unpredictable. The sed command comments out the swap line in /etc/fstab so swap stays off after a reboot.

Step 1.3: Enable Kernel Modules and IP Forwarding

Kubernetes networking relies on two kernel modules and three kernel settings:

• overlay: the filesystem driver containerd uses to stack container image layers.
• br_netfilter: lets iptables see traffic crossing the Linux bridge, which is how Kubernetes Services and network policies work.
• net.bridge.bridge-nf-call-iptables and ip6tables: send bridged IPv4/IPv6 traffic through iptables.
• net.ipv4.ip_forward: allows the node to route packets between pods on different nodes.

modprobe loads the modules now. The file in /etc/modules-load.d/ loads them again on every boot, and the file in /etc/sysctl.d/ does the same for the kernel settings.

Phase 2: Install the K3s Control Plane

Where to run: control plane only (10.50.55.155).

Step 2.1: Install the K3s Server

The official install script downloads K3s, installs it as a systemd service named k3s, and starts the Kubernetes control plane. INSTALL_K3S_EXEC passes extra flags to the server:

• –disable traefik: K3s ships with the Traefik ingress controller. This lab doesn’t use ingress, so we skip it to save resources.
• –node-ip 10.50.55.155: pins the address Kubernetes advertises for this node. This matters on servers with more than one network interface.
• –flannel-backend=vxlan: pod traffic between nodes is wrapped in VXLAN over UDP 8472. This is already the default; it is set explicitly for clarity.

Step 2.2: Let Your User Run kubectl

K3s writes its admin kubeconfig to /etc/rancher/k3s/k3s.yaml, which only root can read. These commands copy it to your home directory, make you the owner, and point KUBECONFIG at it, now and in every future shell.

Step 2.3: Get the Node Join Token

Worker nodes need a shared secret to join the cluster. K3s stores it in node-token.

Output (format)

The token has two parts. K10<ca-hash> lets the joining node verify the server’s TLS certificate. The part after ::server: is the actual password.

Security: Treat the whole token as a credential. Never publish the part after ::server:. Anyone who has it can join a machine to your cluster.

Phase 3: Join the Worker Nodes

Where to run: each worker, in its own terminal. Replace <TOKEN> with the full token from Step 2.3.

When K3S_URL is set, the same install script runs in agent mode. The node connects to the API server at https://10.50.55.155:6443, authenticates with the token, and starts running pods. –node-ip again pins the address each node advertises.

Step 3.1: Join Worker Node 1

Step 3.2: Join Worker Node 2

Phase 4: Install OpenEverest

Where to run: control plane only (10.50.55.155).

Step 4.1: Verify All Nodes Are Ready

Ready means the kubelet on that node is healthy and has working pod networking. Workers show <none> under ROLES; that is normal for K3s agents.

Output

Step 4.2: Install Helm

Helm installs applications from versioned packages called charts. This script detects your OS and architecture and installs the helm binary to /usr/local/bin.

Step 4.3: Confirm the Default StorageClass

Databases need persistent volumes. When OpenEverest creates a database, it requests storage from the default StorageClass, which on K3s is local-path.

Output

What these columns mean for a database:

• WaitForFirstConsumer: the volume is created only after the pod is scheduled, so the data directory lands on the same node as the pod.
• Node-local data: each volume is a directory on one node’s disk. The pod is pinned to that node for its whole life. If the node is lost, the pod cannot move; Galera rebuilds the data on a new node from the surviving replicas.
• ALLOWVOLUMEEXPANSION false: you cannot grow these volumes later, so size them generously.
• RECLAIMPOLICY Delete: deleting the volume claim deletes the data.

If local-path is not marked (default), set it:

Step 4.4: Install OpenEverest with Helm

helm repo add registers the OpenEverest chart repository, and helm repo update downloads its index. helm install creates a release named everest in a new namespace, everest-system, which holds the OpenEverest control plane.

Step 4.5: Verify the OpenEverest Pods

Two pods should reach Running within 1 to 2 minutes:

• everest-server: serves the web console and REST API.
• everest-operator: watches OpenEverest resources and turns them into database clusters by driving the Percona operators.

Output

Step 4.6: Retrieve the Admin Password

OpenEverest stores its user accounts in a Kubernetes Secret. Secret values are base64-encoded, so the command extracts the users.yaml key and decodes it. The output contains the admin user’s initial password.

Phase 5: Open the OpenEverest Web Console

The everest Service is only reachable inside the cluster. To use it from your laptop, chain two forwards: an SSH tunnel from your laptop to the control plane, and a kubectl port-forward from the control plane into the cluster.

Step 5.1: Open an SSH Tunnel from Your Laptop

-L 8080:127.0.0.1:8080 means: anything that connects to port 8080 on my laptop is sent through SSH to port 8080 on the control plane’s loopback interface.

Step 5.2: Port-Forward the UI Service

In the SSH session you just opened, forward local port 8080 on the control plane to the everest Service. Leave this running.

Security: kubectl port-forward listens on 127.0.0.1 by default, which is exactly where the SSH tunnel delivers traffic. Avoid –address 0.0.0.0: it exposes the UI to every host that can reach the control plane.

Step 5.3: Log In

Step 5.4: Create a MySQL Database

In the console, create a new MySQL database with 3 nodes. OpenEverest creates a Percona XtraDB Cluster with three PXC pods and HAProxy in front of them, in the everest namespace. It generates a name for the cluster; in this lab it is mysql-vjg, so substitute your own cluster name in the commands that follow.

Phase 6: Connect to MySQL

Step 6.1: Understand the Kubernetes Services

Before connecting, look at what OpenEverest created. Each Service is a stable network endpoint in front of a group of pods.

Output

Inside the cluster, CoreDNS resolves mysql-vjg-haproxy (or the full form mysql-vjg-haproxy.everest.svc.cluster.local) to the Service. That name does not resolve from your laptop, which is why the next steps use port forwarding.

Step 6.2: Get the MySQL Root Password

OpenEverest stores the database credentials in a Secret named everest-secrets-<cluster-name>. This command decodes the root key into a shell variable, so the password never appears on screen or in your shell history.

Step 6.3: Port-Forward HAProxy on the Control Plane

This forwards port 3306 on the control plane’s loopback interface to the HAProxy Service. Keep this terminal open.

Output

Step 6.4: Tunnel from Your Laptop

Port 3307 here is just a local port on your laptop, chosen so it doesn’t clash with a MySQL server you may already run on 3306. The tunnel sends it to port 3306 on the control plane, where the port-forward from Step 6.3 picks it up.

Step 6.5: Connect with the MySQL Client

In another laptop terminal, connect to the local end of the tunnel. -p with no value makes the client prompt for the password, so it is not saved in your shell history.

The full path of the connection is: laptop port 3307, then SSH, then control plane port 3306, then kubectl port-forward, then HAProxy, then the current writer pod.

Step 6.6: Check the Cluster Status

These queries show Galera’s replication status. Part 2 explains every value; for now, look for Synced, Primary and a cluster size of 3.

SQL

Output

Step 6.7: See Which Node You Are Connected To

@@hostname and @@wsrep_node_name show which PXC pod HAProxy picked for this connection.

SQL

Output

Notice @@read_only = 0. In Percona XtraDB Cluster every node accepts writes; there are no read-only replicas as in asynchronous replication. So why does HAProxy send all writes to one node?

Galera checks each transaction for conflicts at commit time (certification). If two nodes change the same row at the same moment, one of the transactions fails certification and the application gets ERROR 1213 (40001): Deadlock found when trying to get lock, even though no real lock wait happened. Sending all writes to a single node avoids these conflicts. The -replicas endpoint is meant for reads only by convention, not because the nodes are read-only.

Step 6.8: Open a MySQL Shell Inside a Specific Pod

To talk to one particular node without HAProxy, run the MySQL client inside that pod. A PXC pod has more than one container, so -c pxc selects the database container. -it gives you an interactive terminal, and the client prompts for the root password.

SQL

Output

Summary and What’s Next

You now have a 3-node K3s cluster running OpenEverest and a 3-node Percona XtraDB Cluster behind HAProxy, reachable from your laptop. Along the way you saw how K3s nodes join a cluster, why local-path storage pins database pods to nodes, and how HAProxy, headless Services and CoreDNS fit together.

In Part 2, we read Galera’s health indicators in detail and run a live failover test: we delete the writer pod while a client writes every second, then watch HAProxy switch writers, the pod rejoin with an incremental state transfer, and all three nodes end up with identical data.