Downloads

Blog

How to Replace Patroni and Etcd IP/Host Information in PostgreSQL

January 2, 2025

Author

Anil Joshi

Insight for DBAs

PostgreSQL

Share this Post:

As we know, Patroni is a well-established standard for an HA framework for PostgreSQL clusters. From time to time, we need to perform maintenance tasks like upgrading the topology or making changes to the existing setup. Here, we will discuss mainly how we can replace the IP/Host information in Patroni and Etcd layers.

Below, we have a Patroni/Postgres based setup configured with Etcd as Distributed configuration Store (DCS).

Patroni:

shell> patronictl -c /etc/patroni/cluster1-0.yml list<br>+ Cluster: stampede (7440120421538637950) -----------+----+-----------+<br>| Member          | Host       | Role    | State     | TL | Lag in MB |<br>+-----------------+------------+---------+-----------+----+-----------+<br>| cluster1-0      | 172.31.8.2 | Leader  | running   |  1 |           |<br>| cluster115523-1 | 172.31.8.4 | Replica | streaming |  1 |         0 |<br>| cluster123283-2 | 172.31.8.3 | Replica | streaming |  1 |         0 |<br>+-----------------+------------+---------+-----------+----+-----------+<br><br>

shell> patronictl -c /etc/patroni/cluster1-0.yml list + Cluster: stampede (7440120421538637950) -----------+----+-----------+ | Member | Host | Role | State | TL | Lag in MB | +-----------------+------------+---------+-----------+----+-----------+ | cluster1-0 | 172.31.8.2 | Leader | running | 1 | | | cluster115523-1 | 172.31.8.4 | Replica | streaming | 1 | 0 | | cluster123283-2 | 172.31.8.3 | Replica | streaming | 1 | 0 | +-----------------+------------+---------+-----------+----+-----------+

ETCD:

shell> etcdctl  member list<br>5ab37e00821bf348, started, cluster123283-2, http://172.31.8.3:2380, http://172.31.8.3:2379, false<br>6d42e91b01739428, started, cluster1-0, http://172.31.8.2:2380, http://172.31.8.2:2379, false<br>f42fb256e9d14a57, started, cluster115523-1, http://172.31.8.4:2380, http://172.31.8.4:2379, false

shell> etcdctl member list 5ab37e00821bf348, started, cluster123283-2, http://172.31.8.3:2380, http://172.31.8.3:2379, false 6d42e91b01739428, started, cluster1-0, http://172.31.8.2:2380, http://172.31.8.2:2379, false f42fb256e9d14a57, started, cluster115523-1, http://172.31.8.4:2380, http://172.31.8.4:2379, false

Now, we will see the process to change all the IPs in both Etcd/Patroni configurations.

Pause the running cluster.

shell> patronictl -c /etc/patroni/cluster1-0.yml pause<br>Success: cluster management is paused

1	shell> patronictl -c /etc/patroni/cluster1-0.yml pause<br>Success: cluster management is paused

shell> patronictl -c /etc/patroni/cluster1-0.yml list<br>+ Cluster: stampede (7440120421538637950) -----------+----+-----------+<br>| Member          | Host       | Role    | State     | TL | Lag in MB |<br>+-----------------+------------+---------+-----------+----+-----------+<br>| cluster1-0      | 172.31.8.2 | Leader  | running   |  1 |           |<br>| cluster115523-1 | 172.31.8.4 | Replica | streaming |  1 |         0 |<br>| cluster123283-2 | 172.31.8.3 | Replica | streaming |  1 |         0 |<br>+-----------------+------------+---------+-----------+----+-----------+<br>Maintenance mode: on

Then, stop both the Etcd and Patroni service across all nodes.

shell> sudo systemctl stop etcd<br>shell> sudo systemctl stop patroni

1	shell> sudo systemctl stop etcd<br>shell> sudo systemctl stop patroni

Note – The cluster is still operational and postgresql will work fine and serve the traffic however the Patroni management and failover functionality will be disabled.

Here, we also need to clean the Etcd data directory on each node. This will auto restore from the Postgres directory dynamically when we bootstrapped the Etcd again with fresh data/information.

shell> cd /var/lib/etcd/postgres.etcd/<br>shell> sudo rm -rf /var/lib/etcd/postgres.etcd/*

1	shell> cd /var/lib/etcd/postgres.etcd/<br>shell> sudo rm -rf /var/lib/etcd/postgres.etcd/*

Next, we will replace the new IP’s in each Etcd configuration(“/etc/etcd/etcd.conf”).

1) On Node0 replace Old (172.31.8.2) with New (172.31.8.5).

ETCD_NAME=cluster1-0<br>ETCD_INITIAL_CLUSTER="cluster1-0=http://172.31.8.5:2380"<br>ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^"<br>ETCD_INITIAL_CLUSTER_STATE="new"<br>ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.5:2380"<br>ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd"<br>ETCD_LISTEN_PEER_URLS="http://172.31.8.5:2380"<br>ETCD_LISTEN_CLIENT_URLS="http://172.31.8.5:2379,http://localhost:2379"<br>ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.5:2379"<br>

ETCD_NAME=cluster1-0 ETCD_INITIAL_CLUSTER="cluster1-0=http://172.31.8.5:2380" ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.5:2380" ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.8.5:2380" ETCD_LISTEN_CLIENT_URLS="http://172.31.8.5:2379,http://localhost:2379" ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.5:2379"

2) Start the Initial Bootstrap node.

Systemctl start etcd<br>

1	Systemctl start etcd<br>

3) On Node1 replace Old (172.31.8.3, 172.31.8.2) with New (172.31.8.6, 172.31.8.5).

ETCD_NAME=cluster123283-2<br>ETCD_INITIAL_CLUSTER="cluster123283-2=http://172.31.8.6:2380,cluster1-0=http://172.31.8.5:2380"<br>ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^"<br>ETCD_INITIAL_CLUSTER_STATE="existing"<br>ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.6:2380"<br>ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd"<br>ETCD_LISTEN_PEER_URLS="http://172.31.8.6:2380"<br>ETCD_LISTEN_CLIENT_URLS="http://172.31.8.6:2379,http://localhost:2379"<br>ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.6:2379"<br>

ETCD_NAME=cluster123283-2 ETCD_INITIAL_CLUSTER="cluster123283-2=http://172.31.8.6:2380,cluster1-0=http://172.31.8.5:2380" ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^" ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.6:2380" ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.8.6:2380" ETCD_LISTEN_CLIENT_URLS="http://172.31.8.6:2379,http://localhost:2379" ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.6:2379"

Note – Since member [cluster1-0] is UP now, we will use only this initial member node[172.31.8.5] for the “ETCD_INITIAL_CLUSTER” along with the Node1[172.31.8.6].

 

4) Now add this member [cluster123283-2] to the existing cluster. Run the below command on Node0.

shell> etcdctl member add  cluster123283-2 --peer-urls=http://172.31.8.6:2380<br><br>Output:<br>Member 869ab1ea478463c6 added to cluster 271846978ccf8ae3<br>ETCD_NAME="cluster122591-1"<br>ETCD_INITIAL_CLUSTER="cluster1-0=http://192.168.48.5:2380,cluster122591-1=http://192.168.48.6:2380"<br>ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.48.6:2380"<br>ETCD_INITIAL_CLUSTER_STATE="existing"<br>

shell> etcdctl member add cluster123283-2 --peer-urls=http://172.31.8.6:2380 Output: Member 869ab1ea478463c6 added to cluster 271846978ccf8ae3 ETCD_NAME="cluster122591-1" ETCD_INITIAL_CLUSTER="cluster1-0=http://192.168.48.5:2380,cluster122591-1=http://192.168.48.6:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.48.6:2380" ETCD_INITIAL_CLUSTER_STATE="existing"

5) Then, start the Etcd service on Node2.

shell> systemctl start etcd<br>

1	shell> systemctl start etcd<br>

6) On Node2 replace Old (172.31.8.3, 172.31.8.2, 172.31.8.4) with New (172.31.8.6, 172.31.8.5,172.31.8.7).

ETCD_NAME=cluster115523-1<br>ETCD_INITIAL_CLUSTER="cluster115523-1=http://172.31.8.7:2380,cluster1-0=http://172.31.8.5:2380,cluster123283-2=http://172.31.8.6:2380"<br>ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^"<br>ETCD_INITIAL_CLUSTER_STATE="existing"<br>ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.7:2380"<br>ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd"<br>ETCD_LISTEN_PEER_URLS="http://172.31.8.7:2380"<br>ETCD_LISTEN_CLIENT_URLS="http://172.31.8.7:2379,http://localhost:2379"<br>ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.7:2379"<br>

ETCD_NAME=cluster115523-1 ETCD_INITIAL_CLUSTER="cluster115523-1=http://172.31.8.7:2380,cluster1-0=http://172.31.8.5:2380,cluster123283-2=http://172.31.8.6:2380" ETCD_INITIAL_CLUSTER_TOKEN="verysecretpassword1^" ETCD_INITIAL_CLUSTER_STATE="existing" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.8.7:2380" ETCD_DATA_DIR="/var/lib/etcd/postgres.etcd" ETCD_LISTEN_PEER_URLS="http://172.31.8.7:2380" ETCD_LISTEN_CLIENT_URLS="http://172.31.8.7:2379,http://localhost:2379" ETCD_ADVERTISE_CLIENT_URLS="http://172.31.8.7:2379"

7) Now add this member [cluster115523-1] to the existing cluster. Run the below command on Node0.

shell> etcdctl member add  cluster115523-1 --peer-urls=http://172.31.8.7:2380<br><br>Output:<br>Member dd8eb33294f33cf8 added to cluster 271846978ccf8ae3<br>ETCD_NAME="cluster1685-1"<br>ETCD_INITIAL_CLUSTER="cluster1-0=http://192.168.48.5:2380,cluster122591-1=http://192.168.48.6:2380,cluster1685-1=http://192.168.48.7:2380"<br>ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.48.7:2380"<br>ETCD_INITIAL_CLUSTER_STATE="existing"<br>

shell> etcdctl member add cluster115523-1 --peer-urls=http://172.31.8.7:2380 Output: Member dd8eb33294f33cf8 added to cluster 271846978ccf8ae3 ETCD_NAME="cluster1685-1" ETCD_INITIAL_CLUSTER="cluster1-0=http://192.168.48.5:2380,cluster122591-1=http://192.168.48.6:2380,cluster1685-1=http://192.168.48.7:2380" ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.48.7:2380" ETCD_INITIAL_CLUSTER_STATE="existing"

8) Then, start the Etcd service on Node3.

shell> systemctl start etcd<br>

1	shell> systemctl start etcd<br>

9) At this stage, the Etcd nodes will be ready with the new IP changes.

shell> etcdctl  member list<br><br>119ded3078b62b37, started, cluster122024-1, http://172.22.0.7:2380, http://172.22.0.7:2379, false<br>ca08be8df731dada, started, cluster1-0, http://172.22.0.5:2380, http://172.22.0.5:2379, false<br>e7ed45e8c76a64c6, started, cluster111821-1, http://172.22.0.6:2380, http://172.22.0.6:2379, false

shell> etcdctl member list 119ded3078b62b37, started, cluster122024-1, http://172.22.0.7:2380, http://172.22.0.7:2379, false ca08be8df731dada, started, cluster1-0, http://172.22.0.5:2380, http://172.22.0.5:2379, false e7ed45e8c76a64c6, started, cluster111821-1, http://172.22.0.6:2380, http://172.22.0.6:2379, false

shell> etcdctl endpoint status --cluster=true --write-out=table<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>| http://72.22.0.7:2379 | 119ded3078b62b37 |  3.5.12 |   61 kB |     false |      false |         3 |         63 |                 63 |        |<br>| http://172.22.0.5:2379 | ca08be8df731dada |  3.5.12 |   61 kB |     false |      false |         3 |         63 |                 63 |        |<br>| http://172.22.0.6:2379 | e7ed45e8c76a64c6 |  3.5.12 |   61 kB |      true |      false |         3 |         63 |                 63 |        |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>

shell> etcdctl endpoint status --cluster=true --write-out=table +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | http://72.22.0.7:2379 | 119ded3078b62b37 | 3.5.12 | 61 kB | false | false | 3 | 63 | 63 | | | http://172.22.0.5:2379 | ca08be8df731dada | 3.5.12 | 61 kB | false | false | 3 | 63 | 63 | | | http://172.22.0.6:2379 | e7ed45e8c76a64c6 | 3.5.12 | 61 kB | true | false | 3 | 63 | 63 | | +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Next, let’s make the changes with respect to Patroni configurations. One by one, we can edit the Patroni configuration file, whereas we found the old IP traces with the changes below.

On Node0 replace Old (172.31.8.2) with New (172.31.8.5) in (“/etc/patroni/cluster1-0.yml”).

restapi:<br> connect_address: 172.31.8.5:8008<br> <br> <br> etcd3:<br> host: 172.31.8.5:2379<br> <br> postgresql:<br> connect_address: 172.31.8.5:5432<br><br><br> pg_hba:  # Add following lines to pg_hba.conf after running 'initdb'<br> - host replication replicator 172.31.8.5/24 md5<br> - host replication replicator 127.0.0.1/32 trust<br> - host all all 172.31.8.5/24 md5<br> - host all all 0.0.0.0/0 md5

restapi: connect_address: 172.31.8.5:8008 etcd3: host: 172.31.8.5:2379 postgresql: connect_address: 172.31.8.5:5432 pg_hba: # Add following lines to pg_hba.conf after running 'initdb' - host replication replicator 172.31.8.5/24 md5 - host replication replicator 127.0.0.1/32 trust - host all all 172.31.8.5/24 md5 - host all all 0.0.0.0/0 md5

On Node1 replace Old (172.31.8.3) with New (172.31.8.6) in (“/etc/patroni/cluster123283-2.yml”).

restapi:<br> connect_address: 172.31.8.6:8008<br> <br> <br> etcd3:<br> host: 172.31.8.6:2379<br> <br> postgresql:<br> connect_address: 172.31.8.6:5432<br><br> pg_hba:  # Add following lines to pg_hba.conf after running 'initdb'<br> - host replication replicator 172.31.8.6/24 md5<br> - host replication replicator 127.0.0.1/32 trust<br> - host all all 172.31.8.6/24 md5<br> - host all all 0.0.0.0/0 md5

restapi: connect_address: 172.31.8.6:8008 etcd3: host: 172.31.8.6:2379 postgresql: connect_address: 172.31.8.6:5432 pg_hba: # Add following lines to pg_hba.conf after running 'initdb' - host replication replicator 172.31.8.6/24 md5 - host replication replicator 127.0.0.1/32 trust - host all all 172.31.8.6/24 md5 - host all all 0.0.0.0/0 md5

On Node2 replace Old (172.31.8.4) with New (172.31.8.7) in (“/etc/patroni/cluster/cluster115523-1.yml”).

restapi:<br> connect_address: 172.31.8.7:8008<br> <br> <br> etcd3:<br> host: 172.31.8.7:2379<br> <br> postgresql:<br> connect_address: 172.31.8.7:5432<br><br> pg_hba:  # Add following lines to pg_hba.conf after running 'initdb'<br> - host replication replicator 172.31.8.7/24 md5<br> - host replication replicator 127.0.0.1/32 trust<br> - host all all 172.31.8.7/24 md5<br> - host all all 0.0.0.0/0 md5

restapi: connect_address: 172.31.8.7:8008 etcd3: host: 172.31.8.7:2379 postgresql: connect_address: 172.31.8.7:5432 pg_hba: # Add following lines to pg_hba.conf after running 'initdb' - host replication replicator 172.31.8.7/24 md5 - host replication replicator 127.0.0.1/32 trust - host all all 172.31.8.7/24 md5 - host all all 0.0.0.0/0 md5

Once the changes are made on all Postgres/Patroni nodes, we can simply start the Patroni service on all nodes and resume Patroni management.

shell> systemctl start patroni<br>

1	shell> systemctl start patroni<br>

Then we can also resume the paused service again from Node0:

shell> patronictl -c /etc/patroni/cluster1-0.yml resume<br><br>Output:<br>Success: cluster management is resumed

1	shell> patronictl -c /etc/patroni/cluster1-0.yml resume<br><br>Output:<br>Success: cluster management is resumed

Now we can see the Patroni status below with the new Host/IP changes.

shell> patronictl -c /etc/patroni/cluster1-0.yml list<br>Output:<br><br>[root@node0 /]# patronictl list<br>+ Cluster: stampede (7442986751514364015) -+-----------+----+-----------+-----------------+----------------------------------------------+<br>| Member          | Host         | Role    | State     | TL | Lag in MB | Pending restart | Pending restart reason                       |<br>+-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+<br>| cluster1-0      | 192.168.48.5 | Leader  | running   |  1 |           |                <br>|                 |              |         |           |    |           |                 -----------+----------------------------------------------+<br>| cluster122591-1 | 192.168.48.6 | Replica | streaming |  1 |         0 |                 |                                              |<br>+-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+<br>| cluster1685-1   | 192.168.48.7 | Replica | streaming |  1 |         0 |                 |                                              |<br>+-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+<br>

shell> patronictl -c /etc/patroni/cluster1-0.yml list Output: [root@node0 /]# patronictl list + Cluster: stampede (7442986751514364015) -+-----------+----+-----------+-----------------+----------------------------------------------+ | Member | Host | Role | State | TL | Lag in MB | Pending restart | Pending restart reason | +-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+ | cluster1-0 | 192.168.48.5 | Leader | running | 1 | | | | | | | | | -----------+----------------------------------------------+ | cluster122591-1 | 192.168.48.6 | Replica | streaming | 1 | 0 | | | +-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+ | cluster1685-1 | 192.168.48.7 | Replica | streaming | 1 | 0 | | | +-----------------+--------------+---------+-----------+----+-----------+-----------------+----------------------------------------------+

Final thought

The downtime/impact would be low as we put the cluster in “pause” mode which still allows the traffic OR workload to serve without any impact. Also, the backend/Postgres instance wouldn’t have any isolated impact by stopping Patroni as the setup is already in maintenance mode. Only the automatic failover and management of the cluster will be compromised for that duration.

Learn how to elevate PostgreSQL to an enterprise-grade solution with powerful extensions and tools. Our experts reveal the best ways to optimize performance, add functionality, ensure high availability, and automate key tasks.

Download now: PostgreSQLExtension Handbook