We have been promoting and using Patroni as the best high availability framework for PostgreSQL, and Etcd was the preferred/recommended DCS for the Patroni cluster. Both Patroni and Etcd have been part of PostgreSQL distribution from Percona for years now. But one area where we were stuck was the Etcd version, and we continued to use Etcd 3.3 for many years. The reason for this was that Etcd 3.3 was the last version to have API version two as the default, and we didn’t want to break different topologies in user environments and other tools around.
I see many user environments continue to use very old versions of Patroni and Etcd. They continue to be there because everything works fine for them, and there is no compelling reason for upgrading. But we should not forget that Etcd 3.3 is already too old, and its end of life was two years ago. In at least a couple of cases, it was captured in security audits.
The time has come to move on. Recently, we at Percona also moved to Etcd 3.5. Read the announcement here.
This blog post aims to encourage users to upgrade to the latest version of Etcd (3.5.xx). As a first step, let me explain a few differences and changes in our approaches to improving Patroni users’ overall experiences.
Etcd API version change
Etcd version 3.3 was the last version to have API version two as the default. A new version like 3.5 has API version three as the default. Unless there is any compelling reason to stick to an older version of APIs, it would be the right time to switch to a new API version.
As the Patroni documentation says, the switch is not easy because the old key will not be visible in the new API version.
Reference: https://patroni.readthedocs.io/en/latest/yaml_configuration.html#etcdv3
This may not be a big deal for a new cluster, but it could be challenging for existing installations, tooling, automation, and monitoring.
Etcd configuration in YAML
We have been using the environment variable approach to configure and set up Etcd for years. However, we found that we have a few challenges in our customer environment with environmental variables, especially for troubleshooting. So we decided to switch to YAML file for configurations, which is explicit and clear.
Here is the sample YAML configuration for Etcd.
|
1 2 3 4 5 6 7 8 9 10 |
$ cat /etc/etcd/etcd.conf.yaml name: 'pg1' data-dir: /var/lib/etcd initial-advertise-peer-urls: http://10.3.188.140:2380 listen-peer-urls: http://10.3.188.140:2380 advertise-client-urls: http://10.3.188.140:2379 listen-client-urls: http://10.3.188.140:2379,http://localhost:2379 initial-cluster: pg1=http://10.3.188.140:2380,pg2=http://10.3.188.138:2380,pg3=http://10.3.188.104:2380 initial-cluster-state: new initial-cluster-token: pgcluster |
Bootstrapping Etcd cluster
Historically, we were always bootstrapping an Etcd cluster as a single node cluster and then keep adding additional nodes to the cluster. We still continue to recommend that to novice users, as mentioned in the documentation.
In addition to this, I want to let those who feel that this multi-step bootstrapping is cumbersome, it is also possible to start the cluster nodes at a time as demonstrated below.
Patroni configuration change
The change for API v3 is very minimal on Patroni’s side: https://patroni.readthedocs.io/en/latest/yaml_configuration.html#etcdv3. We could just specify “etcd3” instead of the old “etcd” in the configuration file.
Upgrading Etcd version 3.3 to 3.5
Now, coming back to the main topic of the blog post, how to get your Patroni cluster upgraded to a new version of Etcd while switching from APIv2 to v3. This content covers only the Etcd installation specific to a single Patroni cluster. These steps are simple and straightforward. But if the same Etcd cluster is serving multiple Patroni clusters or serving any service other than Patroni, please don’t use these steps.
Step one: Put the Patroni into maintenance mode.
|
1 2 |
$ patronictl -c /etc/patroni/patroni.yml pause Success: cluster management is paused |
This allows us to take down the Patroni service without affecting PostgreSQL, so the application can remain unaffected during this maintenance.
Step two: Verify the maintenance mode and shut down Patroni in all nodes of the cluster.
|
1 2 3 4 5 6 7 8 9 10 11 |
$ patronictl -c /etc/patroni/patroni.yml list + Cluster: stampede (7379471919345521402) -+-----------+ | Member | Host | Role | State | TL | Lag in MB | +--------+------+---------+-----------+----+-----------+ | pg1 | pg1 | Replica | streaming | 1 | 0 | | pg2 | pg2 | Leader | running | 1 | | | pg3 | pg3 | Replica | streaming | 1 | 0 | +--------+------+---------+-----------+----+-----------+ Maintenance mode: on $ sudo systemctl stop patroni |
Since the Patroni is already in maintenance mode. Stopping the Patroni service doesn’t have any effect on the PostgreSQL cluster.
Step three: Shutdown Etcd service on all nodes.
|
1 |
sudo systemctl stop etcd |
Step four: Clean up Etcd data directory.
We don’t need the data directory of the older version and its contents. Cleanup the Etcd data directory as root.
|
1 2 3 |
cd /var/lib/etcd/postgres.etcd/ sudo rm -rf /var/lib/etcd/postgres.etcd/* ls |
** The data directory location might be different in your system. Please verify before doing it.
Now, novice users of Patroni might be worrying about configuration loss. But Patroni is designed to keep a copy of its dynamic configuration in a file patroni.dynamic.json within the PostgreSQL data directory. Patroni is smart enough to restore it if it sees an empty DCS.
Step five: Remove the old version of etcd (3.30) and install the new version of Etcd (3.5) binaries in each node of the cluster.
This step depends on the Linux distribution and repository/package used. Just for example:
|
1 2 3 4 5 6 7 8 9 10 |
$ sudo dnf remove etcd Dependencies resolved. ========================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================= Removing: etcd x86_64 3.3.11-4.el8 @ppg-11-release-x86_64 42 M … $ sudo dnf install etcd |
Step six: Create a fresh Etcd cluster.
|
1 2 3 4 5 6 7 8 9 |
name: 'pg1' data-dir: /var/lib/etcd initial-advertise-peer-urls: http://10.3.188.140:2380 listen-peer-urls: http://10.3.188.140:2380 advertise-client-urls: http://10.3.188.140:2379 listen-client-urls: http://10.3.188.140:2379,http://localhost:2379 initial-cluster: pg1=http://10.3.188.140:2380,pg2=http://10.3.188.138:2380,pg3=http://10.3.188.104:2380 initial-cluster-state: new initial-cluster-token: pgcluster |
Please refer to the video demo linked above in the “Bootstrapping Etcd cluster” section.
Step seven: Start and verify the Etcd cluster.
|
1 2 3 4 5 |
$ sudo systemctl start etcd $ etcdctl member list 8286431a18928d1f, started, pg2, http://10.3.188.138:2380, http://10.3.188.138:2379, false e054f84d75eb4097, started, pg3, http://10.3.188.104:2380, http://10.3.188.104:2379, false fcba16da26d50257, started, pg1, http://10.3.188.140:2380, http://10.3.188.140:2379, false |
Step eight: Modify the Patroni configuration to use Etcd API version three.
Instead of the “etcd:” section in the Patroni configuration, use “etcd3:” in the configuration file. For example:
|
1 2 |
etcd3: hosts: pg1:2379,pg2:2379,pg3:2379 |
Step nice: Start the Patroni on all the nodes one by one.
|
1 |
$sudo systemctl start patroni |
Patroni will automatically copy the contents of the file patroni.dynamic.json to Keys and values in the new ETCD cluster.
We should be able to verify that all Patroni-related configurations are copied back to the Etcd.
|
1 |
$ patronictl -c /etc/patroni/patroni.yml show-config |
Step ten: Take the Patroni cluster out of maintenance mode.
|
1 2 |
$ patronictl -c /etc/patroni/patroni.yml resume Success: cluster management is resumed |
Now Patroni will take over the PostgreSQL cluster, which was running all this time as an independent cluster, and start managing it.
Our PostgreSQL Performance Tuning eBook condenses years of database expertise into a practical guide for optimizing your PostgreSQL databases. Inside, you’ll discover our most effective PostgreSQL performance strategies derived from real-world experience.
Download now: Elevate your PostgreSQL Performance
About the Author
Jobin AugustineJobin Augustine is a PostgreSQL expert and Open Source advocate and has more than 21 years of working experience as a consultant, architect, administrator, writer, and trainer in PostgreSQL, Oracle and other database technologies. He has always been an active participant in the Open Source communities, and his main focus area is database performance and optimization. Jobin is a regular face in many of the PostgreSQL conferences and has presented at various conferences for the last 6+ years. He is a contributor to multiple Open Source Projects, an active blogger, and loves coding in C++ and Python. Jobin holds a Master's in Computer Applications; Currently, he is the PostgreSQL Escalation Specialist for Percona. Previously, he worked at OpenSCG for two years as an architect and was part of the BigSQL core team. Previous to his work at OpenSCG, Jobin worked at Dell as Database Senior Advisor for ten years and another five years with TCS/CMC.
