We recently identified that a batch of our Percona Server for PostgreSQL RPM packages were inadvertently compiled with the debug assertion flag (–enable-cassert) enabled.
While these assertions are invaluable for our developers during the testing phase, they are not intended for production use. We have since rebuilt the packages and strongly recommend that all users of the affected versions update to the latest version immediately.
Why Assertions Don’t Belong in Production
An “assertion” is a sanity check embedded in the code. It says: “The developer assumes that at this exact point, Condition X must be true. If it isn’t, something is fundamentally wrong.“
When you enable these in a production environment, several critical issues arise:
1. Severe Performance Degradation
PostgreSQL performs millions of operations per second. With assertions enabled, the CPU must constantly stop and verify the internal state of the database.
- The Result: You will likely experience a significant increase in CPU utilization and a corresponding decrease in transaction throughput. In many cases, performance can degrade by 20% to 50% depending on the workload.
2. The “Fail-Fast” Problem (Unexpected Downtime)
In a standard production build, if a non-critical inconsistency occurs, PostgreSQL might log a warning or attempt to recover gracefully.
- The Result: With assertions enabled, the instruction is to abort immediately. If an assertion fails, the entire backend process crashes as quickly as possible, so the coredump/debugger is as close to the originating issue as possible. While this is beneficial for catching bugs in a lab, it can lead to potential service restarts, dropped connections, and historically false positives in a live environment.
3. Increased Binary Size and Memory Overhead
The inclusion of debug symbols and assertion logic increases the footprint of the PostgreSQL binaries. This can lead to less efficient use of instruction caches and slightly higher memory consumption per process.
What Happened?
During an update to our build pipeline, a configuration flag used for internal testing was incorrectly applied to the production RPM build scripts. This caused the compiler to include code paths that are normally stripped out for performance and stability.
Recommended Action:
- Check your version: If you installed or updated within the last 4 months, you are likely affected.
- Affected versions:
- 18.1
- 17.6 & 17.7
- 16.10 & 16.11
- 15.14 & 15.15
- 14.19 & 14.20
- 13.22 & 13.23
- Manual check: “pg_config –configure”
- If you see “–enable-cassert” in the output, you’re affected!
- Affected versions:
- Update immediately to the newest released RPMs or images.
The package numbers have changed as part of the rebuild, not the PostgreSQL versions. Please double-check pg_config to ensure you’re not affected.
We apologize for this oversight and are implementing additional automated checks in our CI/CD pipeline to prevent debug flags from reaching our production repositories in the future. If you have any questions or require further assistance, please don’t hesitate to contact us.
Latest fixed major releases:
https://docs.percona.com/postgresql/18/release-notes/release-notes-v18.1.1.html
https://docs.percona.com/postgresql/17/release-notes/release-notes-v17.7.1.html
https://docs.percona.com/postgresql/16/release-notes/release-notes-v16.11.html
https://docs.percona.com/postgresql/15/release-notes-v15.15.html
https://docs.percona.com/postgresql/14/release-notes/release-notes-v14.20.html
https://docs.percona.com/postgresql/13/release-notes/release-notes-v13.23.html
About the Author
Kai WagnerI'm Kai and I work as a Senior Engineering Manager for PostgreSQL at Percona. I'm passionate about Open Source, transparency and free speech. Actively engaged and a public speaker for many Open Source projects such as Ceph, openATTIC, Linux Kernel and now PostgreSQL. I believe an open world is a better world. Whenever I'm not sitting in front of my computer, I either spent time with the family or work on our house.
