We recently identified that a batch of our Percona Server for PostgreSQL RPM packages were inadvertently compiled with the debug assertion flag (–enable-cassert) enabled.
While these assertions are invaluable for our developers during the testing phase, they are not intended for production use. We have since rebuilt the packages and strongly recommend that all users of the affected versions update to the latest version immediately.
An “assertion” is a sanity check embedded in the code. It says: “The developer assumes that at this exact point, Condition X must be true. If it isn’t, something is fundamentally wrong.“
When you enable these in a production environment, several critical issues arise:
PostgreSQL performs millions of operations per second. With assertions enabled, the CPU must constantly stop and verify the internal state of the database.
In a standard production build, if a non-critical inconsistency occurs, PostgreSQL might log a warning or attempt to recover gracefully.
The inclusion of debug symbols and assertion logic increases the footprint of the PostgreSQL binaries. This can lead to less efficient use of instruction caches and slightly higher memory consumption per process.
During an update to our build pipeline, a configuration flag used for internal testing was incorrectly applied to the production RPM build scripts. This caused the compiler to include code paths that are normally stripped out for performance and stability.
The package numbers have changed as part of the rebuild, not the PostgreSQL versions. Please double-check pg_config to ensure you’re not affected.
We apologize for this oversight and are implementing additional automated checks in our CI/CD pipeline to prevent debug flags from reaching our production repositories in the future. If you have any questions or require further assistance, please don’t hesitate to contact us.
Latest fixed major releases:
https://docs.percona.com/postgresql/18/release-notes/release-notes-v18.1.1.html
https://docs.percona.com/postgresql/17/release-notes/release-notes-v17.7.1.html
https://docs.percona.com/postgresql/16/release-notes/release-notes-v16.11.html
https://docs.percona.com/postgresql/15/release-notes-v15.15.html
https://docs.percona.com/postgresql/14/release-notes/release-notes-v14.20.html
https://docs.percona.com/postgresql/13/release-notes/release-notes-v13.23.html
Resources
RELATED POSTS
Hi Kai,
thank you for sharing the insights, how that happened. Anyway, “We recently identified” sounds not “fully honest” to me, as it was users who identified the issue and reported it on GitHub: https://github.com/percona/postgres/issues/595
Kind regards
Aaron
Hi, the GitHub issue was created and reported after we had already been working on it, so my comment and blog are honest. Thank you for sharing your concerns. If you, for example, check the PG 18 repository, you will notice that we had already updated the repository on the 22nd, while the issue in GitHub was created on the 23rd.
Hi Kai,
thank you for your reply and pointing that out. It looked so obvious due to the close timeframe. But the obvious isn’t always the truth. Did not ment to offend you.
Kind regards,
Aaron