This article is, in fact, two topics merged into one publication. Both are related to anonymous statistical data collection within Percona releases of database engines: MySQL, MongoDB, and PostgreSQL.
In the first part of this article, I will share some of our findings and observations and discuss the various conclusions we have drawn from them. The main goal of the first section, however, is to share what we have learned with the community.
In the second section, I will share some details of the new, improved telemetry mechanisms we are now releasing across Percona DBs, and what we expect to learn from them.
Chapter 1
Telemetry findings and observations: sharing back with the community
Several months ago, in a blog post titled “Percona Is Introducing Telemetry Mechanisms Into MySQL, PostgreSQL, and MongoDB,” I announced that we were integrating new telemetry mechanisms into Percona database products. At that time, I assured the community that while the primary benefit of this telemetry is for the Percona product team to make more informed decisions, we would also share some of the gathered statistics with the community.
After nearly nine months of data collection, I would like to share some of our findings here.
Disclaimer: Please note that the graphs I am sharing in this chapter are screenshots from our internal system and dashboards that we are evolving and experimenting on, so they are not always consistent visually.
Key findings for MySQL
Last year was significant for MySQL, marked by the introduction of Innovation Releases—8.1, 8.2, and 8.3—and the designation of versions 8.0.x and 8.4.x as the Long-Term Support releases. Additionally, we witnessed the community edition of MySQL 5.7 reaching its End of Life (EOL) phase.
This raises the crucial question: How is adopting these major versions progressing? Let’s first examine the data provided by our in-product telemetry.
The accompanying graph illustrates only the new installations over the past 90 days.
Image 1: Breakdown of new deployment events by version for Percona Server for MySQL for the last 90 days
Let’s compare this with the statistics from MySQL and MariaDB environments where Percona Monitoring and Management (PMM) is installed. This data source provides a breakdown of active instances over the past 90 days:
Image 2: Breakdown of active instances by version for MySQL and MariaDB monitored by PMM for the last 90 days. The version numbers 5.x, and 8.x show the MySQL versions. The other numbers show MariaDB. Please note that the percentages of MySQL vs. MariaDB versions might not represent the true market presence – it is only what we see through PMM.
We have made two immediate observations.
First, despite MySQL 5.7 reaching its EOL status in November 2023, it remains widely used, with new deployments continuing. This persistence is understandable; many MySQL users I’ve spoken with adhere to the principle of “if it ain’t broke, don’t fix it.” MySQL 5.7 is known for its stability and superior performance compared to versions 8.0 and 8.4. Users not requiring the new features and improvements introduced in later releases often stick with 5.7. The primary drawback of continuing with MySQL 5.7 is the lack of publicly available security fixes.
Second, there are very few deployments on Innovation Releases 8.1, 8.2, and 8.3. This is not surprising, given the short lifespan of these versions. We anticipate a significant shift towards version 8.4 once it is released and gains a reputation for stability among community users. At the time of this blog post, Percona Server for MySQL 8.4 is about to be published.
Percona decisions
- We will continue the MySQL Post-EOL Support program as initially intended and continue to observe the trends and adoption of MySQL 5.7.x
- We will not release Innovation Releases of MySQL versions 9.x. Please read this blog post for the details: No MySQL 9.x Innovation Releases from Percona.
Let’s now look at the types of deployments and the most popular architectures. In this case, we will support our analysis with the data coming from the in-product telemetry, so it only contains information about new deployment events:
Image 3: Breakdown of new deployment events by deployment method for Percona Server for MySQL for the last 90 days
Image 4: Breakdown of new deployment events by hardware architecture for Percona Server for MySQL for the last 90 days
The graphs provide several key insights. First, it is not surprising that most deployments occur in Docker environments. This aligns with the inherent characteristics of such environments, where the dynamic creation and destruction of instances are common and often driven by scaling needs. In the upcoming version of telemetry, we will be able to track the average lifespan of these instances.
The graph in Image 4 highlights the distribution of architectures on which deployments are performed. We are noticing an increasing demand for ARM-based solutions. Although the current presence of ARM deployments is modest, we anticipate significant growth in the near future, supported by our recent and forthcoming releases of ARM packages for RedHat and compatible distributions and Debian/Ubuntu.
Percona decisions
- We will continue investing into Docker-based and Kubernetes-based solutions for MySQL. Please see the announcement, Introducing Percona Everest: Revolutionizing Cloud-Native Database Management.
- We will continue investing in ARM-based packages and Docker images, with the immediate goal of supporting Debian and Ubuntu Linux distributions.
Key findings for MongoDB
For MongoDB, let’s begin by analyzing the versions that have been deployed both recently and historically.
The initial data point involves examining MongoDB version numbers in deployment events over the past 90 days.
Image 5: Breakdown of new deployment events by version for Percona Server for MongoDB for the last 90 days
As illustrated, version 6.0.x is the predominant choice, accounting for nearly 87% of all new deployments of Percona Server for MongoDB. Deployments of version 5.0 and earlier have become virtually obsolete.
Next, compare this with data from Percona Monitoring and Management, which shows the active MongoDB instances. We will now examine the version dynamics over the past 12 months.
Image 6: Dynamics of active instances by version for MongoDB monitored by PMM for the last 12 months
An interesting observation from these two graphs is that while the EOL version MongoDB 4.4.x remains present in existing deployments, it is no longer being newly deployed, and its prevalence is steadily declining. A similar trend is seen with version 5.0.x, although the server base for MongoDB 5.0.x remains relatively stable.
Moreover, version 6.0.x has rapidly gained high adoption, while version 7.0.x is being adopted more cautiously. That’s surprising as upgrading between major MongoDB versions is relatively straightforward and low risk. On the one hand, 7.0.x brings a lot of value with performance, security, and developer experience. On the other hand, you must be aware of some risks we’ve shared within five changes you should know in MongoDB 7. We’re very interested in the reasons and risks you see in adopting the new version. If you’re willing to share, let us know in the MongoDB forum or contact us.
Percona decisions
- While we were considering creating a MongoDB 4.x post-EOL program similar to the program for MySQL, given MongoDB’s different dynamics, this program seems to have little chance of success. Consequently, we will not run it.
- MongoDB 5.0 reaches EOL at the end of October 2024. As we shared, it’s still widely present in your environments. We’re evaluating the post-EOL program for 5.0. If you’re uncomfortable with upgrading to a newer version and prefer to stay longer, we’re here to help. Let’s have a chat!
- Observing the stable and continuous activity of MongoDB 5.0.x, we’ve released the last patch, 5.0.28-24, which contained not only bug fixes but also an innovation: KMIP key state polling to reduce mean time to resolve (MTTR) compromised encryption key incidents.
Next, let’s examine the deployment methods and the popularity of hardware architectures for MongoDB.
Image 7: Breakdown of new deployment events by deployment method for Percona Server for MongoDB for the last 90 days
Image 8: Breakdown of new deployment events by hardware architecture for Percona Server for MongoDB for the last 90 days
Similar to MySQL, most new MongoDB deployments are carried out on Docker, which is not unexpected.
However, it is somewhat concerning that the number of deployments on ARM architecture has decreased in the last quarter. While ARM-based deployments had stabilized around 3%, this figure has recently dropped to below 1%. It remains unclear whether this fluctuation is a periodic trend or indicative of a more sustained decline in the use of ARM hardware for MongoDB deployments.
Percona decisions
- Like in the case of MySQL, we will continue investment into Docker-based and Kubernetes-based solutions MongoDB.
- We will continue observing the ARM-based deployments and react accordingly.
Key findings for PostgreSQL
Now, let’s turn our attention to PostgreSQL data. As always, we will begin by examining the version breakdown.
Image 9: Breakdown of new deployment events by version for Percona Server for PostgreSQL for last 90 days
Image 10: Weekly breakdown of the percentage of PostgreSQL major versions for the last 12 months
These graphs provide complementary insights and reveal some interesting trends. PostgreSQL 14 appears to have reached a stable equilibrium, showing neither significant growth nor decline. In contrast, versions 15 and 16 are rapidly gaining traction, gradually diminishing the presence of older versions such as 13, 12, 11, and earlier. Versions 10 and 9 are present only marginally.
What does this indicate? It illustrates how the approaching EOL dates for major PostgreSQL versions influence the dynamics of new deployments and market presence. Notably, PostgreSQL 12 is set to reach its EOL in November 2024 and PostgreSQL 13 in November 2025. Both versions are already showing signs of decreased presence. This pattern reflects a healthy trend, where legacy versions do not linger excessively, and users are proactively upgrading their environments ahead of EOL dates.
Percona decisions
- We do not see a need for planning post-EOL programs for PostgreSQL.
Are there differences between PostgreSQL, MySQL, and MongoDB regarding Docker-based deployments and hardware architecture selection?
Image 11: Breakdown of new deployment events by deployment method for Percona Server for PostgreSQL for the last 90 days
Image 12: Breakdown of new deployment events by hardware architecture for Percona Server for PostgreSQL for the last 90 days
While the presence of ARM architecture is noteworthy, the more intriguing graph depicts the distribution between package-based and Docker-based deployments. The high volume of Docker deployments for Percona Server for PostgreSQL is particularly noteworthy, given that this deployment method was introduced only recently. We anticipate that the trend will continue to reflect an increase in Docker-based deployments in the future.
Percona decisions
- We will focus on making a broader list of extensions more easily available to the Docker-based and Kubernetes-based customers.
- As we anticipate growing popularity for Percona Everest, we aim to ensure that ARM architectures, known for their cost-effectiveness, are supported. This will provide our customers with an attractive option for deploying Percona Everest in a more economical way, offering significant potential for cost savings.
Most popular operating systems
Finally, let’s examine the popularity of operating systems across our databases. The following graphs present data on new deployments of package-based versions of Percona databases over the past 90 days.
Image 13, 14, 15: Most popular OSes used for deployment of MySQL, MongoDB, and PostgreSQL during the last 90 days
In all three cases, Ubuntu emerges as the dominant operating system. Percona database users predominantly favor popular free Linux distributions over paid options. However, there are notable differences in preferences depending on the database technology:
- MySQL: A significant portion of deployments occurs on RHEL.
- PostgreSQL: Users prefer Oracle Linux as a RHEL-compatible alternative, while MySQL and MongoDB users lean towards Rocky Linux.
- CentOS 7: It is still commonly used for MySQL and PostgreSQL deployments, despite this version being past its end-of-life date.
Percona decisions
- Given the high presence of RHEL for MySQL, we decided to further encourage this trend by acquiring official RHEL9 certification for Percona software for MySQL version 8.0.x:
- We are in the process of providing builds for Ubuntu 24.04 for all Percona Database technologies
- With the upcoming patched PostgreSQL server we are going to acquire RHEL certification for the Percona Distribution for PostgreSQL starting with the PostgreSQL 17 to make it more attractive for the RHEL users.
- We will explore more operating systems and architectures required by the enterprises that could make these open source products easier to be adopted
Chapter 2
New and improved telemetry mechanisms
I am pleased to announce that we are enhancing the telemetry mechanisms to gather more comprehensive data, which will enable us to make even more informed product decisions.
Why did we decide to create a new version of the telemetry?
The existing telemetry mechanism, released in late 2023, has two significant limitations:
- Limited to Deployment Events: Telemetry is only activated during deployment events. While this provides valuable insights into what is being installed, it does not offer any information about the lifespan of the deployed instances. This limitation is particularly critical in Docker environments, where deployment dynamics are far more fluid than package-based deployments, potentially skewing the statistics.
- Lack of Feature Usage Insights: Although we can track which database versions are deployed, the existing telemetry does not capture data on the usage of major functionalities. This gap makes it challenging to understand which specific product areas are most popular, thereby complicating efforts to focus on features of particular interest to users.
What will we collect in future versions of Percona Pillars?
We have identified two critical areas for improvement.
The first improvement involves the upcoming telemetry mechanisms, which will not only activate during deployment but also provide daily updates to Percona containing statistics collected over time. This enhancement will enable us to better understand deployment lifespans, addressing the current mechanism’s first limitation.
The second improvement focuses on expanding the scope of the data collected. In addition to version numbers, the new telemetry will gather information about the major functionalities being utilized. For instance, in MySQL, we will collect the following statistics:
- Storage engine usage (InnoDB vs. MyRocks)
- List of active plugins
- List of active components
- Number and size of databases
- Replication status, including the number and type of replication nodes
- Presence of additional tools such as Percona XtraBackup, Percona Toolkit, Orchestrator, HAProxy, ProxySQL, MySQL Shell, MySQL Router, and Percona Monitoring and Management.
As with the previous iteration of the telemetry project, all collected information is anonymized, ensuring that the source of the statistics cannot be identified once they reach the Percona backend. We do not collect proprietary information, such as host names, IP addresses, database names, table names, or user details, and certainly none of the content within the databases.
Telemetry mechanisms are enabled by default but can be easily disabled. The procedure for disabling telemetry varies depending on the database technology and is detailed in our documentation. Here is how to do that for Percona Server for MySQL 8.0.x: Telemetry and data collection
In contrast to earlier versions of the telemetry project, this iteration allows users to preview the exact content of the statistics before and after sending them to the Percona backend. The telemetry mechanism utilizes two modules: one is a small process running directly on the operating system, and the other is embedded within the database technology. These modules generate files containing the statistics to be sent. The OS module’s role is to package and transmit these statistics securely over HTTPS to the Percona backend. The files are in a human-readable JSON format and can be previewed at the locations specified in our documentation. Here is the link to Percona Server for MySQL 8.0.x Locations of metrics files and telemetry history.
Feedback is welcome
We appreciate the feedback from our users and the community. Please stay in touch with us through the Percona Community Forum.
Hey Guys, firstly great article and it’s always good to base business decisions on solid usage stats. I’m curious on the percentages around docker, specifically how this could be getting skewed by developer stacks vs production environments.
Were a remote dev team and run a full stack local using docker and our own databases locally so we can code wherever and whenever. All teams containers will call in and register the usage stats.
In very large teams you could have hundreds or thousands of developers running local containers vs a small number of production containers – could really skew the stats?
Cheers
Dave
Hi Dave.
Thanks for your comment.
The short-lived docker instances from dev environments are most likely the cause of the Docker domination. We have currently no way of determining for sure if the node is a production or a dev environment.
But we have rolled out the improved version of the telemetry recently and the data is slowly trickling in. We see that when we look at the nodes which have a longer uptime the percentage of package-based vs. docker-based deployments is completely different. It’ll take a moment for the statistics to stabilize, but even from the partial data the breakdown is already looking differently.
I custom-compile all our MySQL (5.7) and PostgreSQL (12) installs and they may not report in to you.
We do have multiple instances of PMM, do they report some telemetry to you?
Hi Mark.
The custom compilations will likely not report anything to us, unless you also substitute them with the build of the telemetry agent. In our releases the telemetry agent is an install dependency, but besides the telemetry the agent does not serve any functional purpose, so I doubt you will want to compile it.
When it comes to PMM, the locally deployed PMM servers regularly send version statistics to our back-end, so unless you disable this functionality or unless your PMM servers are air-gaped we will see anonymized stats coming from your deployments. This document provides more information about PMM telemetry: https://docs.percona.com/percona-monitoring-and-management/how-to/configure.html#telemetry
Regards!