Customer Advisory on Heartbleed OpenSSL Security Bug from our CEO
- Preamble from the CEO
- Consulting & Support Customers
- Remote DB Administration (RDBA) Customers
- Percona Cloud Tools (PCT) Website
- Percona Cloud Tools Client
- Percona Server
- Other Percona Software
- Partner Software Products
- Third-Party Utilities
- Customer VPNs
- Percona Public Websites
Preamble from the CEO
Dear Percona Customers & Users,
The Heartbleed OpenSSL security bug (formally known as CVE-2014-0160) is the equivalent of an earthquake or tsunami for internet security. It is a really big deal. Percona's Ernie Souhrada and Dave Busby have blogged here and here on how MySQL users should respond. Also notable is a 9-minute video at https://vimeo.com/91425662 produced by Elastica, Inc. It clearly diagrams how Heartbleed works.
But beyond this, our customers are asking what specific risks they face from Heartbleed due to their patronage of Percona. Given the magnitude of Heartbleed, I've decided we should respond with a public statement to explain Heartbleed's impact on Percona products, services, and systems.
The good news is that Percona's exposure to Heartbleed thankfully appears to have been much less than what many other organizations have suffered. The bad news is that in some systems and scenarios, there remains the theoretical chance that data could have been compromised. We’ve had no evidence to suggest this has happened. But as Heartbleed exploits leave no trace, Percona is taking a very conservative approach in implementing changes, to be as safe as possible. Below is the rundown on all Percona systems, products, and services.
This is a dynamic, fast moving situation unlike anything the online world or the MySQL community has faced in recent memory. Percona has extensive tests underway to better understand Heartbleed, since reports vary as to its full implications. As we learn more from our tests and from vendors, we will add those findings to this document.
Customers needing further clarification or help with any of these issues should open a ticket at Percona's customer portal https://customers.percona.com, or email me directly at CEO (at) Percona (dot) com
Founder & CEO, Percona
April 10, 2014
Consulting & Support Customers
https://customers.percona.com/ is the site via which Percona communicates with our consulting and support customers. A vulnerable version of OpenSSL was never installed on this system. Email exchanges with our engineers were not at risk while on Percona servers.
When our engineers login to customer systems, they use Secure Shell (SSH). Although based on OpenSSL, SSH does not use the Transport Layer Security (TLS) portion of OpenSSL and in particular the TLS Heartbeat extension that Heartbleed attacks. This means that no traffic encrypted by SSH was at risk when Percona engineers worked on any customer system.
Remote DB Administration (RDBA) Customers
https://rdba.percona.com/ is the site via which Percona RDBA customers interact with Percona. An OpenSSL version vulnerable to Heartbleed was in use here since 2012. Within a day of Heartbleed's announcement we upgraded our SSL version, plus installed new host and customer SSL certificates. As a precaution, we rotated key passwords. We have also analyzed and issued advisories for SSL vulnerabilities on our customer systems.
Overall, we deem the likelihood of compromise of RDBA data as limited. This is because security redundancy built into our RDBA operation such as: extensive use of two-factor authentication; extreme password hashing; hiding from the internet of many key components; secure shell (SSH) usage not vulnerable to Heartbleed; plus, various Percona RDBA operational practices that limit risk. However, the theoretical possibility remains that, due to Heartbleed, some traffic could have been sniffed and decrypted. It is impossible to know whether or not this actually happened although we've seen no secondary evidence to suggest that it has.
Percona Cloud Tools (PCT) Website
https://cloud.percona.com is a public beta of our new cloud monitoring tool for MySQL DBAs. This tool is accessible using three authentication methods: Google account; customers.percona.com accounts; and, cloud.percona.com (local) accounts. An OpenSSL version vulnerable to Heartbleed was in use by us since mid-2013. As with the RDBA customer portal, we quickly upgraded our SSL version and undertook similar fixes.
We have no evidence to suggest that any PCT login credentials were ever compromised although it remains a theoretical possibility that local accounts could have been leaked. As a precaution, we are resetting the password for all local accounts. No authentication details for any other access methods are stored on PCT servers; therefore, a password reset is not necessary for them.
We will be requiring password resets on https://customers.percona.com/ from those accounts who are also users of Percona PCT and RDBA services because these were theoretically vulnerable to Heartbleed.
Percona Cloud Tools Client
The Percona Cloud Tools client (pt-agent) is installed locally on a monitored server and communicates with our PCT server. The client is not directly affected by Heartbleed. However, if Heartbleed compromised to our PCT server, then theoretically traffic between the client and PCT could have been sniffed and decrypted. As a precaution, we will advise certain users (i.e., those that have used their API key to communicate with the PCT server) to change their API key. We will send you instructions on how to do this.
Percona Server, our high performance variant of MySQL, normally relies on whichever OpenSSL library you have installed on your servers. This is called ‘dynamic linking’.
However, please understand that, by default, OpenSSL is disabled in Percona Server, at least in our builds for most operating systems. (Given the urgency of this matter, we cannot verify this is the case for 100% of our builds.) So, if you have not enabled OpenSSL on your installation, you are probably not vulnerable to Heartbleed.
You may also be spared from an external Heartbleed attack if your Percona Server installation does not directly connect to the Internet but is behind a firewall or accessible only via a VPN. Note, however, that attacks may still come from your internal network in this case.
Thus, if your operating system uses a vulnerable OpenSSL library AND you have OpenSSL enabled in Percona Server AND your site is accessible directly from the Internet (though note as above that internal attacks are also possible), then you are at risk. You must update your operating system’s OpenSSL to the latest release, meaning one that contains the Heartbleed fix, and then shutdown and restart Percona Server.
There is always a chance that some packaging incompatibility will arise when making such a sensitive upgrade. In general, Percona recommends caution and testing when upgrading. You may want to contact Percona Consulting or Support in advance for assistance.
Please note: if you are one of the very few users running a statically compiled version of Percona Server, you are affected by Heartbleed and must make two fixes as described below (see ‘Three fixes/action items to make’).
To check whether you have a statically compiled version there are two methods:
- If the text ‘static’ appears in your original installation package name, it means you have a static binary
- You can also determine if you have a static library by inspecting the mysqld binary which is located in your basedir
If you get the output as shown below, your mysqld uses dynamic linking (and you are not affected by the Heartbleed issue):
# ldd /usr/sbin/mysqld|grep ssl libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f45b49dc000)
However, if the ldd command output is empty, then this means your mysqld uses static linking (and your Percona Server installation is potentially affected by Heartbleed), which means you will have to take the three actions specified below.
Three fixes/action items to make: First, update your system’s OpenSSL version to the latest release. Second, switch to Percona Server with dynamically linked OpenSSL packages. Dynamically linked versions of Percona Server may be downloaded here. Third, shutdown and restart Percona Server.
Please note that Percona no longer builds versions of Percona Server with statically linked OpenSSL. All builds are now dynamically linked to OpenSSL.
Other Percona Software
Percona XtraDB Cluster, our high availability solution based on Codership's Galera technology, is the same as Percona Server with regard to Heartbleed.
Percona XtraBackup, the only free and open source backup utility for MySQL, does not use OpenSSL and is therefore not vulnerable to Heartbleed. Data encryption is not done via OpenSSL but via another encryption library. This means that no encrypted backups you have created are impacted in any way by Heartbleed.
Percona Toolkit, our Swiss Army knife for DBAs, uses OpenSSL but only for functionality that is not vulnerable to Heartbleed.
Percona Monitoring Plugins, used with Nagios and Cacti monitoring utilities, are not in themselves vulnerable to Heartbleed. However, they may run on servers which are vulnerable to Heartbleed exploit. Upgrading OpenSSL on the server, as described under the Percona Server section, will fix any vulnerability with these plugins.
Percona Data Recovery Tool for InnoDB, which recovers data from crashed MySQL databases, is not affected by Heartbleed.
Percona Playback, a load testing tool, is not affected by Heartbleed.
Partner Software Products
As part of our Support offerings, Percona redistributes versions of MONyog Ultimate, which monitors MySQL installations, and Severalnines ClusterControl Enterprise Edition, which is used to install and maintain MySQL and Percona Cluster servers.
The MONyog product ships with the OpenSSL library. However, it uses either direct connections or SSH encrypted connections when communicating with monitored datasources, meaning the vulnerable TLS portion of OpenSSL is never utilized. WebYog, the product vendor, plans to be extremely cautious and release a new version of MONyog to include an updated, Heartbleed-safe version of OpenSSL. This is expected within the next few days. Once available, Percona will notify all affected customers.
The Severalnines product relies on whatever OpenSSL library you have installed on your servers, so the same principles on vulnerability and repair apply as for Percona Server.
Percona makes use of various third party web sites, some of which have announced concerns relating to Heartbleed vulnerability. We are monitoring each of these sites and will respond appropriately as information becomes available.
Some customers provide VPNs for Percona staff to use when accessing their systems, in lieu of SSH access. These VPNs are controlled by the respective customers, not Percona. There are many variations of VPNs. Percona recommends that each customer check their VPN to determine if it is affected by Heartbleed and advise us if protocols for accessing their VPN have been changed.
Percona Public Websites
http://www.percona.com is our main website. A vulnerable version of OpenSSL was never installed on this system.
http://www.percona.com/blog is our blog on MySQL topics written by Percona engineers. A vulnerable version of OpenSSL was never installed on this system. No user accounts are believed to have ever been at risk.
http://www.percona.com/forums/ is Percona's interactive forum for user Q&A on MySQL topics. A vulnerable version of OpenSSL was never installed on this system. No user accounts are believed to have ever been at risk.
https://tools.percona.com is Configuration Wizard & Query Advisor. A vulnerable version of OpenSSL was never installed on this system. No user accounts are believed to have ever been at risk.