Percona Toolkit 3.6.0 has been released on June 12, 2024. The most important updates in this version are:
- The possibility to resume pt-online-schema-change if it is interrupted.
- eu-stack support in pt-pmp that significantly improves this tool’s performance and decreases the load it causes on production servers.
- New tool pt-eustack-resolver
- Packages for Ubuntu 24.04 (Noble Numbat)
Other changes include:
- Improvements in pt-galera-log-explainer, pt-k8s-debug-collector, pt-show-grants, pt-summary.
- Bug fixes.
In this blog, I will outline the most significant changes. A full list of improvements and bug fixes can be found in the release notes.
pt-online-schema-change
The tool has the following new options:
- –where that instructs pt-online-schema-change to copy only rows matching specified WHERE clause.
- –history to write progress into the history table.
- –history-table in case you want to override the default name percona.pt_osc_history.
- –resume allows you to resume previously failed jobs stored in the history table.
- –binary-index modifies the behavior of –history such that the history table’s upper and lower boundary columns are created with the BLOB data type. This option works similarly to option –binary-index in the pt-table-checksum.
In the next few weeks, we will release a blog post about resume support in pt-online-schema-change.
In this version, we also fixed PT-1911 – pt-online-schema-change can’t use options port. Now, the option –port is supported the same way as in other tools.
pt-show-grants
We started working on better compatibility with MySQL 8.0 and newer versions. One of the changes is to use option –print_identified_with_as_hex when connecting to a MySQL server that supports this option. This allows the display of hashes for passwords that use caching_sha2_password and sha256_password authentication plugins.
The tool also does not exclude creating an additional CREATE USER statement when dumping grants of the MariaDB server, so such grants could be imported into MySQL 8.0, which does not support user passwords in the GRANT statements. The fix was contributed by Marco Tusa.
pt-pmp
pt-pmp is a profiler tool that creates and summarizes full stack traces of processes on Linux. pt-pmp was inspired by http://poormansprofiler.org and has helped Percona’s Support team resolve many performance issues.
In version 3.6.0, we added an improved pt-pmp port with eu-stack support, contributed by Alexey Stroganov. This option is available via a new option: –dumper (-d). It could be set either to gdb (default), eu (eu-stack), or pteu (pt-eustack-resolver).
eu-stack, unlike gdb, does not cause stalls in the running process it is applied to, so using pt-pmp with eu-stack is much safer in production. Additionally, this increases pt-pmp performance around seven times for dumper eu and 65 times for dumper pteu with no information loss. You need to install elfutils to explore these options.
We plan to publish a blog on eu-stack support in pt-pmp and a new tool, pt-eustack-resolver, soon.
pt-k8s-debug-collector
Users of Percona Operator for PostgreSQL do not need to have psql client installed to collect summaries anymore. After the fix for PT-2233, pt-k8s-debug-collector now uses the client, which is available in the cluster node. Before, it used psql from the client machine.
After the fix for PT-2299, pt-k8s-debug-collector stores details about SSL certificate metadata for all operators. This helps troubleshoot SSL connection issues, such as if one of the certificates is out of date.
pt-stalk
A few years ago, pt-stalk could profile mysqld processes with OProfile’s opcontrol. This tool was then deprecated and removed from the most currently supported operating systems, which meant that pt-stalk lost a helpful feature. In version 3.6.0, we accepted a pull request from Fernando Ipar that adds similar support but uses a fully supported performance profiler operf.
pt-summary
pt-summary already collects dmidecode information and prints memory and system information out of it. A contribution by Alexey Kopytov adds CPU cache configuration information, based on the dmidecode data, to the tool’s output.
pt-mongodb-summary
pt-mongodb-summary now supports MongoDB 6.0 and newer.
pt-galera-log-explainer
One of the most significant additions to the pt-galera-log-explainer is custom regular expressions. You can now specify them via option –custom-regexes. You also have two options for processing matches; printing a matched string or displaying a custom message.
For example, the following command on the test file node1.20230315.log will print matched strings:
|
1 2 3 4 5 6 7 8 9 10 |
$ ./bin/pt-galera-log-explainer list --events --custom-regexes='Page cleaner took [0-9]*ms to flush 2000=' src/go/pt-galera-log-explainer/tests/logs/merge_rotated_daily/node1.20230315.log identifier node1 current path .../tests/logs/merge_rotated_daily/node1.20230315.log last known ip last known name mysql version 2023-03-15T10:12:43.720576+02:00 Page cleaner took 5165ms to flush 2000 2023-03-15T10:12:51.514442+02:00 Page cleaner took 5793ms to flush 2000 2023-03-15T10:13:22.185343+02:00 Page cleaner took 4063ms to flush 2000 |
This snippet prints any custom text you want:
|
1 2 3 4 5 6 7 8 9 |
$ ./bin/pt-galera-log-explainer list --events --custom-regexes='Page cleaner took [0-9]*ms to flush 2000=DOWNTIME!!!' src/go/pt-galera-log-explainer/tests/logs/merge_rotated_daily/node1.20230315.log identifier node1 current path .../tests/logs/merge_rotated_daily/node1.20230315.log last known ip last known name mysql version 2023-03-15T10:12:43.720576+02:00 (repeated x1)DOWNTIME!!! 2023-03-15T10:13:22.185343+02:00 DOWNTIME!!! |
The option –skip-merge tells the tool not to merge log files together. This is useful when analyzing logs from different nodes with the same name.
For the PXC operator nodes, instead, the pt-galera-log-explainer introduced a merging mechanism that allows merging log files collected by the pt-k8s-debug-collector multiple times in the same cluster but taken when pvc is destroyed. These logs were originally treated as logs from different nodes, but now they are merged using podname+deployment+namespace as an identificator.
pt-galera-log-explainer now automatically detects Percona Operator for PXC, so the option –pxc-operator is not required.
A new command, whois, prints information about nodes using any type of information, such as nodename, ip, or uuid. By default, the tool autodetects the correct type. If it fails to detect correctly, specify type with option –type.
Here is the sample output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
$ ./bin/pt-galera-log-explainer whois 10.16.27.149 src/go/pt-galera-log-explainer/tests/logs/operator_ambiguous_ips/* INF Detected logs coming from Percona XtraDB Cluster Operator, enabling --pxc-operator ip: └── 10.16.27.149 ├── nodename: │ └── cluster1-1 (2023-05-21 00:55:34.59855 +0000 UTC) │ ├── ip: │ │ ├── 10.16.27.98 (0001-01-01 00:00:00 +0000 UTC) │ │ ├── 10.16.27.195 (2023-05-10 09:06:21.290854 +0000 UTC) │ │ ├── 10.16.27.93 (2023-05-10 10:49:15.965568 +0000 UTC) │ │ ├── 10.16.27.67 (2023-05-10 11:43:19.838842 +0000 UTC) │ │ └── 10.16.27.203 (2023-05-21 01:21:12.237121 +0000 UTC) │ │ │ └── uuid: │ ├── 09afeef6-a69d (2023-05-10 09:06:21.310966 +0000 UTC) │ ├── 106cd5a8-8e1c (2023-05-10 09:42:20.096709 +0000 UTC) │ ├── 6a146d09-8747 (2023-05-10 10:49:15.98352 +0000 UTC) │ ├── f7946b60-bf31 (2023-05-10 11:43:19.859843 +0000 UTC) │ ├── 2cc76c37-becc (2023-05-10 11:51:58.610014 +0000 UTC) │ ├── 2cc76c37-becd (2023-05-12 19:13:56.828375 +0000 UTC) │ ├── 2cc76c37-bece (2023-05-12 19:29:34.102395 +0000 UTC) │ ├── 2cc76c37-becf (2023-05-16 02:56:58.102204 +0000 UTC) │ ├── 96435e8a-bab8 (2023-05-16 02:58:05.880842 +0000 UTC) │ ├── d0e11ff4-be29 (2023-05-16 02:59:44.222891 +0000 UTC) │ ├── 215101e1-b61d (2023-05-16 03:01:59.175607 +0000 UTC) │ ├── 87e7065b-bf25 (2023-05-16 03:04:51.285176 +0000 UTC) │ ├── 87e7065b-bf26 (2023-05-16 07:52:26.432272 +0000 UTC) │ ├── 3c016ef3-af4c (2023-05-18 08:51:06.97503 +0000 UTC) │ ├── 5fd057e4-bab5 (2023-05-18 11:15:16.987418 +0000 UTC) │ ├── e2239bca-93a3 (2023-05-18 13:13:27.582217 +0000 UTC) │ ├── 250ac3d5-8380 (2023-05-18 13:15:19.825195 +0000 UTC) │ ├── 70a8263e-989f (2023-05-18 13:17:26.686853 +0000 UTC) │ ├── 7a3b782e-96c0 (2023-05-18 14:00:39.734694 +0000 UTC) │ ├── e123e2f3-ace4 (2023-05-21 01:21:12.258721 +0000 UTC) │ ├── e123e2f3-ace5 (2023-05-24 09:08:00.784586 +0000 UTC) │ ├── c943db75-9035 (2023-05-25 04:36:13.482766 +0000 UTC) │ ├── c943db75-9036 (2023-05-28 08:23:24.701198 +0000 UTC) │ ├── 8e6f32b6-bf89 (2023-05-28 08:55:54.185342 +0000 UTC) │ │ └── nodename: │ │ └── unspecified (2023-05-29 07:16:49.686673 +0000 UTC) │ │ │ └── 66e2b7bf-8000 (2023-05-29 07:20:31.719983 +0000 UTC) │ │ └── uuid: └── 4ca2c784-a878 (2023-05-21 00:55:34.618928 +0000 UTC) |
Community contributions:
This release includes contributions from Community and Percona Engineers who do not actively work on the project. We want to thank:
- Alexey Stroganov for eu-stack support in pt-pmp.
- Satoshi MITANI for fixing the wide-character error in pt-online-schema-change.
- Fernando Ipar for operf support in pt-stalk.
- Alexey Kopytov for CPU cache processing in pt-summary.
- Jobin Augustine (Percona Support) for removing the requirement to have a psql client on the node that runs pt-k8s-debug-collector.
- Marco Tusa (Percona Technical Leads) for adding CREATE USER command in pt-show-grants when dumping grants for MariaDB servers.
- Yoann La Cancellera (Percona Support) for all improvements and bug fixes to pt-galera-log-explainer.
About the Author
Sveta SmirnovaSveta joined Percona in 2015. Her main professional interests are problem solving, working with tricky issues, bugs, finding patterns that can solve typical issues quicker and teaching others how to deal with MySQL issues, bugs and gotchas effectively. Before joining Percona Sveta worked as a Support Engineer in the MySQL Bugs Analysis Support Group in MySQL AB-Sun-Oracle. She is the author of the books "MySQL Troubleshooting" and "MySQL Cookbook, 4th Edition".
