Debug

Cluster-level logging

Cluster-level logging involves collecting logs from all Percona XtraDB Cluster Pods in the cluster to some persistent storage. This feature gives the logs a lifecycle independent of nodes, Pods and containers in which they were collected. Particularly, it ensures that Pod logs from previous failures are available for later review.

Log collector is turned on by the logcollector.enabled key in the deploy/cr.yaml configuration file (true by default).

The Operator collects logs using Fluent Bit Log Processor, which supports many output plugins and has broad forwarding capabilities. If necessary, Fluent Bit filtering and advanced features can be configured via the logcollector.configuration key in the deploy/cr.yaml configuration file.

Logs are stored for 7 days and then rotated.

Collected logs can be examined using the following command:

kubectl logs cluster1-pxc-1 -c logs

Note

Technically, logs are stored on the same Persistent Volume, which is used with the corresponding Percona XtraDB Cluster Pod. Therefore collected logs can be found in DATADIR (var/lib/mysql/).

Special debug images

For the cases when Pods are failing for some reason or just show abnormal behavior, the Operator can be used with a special debug images. Percona XtraDB Cluster debug image has the following specifics:

  • it avoids restarting on fail,
  • it contains additional tools useful for debugging (sudo, telnet, gdb, etc.),
  • it has debug mode enabled for the logs.

There are debug versions for all Percona XtraDB Cluster images: they have same names as normal images with a special -debug suffix in their version tag: for example, percona-xtradb-cluster:8.0.21-12.1-debug.

Particularly, using such image is useful if the container entry point fails (e.g. mysqld crashes). In such a situation, Pod is continuously restarting. Continuous restarts prevent to get console access to the container, and so a special approach is needed to make fixes.

To use the debug image instead of the normal one, find the needed image name in the list of certified images and set it for the proper key in the deploy/cr.yaml configuration file. For example, set the following value of the pxc.image key to use the Percona XtraDB Cluster debug image:

  • percona/percona-xtradb-cluster:8.0.21-12.1-debug for Percona XtraDB Cluster 8.0,
  • percona/percona-xtradb-cluster:5.7.32-31.47-debug for Percona XtraDB Cluster 5.7.

The Pod should be restarted to get the new image.

Note

When the Pod is continuously restarting, you may have to delete it to apply image changes.

Table Of Contents

Previous topic

Crash Recovery

Next topic

Custom Resource options

Contact Us

For free technical help, visit the Percona Community Forum.
To report bugs or submit feature requests, open a JIRA ticket.
For paid support and managed or professional services, contact Percona Sales.