Cluster-level logging involves collecting logs from all Percona XtraDB Cluster Pods in the cluster to some persistent storage. This feature gives the logs a lifecycle independent of nodes, Pods and containers in which they were collected. Particularly, it ensures that Pod logs from previous failures are available for later review.
Log collector is turned on by the
logcollector.enabled key in the
deploy/cr.yaml configuration file (
true by default).
The Operator collects logs using Fluent Bit Log Processor,
which supports many output plugins and has broad forwarding capabilities.
If necessary, Fluent Bit filtering and advanced features can be configured via
logcollector.configuration key in the
Logs are stored for 7 days and then rotated.
Collected logs can be examined using the following command:
kubectl logs cluster1-pxc-1 -c logs
Technically, logs are stored on the same Persistent Volume, which is
used with the corresponding Percona XtraDB Cluster Pod. Therefore collected
logs can be found in
Special debug images¶
For the cases when Pods are failing for some reason or just show abnormal behavior, the Operator can be used with a special debug images. Percona XtraDB Cluster debug image has the following specifics:
- it avoids restarting on fail,
- it contains additional tools useful for debugging (sudo, telnet, gdb, etc.),
- it has debug mode enabled for the logs.
There are debug versions for all Percona XtraDB Cluster images: they have same names as normal images with a special
-debug suffix in their version tag: for example,
Particularly, using such image is useful if the container entry point fails
mysqld crashes). In such a situation, Pod is continuously restarting.
Continuous restarts prevent to get console access to the container,
and so a special approach is needed to make fixes.
To use the debug image instead of the normal one, find the needed image name
in the list of certified images and set it
for the proper key in the
deploy/cr.yaml configuration file. For example,
set the following value of the
pxc.image key to use the Percona XtraDB
Cluster debug image:
percona/percona-xtradb-cluster:8.0.21-12.1-debugfor Percona XtraDB Cluster 8.0,
percona/percona-xtradb-cluster:5.7.32-31.47-debugfor Percona XtraDB Cluster 5.7.
The Pod should be restarted to get the new image.
When the Pod is continuously restarting, you may have to delete it to apply image changes.