Perfomance Schema Instrumentation

To improve monitoring Percona XtraDB Cluster has implemented an infrastructure to expose Galera instruments (mutexes, cond-variables, files, threads) as a part of PERFOMANCE_SCHEMA.

Although mutexes and condition variables from wsrep were already part of PERFORMANCE_SCHEMA threads weren’t.

Mutexes, condition variables, threads, and files from Galera library also were not part of the PERFORMANCE_SCHEMA.

You can see the complete list of available instruments by running:

mysql> SELECT * FROM performance_schema.setup_instruments WHERE name LIKE '%galera%' OR name LIKE '%wsrep%';

| NAME                                                     | ENABLED | TIMED |
| wait/synch/mutex/sql/LOCK_wsrep_ready                    | NO      | NO    |
| wait/synch/mutex/sql/LOCK_wsrep_sst                      | NO      | NO    |
| wait/synch/mutex/sql/LOCK_wsrep_sst_init                 | NO      | NO    |
| stage/wsrep/wsrep: in rollback thread                    | NO      | NO    |
| stage/wsrep/wsrep: aborter idle                          | NO      | NO    |
| stage/wsrep/wsrep: aborter active                        | NO      | NO    |
73 rows in set (0.00 sec)

Some of the most important are:

  • Two main actions that Galera does are REPLICATION and ROLLBACK. Mutexes, condition variables, and threads related to this are part of PERFORMANCE_SCHEMA.
  • Galera internally uses monitor mechanism to enforce ordering of events. These monitor control events apply and are mainly responsible for the wait between different action. All such monitor mutexes and condition variables are covered as part of this implementation.
  • There are lot of other miscellaneous action related to receiving of package and servicing messages. Mutexes and condition variables needed for them are now visible too. Threads that manage receiving and servicing are also being instrumented.

This feature has exposed all the important mutexes, condition variables that lead to lock/threads/files as part of this process.

Besides exposing file it also tracks write/read bytes like stats for file. These stats are not exposed for Galera files as Galera uses mmap.

Also, there are some threads that are short-lived and created only when needed especially for SST/IST purpose. They are also tracked but come into PERFORMANCE_SCHEMA tables only if/when they are created.

Stage Info from Galera specific function which server updates to track state of running thread is also visible in PERFORMANCE_SCHEMA.

What is not exposed ?

Galera uses customer data-structure in some cases (like STL structures). Mutexes used for protecting these structures which are not part of mainline Galera logic or doesn’t fall in big-picture are not tracked. Same goes with threads that are gcomm library specific.

Galera maintains a process vector inside each monitor for its internal graph creation. This process vector is 65K in size and there are two such vectors per monitor. That is 128K * 3 = 384K condition variables. These are not tracked to avoid hogging PERFORMANCE_SCHEMA limits and sidelining of the main crucial information.

Contact Us

For free technical help, visit the Percona Community Forum.
To report bugs or submit feature requests, open a JIRA ticket.
For paid support and managed or professional services, contact Percona Sales.