Valeriy Kravchuk talked about while troubleshooting MariaDB server performance problems it is important to find out where the time is spent in the mysqld process, on-CPU and off-CPU. The process of investigation should have as small influence as possible on the server we try to troubleshoot.

Performance_schema introduced in MySQL 5.5 (and inherited from MySQL 5.6 by MariaDB) is supposed to provide detailed enough instrumentation for most cases. But it comes with a cost, requires careful sizing of performance counters, and the process of instrumenting the code is not yet complete even for MySQL 8, to say nothing about MariaDB with its 3rd party storage engines, plugins and libraries like Galera.

This is when perf profiler and, on recent Linux kernels (4.9+) eBPF and bpftrace tools come handy.

Specifically, perf profiler and ftrace interface can be easily used while studying MariaDB performance problems. Basic usage steps are presented and several typical real life use cases (including adding dynamic probes to almost any line of MariaDB code) are discussed.

On Linux 4.9+ eBPF is probably the most powerful and least intrusive way to study performance problems. Basic usage of , bcc tools and bpftrace, as well as main bpftrace features and commands are demonstrated.

One of the ways to present and study stack samples collected by perf or bpftrace, Flame Graphs, is also presented with examples coming from my experince as a support engineer.