A few thousand gcache.page.* files in a Percona XtraDB Cluster (PXC) data directory is not something you see every day. We came across a case where these files had been accumulating over time and slowly consuming disk space. So, let’s dig into what happened.
At first glance, it looked like GCache had simply stopped cleaning itself up. The investigation started by answering two simple questions: when did the files start appearing, and what changed in the cluster at that time?
The oldest files showed the issue started on July 9.
|
1 2 3 4 5 6 7 8 |
[hostx] percona@hostx: ~ $ ls -lh /var/lib/mysql/mysql-data/gcache.page.* -rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000000 -rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000001 -rw-r----- 1 mysql mysql 128M Jul 9 21:42 /var/lib/mysql/mysql-data/gcache.page.000002 ... -rw-r----- 1 mysql mysql 128M Jul 26 07:45 /var/lib/mysql/mysql-data/gcache.page.005976 -rw-r----- 1 mysql mysql 128M Jul 26 07:50 /var/lib/mysql/mysql-data/gcache.page.005977 -rw-r----- 1 mysql mysql 128M Jul 26 07:56 /var/lib/mysql/mysql-data/gcache.page.005978 |
The newest files showed they stopped being created on July 26, which immediately provided a timeline to investigate.
The creation of GCache page files wasn’t random. It started at a specific point in time and stopped after the next MySQL restart.
Normally, you don’t see thousands of GCache page files unless Galera cannot reclaim old pages or an exceptionally large writeset forces additional page allocation. In this environment, the GCache ring file was around 60 GB, making the large writeset theory very unlikely – actually Impossible! Because there’s a hard limit of the largest transaction size at 2GB.
That pushed the investigation toward the error log, where Galera was found logging:
|
1 2 |
2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494 2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytes |
This was the first strong clue. When gcache.freeze_purge_at_seqno is active, Galera stops reclaiming old GCache pages. As replication continues, new page files are allocated while existing ones remain on disk.
Looking a few seconds later in the error log revealed a cluster partition. The timing is difficult to ignore: Galera froze GCache purging, created the first page file, and then recorded the membership change.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
2026-07-09T21:41:04.617132Z 91699 [Note] [MY-000000] [Galera] Freezing gcache purge at 16198595494 2026-07-09T21:41:04.622051Z 0 [Note] [MY-000000] [Galera] Created page /var/lib/mysql/mysql-data/gcache.page.000000 of size 134217728 bytes 2026-07-09T21:41:14.537019Z 53033 [Note] [MY-010559] [Repl] Multi-threaded replica statistics for channel '': seconds elapsed = 122; events assigned = 110401704; worker queues filled over overrun level = 0; waited due a Worker queue full = 0; waited due the total size = 0; waited at clock conflicts = 7659954362300 waited (count) when Workers occupied = 47189196 waited when Workers occupied = 22964040282100 2026-07-09T21:42:02.139966Z 91770 [Note] [MY-010914] [Server] Aborted connection 91770 to db: 'unconnected' user: 'percona' host: '10.1.163.3' (Got an error reading communication packets). 2026-07-09T21:42:17.061513Z 0 [Note] [MY-000000] [Galera] forgetting c3acab8a-8a74 (ssl://10.1.21.5:4567) 2026-07-09T21:42:17.061596Z 0 [Note] [MY-000000] [Galera] Node 7e3c8f1e-ad75 state primary 2026-07-09T21:42:17.061615Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view_id(PRIM,7e3c8f1e-ad75,11) memb { 7e3c8f1e-ad75,1 } joined { } left { } partitioned { c3acab8a-8a74,1 } ) |
Although the logs do not explicitly state why purge was frozen, the sequence of events strongly suggests that Galera retained the writesets so the partitioned node could potentially perform an IST when it rejoined.
Further investigation into the codebase and documentation hinted that it is practically impossible that Galera can actually invoke the gcache pages purge pause and only practical way to do it is using:
|
1 |
SET GLOBAL wsrep_provider_options='gcache.freeze_purge_at_seqno=XYZ'; |
Reference: https://github.com/percona/galera/pull/132
Related reading: No SST node rejoins in PXC
Rather than stopping with a theory, Peter Sylvester (SoS) reproduced the behavior in the lab by manually setting gcache.freeze_purge_at_seqno and generating workload with Sysbench. The result matched what we observed in the production.
Additionally, even though gcache.keep_pages_count=3, Galera continued creating additional page files because purging was frozen.
|
1 2 3 4 5 6 7 8 9 10 |
mysql> show global status like 'wsrep_last_committed'; +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | wsrep_last_committed | 94559 | +----------------------+-------+ 1 row in set (0.00 sec) mysql> SET GLOBAL wsrep_provider_options="gcache.freeze_purge_at_seqno=94559"; Query OK, 0 rows affected (0.00 sec) |
Sysbench was then run on the cluster’s source host to generate logs…
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@CENTOS9-1 ~]# sysbench oltp_read_write --db-driver=mysql --mysql-db=sysbench --mysql-user=sysbench --mysql-password=password --mysql-port=3306 --table_size=1000 --tables=4 --threads=4 --rand-type=uniform --range_size=100 --time=0 --rate=0 --report_interval=5 run WARNING: Both event and time limits are disabled, running an endless test sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3) Running the test with following options: Number of threads: 4 Report intermediate results every 5 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 5s ] thds: 4 tps: 311.80 qps: 6246.62 (r/w/o: 4373.81/1248.40/624.40) lat (ms,95%): 16.71 err/s: 0.00 reconn/s: 0.00 |
Following page files are present in datadir
|
1 2 3 4 5 6 7 8 9 |
[root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache* -rw-r----- 1 mysql mysql 11M Aug 6 14:14 /var/lib/mysql/galera.cache -rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000 -rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004 -rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005 -rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000006 |
After the load completed, the mysqld was restarted to observe if the page files were then cleared! But they were not.
|
1 2 3 4 5 6 7 8 9 10 11 |
[root@CENTOS9-3 ~]# systemctl stop mysql [root@CENTOS9-3 ~]# systemctl start mysql [root@CENTOS9-3 ~]# ls -lh /var/lib/mysql/*cache* -rw-r----- 1 mysql mysql 11M Aug 6 14:17 /var/lib/mysql/galera.cache -rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000000 -rw-r----- 1 mysql mysql 10M Aug 6 14:14 /var/lib/mysql/gcache.page.000001 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000002 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000003 -rw-r----- 1 mysql mysql 10M Aug 6 14:15 /var/lib/mysql/gcache.page.000004 -rw-r----- 1 mysql mysql 10M Aug 6 14:16 /var/lib/mysql/gcache.page.000005 -rw-r----- 1 mysql mysql 10M Aug 6 14:17 /var/lib/mysql/gcache.page.000006 |
Later the MySQL was restarted, expecting the startup to reclaim the unused pages. It didn’t. Every page file remained on disk after a restart. Even after the explicit configuration of gcache.freeze_purge_at_seqno=-1.
This suggests that startup recovery does not automatically remove these accumulated page files simply because gcache.freeze_purge_at_seqno has been cleared. At least in testing, once purge has been frozen and page files accumulate, restarting MySQL alone is not enough to reclaim the space.
We have a new bug in place for getting this behaviour sorted: PXC-5323
To answer that, MySQL was stopped, the local GCache files were removed and the node was started again.
Galera recreated the required cache structures automatically. More importantly, the node successfully completed an IST; and in our tests, deleting the local page files did not force an SST.
Before removing the files, make sure the node is stopped and the cluster has another healthy node that can provide the required writesets. Always validate this behavior in your own environment before using it operationally. Our testing showed that an IST was sufficient.
Based on both the production logs and our lab testing, the accumulation of gcache.page.* files was caused by GCache purging being frozen. Restarting MySQL did not reclaim the accumulated files in our testing. The behavior where a PXC node continuously creates new GCache page files is already fixed under: PXC-4495.
The practical workaround is to stop MySQL, remove the local galera.cache and gcache.page.* files, and start the node again. In our testing, the node rejoined the cluster using IST without requiring a full SST. As always, make sure another healthy node has the required writesets before performing this cleanup.
Resources
RELATED POSTS
