We have a 3 node Galera Cluster running on Kubernetes, behind 2 HAProxy PODs configured so, all queries are executed on the first POD/node of the cluster if available, and the other 2 nodes, provide HA (HA Proxy backend backup nodes). In the config file, gcache.size is configured to 5 GB, and when a new node is deployed, galera.cache file is 5.1GB so, it seems to get that configuration correctly. However, what we are seeing is galera.cache growing in size up to 80 GBs or more for that first node of the cluster. As far as we know, this file should not increase in size. The problem is also reproduced when scaling the cluster down to one only node. It does not stop growing. The version deployed is 10.3.22 (10.3.22-debian-10-r1 Bitnami Docker image) These are the wsrep provider options specified in my.cnf:
wsrep_on=ON wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so wsrep_provider_options="gcache.size=5G" wsrep_sst_method=mariabackup wsrep_slave_threads=4 wsrep_cluster_address=gcomm:// wsrep_cluster_name=galera wsrep_sst_auth="root:" innodb-flush-log-at-trx-commit=2 # MYISAM REPLICATION SUPPORT # wsrep_replicate_myisam=ON
We've been dealing with this situation for some time now, we can remove the first node(POD) and then the galera.cache is recreated so, we free disk space. The first node syncs through IST with any of the other 2 nodes and the HAProxy points to a backup node meanwhile, and then back to the first node when recovered, with no downtime. However, we want to avoid to do this. We can't figure out why galera.cache file size increases, there is no documentation nor bug we could find talking about any similar issue. Any help will be much appreciated!