1

We have a 3 node Galera Cluster running on Kubernetes, behind 2 HAProxy PODs configured so, all queries are executed on the first POD/node of the cluster if available, and the other 2 nodes, provide HA (HA Proxy backend backup nodes). In the config file, gcache.size is configured to 5 GB, and when a new node is deployed, galera.cache file is 5.1GB so, it seems to get that configuration correctly. However, what we are seeing is galera.cache growing in size up to 80 GBs or more for that first node of the cluster. As far as we know, this file should not increase in size. The problem is also reproduced when scaling the cluster down to one only node. It does not stop growing. The version deployed is 10.3.22 (10.3.22-debian-10-r1 Bitnami Docker image) These are the wsrep provider options specified in my.cnf:

wsrep_on=ON
wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
wsrep_provider_options="gcache.size=5G"
wsrep_sst_method=mariabackup
wsrep_slave_threads=4
wsrep_cluster_address=gcomm://
wsrep_cluster_name=galera
wsrep_sst_auth="root:"
innodb-flush-log-at-trx-commit=2
# MYISAM REPLICATION SUPPORT #
wsrep_replicate_myisam=ON

We've been dealing with this situation for some time now, we can remove the first node(POD) and then the galera.cache is recreated so, we free disk space. The first node syncs through IST with any of the other 2 nodes and the HAProxy points to a backup node meanwhile, and then back to the first node when recovered, with no downtime. However, we want to avoid to do this. We can't figure out why galera.cache file size increases, there is no documentation nor bug we could find talking about any similar issue. Any help will be much appreciated!

Improve this question
7
  • 2
    You should probably report this in the mariadb bug tracker. That said, you are somewhat behind with your updates, so you might first want to update to the most recent 10.3 version (currently 10.3.32) to see if the problem goes away by itself.
    – dbdemon
    Commented Jan 5, 2022 at 17:22
  • Hi @dbdemon, I registered the issue: jira.mariadb.org/browse/MDEV-27457 After trying with a 10.4 version and still getting the issue, I am wondering if it may be related to the way we configure the HAProxy to point to a single node. But it still does not make much sense, the galera.cache file should not grow over the specified size in my.cnf, and that's Galera MariaDB circular buffer file not working as expected.
    – Alex Núñez
    Commented Jan 12, 2022 at 10:34
  • Do you need to replicate MyISAM tables? That is the only unusual setting I see in the configuration you posted. If it's not strictly needed, then try turning it off. MyISAM support is only experimental anyway. (You don't need wsrep_replicate_myisam=ON for system tables in the mysql, information_schema databases as this is replicated anyway).
    – dbdemon
    Commented Jan 12, 2022 at 10:46
  • I removed that setting and redeployed, but nothing changed, galera.cache is still growing.
    – Alex Núñez
    Commented Jan 12, 2022 at 17:26
  • Did anybody ever see galera.cache growing over the specified size in my.cnf? Any suggestion about where else to post this issue? Also tried in the Bitnami Charts github but, no luck so far. Where else could I get support from? I'm really stuck with this issue!
    – Alex Núñez
    Commented Jan 12, 2022 at 17:31

1 Answer 1

Reset to default
1

Solved after upgrading Galera library libgalera_smm.so to 26.4.9 version. The galera.cache does not grow over the specified 128MB gcache.size, using MariaDB 10.4.12 and that Galera version.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.