MySQL NDB Cluster Community - High memory utilization by mysqld only on source/replica nodes

Question

I have two MySQL NDB community clusters 8.0.30, each with 3 VMs
Cluster#1_VM1: mysqld + ndb_mgmd
Cluster#1_VM2: mysqld + ndb
Cluster#1_VM3: mysqld + ndb

Cluster#2_VM1: mysqld + ndb_mgmd
Cluster#2_VM2: mysqld + ndb
Cluster#2_VM3: mysqld + ndb

I have bidirectional replication between Cluster#1_VM2 and Cluster#2_VM2.
VM2/3 have identical parameters.

The choice of such architecture might not be perfect but I think it was not relevant here.

mysqld process consumes a lot of memory on VMs with replication. And replication seems to be the only difference between those VMs. (Though I'm not quite sure how is load distributed among mysqld processes) I would assume that ndbd would be the one to use most of the RAM since it has 8GB of DataMemory. But somehow mysqld is also utilizing a lot of RAM too.

I've checked several other questions here (e.g. High Memory Usage on Replica - Mysql 8 ) Most of them are related to innodb and the queries for e.g. buffer utilization show nothing that could show me where that high memory utilization comes from.

So far I've checked

• buffer utilizations for innoDB (I have NDB, I know)
• engine status for innodb and ndb - no hint of high mem usage, or at least not clearly visible

All I know for now is that it grows over time after restart of the process. But don't know yet how to limit/control the amount of memory consumed by mysqld in this case.

Answer 1

The MySQL Server on the source cluster will consume memory when writing binlog of all changes that happens in NDB.

This is done by a background thread which subscribes to change notifications on all tables which are configured to be replicated. The change notifications are buffered in memory (the event buffer) until the background thread has completed writing them to the binlog. All changes during an NDB epoch (which is 100 ms by default) is grouped together and written in a binlog transaction. This buffering consumes some memory and might grow as traffic increases, but should otherwise not grow overtime. More info about the event buffer and messages written to the cluster log can be found here https://dev.mysql.com/doc/mysql-cluster-excerpt/8.0/en/mysql-cluster-logs-event-buffer.html

The background thread interacts with different subsystems inside the MySQL server in order to write to binlog, keep the data dictionary in sync, writing to the mysql.ndb_binlog_index table in InnoDB and some other tasks. There are also two other background threads.

You can check the memory usage of those threads with performance_schema, using queries like below:

mysql> SELECT PFS_THR.NAME, EVENT_NAME, PFS_MEM.CURRENT_NUMBER_OF_BYTES_USED
  FROM performance_schema.memory_summary_by_thread_by_event_name PFS_MEM
  JOIN performance_schema.threads PFS_THR
  ON PFS_MEM.THREAD_ID = PFS_THR.THREAD_ID
  WHERE PFS_THR.NAME LIKE '%ndb_%';

mysql> SELECT PFS_THR.NAME, EVENT_NAME, PFS_MEM.CURRENT_NUMBER_OF_BYTES_USED
  FROM performance_schema.memory_summary_by_thread_by_event_name PFS_MEM
  JOIN performance_schema.threads PFS_THR
  ON PFS_MEM.THREAD_ID = PFS_THR.THREAD_ID
  WHERE PFS_THR.NAME LIKE '%ndb_%' AND
  EVENT_NAME LIKE '%THD::main_mem_root%';

The thing to look for is if memory usage grows over time.

Since you have replication in both directions (usually called active-active replication) there will be another background thread on the second cluster doing the same things for changes going to the first cluster.

For the actual replication there are the standard MySQL replication threads which pull the binlog from the source and applies it on the replica, see https://dev.mysql.com/doc/refman/8.0/en/replication-threads.html Those threads can be monitored in the similar way using performance_schema.