1

I have 5 mysql servers in master-master group replication setup. (MySQL version 8.0.26) 2 nodes are having 1vCore and 2GB RAM and 20GB HDD configurations, and rest 3 nodes are having about 32GB RAM and 6vCores.

There are 5 InnoDB databases having 45 tables and mysqldump sql file is around 150MB, for all db combined.

I have following configuration for those 2 nodes of MySQL:

max_connections = 900
wait_timeout = 60
interactive_timeout = 60
log_timestamps = SYSTEM

innodb_log_buffer_size = 256M
innodb_log_file_size = 1G
innodb_write_io_threads = 16
innodb_flush_log_at_trx_commit = 0
innodb_doublewrite = 0

# Disabling other storage engines, use only InnoDB
disabled_storage_engines = "MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"

My springboot java application is not optimised and somehow creates many connections to database for quick queries. It throws exception when max_connections are 250 or lower and works fine when it's between 600 and 900.

The problem is that these 2 nodes, always goes out of memory and the VM stops responding, and then automatically reboots. MySQL works fine after reboot and slowly it fills up the memory and same things repeats.

mysqltuner perl script suggest memory allocation based on multiplication of the max_connections and other values. But that is not practical as it will happen when server is running at full capacity.

I have tried many configurations but still the same problem is there. Here's the link of configuration that I have tried - Dedicated mysql server slow response

EDIT 1

  • Also, I would like to mention that, they are acting as backup servers in group replication and application do not connect to the servers until required. The memory overflow is caused only by group replication syncing.
  • The server runs normally till 5 hours then reboot itself
  • After changing the flags to recommended ones by Rick James below is the mysqltuner report

MySQLTuner 1.8.1 - Major Hayden [email protected] Bug reports, feature requests, and downloads at http://mysqltuner.pl/ Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials passed on the command line
[OK] Currently running supported MySQL version 8.0.26-0ubuntu0.20.04.2
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(22K)
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[OK] Log file /var/log/mysql/error.log is readable.
[!!] /var/log/mysql/error.log contains 47 warning(s).
[!!] /var/log/mysql/error.log contains 6 error(s).
[--] 6 start(s) detected in /var/log/mysql/error.log
[--] 1) 2021-09-16T11:38:56.901372+05:30 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 2) 2021-09-16T11:38:56.901295+05:30 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 3) 2021-09-16T10:38:33.120014+05:30 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 4) 2021-09-16T10:38:33.119526+05:30 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 5) 2021-09-16T09:44:28.059734+05:30 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26-0ubuntu0.20.04.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 6) 2021-09-16T09:44:28.059267+05:30 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 1 shutdown(s) detected in /var/log/mysql/error.log
[--] 1) 2021-09-16T11:38:40.998090+05:30 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.26-0ubuntu0.20.04.2)  (Ubuntu).
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in InnoDB tables: 3.7G (Tables: 60)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8
 
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 13m 22s (34 q [0.042 qps], 33 conn, TX: 68K, RX: 619M)
[--] Reads / Writes: 0% / 100%
[--] Binary logging is enabled (GTID MODE: ON)
[--] Physical Memory     : 1.9G
[--] Max MySQL memory    : 7.0G
[--] Other process memory: 0B
[--] Total buffers: 680.0M global + 65.1M per thread (100 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 810.3M (40.77% of installed RAM)
[!!] Maximum possible memory usage: 7.0G (362.00% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (0/34)
[OK] Highest usage of available connections: 2% (2/100)
[!!] Aborted connections: 9.09%  (3/33)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[--] Query cache have been removed in MySQL 8
[OK] No Sort requiring temporary tables
[OK] No joins without indexes
[OK] Temporary tables created on disk: 0% (0 on disk / 5 total)
[OK] Thread cache hit rate: 96% (1 created / 33 connections)
[OK] Table cache hit rate: 99% (413K hits / 414K requests)
[OK] table_definition_cache(2000) is upper than number of tables(391)
[OK] Open file limit used: 0% (9/10K)
[OK] Table locks acquired immediately: 100% (5 immediate / 5 locks)
[OK] Binlog cache memory access: 98.63% (199686 Memory / 202457 Total)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] MyISAM Metrics are disabled on last MySQL versions.
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 640.0M/3.7G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (320 %): 1.0G * 2/640.0M should be equal to 25%
[OK] InnoDB buffer pool instances: 1
[--] Number of InnoDB Buffer Pool Chunk : 5 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.95% (5081504 hits/ 5083928 total)
[!!] InnoDB Write Log efficiency: 72.9% (1546641 hits/ 2121634 total)
[OK] InnoDB log waits: 0.00% (0 waits / 574993 writes)
 
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine not available.
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mysql/error.log file
    Control error line(s) into /var/log/mysql/error.log file
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Reduce your overall MySQL memory footprint for system stability
    Dedicate this server to your database for highest performance.
    Reduce or eliminate unclosed connections and network issues
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    innodb_buffer_pool_size (>= 3.7G) if possible.
    innodb_log_file_size should be (=80M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
6
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
    – Community Bot
    Sep 15, 2021 at 13:26
  • Please post the complete TEXT mysqltuner report and how many hours will your 2 nodes run before they automatically reboot, usually? Sep 15, 2021 at 21:23
  • @WilsonHauck it takes about 5 hours to do an auto reboot of those 2 nodes. Also, I have attached the report in question after the changes suggested by Rick James. Also, I would like to mention that, they are acting as backup servers in group replication and application do not connect to the servers until required. The memory overflow is caused only by group replication syncing. Sep 16, 2021 at 6:35
  • Additional information request, please. At 4 hours of uptime-from 1 nodes Any SSD or NVME devices on MySQL Host? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; \F) complete www.MySQLTuner.pl G) SHOW ENGINE INNODB STATUS; AND Optional very helpful information, if available includes - htop OR top for most active apps, ulimit -a for a Linux/Unix list of limits, for server workload tuning analysis to provide suggestions. Sep 16, 2021 at 17:51
  • 1
    Hello @WilsonHauck, I have dropped the idea of having low end resources for MySQL backup server and now have just 1 single 32GB RAM standalone server, and it just takes 1.4GB RAM at max load. Some answers to your questions are, all of them are having SSDs, uptime is normally 5 hours since it reboots, just MySQL is installed in a fresh VMs, nothing else at all. Rest all details I'll inform, when I'll setup the group replication again in future. Sep 17, 2021 at 6:16

1 Answer 1

2

If you have only 2GB of RAM,

innodb_log_buffer_size = 8M
max_connections = 100
innodb_write_io_threads = 4
innodb_buffer_pool_size = 600M

The 3 servers with 32GB can have higher settings.

Using lots of threads may actually be slowing down things in your app. It is certainly helping lead to memory overflow.

You say "5 mysql nodes master-master" -- Does that mean 5 pairs of servers (10 servers total)? Or 5 servers in circular replication (a disaster waiting to happen)?

If you insist on lots of connections, you need to get more RAM than 2GB and probably more CPU cores. If the connections are CPU-bound, there is no benefit in having more connections than cores.

4
  • I have updated the configurations to your suggested configurations. I will wait for 24 hours to see if it still gets out of memory and reboots. Also, I have 5 servers in total which are all configured as MASTER nodes in group replication like explained here digitalocean.com/community/tutorials/… Sep 16, 2021 at 6:18
  • @SunilKumar - How far apart are the 5 servers? (Same building versus cross-country)
    – Rick James
    Sep 16, 2021 at 16:41
  • Hello @Rick, the server still rebooted 2 times after implementing your configuration suggestion. I have dropped the idea of having smaller RAM backup server since I'm facing this issue from past 1-2 months. For the question you asked, those 3 servers are located in different buildings in 1 mile radius, in India, 1 server 2GB RAM is a VM located in Singapore, and 5th node of 2GB RAM is located in India, but roughly 100 miles away. Sep 17, 2021 at 6:09
  • Currently I'm using one of the 32GB RAM server as a standalone MySQL server (no group replication) and it just takes about 1.4GB RAM at max load. In future, I'll setup MASTER-SLAVE configuration again, most probably on more resources VMs. I'll inform if the same issue comes again in future in that setup. Sep 17, 2021 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.