Can you help me understand what was happen on MariaDB server. I wasn't able to connect to this server because there was 'Too many connections'. This is one of our slave in replication so non of main server changes were not applied on this replica.
I had to kill (-9) MariaDB server. I saw in error logs:
BUFFER POOL AND MEMORY
----------------------
InnoDB: ###### Diagnostic info printed to the standard error stream
2023-06-28 13:15:46 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789684688640 has waited at trx0rseg.ic line 50 for 272.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd1098 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:15:46 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789781440256 has waited at trx0undo.ic line 164 for 271.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887d195a8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file trx0undo.ic line 180
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:15:46 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789808133888 has waited at srv0srv.cc line 2096 for 250.00 seconds the semaphore:
X-lock on RW-latch at 0x7f28d1cbf440 created in file dict0dict.cc line 1107
a writer (thread id 139789701474048) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0purge.cc line 853
Last time write locked in file dict0stats.cc line 2457
2023-06-28 13:15:46 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789701474048 has waited at trx0trx.cc line 1157 for 251.00 seconds the semaphore:
Mutex at 0x7f28d1cc7fd8, Mutex REDO_RSEG created trx0rseg.cc:168, lock var 2
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789782349568 has waited at trx0rseg.ic line 50 for 176.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd13c8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789709866752 has waited at dict0dict.cc line 7209 for 12.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789684688640 has waited at trx0rseg.ic line 50 for 272.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd1098 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781440256 has waited at trx0undo.ic line 164 for 271.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887d195a8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file trx0undo.ic line 180
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781743360 has waited at dict0dict.cc line 1160 for 169.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789808133888 has waited at srv0srv.cc line 2096 for 250.00 seconds the semaphore:
X-lock on RW-latch at 0x7f28d1cbf440 created in file dict0dict.cc line 1107
a writer (thread id 139789701474048) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0purge.cc line 853
Last time write locked in file dict0stats.cc line 2457
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781137152 has waited at dict0load.cc line 2747 for 219.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:15:46 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789701474048 has waited at trx0trx.cc line 1157 for 251.00 seconds the semaphore:
Mutex at 0x7f28d1cc7fd8, Mutex REDO_RSEG created trx0rseg.cc:168, lock var 2
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending reads 0, writes 0
InnoDB: ###### Diagnostic info printed to the standard error stream
2023-06-28 13:16:17 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789684688640 has waited at trx0rseg.ic line 50 for 303.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd1098 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:16:17 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789781440256 has waited at trx0undo.ic line 164 for 302.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887d195a8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file trx0undo.ic line 180
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:16:17 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789808133888 has waited at srv0srv.cc line 2096 for 281.00 seconds the semaphore:
X-lock on RW-latch at 0x7f28d1cbf440 created in file dict0dict.cc line 1107
a writer (thread id 139789701474048) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0purge.cc line 853
Last time write locked in file dict0stats.cc line 2457
2023-06-28 13:16:17 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789781137152 has waited at dict0load.cc line 2747 for 250.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:16:17 139789791348480 [Warning] InnoDB: A long semaphore wait:
--Thread 139789701474048 has waited at trx0trx.cc line 1157 for 282.00 seconds the semaphore:
Mutex at 0x7f28d1cc7fd8, Mutex REDO_RSEG created trx0rseg.cc:168, lock var 2
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789782349568 has waited at trx0rseg.ic line 50 for 207.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd13c8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789709866752 has waited at dict0dict.cc line 7209 for 43.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789684688640 has waited at trx0rseg.ic line 50 for 303.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887cd1098 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file not yet reserved line 0
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781440256 has waited at trx0undo.ic line 164 for 302.00 seconds the semaphore:
X-lock on RW-latch at 0x7f2887d195a8 created in file buf0buf.cc line 1471
a writer (thread id 0) has reserved it in mode SX
number of readers 0, waiters flag 1, lock_word: 10000000
Last time read locked in file trx0undo.ic line 180
Last time write locked in file buf0flu.cc line 1236
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781743360 has waited at dict0dict.cc line 1160 for 200.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789808133888 has waited at srv0srv.cc line 2096 for 281.00 seconds the semaphore:
X-lock on RW-latch at 0x7f28d1cbf440 created in file dict0dict.cc line 1107
a writer (thread id 139789701474048) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file row0purge.cc line 853
Last time write locked in file dict0stats.cc line 2457
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789781137152 has waited at dict0load.cc line 2747 for 250.00 seconds the semaphore:
Mutex at 0x7f28d1cbf4c0, Mutex DICT_SYS created dict0dict.cc:1096, lock var 2
2023-06-28 13:16:17 139789791348480 [Note] InnoDB: A semaphore wait:
--Thread 139789701474048 has waited at trx0trx.cc line 1157 for 282.00 seconds the semaphore:
Mutex at 0x7f28d1cc7fd8, Mutex REDO_RSEG created trx0rseg.cc:168, lock var 2
Can you tell me what was happened?
SHOW ENGINE INNODB STATUS;
andSHOW FULLPROCESSLIST;
Those may help in analyzing. Meanwhile, increasemax_connections
(but not too high) and/or reduce the number of connections that clients can create. (Eg, Apache has a limit that is sometimes too high.)