1

I am having the hardest time tracking down this problem.

Every few minutes or so my database server crashes so all 3 WordPress sites on my WHM cPanels all crash at the same time. When it does, I get the following errors on phpMyAdmin. error reporting

My status monitor also looks like this: status graphs

Just before the crash the numbers look like:

Questions: 910
Connections: 30
Processes: 26
Bytes sent: 59.6 MiB
Average load: 100%
Buffered memory: 1.2 GiB
Cached memory: 562.6 MiB
Free memory: 77.3 MiB
Cached swap: 25.9 MiB
Used swap: 164.2 MiB
Free swap: 833.9 MiB

Can somebody point me in the right direction on how to solve this?

Edit: As per JYOTI RAJAI's comment, I tracked down the logs. For yesterday I had over 60,000 instances of this error message:

2020-06-04T19:06:51.057890Z 68060 [ERROR] /usr/sbin/mysqld: Table './<database>/wp_options' is marked as crashed and should be repaired

Then this started happening:

2020-06-04T19:06:51.995679Z 68061 [Note] Found 628 of 627 rows when repairing './<database>/wp_options'
2020-06-04T19:07:45.832221Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-04T19:07:45.835576Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 19188 ...
2020-06-04T19:07:45.868836Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-04T19:07:45.868876Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-04T19:07:45.868881Z 0 [Note] InnoDB: Uses event mutexes
2020-06-04T19:07:45.868885Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-04T19:07:45.868889Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-04T19:07:45.868893Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-04T19:07:45.870568Z 0 [Note] InnoDB: Number of pools: 1
2020-06-04T19:07:45.874033Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-04T19:07:45.877335Z 0 [Note] InnoDB: Initializing buffer pool, total size = 52M, instances = 1, chunk size = 52M
2020-06-04T19:07:45.893666Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-06-04T19:07:45.902361Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-06-04T19:07:45.920678Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-04T19:07:45.926440Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 187614470
2020-06-04T19:07:45.926465Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 187622785
2020-06-04T19:07:45.928429Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-06-04T19:07:45.928444Z 0 [Note] InnoDB: Starting crash recovery.
2020-06-04T19:07:45.996699Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
2020-06-04T19:07:45.997811Z 0 [Note] InnoDB: Apply batch completed
2020-06-04T19:07:46.183292Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-04T19:07:46.183331Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-06-04T19:07:46.183398Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-06-04T19:07:46.224593Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-06-04T19:07:46.225793Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-06-04T19:07:46.225824Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-06-04T19:07:46.226142Z 0 [Note] InnoDB: Waiting for purge to start
2020-06-04T19:07:46.276311Z 0 [Note] InnoDB: 5.7.30 started; log sequence number 187622785
2020-06-04T19:07:46.277779Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-06-04T19:07:46.279417Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-06-04T19:07:46.294519Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200604 12:07:46
2020-06-04T19:07:46.295492Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-06-04T19:07:46.295506Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-06-04T19:07:46.298040Z 0 [Warning] CA certificate ca.pem is self signed.
2020-06-04T19:07:46.298098Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-06-04T19:07:46.299492Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-06-04T19:07:46.299534Z 0 [Note] IPv6 is available.
2020-06-04T19:07:46.299554Z 0 [Note]   - '::' resolves to '::';
2020-06-04T19:07:46.299560Z 0 [Note] Server socket created on IP: '::'.
2020-06-04T19:07:46.360571Z 0 [Note] Event Scheduler: Loaded 0 events
2020-06-04T19:07:46.365364Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2020-06-04T19:07:50.924930Z 3 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './<database>/wp_redirection_404.MYI'; try to repair it
2020-06-04T19:07:50.924962Z 3 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './<database>/wp_redirection_404.MYI'; try to repair it
2020-06-04T19:07:50.924967Z 3 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './<database>/wp_redirection_404.MYI'; try to repair it
2020-06-04T19:07:50.924971Z 3 [ERROR] Got an error from thread_id=3, /export/home2/pb2/build/sb_1-38465026-1584986589.6/rpm/BUILD/mysql-5.7.30/mysql-5.7.30/storage/myisam/mi_write.c:232
2020-06-04T19:07:50.924976Z 3 [ERROR] MySQL thread id 3, OS thread handle 140419102160640, query id 527 localhost <username> update
INSERT INTO `wp_redirection_404` (`url`, `created`, `ip`, `agent`) VALUES ('/autodiscover/autodiscover.xml', '2020-06-04 19:07:50', '67.207.43.162', 'Microsoft Office/14.0 (Windows NT 6.2; Microsoft Outlook 14.0.7248; Pro)')
2020-06-04T19:08:00.600440Z 24 [ERROR] /usr/sbin/mysqld: Table './<database>/wp_redirection_404' is marked as crashed and should be repaired
... ( repeat single line above 54 times ) ...
2020-06-04T19:08:46.589575Z 79 [ERROR] /usr/sbin/mysqld: Table './<database>/wp_redirection_404' is marked as crashed and last (automatic?) repair failed
... ( repeat single line above 1848 times ) ... 

The reboot sequence kept happening every few minutes. I uninstalled the WordPress plugin that included the table wp_redirection_404 before asking this question because the crashes kept happening. But that part of the log looks like this:

2020-06-05T01:31:32.534229Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 19731 ...
2020-06-05T01:31:32.594714Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-05T01:31:32.594773Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-05T01:31:32.594781Z 0 [Note] InnoDB: Uses event mutexes
2020-06-05T01:31:32.594788Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-05T01:31:32.594794Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-05T01:31:32.594800Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-05T01:31:32.595445Z 0 [Note] InnoDB: Number of pools: 1
2020-06-05T01:31:32.595713Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-05T01:31:32.599173Z 0 [Note] InnoDB: Initializing buffer pool, total size = 52M, instances = 1, chunk size = 52M
2020-06-05T01:31:32.609154Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-06-05T01:31:32.611283Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-06-05T01:31:32.656917Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-05T01:31:32.680961Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 188393877
2020-06-05T01:31:32.680980Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 188393886
2020-06-05T01:31:32.680989Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-06-05T01:31:32.680994Z 0 [Note] InnoDB: Starting crash recovery.
2020-06-05T01:31:33.418849Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-05T01:31:33.418872Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-06-05T01:31:33.418911Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-06-05T01:31:33.437853Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-06-05T01:31:33.438830Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-06-05T01:31:33.438856Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-06-05T01:31:33.439142Z 0 [Note] InnoDB: Waiting for purge to start
2020-06-05T01:31:33.496447Z 0 [Note] InnoDB: 5.7.30 started; log sequence number 188393886
2020-06-05T01:31:33.496732Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-06-05T01:31:33.509354Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-06-05T01:31:33.522014Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-06-05T01:31:33.522028Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-06-05T01:31:33.546564Z 0 [Warning] CA certificate ca.pem is self signed.
2020-06-05T01:31:33.546668Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-06-05T01:31:33.559840Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200604 18:31:33
2020-06-05T01:31:33.581455Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-06-05T01:31:33.581504Z 0 [Note] IPv6 is available.
2020-06-05T01:31:33.581530Z 0 [Note]   - '::' resolves to '::';
2020-06-05T01:31:33.581535Z 0 [Note] Server socket created on IP: '::'.
2020-06-05T01:31:34.139900Z 0 [Note] Event Scheduler: Loaded 0 events
2020-06-05T01:31:34.140388Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2020-06-05T01:37:46.181017Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-05T01:37:46.182802Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 21146 ...
2020-06-05T01:37:46.190432Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-05T01:37:46.190465Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-05T01:37:46.190470Z 0 [Note] InnoDB: Uses event mutexes
2020-06-05T01:37:46.190473Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-05T01:37:46.190477Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-05T01:37:46.190480Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-05T01:37:46.190795Z 0 [Note] InnoDB: Number of pools: 1
2020-06-05T01:37:46.190921Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-05T01:37:46.192518Z 0 [Note] InnoDB: Initializing buffer pool, total size = 52M, instances = 1, chunk size = 52M
2020-06-05T01:37:46.195871Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-06-05T01:37:46.197148Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-06-05T01:37:46.223869Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-05T01:37:46.229362Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 188416734
2020-06-05T01:37:46.229378Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 188416743
2020-06-05T01:37:46.229385Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-06-05T01:37:46.229389Z 0 [Note] InnoDB: Starting crash recovery.
2020-06-05T01:37:46.442722Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-05T01:37:46.442754Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-06-05T01:37:46.442814Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-06-05T01:37:46.463571Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-06-05T01:37:46.464893Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-06-05T01:37:46.464915Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-06-05T01:37:46.465319Z 0 [Note] InnoDB: Waiting for purge to start
2020-06-05T01:37:46.515572Z 0 [Note] InnoDB: 5.7.30 started; log sequence number 188416743
2020-06-05T01:37:46.515974Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-06-05T01:37:46.516370Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-06-05T01:37:46.522591Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-06-05T01:37:46.522617Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-06-05T01:37:46.529113Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200604 18:37:46
2020-06-05T01:37:46.529627Z 0 [Warning] CA certificate ca.pem is self signed.
2020-06-05T01:37:46.529722Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-06-05T01:37:46.531474Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-06-05T01:37:46.531537Z 0 [Note] IPv6 is available.
2020-06-05T01:37:46.531554Z 0 [Note]   - '::' resolves to '::';
2020-06-05T01:37:46.531562Z 0 [Note] Server socket created on IP: '::'.
2020-06-05T01:37:46.591272Z 0 [Note] Event Scheduler: Loaded 0 events
2020-06-05T01:37:46.591449Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2020-06-05T01:43:27.914157Z 224 [Note] Aborted connection 224 to db: '<database>' user: '<username>' host: 'localhost' (Got an error writing communication packets)
2020-06-05T01:43:37.458641Z 226 [Note] Aborted connection 226 to db: '<database>' user: '<username>' host: 'localhost' (Got an error reading communication packets)
2020-06-05T01:44:16.641164Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-05T01:44:16.642888Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 22406 ...
2020-06-05T01:44:16.651991Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-05T01:44:16.652029Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-05T01:44:16.652033Z 0 [Note] InnoDB: Uses event mutexes
2020-06-05T01:44:16.652036Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-05T01:44:16.652039Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-05T01:44:16.652042Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-05T01:44:16.652439Z 0 [Note] InnoDB: Number of pools: 1
2020-06-05T01:44:16.652636Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-05T01:44:16.654861Z 0 [Note] InnoDB: Initializing buffer pool, total size = 52M, instances = 1, chunk size = 52M
2020-06-05T01:44:16.659698Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-06-05T01:44:16.661408Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-06-05T01:44:16.686889Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-05T01:44:16.692320Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 188447819
2020-06-05T01:44:16.692336Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 188447828
2020-06-05T01:44:16.692343Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-06-05T01:44:16.692348Z 0 [Note] InnoDB: Starting crash recovery.
2020-06-05T01:44:16.901379Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-05T01:44:16.901404Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-06-05T01:44:16.901443Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-06-05T01:44:16.920059Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-06-05T01:44:16.921170Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-06-05T01:44:16.921187Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-06-05T01:44:16.921533Z 0 [Note] InnoDB: Waiting for purge to start
2020-06-05T01:44:16.971815Z 0 [Note] InnoDB: 5.7.30 started; log sequence number 188447828
2020-06-05T01:44:16.972472Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-06-05T01:44:16.972945Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-06-05T01:44:16.977541Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-06-05T01:44:16.977557Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-06-05T01:44:16.980940Z 0 [Warning] CA certificate ca.pem is self signed.
2020-06-05T01:44:16.981041Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-06-05T01:44:16.983431Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200604 18:44:16
2020-06-05T01:44:16.984694Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-06-05T01:44:16.984746Z 0 [Note] IPv6 is available.
2020-06-05T01:44:16.984759Z 0 [Note]   - '::' resolves to '::';
2020-06-05T01:44:16.984763Z 0 [Note] Server socket created on IP: '::'.
2020-06-05T01:44:17.037645Z 0 [Note] Event Scheduler: Loaded 0 events
2020-06-05T01:44:17.037799Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2020-06-05T01:50:26.051180Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-05T01:50:26.052918Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 23709 ...
2020-06-05T01:50:26.063292Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-05T01:50:26.063324Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-05T01:50:26.063328Z 0 [Note] InnoDB: Uses event mutexes
2020-06-05T01:50:26.063349Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-05T01:50:26.063356Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-05T01:50:26.063363Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-05T01:50:26.063641Z 0 [Note] InnoDB: Number of pools: 1
2020-06-05T01:50:26.063764Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-05T01:50:26.065589Z 0 [Note] InnoDB: Initializing buffer pool, total size = 52M, instances = 1, chunk size = 52M
2020-06-05T01:50:26.068952Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-06-05T01:50:26.070314Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-06-05T01:50:26.095510Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-06-05T01:50:26.102759Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 188478220
2020-06-05T01:50:26.102775Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 188478229
2020-06-05T01:50:26.102783Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-06-05T01:50:26.102787Z 0 [Note] InnoDB: Starting crash recovery.
2020-06-05T01:50:26.442663Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-05T01:50:26.442688Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-06-05T01:50:26.442723Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-06-05T01:50:26.461271Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-06-05T01:50:26.462169Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-06-05T01:50:26.462182Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-06-05T01:50:26.462476Z 0 [Note] InnoDB: Waiting for purge to start
2020-06-05T01:50:26.512694Z 0 [Note] InnoDB: 5.7.30 started; log sequence number 188478229
2020-06-05T01:50:26.513183Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-06-05T01:50:26.513601Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-06-05T01:50:26.518699Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-06-05T01:50:26.518716Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-06-05T01:50:26.520859Z 0 [Warning] CA certificate ca.pem is self signed.
2020-06-05T01:50:26.520984Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-06-05T01:50:26.526583Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200604 18:50:26
2020-06-05T01:50:26.528066Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
2020-06-05T01:50:26.528151Z 0 [Note] IPv6 is available.
2020-06-05T01:50:26.528172Z 0 [Note]   - '::' resolves to '::';
2020-06-05T01:50:26.528180Z 0 [Note] Server socket created on IP: '::'.
2020-06-05T01:50:26.666477Z 0 [Note] Event Scheduler: Loaded 0 events
2020-06-05T01:50:26.666681Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.30'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2020-06-05T01:50:26.667545Z 0 [Note] Giving 1 client threads a chance to die gracefully
2020-06-05T01:50:26.667577Z 0 [Note] Shutting down slave threads
2020-06-05T01:50:28.667748Z 0 [Note] Forcefully disconnecting 0 remaining clients
2020-06-05T01:50:28.667795Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-06-05T01:50:28.667906Z 0 [Note] Binlog end
2020-06-05T01:50:28.668640Z 0 [Note] Shutting down plugin 'ngram'
2020-06-05T01:50:28.668663Z 0 [Note] Shutting down plugin 'partition'
2020-06-05T01:50:28.668670Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2020-06-05T01:50:28.668677Z 0 [Note] Shutting down plugin 'ARCHIVE'
2020-06-05T01:50:28.668683Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2020-06-05T01:50:28.668724Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2020-06-05T01:50:28.668734Z 0 [Note] Shutting down plugin 'MyISAM'
2020-06-05T01:50:28.668752Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2020-06-05T01:50:28.668758Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2020-06-05T01:50:28.668764Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2020-06-05T01:50:28.668769Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2020-06-05T01:50:28.668774Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2020-06-05T01:50:28.668779Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2020-06-05T01:50:28.668784Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2020-06-05T01:50:28.668789Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2020-06-05T01:50:28.668794Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2020-06-05T01:50:28.668799Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2020-06-05T01:50:28.668804Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2020-06-05T01:50:28.668809Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2020-06-05T01:50:28.668814Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2020-06-05T01:50:28.668819Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2020-06-05T01:50:28.668824Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2020-06-05T01:50:28.668829Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2020-06-05T01:50:28.668834Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2020-06-05T01:50:28.668839Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2020-06-05T01:50:28.668844Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2020-06-05T01:50:28.668850Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2020-06-05T01:50:28.668855Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2020-06-05T01:50:28.668860Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2020-06-05T01:50:28.668866Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2020-06-05T01:50:28.668870Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2020-06-05T01:50:28.668876Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2020-06-05T01:50:28.668881Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2020-06-05T01:50:28.668886Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2020-06-05T01:50:28.668891Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2020-06-05T01:50:28.668896Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2020-06-05T01:50:28.668901Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2020-06-05T01:50:28.668906Z 0 [Note] Shutting down plugin 'InnoDB'
2020-06-05T01:50:28.668988Z 0 [Note] InnoDB: FTS optimize thread exiting.
2020-06-05T01:50:28.669176Z 0 [Note] InnoDB: Starting shutdown...
2020-06-05T01:50:28.770277Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-06-05T01:50:28.770545Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200604 18:50:28
2020-06-05T01:50:30.387663Z 0 [Note] InnoDB: Shutdown completed; log sequence number 188479684
2020-06-05T01:50:30.389209Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-06-05T01:50:30.389248Z 0 [Note] Shutting down plugin 'MEMORY'
2020-06-05T01:50:30.389258Z 0 [Note] Shutting down plugin 'CSV'
2020-06-05T01:50:30.389262Z 0 [Note] Shutting down plugin 'sha256_password'
2020-06-05T01:50:30.389270Z 0 [Note] Shutting down plugin 'mysql_native_password'
2020-06-05T01:50:30.389409Z 0 [Note] Shutting down plugin 'binlog'
2020-06-05T01:50:30.389715Z 0 [Note] /usr/sbin/mysqld: Shutdown complete

2020-06-05T01:50:31.371190Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-05T01:50:31.371612Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.30) starting as process 23807 ...

Hopefully this means the databases have fixed themselves. When I checked on them this morning, they were up for 11 hours, 57 minutes, and 17 seconds.

4
  • Can you get mysql error logs ? – JYOTI RAJAI Jun 5 '20 at 8:54
  • Additional information request. RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; for server workload tuning analysis to provide suggestions. – Wilson Hauck Jun 5 '20 at 17:29
  • "Questions" -- total? Can't be. "per second"? per hour"? What. Junk! What do the colors represent for memory? Which process? All processes? – Rick James Jun 9 '20 at 3:27
  • How many cores? How much RAM? – Rick James Jun 9 '20 at 3:28
1

Plan A: Use myisamcheck to fix the "marked as crashed and needs repair" tables.

Plan B: Don't use ENGINE=MyISAM; switch to InnoDB. Be sure to adjust key_buffer_size and innodb_buffer_pool_size.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.