1

I'm dumb, I did the stupidest thing I could. On one of my servers, the SSD was full and I wasn't aware of it.

The high-traffic website became working really slow, then started displaying errors, because even the database couldn't save new rows.

Before I figured what's going on, I performed a hard reboot. And then the mysqld service wasn't be able to start.

I started to delete some non important files (it turned out 80% of the space was trash), so now I have 80% of free space.

Unfortunately, every time I try to run "service mysqld restart" or start, it starts and then stops immediately.

My configuration: - Debian 7.5 + DirectAdmin (Apache, MySQL), - The database is huge and includes both MyISAM and InnoDB tables.

I tried to make an export, but cannot run mysqldump without working mysqld service.

I copied my tables from /usr/local/mysql/data and also have a backup, but in SQL format - so I cannot import it. But the best option is to run current tables, because the backup is old.

I've tried so far:

  • myisamchk -r -v *.MYI
  • Setting "innodb_force_recovery=3" (first I tried 1 and 2)
  • Many hard reboots and all services restarts
  • Change name of the corrupted database folder, but then I've seen errors in my log (couldn't find table files etc.)

MySQL log doesn't explain anything:

150813 15:48:13 mysqld_safe mysqld from pid file /usr/local/mysql/data/ns390143.pid ended
150813 15:48:21 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
2015-08-13 15:48:21 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-08-13 15:48:21 2954 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
2015-08-13 15:48:21 2954 [Note] Plugin 'FEDERATED' is disabled.
2015-08-13 15:48:21 2954 [Note] InnoDB: The InnoDB memory heap is disabled
2015-08-13 15:48:21 2954 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-08-13 15:48:21 2954 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-08-13 15:48:21 2954 [Note] InnoDB: Using Linux native AIO
2015-08-13 15:48:21 2954 [Note] InnoDB: Using CPU crc32 instructions
2015-08-13 15:48:21 2954 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-08-13 15:48:21 2954 [Note] InnoDB: Completed initialization of buffer pool
2015-08-13 15:48:21 2954 [Note] InnoDB: Highest supported file format is Barracuda.
2015-08-13 15:48:21 2954 [Note] InnoDB: 128 rollback segment(s) are active.
2015-08-13 15:48:21 2954 [Note] InnoDB: Waiting for purge to start
2015-08-13 15:48:21 2954 [Note] InnoDB: 5.6.12 started; log sequence number 28646291749
2015-08-13 15:48:21 2954 [ERROR] Aborting

2015-08-13 15:48:21 2954 [Note] Binlog end
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'partition'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'ARCHIVE'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_METRICS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_CMP'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_LOCKS'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'INNODB_TRX'
2015-08-13 15:48:21 2954 [Note] Shutting down plugin 'InnoDB'
2015-08-13 15:48:21 2954 [Note] InnoDB: FTS optimize thread exiting.
2015-08-13 15:48:21 2954 [Note] InnoDB: Starting shutdown...
2015-08-13 15:48:23 2954 [Note] InnoDB: Shutdown completed; log sequence number 28646291759
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'BLACKHOLE'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'CSV'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'MRG_MYISAM'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'MyISAM'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'MEMORY'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'sha256_password'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'mysql_old_password'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'mysql_native_password'
2015-08-13 15:48:23 2954 [Note] Shutting down plugin 'binlog'
2015-08-13 15:48:23 2954 [Note]

My questions are:

  1. How to fix it? And is it possible?

  2. If not, how can I import .sql file backup?

  • You have a mysqldump SQL file? – Vérace Aug 13 '15 at 14:32
  • @Verace, Yes I have. I think the database is OK, but my config (my.cnf) has been reset. Now trying to make this work. I can import/export the DB, but cannot connect from the website (phpMyAdmin throws #2002 error). – Bat_ Aug 13 '15 at 15:43
  • Please merge your accounts. Once this is done, you will be able to edit your question and comment on any answers. Please don't add responses as "answers" in the meantime, thank you. – Paul White Aug 13 '15 at 16:47
  • In response to the OP's "answer" (I can import/export the DB). Maybe a fresh install with a fresh mysqldump? Maybe even a reinstall of the OS if you can spare the downtime? And in future, make sure you monitor your disk space - df -h on Linux. – Vérace Aug 13 '15 at 16:51
  • What is on the "full" drive? What can eventually be moved elsewhere? – Rick James Aug 17 '15 at 5:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.