mysqld_safe mysqld from pid file /data/mysql/mysqld.pid ended

Question

MySQL is restarting by itself, but it's crashing most of the time. I am unable to restart the instance due to a pid issue. Tried to restart MySQL

1. moving original ib* files to _old - this time MySQL is started
2. stopped MySQL and moved the _old to original state
3. restarted instance again - it's throwing an error pid ended

MySQL version is 5.6.21

From error log :

150713 11:39:42 mysqld_safe Number of processes running now: 0 150713 11:39:42 mysqld_safe mysqld restarted 2015-07-13 11:39:42 0 [Warning] Using unique option prefix lower-case-table-name instead of lower_case_table_names is deprecated and will be removed in a future release. Please use the full name instead. 2015-07-13 11:39:42 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2015-07-13 11:39:42 21262 [Note] Plugin 'FEDERATED' is disabled. 2015-07-13 11:39:42 7fac432cf780 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. 2015-07-13 11:39:42 21262 [Note] InnoDB: Using atomics to ref count buffer pool pages 2015-07-13 11:39:42 21262 [Note] InnoDB: The InnoDB memory heap is disabled 2015-07-13 11:39:42 21262 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2015-07-13 11:39:42 21262 [Note] InnoDB: Memory barrier is not used 2015-07-13 11:39:42 21262 [Note] InnoDB: Compressed tables use zlib 1.2.3 2015-07-13 11:39:42 21262 [Note] InnoDB: Using Linux native AIO 2015-07-13 11:39:42 21262 [Note] InnoDB: Using CPU crc32 instructions 2015-07-13 11:39:42 21262 [Note] InnoDB: Initializing buffer pool, size = 16.0G 2015-07-13 11:39:43 21262 [Note] InnoDB: Completed initialization of buffer pool 2015-07-13 11:39:43 21262 [ERROR] InnoDB: Space id in fsp header 1416128883,but in the page header 824195850 06:09:43 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=2097152 max_used_connections=0 max_threads=1000 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2325895 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /opt/mysql/server-5.6/bin/mysqld(my_print_stacktrace+0x35)[0x8f8bb5] /opt/mysql/server-5.6/bin/mysqld(handle_fatal_signal+0x494)[0x67aba4] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f8cdbd3d340] /opt/mysql/server-5.6/bin/mysqld[0xb3f6bb] /opt/mysql/server-5.6/bin/mysqld[0x9ba3c7] /opt/mysql/server-5.6/bin/mysqld[0xa65793] /opt/mysql/server-5.6/bin/mysqld[0xaa7016] /opt/mysql/server-5.6/bin/mysqld[0xaa921c] /opt/mysql/server-5.6/bin/mysqld[0xa17148] /opt/mysql/server-5.6/bin/mysqld[0x960b3d] /opt/mysql/server-5.6/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5baa58] /opt/mysql/server-5.6/bin/mysqld[0x705bf1] /opt/mysql/server-5.6/bin/mysqld(_Z11plugin_initPiPPci+0xbf6)[0x70a8a6] /opt/mysql/server-5.6/bin/mysqld[0x5ad89d] /opt/mysql/server-5.6/bin/mysqld(_Z11mysqld_mainiPPc+0x3e5)[0x5b2855] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f8cda921ec5] /opt/mysql/server-5.6/bin/mysqld[0x5a4bd9] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 150713 11:44:34 mysqld_safe mysqld from pid file /data/mysql/mysqld.pid ended

Answer 1

Make a clean attempt:
remove ibdata and iblog files (present)
copy ibdata file (original)
chown mysql:mysql /datadir/ -R
start mysql

and then check the error log once again.

Make sure your ibdata and iblog files are placed in correct location. Share your innodb parameters. I saw a bug report similar to your description: http://bugs.mysql.com/bug.php?id=69623 (Though version appears older)

For now, if you have backup + binary-logs then you can restore from backup and play binary logs again to catchup to latest.

Monitor for next crash and if it appears try to identify what actions caused crash. Check binary-log for DML/DDL... Upon crashes you might want to enable general query log to further review what was going on before the crash.

Answer 2

Yeah i have tried with above steps. but still i am facing same error like "mysqld from pid file /mysqld.pid ended" Innodb variables innodb_use_native_aio = 0 innodb_buffer_pool_size = 16G innodb_log_buffer_size = 8M innodb_log_file_size = 128M innodb_file_per_table = 1 innodb_max_dirty_pages_pct = 90 innodb_read_io_threads = 16 innodb_write_io_threads = 16 innodb_thread_concurrency = 24 innodb_additional_mem_pool_size = 20M innodb_file_per_table