So for the first time our production system produced the below assertion failure. The system came up without issues after mysqld automatically restarted itself and rolled back some transactions..
But why did this happen suddenly on a stable system?
Uptime is over 5 months, Mysqld was up for about 4 straight months with no other errors. Nothing in terms of file system errors, memory errors.
The system is a Ubuntu 10.4 64bit with ~16gb ram. .. running mysql 5.1.41 all Innodb tables.
can someone explain a little more what this assertion failure means more specifically?
120127 7:57:00 InnoDB: Assertion failure in thread 139886060934912 in file ../../../storage/innobase/rem/rem0rec.c line 337 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: about forcing recovery. 120127 7:57:00 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=536870912 read_buffer_size=131072 max_used_connections=107 max_threads=151 threads_connected=49 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 854399 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0x7f39ac188290 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f39c2f88e58 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f3cb33671a9] /usr/sbin/mysqld(handle_segfault+0x404) [0x7f3cb3074884] /lib/libpthread.so.0(+0xf8f0) [0x7f3cb28e58f0] /lib/libc.so.6(gsignal+0x35) [0x7f3cb135fa75] /lib/libc.so.6(abort+0x180) [0x7f3cb13635c0] /usr/sbin/mysqld(rec_get_offsets_func+0xe0) [0x7f3cb32a9e50] /usr/sbin/mysqld(row_search_for_mysql+0x56e) [0x7f3cb32be4ce] /usr/sbin/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x7c) [0x7f3cb32271ec] /usr/sbin/mysqld(+0x3ccb0e) [0x7f3cb30e1b0e] /usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x81) [0x7f3cb30dfeb1] /usr/sbin/mysqld(+0x3cb3fd) [0x7f3cb30e03fd] /usr/sbin/mysqld(JOIN::exec()+0xbac) [0x7f3cb30f55ec] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x142) [0x7f3cb30f1302] /usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x7f3cb30f6c04] /usr/sbin/mysqld(+0x36d9fa) [0x7f3cb30829fa] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x526) [0x7f3cb3086806] /usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x35f) [0x7f3cb308bcbf] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd69) [0x7f3cb308ca39] /usr/sbin/mysqld(do_command(THD*)+0xea) [0x7f3cb308d61a] /usr/sbin/mysqld(handle_one_connection+0x23d) [0x7f3cb307f64d] /lib/libpthread.so.0(+0x69ca) [0x7f3cb28dc9ca] /lib/libc.so.6(clone+0x6d) [0x7f3cb141270d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x7f3ccb4021b0 = /* load collection com.lus111.db.bean.BnSource.inputSources */ select inputsourc0_.CHILD_SOURCE_UID as CHILD1_1_, inputsourc0_.PARENT_SOURCE_UID as PARENT2_1_, bnsource1_.UID as UID22_0_, bnsource1_.VERSION as VERSION22_0_, bnsource1_.MODIFIED_DATE as MODIFIED3_22_0_, bnsource1_.MODIFIED_BY as MODIFIED4_22_0_, bnsource1_.CREATED_DATE as CREATED5_22_0_, bnsource1_.CREATED_BY as CREATED6_22_0_, bnsource1_.IS_PUBLISHED as IS7_22_0_, bnsource1_.FORECAST_NAME_TRACKER as FORECAST8_22_0_, bnsource1_.CLIENT_UID as CLIENT9_22_0_, bnsource1_.COMPANY_UID as COMPANY10_22_0_, bnsource1_.LTUSER_UID as LTUSER11_22_0_, bnsource1_.FILE_UPLOAD_PDO_UID as FILE12_22_0_, bnsource1_.REPORT_UID as REPORT13_22_0_, bnsource1_.SOURCE_GENERATOR_UID as SOURCE14_22_0_ from Source_Inputs inputsourc0_ left outer join Source bnsource1_ on inputsourc0_.PARENT_SOURCE_UID=bnsource1_.UID where inputsourc0_.CHILD_SOURCE_UID=17950 thd->thread_id=264341 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. pure virtual method called terminate called without an active exception 120127 7:57:03 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Log scan progressed past the checkpoint lsn 224 4208337564 120127 7:57:06 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 224 4213580288
.. then eventually it recovered.
Do I need to worry more about this?