Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

We have a high-end server, 128GB RAM, 32 Core , Xeon, SSD RAID 10 - running Ubuntu 12.04 with MySQL 5.5.28.

During high load, randomly we got the below error.

MySQLcheck, innochecksum shows no corruption.

Below is the crash log and my.cnf.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com
-- InnoDB: error clustered record for sec rec not found
InnoDB: index `FK_AccountLine_Statement` of table `MyCompany_staging`.`AccountLine`
InnoDB: sec index record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 8; hex 8000000000637b3c; asc      c{<;;
 1: len 8; hex 800000000fedcdfb; asc         ;;

InnoDB: clust index record PHYSICAL RECORD: n_fields 37; compact format; info bits 0
 0: len 8; hex 800000000fedcdc8; asc         ;;
 1: len 6; hex 00001464ddd9; asc    d  ;;
 2: len 7; hex 440011000d1802; asc D      ;;
 3: len 4; hex 80000000; asc     ;;
 4: len 25; hex 4e65742043617368202d20456e64696e672042616c616e6365; asc Net Cash - Ending Balance;;
 5: len 4; hex 80000000; asc     ;;
 6: len 1; hex 00; asc  ;;
 7: len 8; hex 8000000000000002; asc         ;;
 8: len 8; hex 8000000000000005; asc         ;;
 9: len 8; hex 8000000000ccdcf8; asc         ;;
 10: len 4; hex 8000001f; asc     ;;
 11: SQL NULL;
 12: len 8; hex 8000124597556740; asc    E Ug@;;
 13: len 8; hex 80001245a50aecc0; asc    E    ;;
 14: len 8; hex 8000000000000003; asc         ;;
 15: len 8; hex 00000000c07a0041; asc      z A;;
 16: len 1; hex 80; asc  ;;
 17: len 8; hex 8000124cd06715b4; asc    L g  ;;
 18: len 21; hex 6576616e2e6a656875406c7573696768742e636f6d; asc fake.email@MyCompany.com;;
 19: SQL NULL;
 20: SQL NULL;
 21: len 8; hex 80000000000c8660; asc        `;;
 22: SQL NULL;
 23: SQL NULL;
 24: SQL NULL;
 25: SQL NULL;
 26: SQL NULL;
 27: SQL NULL;
 28: len 1; hex 80; asc  ;;
 29: SQL NULL;
 30: SQL NULL;
 31: len 1; hex 80; asc  ;;
 32: len 1; hex 80; asc  ;;
 33: len 1; hex 80; asc  ;;
 34: SQL NULL;
 35: SQL NULL;
 36: SQL NULL;

TRANSACTION 15933742, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 0
28 lock struct(s), heap size 3112, 107 row lock(s), undo log entries 141
MySQL thread id 5296, OS thread handle 0x7f7d8ac90700, query id 1227065921 11.38.34.150 MyCompany Sending data
/* load one-to-many com.MyCompany.db.bean.BnFinancialStatement.accountLines */ select accountlin0_.STATEMENT_UID as STATEMENT18_2_, accountlin0_.UID as UID2_, accountlin0_.DISPLAY_ORDER as DISPLAY7_2_, accountlin0_.UID as UID0_1_, accountlin0_.VERSION as VERSION0_1_, accountlin0_.MODIFIED_DATE as MODIFIED3_0_1_, accountlin0_.MODIFIED_BY as MODIFIED4_0_1_, accountlin0_.CREATED_DATE as CREATED5_0_1_, accountlin0_.CREATED_BY as CREATED6_0_1_, accountlin0_.DISPLAY_ORDER as DISPLAY7_0_1_, accountlin0_.NAME as NAME0_1_, accountlin0_.INDENT_LEVEL as INDENT9_0_1_, accountlin0_.IS_FLIP_SIGN as IS10_0_1_,

Here is the my.cnf:

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock


[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
skip-name-resolve
innodb_file_per_table
default_storage_engine=InnoDB

user            = mysql
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /data/mysql
tmpdir          = /tmp
skip-external-locking

key_buffer              = 512M
max_allowed_packet      = 128M
thread_stack            = 192K
thread_cache_size       = 64

myisam-recover         = BACKUP
max_connections        = 500
table_cache            = 812
table_definition_cache = 812

#query_cache_limit       = 4M
#query_cache_size        = 512M
join_buffer_size        = 512K


innodb_additional_mem_pool_size = 20M
innodb_buffer_pool_size = 96G
#innodb_file_io_threads = 4
#innodb_thread_concurrency = 12
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120

log_error                = /var/log/mysql/error.log

long_query_time =       5
slow_query_log  =       1
slow_query_log_file     =       /var/log/mysql/slowlog.log

[mysqldump]
quick
quote-names
max_allowed_packet      = 16M

[mysql]

[isamchk]
key_buffer              = 16M

Memory usage, disk space is fine. This only happens during high I/O.

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.