We have a high-end server, 128GB RAM, 32 Core , Xeon, SSD RAID 10 - running Ubuntu 12.04 with MySQL 5.5.28.
During high load, randomly we got the below error.
MySQLcheck, innochecksum shows no corruption.
Below is the crash log and my.cnf.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com
-- InnoDB: error clustered record for sec rec not found
InnoDB: index `FK_AccountLine_Statement` of table `MyCompany_staging`.`AccountLine`
InnoDB: sec index record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
0: len 8; hex 8000000000637b3c; asc c{<;;
1: len 8; hex 800000000fedcdfb; asc ;;
InnoDB: clust index record PHYSICAL RECORD: n_fields 37; compact format; info bits 0
0: len 8; hex 800000000fedcdc8; asc ;;
1: len 6; hex 00001464ddd9; asc d ;;
2: len 7; hex 440011000d1802; asc D ;;
3: len 4; hex 80000000; asc ;;
4: len 25; hex 4e65742043617368202d20456e64696e672042616c616e6365; asc Net Cash - Ending Balance;;
5: len 4; hex 80000000; asc ;;
6: len 1; hex 00; asc ;;
7: len 8; hex 8000000000000002; asc ;;
8: len 8; hex 8000000000000005; asc ;;
9: len 8; hex 8000000000ccdcf8; asc ;;
10: len 4; hex 8000001f; asc ;;
11: SQL NULL;
12: len 8; hex 8000124597556740; asc E Ug@;;
13: len 8; hex 80001245a50aecc0; asc E ;;
14: len 8; hex 8000000000000003; asc ;;
15: len 8; hex 00000000c07a0041; asc z A;;
16: len 1; hex 80; asc ;;
17: len 8; hex 8000124cd06715b4; asc L g ;;
18: len 21; hex 6576616e2e6a656875406c7573696768742e636f6d; asc fake.email@MyCompany.com;;
19: SQL NULL;
20: SQL NULL;
21: len 8; hex 80000000000c8660; asc `;;
22: SQL NULL;
23: SQL NULL;
24: SQL NULL;
25: SQL NULL;
26: SQL NULL;
27: SQL NULL;
28: len 1; hex 80; asc ;;
29: SQL NULL;
30: SQL NULL;
31: len 1; hex 80; asc ;;
32: len 1; hex 80; asc ;;
33: len 1; hex 80; asc ;;
34: SQL NULL;
35: SQL NULL;
36: SQL NULL;
TRANSACTION 15933742, ACTIVE 1 sec starting index read
mysql tables in use 2, locked 0
28 lock struct(s), heap size 3112, 107 row lock(s), undo log entries 141
MySQL thread id 5296, OS thread handle 0x7f7d8ac90700, query id 1227065921 11.38.34.150 MyCompany Sending data
/* load one-to-many com.MyCompany.db.bean.BnFinancialStatement.accountLines */ select accountlin0_.STATEMENT_UID as STATEMENT18_2_, accountlin0_.UID as UID2_, accountlin0_.DISPLAY_ORDER as DISPLAY7_2_, accountlin0_.UID as UID0_1_, accountlin0_.VERSION as VERSION0_1_, accountlin0_.MODIFIED_DATE as MODIFIED3_0_1_, accountlin0_.MODIFIED_BY as MODIFIED4_0_1_, accountlin0_.CREATED_DATE as CREATED5_0_1_, accountlin0_.CREATED_BY as CREATED6_0_1_, accountlin0_.DISPLAY_ORDER as DISPLAY7_0_1_, accountlin0_.NAME as NAME0_1_, accountlin0_.INDENT_LEVEL as INDENT9_0_1_, accountlin0_.IS_FLIP_SIGN as IS10_0_1_,
Here is the my.cnf:
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
skip-name-resolve
innodb_file_per_table
default_storage_engine=InnoDB
user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /data/mysql
tmpdir = /tmp
skip-external-locking
key_buffer = 512M
max_allowed_packet = 128M
thread_stack = 192K
thread_cache_size = 64
myisam-recover = BACKUP
max_connections = 500
table_cache = 812
table_definition_cache = 812
#query_cache_limit = 4M
#query_cache_size = 512M
join_buffer_size = 512K
innodb_additional_mem_pool_size = 20M
innodb_buffer_pool_size = 96G
#innodb_file_io_threads = 4
#innodb_thread_concurrency = 12
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
log_error = /var/log/mysql/error.log
long_query_time = 5
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slowlog.log
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
Memory usage, disk space is fine. This only happens during high I/O.
