5

I have an mysql cluster which have 4 api nodes, 2 management nodes and 4 data nodes. Today, I was having problems while trying to connect database and all queries was hanging at "Opening tables" state. After inspecting logs i have found these erros on logs:

Api Node Erros:

2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 5 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 6 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 7 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: Data node: 8 failed, subscriber bitmask 00
2015-08-20 19:44:14 15540 [Note] NDB Schema dist: cluster failure at epoch 3313124/17.
2015-08-20 19:44:14 15540 [Note] NDB Binlog: ndb tables initially read only on reconnect.
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Got temporary error 4028 'Node failure caused abort of transaction' from NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] /opt/mysql/server-5.6/bin/mysqld: Sort aborted: Got temporary error 4028 'Node failure caused abort of transaction' from NDBCLUSTER
2015-08-20 19:44:14 15540 [ERROR] Got error 4010 when reading table './database_name/table'
2015-08-20 19:44:14 15540 [Note] NDB Binlog: cluster failure for ./database_name/table_name at epoch 3313124/17.

mysql> show processlists;

Id  User    Host    db  Command Time    State   Info
1   system user     NULL    Daemon  1497    Waiting for ndbcluster to start NULL

Data Node Erros:

2015-08-20 19:44:14 [ndbd] ERROR -- c_gcp_list.seize() failed: gci: 14229759227592721 nodes: 0000000000000000000000000000040000000000000000000000000000001a00
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2000b from: 0fa2000b
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2000c from: 0fa2000c
2015-08-20 19:44:14 [ndbd] WARNING -- ACK wo/ gcp record (gci: 3313124/17) ref: 0fa2008a from: 0fa2008a

Management Node Erros:

2015-08-20 19:44:14 [MgmtSrvr] INFO     -- Node 5: Disconnecting lagging nodes '0000000000000000000000000000000000000000000000000000000000000200',
2015-08-20 19:44:14 [MgmtSrvr] WARNING  -- Node 5: Disconnecting node 9 because it has exceeded MaxBufferedEpochs (100 > 100), epoch 3313119/4

Detailed logs and configurations

Data node config:

https://gist.github.com/sdemircan/730fa49fcc14b4376c42

Api node config:

https://gist.github.com/sdemircan/f9d230d32700b86564fd

Management node config:

https://gist.github.com/sdemircan/d6fbd54799daaae01bf2

Api Node logs:

https://gist.github.com/sdemircan/2d62b1c92176de9de9d3

Data Node logs:

https://gist.github.com/sdemircan/d0c97b82457a9c33deaa

Data Node logs:

https://gist.github.com/sdemircan/3faa1e41367bc7655210

Management Node logs:

https://gist.github.com/sdemircan/a026ac57757fafdafaa9

What might bring MaxBufferedEpochs to upper limit?

1

1 Answer 1

0

You may have had a large transaction, a query retrieving many rows, pull back a lot of data and saturate the network connection for node 9.

On reconnect the API nodes are readonly so it has to be a query executed before this

NDB Binlog: ndb tables initially read only on reconnect

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.