Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

Over the weekend our cluster we set up for evaluation had one of its data nodes lose network connectivity briefly. This was while another data node was in the middle of a restart (which had been going for 22 hours in phase 5, but that's a separate issue.) I'm seriously glad this happened before going production.

This led to all of the data nodes going offline:

2012-11-23 19:24:19 [ndbd] ALERT    -- Node 1: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.

The question is: What can I do to get the cluster going again? Alternatively, what can I look at to see what is going on? The logs are essentially empty (as you'll see below). What is safe to do at this point?

Details:

I am trying to get the cluster back online but am having little luck. I started 3 of the data nodes the normal way (ran "ndbmtd") and the data node that went down mid-restart with "ndbmtd --initial". All three "ndbmtd" nodes logged that they completed phase 3, meaning they're in phase 4, and the one "ndbmtd --initial" node completed phase 1 at the same time.

That was all approximately 50 hours ago.

Here's what I have in the --initial node's log:

2012-11-24 07:49:54 [ndbd] INFO     -- Start phase 1 completed
jbalock thr: 0 waiting for lock, contentions: 1 spins: 1
jbalock thr: 0 waiting for lock, contentions: 2 spins: 2
jbalock thr: 0 waiting for lock, contentions: 3 spins: 3
jbalock thr: 0 waiting for lock, contentions: 4 spins: 4
jbalock thr: 0 waiting for lock, contentions: 5 spins: 5
jbalock thr: 0 waiting for lock, contentions: 6 spins: 6
jbalock thr: 0 waiting for lock, contentions: 7 spins: 7
jbalock thr: 0 waiting for lock, contentions: 8 spins: 9
jbalock thr: 0 waiting for lock, contentions: 9 spins: 36
jbalock thr: 0 waiting for lock, contentions: 10 spins: 37
jbalock thr: 0 waiting for lock, contentions: 11 spins: 38

and here's what's at the end of all of the other data nodes logs:

RESTORE table: 655 32 rows applied
RESTORE table: 655 38 rows applied
Applying undo to LCP: 61183
2012-11-24 07:55:06 [ndbd] INFO     -- Undo head - undo_1.log page: 321844 lsn: 245995614
send lock node 49 waiting for lock, contentions: 200 spins: 363193
send lock node 49 waiting for lock, contentions: 400 spins: 695011
send lock node 49 waiting for lock, contentions: 600 spins: 1194842

Node 49 in the management node.

Finally, here's the pertinent entries in the config.ini (only thing excluded are the data node hostnames):

[ndbd default]
NoOfReplicas           = 2      # Number of replicas
DataMemory             = 30G    # How much memory to allocate for data storage (most data will be on disk, indexed attributes in memory)
IndexMemory            = 10G    # How much memory to allocate for index storage
MaxNoOfOrderedIndexes  = 1024
MaxNoOfAttributes      = 6000   # 6 times MaxNoOfTables
MaxNoOfTables          = 1000
TimeBetweenLocalCheckpoints = 8
ServerPort             = 2202
NoOfFragmentLogFiles   = 600
NoOfFragmentLogParts   = 8

ODirect                = 1
SchedulerSpinTimer     = 400
SchedulerExecutionTimer= 100
MaxNoOfConcurrentOperations = 100000
RealtimeScheduler      = 1

CompressedLCP          = 1

ThreadConfig=ldm={count=8,cpubind=1,2,3,4,5,6,7,8},main={cpubind=9},io={cpubind=9},rep={cpubind=10},tc={count=4,cpubind=11,12,13,14},recv={count=3,cpubind=15,16,17},send={count=3,cpubind=18,19,20}

The hardware is all quad-core 2.4Ghz Xeons, if that matters. top on the three "ndbmtd" data nodes shows that ndbmtd is using 100% CPU on one of its threads. On the "ndbmtd --initial" data node there is almost no activity.

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.