Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

At least a few times a day, my main production server has recently begun failing over. Only, the quorum doesn't move to the other node.

I ran the validation report, and got a lot of info I don't understand. It is an HP ProLiant BL680c G5. Some of the info in the report that I'm wondering about:

Degraded:
HpCISSs
HP MPIO DSM for EVA4x00/6x00/8x00 family of Disk Arrays
Link-Layer Topology Discovery Mapper I/O Driver
Mount Point Manager

Those are just a few. I'm not so worried about "errors", but "degraded" seems to imply that it should be running but isn't doing so well.

System details:
sql - 10.50.4000.0
Windows - Windows NT - 64 Bit

It really is going to be awful if it IS an MPIO issue. That has happened many times, and the DBA team has been accused of modifying the settings! I'm the team lead, and even I have no idea what half of this means (I know what MPIO is, and I recognize all the SAN stuff, but troubleshooting? Nah.)

Interesting new information - right before this began happening, we updated the Firewall Service Module, and rebooted the core switches.

I'm thinking there are settings which don't have the correct defaults?

The adding/removing didn't work. It failed over yesterday evening.

share|improve this question
    
What are the logged cluster events/errors during the failover? –  Thomas Stringer Feb 6 '13 at 0:49
    
"Cluster Agent: The cluster resource FileServer-(server)(Cluster Disk 1) has failed. [SNMP TRAP: 15006 in CPQCLUS.MIB]" "Cluster Agent: The cluster resource SQL Server Agent has become degraded. [SNMP TRAP: 15005 in CPQCLUS.MIB]" "Cluster Agent: The cluster resource SQL Server has become degraded. [SNMP TRAP: 15005 in CPQCLUS.MIB]" "Cluster Agent: The cluster resource Analysis Services has become degraded. [SNMP TRAP: 15005 in CPQCLUS.MIB]" "Cluster Agent: The cluster resource FileServer-(servername)(Cluster Disk 4) has failed. [SNMP TRAP: 15006 in CPQCLUS.MIB]" –  Amanda Feb 6 '13 at 1:29
    
it is really odd, because there aren't a lot of error messages. the only real info i have is from the validation report. Disks 1-4 always fail, but not logged in the same order, and then the quarum just stays on the node it is on. –  Amanda Feb 6 '13 at 1:29
    
Ha! After speaking with the network guy, he thinks that perhaps when the core switch was rebooted, it affected the preferred routes for the NICs. He's going to clean things up a bit this weekend (removing the file shares and recreating them) and we'll see where we are. –  Amanda Feb 8 '13 at 19:52
    
Hi Amanda, you might want to click the "edit" button above and add your new details into your question, to make it easier for others to follow along. Reply with @jcolebrand if you have issues and I can assist (but I prefer to empower others and let them do the edits themselves :D ) –  jcolebrand Feb 8 '13 at 21:14
show 4 more comments

1 Answer

I'm not familiar with this particular error but I've encountered situations when a two-node cluster had multiple failovers due to MPIO issues with the SAN LUNs. More often than not, it was resolved by updating HBA drivers.

One other thing to look for is to ensure that the disk dependencies are properly set. The SQL Server service should be dependent on all that disks that are hosting the DB files and the backups as well as the disk with the drive letter acting as the mountpoint host. I've run into a few hosts where a missing disk dependency caused a disk to go offline before SQL could close out the DB files.

share|improve this answer
    
Microsoft Multi-Path Device Specific Module,Storage volumes, several of these - HP NC373i Multifunction Gigabit Server Adapter #2 are also "degraded" –  Amanda Feb 6 '13 at 1:34
    
I looked at a validation report from another SQL Server cluster, and it had the same "degredations" so I figure these are just normal for our environment. (there were a lot of other items that were "OK") HOWEVER, I compared the Device Managers and found that there are two multi-function gigabit adapters that are offline in the cluster that keeps rebooting. I sent the screenshots to our network guys and i'll see what they say. –  Amanda Feb 6 '13 at 21:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.