Oracle 12c RAC Cluster / CRS fails to start after reboot

Question

Could someone shed some light to help fix the cluster/CRS startup failure

1) /oragrid/app/grid_home/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

2) /oragrid/app/grid_home/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

3) /oragrid/app/grid_home/bin/crsctl start res ora.crsd -init

CRS-2672: Attempting to start 'ora.gpnpd' on 'xxxxxxxxxx'

CRS-2674: Start of 'ora.gpnpd' on 'xxxxxxxxxx' failed

4) /oragrid/app/grid_home/bin/crsctl start cluster

CRS-2672: Attempting to start 'ora.gpnpd' on 'xxxxxxxx'

CRS-2674: Start of 'ora.gpnpd' on 'xxxxxxxx' failed

Answer 1

First you need to analyze log files ( CRS and ASM ). Then check the status of the disks in the ASM and pay attention to the disk where it is located OCR.

Maybe : CRS-1013:The OCR location in an ASM disk group is inaccessible.

Additionally:

# Check disks in ASM Diskgroup

set lines 999;
col diskgroup for a15
col diskname for a16
col path for a35

select a.name DiskGroup,b.name DiskName, b.MOUNT_STATUS, b.total_mb, (b.total_mb-b.free_mb) Used_MB, b.free_mb,b.path,b.header_status
from v$asm_disk b, v$asm_diskgroup a
where a.group_number (+) =b.group_number
order by b.group_number,b.name;

-- MOUNT_STATUS (CLOSED) !!

Then, check the availability of disks, and restart the nodes in the cluster

Answer 2

I'm not a pro but I have tested this on my personal laptop. There is another way to work around like creating a shell script and then set it on crontab for example:

su - root
pwd :- /home/root 

to have an idea about whereis location of oracleasm pkg :- # whereis oracleasm

cat oracleasm_script.sh 



#!/bin/sh
/usr/bin/oracleasm init 
/usr/bin/oracleasm scandisks
/usr/bin/oracleasm listdisks

chown -R oracle:oinstall /dev/oracleasm/disks
chmod -R 777 /dev/oracleasm/disks

1. su - root
2. crontab -e
3. @reboot /root/oracleasm_script.sh > /dev/null 2>&1

Answer 3

ASM manual startup:

[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/bin
[oracle@rac1 bin]$ ./crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
[oracle@rac1 bin]$ 
[oracle@rac1 bin]$ su - root
Password: 
[root@rac1 ~]# cd /u01/app/11.2.0/grid/bin
[root@rac1 bin]# ./crsctl start crs
CRS-4124: Oracle High Availability Services startup failed
CRS-4000: Command Start failed, or completed with errors
[root@rac1 bin]#

[root@rac1 bin]# ./crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@rac1 bin]# 


[root@rac1 bin]# nohup /etc/init.d/init.ohasd run &
press Enter:
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

[root@rac1 bin]# cd /dev/oracleasm/disks
[root@rac1 disks]# ls -lrt
total 0
brw------- 1 root root 8, 17 May  6 10:16 DISK1
brw------- 1 root root 8, 33 May  6 10:16 DISK2
brw------- 1 root root 8, 49 May  6 10:16 DISK3
brw------- 1 root root 8, 65 May  6 10:16 DISK4
brw------- 1 root root 8, 81 May  6 10:16 DISK5

[root@rac1 bin]# ps -ef  | grep css
root      3784     1  0 10:17 ?        00:00:01 /u01/app/grid/11.2.0/bin/cssdmonitor
root      3801     1  0 10:17 ?        00:00:01 /u01/app/grid/11.2.0/bin/cssdagent
root      4189  4107  0 10:26 pts/1    00:00:00 grep css

[root@rac1 disks]# chown -R oracle:dba /dev/oracleasm/disks
[root@rac1 disks]# chmod -R 777 /dev/oracleasm/disks
[root@rac1 disks]# ls -lrt
total 0
brwxrwxrwx 1 oracle dba 8, 17 May  6 10:16 DISK1
brwxrwxrwx 1 oracle dba 8, 33 May  6 10:16 DISK2
brwxrwxrwx 1 oracle dba 8, 49 May  6 10:16 DISK3
brwxrwxrwx 1 oracle dba 8, 65 May  6 10:16 DISK4
brwxrwxrwx 1 oracle dba 8, 81 May  6 10:16 DISK5

[root@rac1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'rac1'
CRS-2681: Clean of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded

[root@rac1 bin]# ps -ef | grep css
root      3784     1  0 10:17 ?        00:00:01 /u01/app/grid/11.2.0/bin/cssdmonitor
root      4372     1  0 10:30 ?        00:00:01 /u01/app/grid/11.2.0/bin/cssdagent
oracle    4387     1  1 10:30 ?        00:00:02 /u01/app/grid/11.2.0/bin/ocssd.bin
root      5347  4107  0 10:33 pts/1    00:00:00 grep css


[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac1 bin]#


Su – oracle

Srvctl start database –d dbname 

Create diskgroup FRA external redundancy ‘ORCL:VOL5’;(in ASM (sqlplus / as sysasm))

Asmcmd lsdg
Alter diskgroup FRA mount; (if not in mount).

http://mbc-dba.blogspot.com/2014/08/crs-4639-crs-4124-oracle-high.html
https://shivanandarao-oracle.com/2012/06/03/crs-4535-cannot-communicate-with-cluster-ready-services/