After adding a 3rd node to our 2 node 12c RAC, one of the existing node is now unable to start the database resource.
Extract from crsctl stat res -t:
1 ONLINE ONLINE rac1 Open,STABLE 2 ONLINE OFFLINE STABLE 3 ONLINE ONLINE rac3 Open,STABLE
Any attempt to start the resource (manual or automatic) gives the following:
C:\Windows\system32>crsctl start res ora.datacdb.db
CRS-2672: Attempting to start 'ora.datacdb.db' on 'rac2'
ORA-12560: TNS:protocol adapter error
CRS-5017: The resource action "ora.datacdb.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "C:\app\12.1.0\grid\log\rac2\agent\crsd\oraagent_system\oraagent_system.log".
CRS-2674: Start of 'ora.datacdb.db' on 'rac2' failed
CRS-2679: Attempting to clean 'ora.datacdb.db' on 'rac2'
CRS-2681: Clean of 'ora.datacdb.db' on 'rac2' succeeded
CRS-2528: Unable to place an instance of 'ora.datacdb.db' as all possible servers are occupied by the resource
CRS-4000: Command Start failed, or completed with errors.
If I try to connect to the instance via a bequeath connection I get either the same TNS error, or a ORA-27101 Shared Realm error.
I already have an Oracle SR open for about a week, but so far it's not yielded anything constructive.