In oracle rac, it is often tedious to diagnose and troubleshoot crs startup issues during a production downtime or any activity
We have to react quickly to understand the cause of issue and try to narrow down issue to resolve the error.
[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Start failed, or completed with errors.
Check if crs daemon is up and running on your system
[root@exdbadm01 oracle]# ps -ef|grep crsd.bin|grep -v 'grep'
root 5209 1 0 20:20 ? 00:00:03 /u01/app/11.2.0/bin/crsd.bin reboot
Check for errors in crsd log and try to narrow down the cause of issue
[root@exdbadm01 oracle]# grep 'error' /u01/app/11.2.0/log/exdbadm01/crsd/crsd.log|tail
2021-06-14 16:21:20.192: [ CRSPE][4139338048]{2:26915:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b64f813f270]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:16:12.605: [ CRSPE][2957285696]{1:14708:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-13 17:16:13.957: [ CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2b71b8015040]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-13 17:16:15.045: [ CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2b71b8002640]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:17:01.733: [ CRSPE][2957285696]{2:42190:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b71b800a680]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:20:33.178: [ CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-21 20:20:36.485: [ CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2aaf30001b80]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-21 20:20:38.458: [ CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2aaf300008f0]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:31:05.427: [ CRSPE][778697024]{1:44648:2} Start action failed with error code: 3
2021-07-21 20:31:05.429: [ CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbx.db 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbx.db'.
Use clusterware commands using crsctl utility
[root@exdbadm01 oracle]# . oraenv
ORACLE_SID = [exdbx] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[root@exdbadm01 oracle]# crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
Verify the inittab.crs script and try to find if there are any modifications
[root@exdbadm01 oracle]# grep '/etc/init.d/init.ohasd' /etc/inittab.crs
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
Check the init.ohasd script for any missing lines
[root@exdbadm01 oracle]# cat /etc/init.d/init.ohasdTry to start the cluster
[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.mdnsd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'exdbadm01'
CRS-2676: Start of 'ora.mdnsd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exdbadm01'
CRS-2676: Start of 'ora.gpnpd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exdbadm01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'exdbadm01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2676: Start of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'exdbadm01'
CRS-2676: Start of 'ora.evmd' on 'exdbadm01' succeeded
CRS-5702: Resource 'ora.evmd' is already running on 'exdbadm01'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'exdbadm01'
CRS-2681: Clean of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'exdbadm01'
CRS-2676: Start of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'exdbadm01'
CRS-2676: Start of 'ora.crsd' on 'exdbadm01' succeeded
Check the status of the clusterware resources
[root@exdbadm01 oracle]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.FRA.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.LISTENER.lsnr
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.MGMT.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.asm
ONLINE ONLINE exdbadm01 Started
ONLINE ONLINE exdbadm02 Started
ora.gsd
OFFLINE OFFLINE exdbadm01
OFFLINE OFFLINE exdbadm02
ora.net1.network
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.ons
ONLINE ONLINE exdbadm01
ONLINE OFFLINE exdbadm02
ora.registry.acfs
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE exdbadm01
ora.cvu
1 ONLINE ONLINE exdbadm01
ora.exdbadm01.vip
1 ONLINE ONLINE exdbadm01
ora.exdbadm02.vip
1 ONLINE ONLINE exdbadm02
ora.exdbx.db
1 ONLINE OFFLINE STARTING
2 ONLINE OFFLINE
ora.oc4j
1 ONLINE ONLINE exdbadm01
ora.scan1.vip
1 ONLINE ONLINE exdbadm01
