In oracle rac, it is often tedious to diagnose and troubleshoot crs startup issues during a production downtime or any activity

We have to react quickly to understand the cause of issue and try to narrow down issue to resolve the error.

[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Start failed, or completed with errors.

Check if crs daemon is up and running on your system

[root@exdbadm01 oracle]#  ps -ef|grep crsd.bin|grep -v 'grep'
root      5209     1  0 20:20 ?        00:00:03 /u01/app/11.2.0/bin/crsd.bin reboot

Check for errors in crsd log and try to narrow down the cause of issue

[root@exdbadm01 oracle]# grep 'error' /u01/app/11.2.0/log/exdbadm01/crsd/crsd.log|tail
2021-06-14 16:21:20.192: [   CRSPE][4139338048]{2:26915:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b64f813f270]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:16:12.605: [   CRSPE][2957285696]{1:14708:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-13 17:16:13.957: [   CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2b71b8015040]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-13 17:16:15.045: [   CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2b71b8002640]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:17:01.733: [   CRSPE][2957285696]{2:42190:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b71b800a680]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:20:33.178: [   CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-21 20:20:36.485: [   CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2aaf30001b80]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-21 20:20:38.458: [   CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2aaf300008f0]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:31:05.427: [   CRSPE][778697024]{1:44648:2} Start action failed with error code: 3
2021-07-21 20:31:05.429: [   CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbx.db 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbx.db'.

Use clusterware commands using crsctl utility

[root@exdbadm01 oracle]# . oraenv
ORACLE_SID = [exdbx] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[root@exdbadm01 oracle]# crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

Verify the inittab.crs script and try to find if there are any modifications

[root@exdbadm01 oracle]# grep '/etc/init.d/init.ohasd' /etc/inittab.crs
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

Check the init.ohasd script for any missing lines

[root@exdbadm01 oracle]# cat /etc/init.d/init.ohasd

Check the runlevels of the init bootloader

[root@exdbadm01 ~]# cat /etc/inittab|grep 'id'
id:3:initdefault:

Check if crs autostart is disabled

[root@exdbadm01 ~]# crsctl config crs
CRS-4622: Oracle High Availability Services autostart is enabled.

[root@exdbadm01 ~]# crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.

Try to start the cluster

[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.mdnsd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'exdbadm01'
CRS-2676: Start of 'ora.mdnsd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exdbadm01'
CRS-2676: Start of 'ora.gpnpd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exdbadm01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'exdbadm01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2676: Start of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'exdbadm01'
CRS-2676: Start of 'ora.evmd' on 'exdbadm01' succeeded
CRS-5702: Resource 'ora.evmd' is already running on 'exdbadm01'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'exdbadm01'
CRS-2681: Clean of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'exdbadm01'
CRS-2676: Start of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'exdbadm01'
CRS-2676: Start of 'ora.crsd' on 'exdbadm01' succeeded

Check the status of the clusterware resources

[root@exdbadm01 oracle]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.FRA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.LISTENER.lsnr
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.MGMT.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.asm
               ONLINE  ONLINE       exdbadm01                Started
               ONLINE  ONLINE       exdbadm02                Started
ora.gsd
               OFFLINE OFFLINE      exdbadm01
               OFFLINE OFFLINE      exdbadm02
ora.net1.network
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.ons
               ONLINE  ONLINE       exdbadm01
               ONLINE  OFFLINE      exdbadm02
ora.registry.acfs
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       exdbadm01
ora.cvu
      1        ONLINE  ONLINE       exdbadm01
ora.exdbadm01.vip
      1        ONLINE  ONLINE       exdbadm01
ora.exdbadm02.vip
      1        ONLINE  ONLINE       exdbadm02
ora.exdbx.db
      1        ONLINE  OFFLINE                               STARTING
      2        ONLINE  OFFLINE
ora.oc4j
      1        ONLINE  ONLINE       exdbadm01
ora.scan1.vip
      1        ONLINE  ONLINE       exdbadm01