TROUBLESHOOT RAC CRS STARTUP ISSUES

In oracle rac, it is often tedious to diagnose and troubleshoot crs startup issues during a production downtime or any activity

We have to react quickly to understand the cause of issue and try to narrow down issue to resolve the error.

[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Start failed, or completed with errors.

Check if crs daemon is up and running on your system

[root@exdbadm01 oracle]#  ps -ef|grep crsd.bin|grep -v 'grep'
root      5209     1  0 20:20 ?        00:00:03 /u01/app/11.2.0/bin/crsd.bin reboot

Check for errors in crsd log and try to narrow down the cause of issue

[root@exdbadm01 oracle]# grep 'error' /u01/app/11.2.0/log/exdbadm01/crsd/crsd.log|tail
2021-06-14 16:21:20.192: [   CRSPE][4139338048]{2:26915:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b64f813f270]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:16:12.605: [   CRSPE][2957285696]{1:14708:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-13 17:16:13.957: [   CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2b71b8015040]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-13 17:16:15.045: [   CRSPE][2957285696]{1:14708:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2b71b8002640]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-13 17:17:01.733: [   CRSPE][2957285696]{2:42190:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm02 1] on [exdbadm02] : local=0, unplanned=00x2b71b800a680]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:20:33.178: [   CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbadm02.vip 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbadm02.vip'.
2021-07-21 20:20:36.485: [   CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER_SCAN1.lsnr 1 1] on [exdbadm01] : local=0, unplanned=00x2aaf30001b80]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER_SCAN1.lsnr' has placement error.
2021-07-21 20:20:38.458: [   CRSPE][778697024]{1:44648:2} Re-evaluation of queued op [START of [ora.LISTENER.lsnr exdbadm01 1] on [exdbadm01] : local=0, unplanned=00x2aaf300008f0]. found it no longer needed:CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
2021-07-21 20:31:05.427: [   CRSPE][778697024]{1:44648:2} Start action failed with error code: 3
2021-07-21 20:31:05.429: [   CRSPE][778697024]{1:44648:2} Sequencer for [ora.exdbx.db 1 1] has completed with error: CRS-0215: Could not start resource 'ora.exdbx.db'.

Use clusterware commands using crsctl utility

[root@exdbadm01 oracle]# . oraenv
ORACLE_SID = [exdbx] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[root@exdbadm01 oracle]# crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

Verify the inittab.crs script and try to find if there are any modifications

[root@exdbadm01 oracle]# grep '/etc/init.d/init.ohasd' /etc/inittab.crs
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

Check the init.ohasd script for any missing lines

[root@exdbadm01 oracle]# cat /etc/init.d/init.ohasd

Try to start the cluster

[root@exdbadm01 oracle]# crsctl start cluster -all
CRS-2672: Attempting to start 'ora.mdnsd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'exdbadm01'
CRS-2676: Start of 'ora.mdnsd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exdbadm01'
CRS-2676: Start of 'ora.gpnpd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exdbadm01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'exdbadm01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2676: Start of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'exdbadm01'
CRS-2676: Start of 'ora.evmd' on 'exdbadm01' succeeded
CRS-5702: Resource 'ora.evmd' is already running on 'exdbadm01'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'exdbadm01'
CRS-2681: Clean of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'exdbadm01'
CRS-2676: Start of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'exdbadm01'
CRS-2676: Start of 'ora.crsd' on 'exdbadm01' succeeded

Check the status of the clusterware resources

[root@exdbadm01 oracle]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.FRA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.LISTENER.lsnr
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.MGMT.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.asm
               ONLINE  ONLINE       exdbadm01                Started
               ONLINE  ONLINE       exdbadm02                Started
ora.gsd
               OFFLINE OFFLINE      exdbadm01
               OFFLINE OFFLINE      exdbadm02
ora.net1.network
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.ons
               ONLINE  ONLINE       exdbadm01
               ONLINE  OFFLINE      exdbadm02
ora.registry.acfs
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       exdbadm01
ora.cvu
      1        ONLINE  ONLINE       exdbadm01
ora.exdbadm01.vip
      1        ONLINE  ONLINE       exdbadm01
ora.exdbadm02.vip
      1        ONLINE  ONLINE       exdbadm02
ora.exdbx.db
      1        ONLINE  OFFLINE                               STARTING
      2        ONLINE  OFFLINE
ora.oc4j
      1        ONLINE  ONLINE       exdbadm01
ora.scan1.vip
      1        ONLINE  ONLINE       exdbadm01

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s