RAC clusterware commands

To stop the cluster resources on all nodes in RAC cluster

[root@exdbadm01 oracle]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.crsd' on 'exdbadm02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'exdbadm01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.cvu' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'exdbadm02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.exdbadm01.vip' on 'exdbadm01'
CRS-2677: Stop of 'ora.DATA.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exdbadm02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.exdbadm02.vip' on 'exdbadm02'
CRS-2677: Stop of 'ora.FRA.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.exdbadm01.vip' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.exdbadm02.vip' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.MGMT.dg' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm01'
CRS-2677: Stop of 'ora.MGMT.dg' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm02'
CRS-2677: Stop of 'ora.asm' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'exdbadm01'
CRS-2677: Stop of 'ora.asm' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.ons' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exdbadm01'
CRS-2677: Stop of 'ora.net1.network' on 'exdbadm01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exdbadm01' has completed
CRS-2677: Stop of 'ora.crsd' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.evmd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm01'
CRS-2677: Stop of 'ora.evmd' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.asm' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exdbadm01'
CRS-2677: Stop of 'ora.oc4j' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'exdbadm02'
CRS-2677: Stop of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exdbadm01'
CRS-2677: Stop of 'ora.ons' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exdbadm02'
CRS-2677: Stop of 'ora.net1.network' on 'exdbadm02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exdbadm02' has completed
CRS-2677: Stop of 'ora.crsd' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.evmd' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm02'
CRS-2677: Stop of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.asm' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exdbadm02'
CRS-2677: Stop of 'ora.ctssd' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exdbadm02'
CRS-2677: Stop of 'ora.cssd' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exdbadm02'
CRS-2677: Stop of 'ora.diskmon' on 'exdbadm02' succeeded

To start cluster resources on local node

[root@exdbadm01 oracle]# crsctl start cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'exdbadm01'
CRS-2676: Start of 'ora.mdnsd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exdbadm01'
CRS-2676: Start of 'ora.gpnpd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exdbadm01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'exdbadm01'
CRS-2676: Start of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'exdbadm01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2676: Start of 'ora.evmd' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'exdbadm01'
CRS-2676: Start of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'exdbadm01'
CRS-2676: Start of 'ora.crsd' on 'exdbadm01' succeeded

To start database instances on all nodes

[oracle@exdbadm01 ~]$ srvctl start database -d exdbx
[oracle@exdbadm01 ~]$

To check the database configuration

[oracle@exdbadm01 ~]$ srvctl config database -d exdbx
Database unique name: exdbx
Database name: exdbx
Oracle home: /data/oracle/11.2.0/db_home
Oracle user: oracle
Spfile: +DATA/exdbx/spfileexdbx.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: exdbx
Database instances: exdbx1,exdbx2
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@exdbadm01 ~]$

To check the status of the cluster resource.Note that this report is very important while troubleshoot of RAC issues

[oracle@exdbadm01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.DATA2.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.FRA.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.LISTENER.lsnr
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.MGMT.dg
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.asm
               ONLINE  ONLINE       exdbadm01                Started
               ONLINE  ONLINE       exdbadm02                Started
ora.gsd
               OFFLINE OFFLINE      exdbadm01
               OFFLINE OFFLINE      exdbadm02
ora.net1.network
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.ons
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
ora.registry.acfs
               ONLINE  ONLINE       exdbadm01
               ONLINE  ONLINE       exdbadm02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       exdbadm02
ora.cvu
      1        ONLINE  ONLINE       exdbadm02
ora.exdbadm01.vip
      1        ONLINE  ONLINE       exdbadm01
ora.exdbadm02.vip
      1        ONLINE  ONLINE       exdbadm02
ora.exdbx.db
      1        ONLINE  ONLINE       exdbadm01                Open
      2        ONLINE  ONLINE       exdbadm02                Open
ora.oc4j
      1        ONLINE  ONLINE       exdbadm02
ora.scan1.vip
      1        ONLINE  ONLINE       exdbadm02

To check the status of the database instances on all nodes

[oracle@exdbadm01 ~]$ srvctl status database -d exdbx
Instance exdbx1 is running on node exdbadm01
Instance exdbx2 is running on node exdbadm02

To stop an instance

[oracle@exdbadm01 ~]$ srvctl stop instance -d exdbx -i exdbx1

To start an instance

[oracle@exdbadm01 ~]$ srvctl start instance -d exdbx -i exdbx1

OCR and votedisk

To check ocr integrity between each nodes and corruption check

[oracle@exdbadm01 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3128
         Available space (kbytes) :     258992
         ID                       : 1861193736
         Device/File Name         :      +MGMT
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

To display all the backup of ocr file

[oracle@exdbadm01 ~]$ ocrconfig -showbackup

exdbadm02     2021/03/28 20:26:49     /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup00.ocr

exdbadm02     2021/03/28 16:26:48     /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup01.ocr

exdbadm02     2021/03/26 22:44:59     /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup02.ocr

exdbadm02     2021/03/28 16:26:48     /u01/app/11.2.0/grid/cdata/exdbadm-scan/day.ocr

exdbadm02     2021/03/15 16:34:25     /u01/app/11.2.0/grid/cdata/exdbadm-scan/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

To check the voting disk

[oracle@exdbadm01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   65371b4d26a54fa0bfce069c3b2d52ae (o/192.168.56.33/MGMT_CD_cell15_stocell) [MGMT]
Located 1 voting disk(s).

To display the olr backup

[oracle@exdbadm01 ~]$ ocrconfig -local -showbackup

exdbadm01     2021/02/05 12:16:03     /u01/app/11.2.0/grid/cdata/exdbadm01/backup_20210205_121603.olr

To get the hostname and current nodename

[oracle@exdbadm01 ~]$ crsctl get hostname
exdbadm01
[oracle@exdbadm01 ~]$ crsctl get nodename
exdbadm01

Network commands in RAC

To check the status of scan

[oracle@exdbadm01 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node exdbadm01

To check scan configuration

[oracle@exdbadm01 ~]$ srvctl config scan
SCAN name: exdbadm-scan, Network: ---
SCAN VIP name: scan1, IP: ---

To check the nodeapps status

[oracle@exdbadm01 ~]$ srvctl status nodeapps
VIP exdbadm01-vip is enabled
VIP exdbadm01-vip is running on node: exdbadm01
VIP exdbadm02-vip is enabled
VIP exdbadm02-vip is running on node: exdbadm02
Network is enabled
Network is running on node: exdbadm01
Network is running on node: exdbadm02
GSD is disabled
GSD is not running on node: exdbadm01
GSD is not running on node: exdbadm02
ONS is enabled
ONS daemon is running on node: exdbadm01
ONS daemon is running on node: exdbadm02

To check the status of network interface

[oracle@exdbadm01 ~]$ oifcfg getif
eth0  192.168.56.0  global  public
eth1  192.168.57.0  global  cluster_interconnect

Other commands

To check the list of nodes in cluster

[oracle@exdbadm01 ~]$ olsnodes -n -s -t
exdbadm01       1       Active  Unpinned
exdbadm02       2       Active  Unpinned

To check ocr integrity with cluster utility

[oracle@exdbadm01 ~]$ cluvfy comp ocr

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+MGMT" available on all the nodes


NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.

To check grid versions

[oracle@exdbadm01 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

[oracle@exdbadm01 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]

[oracle@exdbadm01 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [exdbadm01] is [11.2.0.4.0]

To check crs admin

[oracle@exdbadm01 ~]$ crsctl query crs administrator
CRS Administrator List: *

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s