RAC clusterware commands
To stop the cluster resources on all nodes in RAC cluster
[root@exdbadm01 oracle]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.crsd' on 'exdbadm02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'exdbadm01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.cvu' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.DATA2.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.MGMT.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'exdbadm02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.exdbadm01.vip' on 'exdbadm01'
CRS-2677: Stop of 'ora.DATA.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'exdbadm02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.exdbadm02.vip' on 'exdbadm02'
CRS-2677: Stop of 'ora.FRA.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.exdbadm01.vip' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.exdbadm02.vip' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.DATA2.dg' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.MGMT.dg' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm01'
CRS-2677: Stop of 'ora.MGMT.dg' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm02'
CRS-2677: Stop of 'ora.asm' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'exdbadm01'
CRS-2677: Stop of 'ora.asm' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.ons' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exdbadm01'
CRS-2677: Stop of 'ora.net1.network' on 'exdbadm01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exdbadm01' has completed
CRS-2677: Stop of 'ora.crsd' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.evmd' on 'exdbadm01'
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm01'
CRS-2677: Stop of 'ora.evmd' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.asm' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exdbadm01'
CRS-2677: Stop of 'ora.oc4j' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'exdbadm02'
CRS-2677: Stop of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exdbadm01'
CRS-2677: Stop of 'ora.ons' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'exdbadm02'
CRS-2677: Stop of 'ora.net1.network' on 'exdbadm02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'exdbadm02' has completed
CRS-2677: Stop of 'ora.crsd' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.evmd' on 'exdbadm02'
CRS-2673: Attempting to stop 'ora.asm' on 'exdbadm02'
CRS-2677: Stop of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.asm' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'exdbadm02'
CRS-2677: Stop of 'ora.ctssd' on 'exdbadm02' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'exdbadm02'
CRS-2677: Stop of 'ora.cssd' on 'exdbadm02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'exdbadm02'
CRS-2677: Stop of 'ora.diskmon' on 'exdbadm02' succeeded
To start cluster resources on local node
[root@exdbadm01 oracle]# crsctl start cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'exdbadm01'
CRS-2676: Start of 'ora.mdnsd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'exdbadm01'
CRS-2676: Start of 'ora.gpnpd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exdbadm01'
CRS-2676: Start of 'ora.cssdmonitor' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'exdbadm01'
CRS-2676: Start of 'ora.ctssd' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'exdbadm01'
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'exdbadm01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'exdbadm01'
CRS-2676: Start of 'ora.evmd' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'exdbadm01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'exdbadm01'
CRS-2676: Start of 'ora.asm' on 'exdbadm01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'exdbadm01'
CRS-2676: Start of 'ora.crsd' on 'exdbadm01' succeeded
To start database instances on all nodes
[oracle@exdbadm01 ~]$ srvctl start database -d exdbx
[oracle@exdbadm01 ~]$
To check the database configuration
[oracle@exdbadm01 ~]$ srvctl config database -d exdbx
Database unique name: exdbx
Database name: exdbx
Oracle home: /data/oracle/11.2.0/db_home
Oracle user: oracle
Spfile: +DATA/exdbx/spfileexdbx.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: exdbx
Database instances: exdbx1,exdbx2
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@exdbadm01 ~]$
To check the status of the cluster resource.Note that this report is very important while troubleshoot of RAC issues
[oracle@exdbadm01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.DATA2.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.FRA.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.LISTENER.lsnr
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.MGMT.dg
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.asm
ONLINE ONLINE exdbadm01 Started
ONLINE ONLINE exdbadm02 Started
ora.gsd
OFFLINE OFFLINE exdbadm01
OFFLINE OFFLINE exdbadm02
ora.net1.network
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.ons
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
ora.registry.acfs
ONLINE ONLINE exdbadm01
ONLINE ONLINE exdbadm02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE exdbadm02
ora.cvu
1 ONLINE ONLINE exdbadm02
ora.exdbadm01.vip
1 ONLINE ONLINE exdbadm01
ora.exdbadm02.vip
1 ONLINE ONLINE exdbadm02
ora.exdbx.db
1 ONLINE ONLINE exdbadm01 Open
2 ONLINE ONLINE exdbadm02 Open
ora.oc4j
1 ONLINE ONLINE exdbadm02
ora.scan1.vip
1 ONLINE ONLINE exdbadm02
To check the status of the database instances on all nodes
[oracle@exdbadm01 ~]$ srvctl status database -d exdbx
Instance exdbx1 is running on node exdbadm01
Instance exdbx2 is running on node exdbadm02
To stop an instance
[oracle@exdbadm01 ~]$ srvctl stop instance -d exdbx -i exdbx1
To start an instance
[oracle@exdbadm01 ~]$ srvctl start instance -d exdbx -i exdbx1
OCR and votedisk
To check ocr integrity between each nodes and corruption check
[oracle@exdbadm01 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3128
Available space (kbytes) : 258992
ID : 1861193736
Device/File Name : +MGMT
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
To display all the backup of ocr file
[oracle@exdbadm01 ~]$ ocrconfig -showbackup
exdbadm02 2021/03/28 20:26:49 /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup00.ocr
exdbadm02 2021/03/28 16:26:48 /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup01.ocr
exdbadm02 2021/03/26 22:44:59 /u01/app/11.2.0/grid/cdata/exdbadm-scan/backup02.ocr
exdbadm02 2021/03/28 16:26:48 /u01/app/11.2.0/grid/cdata/exdbadm-scan/day.ocr
exdbadm02 2021/03/15 16:34:25 /u01/app/11.2.0/grid/cdata/exdbadm-scan/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
To check the voting disk
[oracle@exdbadm01 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 65371b4d26a54fa0bfce069c3b2d52ae (o/192.168.56.33/MGMT_CD_cell15_stocell) [MGMT]
Located 1 voting disk(s).
To display the olr backup
[oracle@exdbadm01 ~]$ ocrconfig -local -showbackup
exdbadm01 2021/02/05 12:16:03 /u01/app/11.2.0/grid/cdata/exdbadm01/backup_20210205_121603.olr
To get the hostname and current nodename
[oracle@exdbadm01 ~]$ crsctl get hostname
exdbadm01
[oracle@exdbadm01 ~]$ crsctl get nodename
exdbadm01
Network commands in RAC
To check the status of scan
[oracle@exdbadm01 ~]$ srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node exdbadm01
To check scan configuration
[oracle@exdbadm01 ~]$ srvctl config scan
SCAN name: exdbadm-scan, Network: ---
SCAN VIP name: scan1, IP: ---
To check the nodeapps status
[oracle@exdbadm01 ~]$ srvctl status nodeapps
VIP exdbadm01-vip is enabled
VIP exdbadm01-vip is running on node: exdbadm01
VIP exdbadm02-vip is enabled
VIP exdbadm02-vip is running on node: exdbadm02
Network is enabled
Network is running on node: exdbadm01
Network is running on node: exdbadm02
GSD is disabled
GSD is not running on node: exdbadm01
GSD is not running on node: exdbadm02
ONS is enabled
ONS daemon is running on node: exdbadm01
ONS daemon is running on node: exdbadm02
To check the status of network interface
[oracle@exdbadm01 ~]$ oifcfg getif
eth0 192.168.56.0 global public
eth1 192.168.57.0 global cluster_interconnect
Other commands
To check the list of nodes in cluster
[oracle@exdbadm01 ~]$ olsnodes -n -s -t
exdbadm01 1 Active Unpinned
exdbadm02 2 Active Unpinned
To check ocr integrity with cluster utility
[oracle@exdbadm01 ~]$ cluvfy comp ocr
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all specified nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+MGMT" available on all the nodes
NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Verification of OCR integrity was successful.
To check grid versions
[oracle@exdbadm01 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[oracle@exdbadm01 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.4.0]
[oracle@exdbadm01 ~]$ crsctl query crs softwareversion
Oracle Clusterware version on node [exdbadm01] is [11.2.0.4.0]
To check crs admin
[oracle@exdbadm01 ~]$ crsctl query crs administrator
CRS Administrator List: *