I was trying to mount my DR system to synchronise after an activity.But i faced some issues where i was not able to mount the database because of missing controlfile.
I checked the alertlog and i found these errors on the trace file
Errors in file /oracle/base/diag/rdbms/orcldg/orcldg/trace/orcldg_asmb_6031.trc: ORA-15077: could not locate ASM instance serving a required diskgroup ORA-29701: unable to connect to Cluster Synchronization Service Stopping background process RBAL
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/orcldg/control.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orcldg/control.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-29701: unable to connect to Cluster Synchronization Service
ORA-205 signalled during: alter database mount...
I was clear that ASM was offline and cluster services were not up and running for any of the diskgroups
[oracle@orcldg ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
OFFLINE OFFLINE orcldg STABLE
ora.DATA.dg
OFFLINE OFFLINE orcldg STABLE
ora.FRA.dg
OFFLINE OFFLINE orcldg STABLE
ora.LISTENER.lsnr
ONLINE ONLINE orcldg STABLE
ora.asm
OFFLINE OFFLINE orcldg Instance Shutdown,ST
ABLE
ora.ons
OFFLINE OFFLINE orcldg STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 OFFLINE OFFLINE STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE orcldg STABLE
--------------------------------------------------------------------------------
Then i started the ASM resource manually.Then the services were all UP.
[oracle@orcldg ~]$ srvctl start asm
[oracle@orcldg ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE orcldg STABLE
ora.DATA.dg
ONLINE ONLINE orcldg STABLE
ora.FRA.dg
ONLINE ONLINE orcldg STABLE
ora.LISTENER.lsnr
ONLINE ONLINE orcldg STABLE
ora.asm
ONLINE ONLINE orcldg Started,STABLE
ora.ons
OFFLINE OFFLINE orcldg STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE orcldg STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE orcldg STABLE
--------------------------------------------------------------------------------
Then i started the database and mounted in recovery mode.
SQL> startup mount; ORACLE instance started. Total System Global Area 1140850688 bytes Fixed Size 2923584 bytes Variable Size 419431360 bytes Database Buffers 704643072 bytes Redo Buffers 13852672 bytes Database mounted. SQL> alter database recover managed standby database disconnect from session; Database altered.
The DR was in sync after starting the recovery mode.
MESSAGE
--------------------------------------------------------------------------------
Media Recovery Log +FRA/ORCLDG/ARCHIVELOG/2020_05_29/thread_1_seq_1331.522.10417
22827
Media Recovery Waiting for thread 1 sequence 1332 (in transit)
Root cause:
[root@orcldg oracle]# free -m
total used free shared buffers cached
Mem: 3696 2889 807 1710 44 2169
-/+ buffers/cache: 675 3020
Swap: 2095 0 2095
I didnt had sufficient memory to handle the ASM disk processes.
I had set hardcoded shell script on crontab to run for every 5 minutes.
*/5 * * * * /home/oracle/tblspce.sh > /dev/null
The script processes were piling up and stacked in a queue without getting released from the server.
[oracle@orcldg ~]$ ps -ef|grep oratbl.sh oracle 6645 6519 0 23:50 pts/0 00:00:00 grep oratbl.sh oracle 6647 6519 0 23:50 pts/0 00:00:00 grep oratbl.sh oracle 6649 6519 0 23:50 pts/0 00:00:00 grep oratbl.sh oracle 6651 6519 0 23:50 pts/0 00:00:00 grep oratbl.sh
After killing the processes i was able to start the ASM
This caused the ASM process to hung and not able to identify the controlfile.
ORA-00205: error in identifying control file on standby
