ORA-00205: error in identifying control file on standby

I was trying to mount my DR system to synchronise after an activity.But i faced some issues where i was not able to mount the database because of missing controlfile.

I checked the alertlog and i found these errors on the trace file

Errors in file /oracle/base/diag/rdbms/orcldg/orcldg/trace/orcldg_asmb_6031.trc:
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
Stopping background process RBAL

ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/orcldg/control.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orcldg/control.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-29701: unable to connect to Cluster Synchronization Service
ORA-205 signalled during: alter database mount...

I was clear that ASM was offline and cluster services were not up and running for any of the diskgroups

[oracle@orcldg ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               OFFLINE OFFLINE      orcldg                   STABLE
ora.DATA.dg
               OFFLINE OFFLINE      orcldg                   STABLE
ora.FRA.dg
               OFFLINE OFFLINE      orcldg                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       orcldg                   STABLE
ora.asm
               OFFLINE OFFLINE      orcldg                   Instance Shutdown,ST
                                                             ABLE
ora.ons
               OFFLINE OFFLINE      orcldg                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                               STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orcldg                   STABLE
--------------------------------------------------------------------------------

Then i started the ASM resource manually.Then the services were all UP.

[oracle@orcldg ~]$ srvctl start asm
[oracle@orcldg ~]$  crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       orcldg                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       orcldg                   STABLE
ora.FRA.dg
               ONLINE  ONLINE       orcldg                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       orcldg                   STABLE
ora.asm
               ONLINE  ONLINE       orcldg                   Started,STABLE
ora.ons
               OFFLINE OFFLINE      orcldg                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       orcldg                   STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       orcldg                   STABLE
--------------------------------------------------------------------------------

Then i started the database and mounted in recovery mode.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1140850688 bytes
Fixed Size                  2923584 bytes
Variable Size             419431360 bytes
Database Buffers          704643072 bytes
Redo Buffers               13852672 bytes
Database mounted.
SQL> alter database recover managed standby database disconnect from session;

Database altered.

The DR was in sync after starting the recovery mode.

MESSAGE
--------------------------------------------------------------------------------

Media Recovery Log +FRA/ORCLDG/ARCHIVELOG/2020_05_29/thread_1_seq_1331.522.10417
22827

Media Recovery Waiting for thread 1 sequence 1332 (in transit)

Root cause:

[root@orcldg oracle]# free -m
             total       used       free     shared    buffers     cached
Mem:          3696       2889        807       1710         44       2169
-/+ buffers/cache:        675       3020
Swap:         2095          0       2095

I didnt had sufficient memory to handle the ASM disk processes.

I had set hardcoded shell script on crontab to run for every 5 minutes.

*/5 * * * * /home/oracle/tblspce.sh > /dev/null

The script processes were piling up and stacked in a queue without getting released from the server.

[oracle@orcldg ~]$ ps -ef|grep oratbl.sh
oracle    6645  6519  0 23:50 pts/0    00:00:00 grep oratbl.sh
oracle    6647  6519  0 23:50 pts/0    00:00:00 grep oratbl.sh
oracle    6649  6519  0 23:50 pts/0    00:00:00 grep oratbl.sh
oracle    6651  6519  0 23:50 pts/0    00:00:00 grep oratbl.sh


After killing the processes i was able to start the ASM

This caused the ASM process to hung and not able to identify the controlfile.

ORA-00205: error in identifying control file on standby

Published by kishan

You will know about me.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: