ORA-16038: log 1 sequence# 191 cannot be archived – ORA-19809: limit exceeded for recovery files – ORA-00312: online log 1 thread 1: ‘+DATA/exdbx/onlinelog/group_1.261.1063822485’ – ORA-00312: online log 1 thread 1: ‘+FRA/exdbx/onlinelog/group_1.257.1063822491’

Cause:

RAC database instances were not starting after investigating for long time.

ORA-16038: log 1 sequence# 191 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1: '+DATA/exdbx/onlinelog/group_1.261.1063822485'
ORA-00312: online log 1 thread 1: '+FRA/exdbx/onlinelog/group_1.257.1063822491'
USER (ospid: 17363): terminating the instance due to error 16038
Tue Mar 30 12:12:36 2021
System state dump requested by (instance=1, osid=17363), summary=[abnormal instance termination].
System State dumped to trace file /u01/base/diag/rdbms/exdbx/exdbx1/trace/exdbx1_diag_17302_20210330121236.trc
Dumping diagnostic data in directory=[cdmp_20210330121236], requested by (instance=1, osid=17363), summary=[abnormal instance termination].
Instance terminated by USER, pid = 17363

srvctl throw error with respect to archivelog sequence and ended up with end of file on communicatio channel

[oracle@exdbadm01 ~]$ srvctl start database -d exdbx
PRCR-1079 : Failed to start resource ora.exdbx.db
CRS-5017: The resource action "ora.exdbx.db start" encountered the following error:
ORA-16038: log 1 sequence# 191 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1: '+DATA/exdbx/onlinelog/group_1.261.1063822485'
ORA-00312: online log 1 thread 1: '+FRA/exdbx/onlinelog/group_1.257.1063822491'
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/exdbadm02/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.exdbx.db' on 'exdbadm02' failed
CRS-5017: The resource action "ora.exdbx.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 17363
Session ID: 1 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/exdbadm01/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.exdbx.db' on 'exdbadm01' failed
CRS-2632: There are no more servers to try to place resource 'ora.exdbx.db' on that would satisfy its placement policy

When i checked FRA,sufficient space was available but still error persists.

kish@exdbx<>show parameter db_recovery_file_dest

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      +FRA
db_recovery_file_dest_size           big integer 4868M

Workaround:

After investigating on the root cause, archivelog files were directly removed using OS utility and not cleared using RMAN graceful archive delete



RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;

After running above archivelog delete commands in RMAN,issue got cleared

[oracle@exdbadm01 lib]$ srvctl stop database -d exdbx                                                                 [oracle@exdbadm01 lib]$ srvctl start database -d exdbx
[oracle@exdbadm01 lib]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Mar 30 17:30:28 2021

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

kish@exdbx<>select status from gv$instance;

STATUS
------------
OPEN
OPEN

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s