ORA-16038: log 1 sequence# 191 cannot be archived – ORA-19809: limit exceeded for recovery files – ORA-00312: online log 1 thread 1: ‘+DATA/exdbx/onlinelog/group_1.261.1063822485’ – ORA-00312: online log 1 thread 1: ‘+FRA/exdbx/onlinelog/group_1.257.1063822491’
Cause:
RAC database instances were not starting after investigating for long time.
ORA-16038: log 1 sequence# 191 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1: '+DATA/exdbx/onlinelog/group_1.261.1063822485'
ORA-00312: online log 1 thread 1: '+FRA/exdbx/onlinelog/group_1.257.1063822491'
USER (ospid: 17363): terminating the instance due to error 16038
Tue Mar 30 12:12:36 2021
System state dump requested by (instance=1, osid=17363), summary=[abnormal instance termination].
System State dumped to trace file /u01/base/diag/rdbms/exdbx/exdbx1/trace/exdbx1_diag_17302_20210330121236.trc
Dumping diagnostic data in directory=[cdmp_20210330121236], requested by (instance=1, osid=17363), summary=[abnormal instance termination].
Instance terminated by USER, pid = 17363
srvctl throw error with respect to archivelog sequence and ended up with end of file on communicatio channel
[oracle@exdbadm01 ~]$ srvctl start database -d exdbx
PRCR-1079 : Failed to start resource ora.exdbx.db
CRS-5017: The resource action "ora.exdbx.db start" encountered the following error:
ORA-16038: log 1 sequence# 191 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 1 thread 1: '+DATA/exdbx/onlinelog/group_1.261.1063822485'
ORA-00312: online log 1 thread 1: '+FRA/exdbx/onlinelog/group_1.257.1063822491'
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/exdbadm02/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.exdbx.db' on 'exdbadm02' failed
CRS-5017: The resource action "ora.exdbx.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 17363
Session ID: 1 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/exdbadm01/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.exdbx.db' on 'exdbadm01' failed
CRS-2632: There are no more servers to try to place resource 'ora.exdbx.db' on that would satisfy its placement policyWhen i checked FRA,sufficient space was available but still error persists.
kish@exdbx<>show parameter db_recovery_file_dest
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string +FRA
db_recovery_file_dest_size big integer 4868M
Workaround:
After investigating on the root cause, archivelog files were directly removed using OS utility and not cleared using RMAN graceful archive delete
RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;
After running above archivelog delete commands in RMAN,issue got cleared
[oracle@exdbadm01 lib]$ srvctl stop database -d exdbx [oracle@exdbadm01 lib]$ srvctl start database -d exdbx
[oracle@exdbadm01 lib]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Mar 30 17:30:28 2021
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
kish@exdbx<>select status from gv$instance;
STATUS
------------
OPEN
OPEN