ORA-16038: log 5 sequence# 290 cannot be archivedORA-19502: write error on file “”, block number (block size=)ORA-00312: online log 5 thread 1:

ORA-16038: log 5 sequence# 290 cannot be archivedORA-19502: write error on file “”, block number (block size=)ORA-00312: online log 5 thread 1:

There was an archiver error during startup of the database which prevented the instance to start with ORA-03113. There was no space left on the filesystem which further caused pressure to prevent the instance from coming up.

kIsH@xHydra<>startup;
ORACLE instance started.

Total System Global Area 2415918568 bytes
Fixed Size                  9137640 bytes
Variable Size             553648128 bytes
Database Buffers         1845493760 bytes
Redo Buffers                7639040 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 4796
Session ID: 1 Serial number: 21471

Alert log was flooded with archive errors

ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53xwqs1_.arc", block number 69632 (block size=512)
ARC0 (PID:6786): I/O error 19502 archiving LNO:5 to '/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53xwqs1_.arc'
ARC0 (PID:6786): Stuck archiver: inactive mandatory LAD:1
ARC0 (PID:6786): Stuck archiver condition declared
2023-05-03T11:26:39.855296+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc0_6786.trc:
ORA-16038: log 5 sequence# 290 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 5 thread 1: '/u01/app/oracle/oradata/EXHYDRA/onlinelog/redo02.log'
ORA-00312: online log 5 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo04.log'
ARC0 (PID:6786): Archival error occurred on a closed thread, archiver continuing
2023-05-03T11:26:39.857066+05:30
ORACLE Instance eXhydra, archival error, archiver continuing
2023-05-03T11:27:40.222395+05:30
...
2023-05-03T11:28:40.759985+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
Additional information: 4
Additional information: 122880
Additional information: 4294967295
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
Deleted Oracle managed file /apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc
2023-05-03T11:28:40.765832+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
Additional information: 4
Additional information: 122880
Additional information: 4294967295
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
ARC2 (PID:6790): I/O error 19502 archiving LNO:5 to '/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc'
ARC2 (PID:6790): Stuck archiver: inactive mandatory LAD:1
ARC2 (PID:6790): Stuck archiver condition declared
2023-05-03T11:28:40.779566+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-16038: log 5 sequence# 290 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 5 thread 1: '/u01/app/oracle/oradata/EXHYDRA/onlinelog/redo02.log'
ORA-00312: online log 5 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo04.log'
ARC2 (PID:6790): Archival error occurred on a closed thread, archiver continuing
2023-05-03T11:28:40.779642+05:30
ORACLE Instance eXhydra, archival error, archiver continuing
2023-05-03T11:29:19.007555+05:30
alter database clear logfile group 6
ORA-350 signalled during: alter database clear logfile group 6...

The mount point space is filled with datafiles and redo log files. Even all the archivelogs were deleted and backed up to tape. There are no files to be removed other than mandatory files

[oracle@staxhost oradata]$ df -h apps01
df: apps01: No such file or directory
[oracle@staxhost oradata]$ df -h /apps01
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        30G   30G   61M 100% /apps01
[oracle@staxhost oradata]$ cd /apps01/
[oracle@staxhost apps01]$ ls
oradata  prox2.xml  RECO
[oracle@staxhost apps01]$ cd RECO/
[oracle@staxhost RECO]$ du -sh *
4.8G    EXHYDRA
[oracle@staxhost RECO]$ cd EXHYDRA/
archivelog/                       backupset/                        EF2D7C66D65D5854E053C738A8C0C143/ onlinelog/
autobackup/                       controlfile/                      foreign_archivelog/
[oracle@staxhost RECO]$ cd EXHYDRA/
[oracle@staxhost EXHYDRA]$ du -sh *
0       archivelog
39M     autobackup
0       backupset
20M     controlfile
0       EF2D7C66D65D5854E053C738A8C0C143
0       foreign_archivelog
4.7G    onlinelog

Hence, mounted the database and tried dropping redo log groups. But none of them were archived. The goal is to open the database quickly without adding additional storage.

kIsH@xHydra<>startup mount;
ORACLE instance started.

Total System Global Area 2415918568 bytes
Fixed Size                  9137640 bytes
Variable Size             553648128 bytes
Database Buffers         1845493760 bytes
Redo Buffers                7639040 bytes
Database mounted.

kIsH@xHydra<>alter database drop logfile group 6;
alter database drop logfile group 6
*
ERROR at line 1:
ORA-00350: log 6 of instance eXhydra (thread 1) needs to be archived
ORA-00312: online log 6 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo05.log'


kIsH@xHydra<>alter database clear logfile group 6;
alter database clear logfile group 6
*
ERROR at line 1:
ORA-00350: log 6 of instance eXhydra (thread 1) needs to be archived
ORA-00312: online log 6 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo05.log'

Since, it takes more time to add additional storage, a quick workaround was applied by disabling the archive log and instance opened!

kIsH@xHydra<>alter database noarchivelog;

Database altered.

kIsH@xHydra<>alter database open;

Database altered.

Leave a Reply

%d