ORA-16038: log 5 sequence# 290 cannot be archivedORA-19502: write error on file “”, block number (block size=)ORA-00312: online log 5 thread 1:
There was an archiver error during startup of the database which prevented the instance to start with ORA-03113. There was no space left on the filesystem which further caused pressure to prevent the instance from coming up.
kIsH@xHydra<>startup;
ORACLE instance started.
Total System Global Area 2415918568 bytes
Fixed Size 9137640 bytes
Variable Size 553648128 bytes
Database Buffers 1845493760 bytes
Redo Buffers 7639040 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 4796
Session ID: 1 Serial number: 21471
Alert log was flooded with archive errors
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53xwqs1_.arc", block number 69632 (block size=512)
ARC0 (PID:6786): I/O error 19502 archiving LNO:5 to '/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53xwqs1_.arc'
ARC0 (PID:6786): Stuck archiver: inactive mandatory LAD:1
ARC0 (PID:6786): Stuck archiver condition declared
2023-05-03T11:26:39.855296+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc0_6786.trc:
ORA-16038: log 5 sequence# 290 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 5 thread 1: '/u01/app/oracle/oradata/EXHYDRA/onlinelog/redo02.log'
ORA-00312: online log 5 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo04.log'
ARC0 (PID:6786): Archival error occurred on a closed thread, archiver continuing
2023-05-03T11:26:39.857066+05:30
ORACLE Instance eXhydra, archival error, archiver continuing
2023-05-03T11:27:40.222395+05:30
...
2023-05-03T11:28:40.759985+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
Additional information: 4
Additional information: 122880
Additional information: 4294967295
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
Deleted Oracle managed file /apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc
2023-05-03T11:28:40.765832+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
Additional information: 4
Additional information: 122880
Additional information: 4294967295
ORA-19502: write error on file "/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc", block number 122880 (block size=512)
ARC2 (PID:6790): I/O error 19502 archiving LNO:5 to '/apps01/RECO/EXHYDRA/archivelog/2023_05_03/o1_mf_1_290_l53y0jmr_.arc'
ARC2 (PID:6790): Stuck archiver: inactive mandatory LAD:1
ARC2 (PID:6790): Stuck archiver condition declared
2023-05-03T11:28:40.779566+05:30
Errors in file /u01/app/oracle/diag/rdbms/exhydra/eXhydra/trace/eXhydra_arc2_6790.trc:
ORA-16038: log 5 sequence# 290 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 5 thread 1: '/u01/app/oracle/oradata/EXHYDRA/onlinelog/redo02.log'
ORA-00312: online log 5 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo04.log'
ARC2 (PID:6790): Archival error occurred on a closed thread, archiver continuing
2023-05-03T11:28:40.779642+05:30
ORACLE Instance eXhydra, archival error, archiver continuing
2023-05-03T11:29:19.007555+05:30
alter database clear logfile group 6
ORA-350 signalled during: alter database clear logfile group 6...
The mount point space is filled with datafiles and redo log files. Even all the archivelogs were deleted and backed up to tape. There are no files to be removed other than mandatory files
[oracle@staxhost oradata]$ df -h apps01
df: apps01: No such file or directory
[oracle@staxhost oradata]$ df -h /apps01
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 30G 30G 61M 100% /apps01
[oracle@staxhost oradata]$ cd /apps01/
[oracle@staxhost apps01]$ ls
oradata prox2.xml RECO
[oracle@staxhost apps01]$ cd RECO/
[oracle@staxhost RECO]$ du -sh *
4.8G EXHYDRA
[oracle@staxhost RECO]$ cd EXHYDRA/
archivelog/ backupset/ EF2D7C66D65D5854E053C738A8C0C143/ onlinelog/
autobackup/ controlfile/ foreign_archivelog/
[oracle@staxhost RECO]$ cd EXHYDRA/
[oracle@staxhost EXHYDRA]$ du -sh *
0 archivelog
39M autobackup
0 backupset
20M controlfile
0 EF2D7C66D65D5854E053C738A8C0C143
0 foreign_archivelog
4.7G onlinelog
Hence, mounted the database and tried dropping redo log groups. But none of them were archived. The goal is to open the database quickly without adding additional storage.
kIsH@xHydra<>startup mount;
ORACLE instance started.
Total System Global Area 2415918568 bytes
Fixed Size 9137640 bytes
Variable Size 553648128 bytes
Database Buffers 1845493760 bytes
Redo Buffers 7639040 bytes
Database mounted.
kIsH@xHydra<>alter database drop logfile group 6;
alter database drop logfile group 6
*
ERROR at line 1:
ORA-00350: log 6 of instance eXhydra (thread 1) needs to be archived
ORA-00312: online log 6 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo05.log'
kIsH@xHydra<>alter database clear logfile group 6;
alter database clear logfile group 6
*
ERROR at line 1:
ORA-00350: log 6 of instance eXhydra (thread 1) needs to be archived
ORA-00312: online log 6 thread 1: '/apps01/RECO/EXHYDRA/onlinelog/redo05.log'
Since, it takes more time to add additional storage, a quick workaround was applied by disabling the archive log and instance opened!
kIsH@xHydra<>alter database noarchivelog;
Database altered.
kIsH@xHydra<>alter database open;
Database altered.