ORA-01119: error in creating database file ‘/apps01/oradata/orcl19xsystem02.dbf’ ORA-17610: file ‘/apps01/oradata/orcl19xsystem02.dbf’ does not exist and nosize specified ORA-27037: unable to obtain file status Linux-x86_64 Error: 2: No such file or directoryAdditional information: 3 – ***Restore system tablespace***
Suddenly i woke up with a jerk from a powerful nap after i heard my oncall phone tringing with loud noise. I got a call from the client screaming about application down and insisted to troubleshoot the issue within less time. I was in night shift with one of my junior dba and he had mistakenly removed one of the system tablespace.
He didnot delete it intentionally but it was purely a human error where he pressed up arrow on the keyboard in the server terminal like below
This happens for everyone that someone might have removed old datafile due to decommission of obsolete files.
That junior dba was lazy to type commands and used up arrow to search for previously used commands and pressed enter
[oracle@orcl19x ~]$ sqlplus / as sysdba ^
[oracle@orcl19x ~]$ . oraenv ^
[oracle@orcl19x ~]$ ls -lrt ^
[oracle@orcl19x ~]$ rm -rf /apps01/oradata/orcl19xsystem01.dbf ^ <======= Enter =======> and !!!!!!!!BOOM!!!!
Understanding the situation, i tried to perform some command to verify if the instance is still intact without system tablespace
SQL> alter tablespace system add datafile '/apps01/oradata/orcl19xsystem02.dbf';
alter tablespace system add datafile '/apps01/oradata/orcl19xsystem02.dbf'
*
ERROR at line 1:
ORA-01119: error in creating database file
'/apps01/oradata/orcl19xsystem02.dbf'
ORA-17610: file '/apps01/oradata/orcl19xsystem02.dbf' does not exist and no
size specified
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
10 seconds –> After aborting the instance
[oracle@orcl19x ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Sat May 15 18:46:06 2021
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL> shu abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 1577058304 bytes
Fixed Size 2924832 bytes
Variable Size 1157631712 bytes
Database Buffers 402653184 bytes
Redo Buffers 13848576 bytes
Database mounted.
30 seconds —> Checked the tablespace backup for system
RMAN> list backup of tablespace system;
using target database control file instead of recovery catalog
List of Backup Sets
===================
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
3 Full 357.08M DISK 00:03:18 25-FEB-21
BP Key: 3 Status: AVAILABLE Compressed: YES Tag: TAG20210225T102357
Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/03vo2glu_1_1
List of Datafiles in backup set 3
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 15059775 25-FEB-21 /apps01/oradata/orcl19xsystem01.dbf
9 Full 15059775 25-FEB-21 /apps01/oradata/orcl19x/SYSTEM9302.dbf
BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
9 Full 213.52M DISK 00:02:23 15-MAY-21
BP Key: 9 Status: AVAILABLE Compressed: YES Tag: TAG20210515T182653
Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1
List of Datafiles in backup set 9
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- --------- ----
1 Full 15876175 15-MAY-21 /apps01/oradata/orcl19xsystem01.dbf
9 Full 15876175 15-MAY-21 /apps01/oradata/orcl19x/SYSTEM9302.dbf
40 seconds —-> Checked the file# for system tb
SQL> select name,file# from v$datafile where name like '%system%';
NAME FILE#
--------------- ------
/apps01/oradata 1
/orcl19xsystem0
1.dbf
3rd minute —–> as the size of system tablespace was small, i could able to restore datafile faster from backups
[oracle@orcl19x ~]$ rman target /
Recovery Manager: Release 12.1.0.2.0 - Production on Sat May 15 18:59:16 2021
Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCL19X (DBID=4048647388, not open)
RMAN> restore datafile 1;
Starting restore at 15-MAY-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=21 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /apps01/oradata/orcl19xsystem01.dbf
channel ORA_DISK_1: reading from backup piece /apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1
channel ORA_DISK_1: piece handle=/apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1 tag=TAG20210515T182653
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:56
Finished restore at 15-MAY-21
4th minute —> Checked if datafile scn is consistent with controlfile scn
SQL> select file#,checkpoint_change#,status,recover,bytes/1073741824,tablespace_name from v$datafile_header;
FILE# CHECKPOINT_CHANGE# STATUS REC BYTES/1073741824 TABLESPACE_NAME
---------- ------------------ ------- --- ---------------- --------------------
1 15876175 ONLINE YES .791015625 SYSTEM
2 15876236 ONLINE NO .01574707 EXAMPLE
3 15876236 ONLINE NO .8203125 SYSAUX
4 15876236 ONLINE NO .151367188 UNDOTBS1
5 15876236 ONLINE NO 1.22070313 EXAMPLE
6th minute —–> Checked for the backup of archivelog from broken scn to last scn to be recovered
RMAN> list backup of archivelog scn between 15876175 and 15876236;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
12 12.00K DISK 00:00:00 15-MAY-21
BP Key: 12 Status: AVAILABLE Compressed: YES Tag: TAG20210515T182938
Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/0cvuu88j_1_1
List of Archived Logs in backup set 12
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 96 15876133 15-MAY-21 15876236 15-MAY-21
7th minute —> recovered the archives until last scn
RMAN> run
2> {
3> set until scn 15876236;
4> recover datafile 1;
5> }
executing command: SET until clause
Starting recover at 15-MAY-21
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1 device type=DISK
starting media recovery
media recovery complete, elapsed time: 00:00:04
Finished recover at 15-MAY-21
8th minute —> datafile is back as earlier
SQL> select file#,checkpoint_change#,status,recover,bytes/1073741824,tablespace_name from v$datafile_header;
FILE# CHECKPOINT_CHANGE# STATUS REC BYTES/1073741824 TABLESPACE_NAME
---------- ------------------ ------- --- ---------------- --------------------
1 15977664 ONLINE NO .791015625 SYSTEM
2 15876236 ONLINE NO .01574707 EXAMPLE
3 15876236 ONLINE NO .8203125 SYSAUX
4 15876236 ONLINE NO .151367188 UNDOTBS1
5 15876236 ONLINE NO 1.22070313 EXAMPLE
9th minute —> opened the database
SQL> alter database open;
Database altered.
10th minute —> client asked for RCA
And 11th minute —> My friend woke me up and i shouted to take a fresh backup as it was a bad dream 😉