ORA-01119: error in creating database file ‘/apps01/oradata/orcl19xsystem02.dbf’ ORA-17610: file ‘/apps01/oradata/orcl19xsystem02.dbf’ does not exist and nosize specified ORA-27037: unable to obtain file status Linux-x86_64 Error: 2: No such file or directoryAdditional information: 3 – ***Restore system tablespace***

Suddenly i woke up with a jerk from a powerful nap after i heard my oncall phone tringing with loud noise. I got a call from the client screaming about application down and insisted to troubleshoot the issue within less time. I was in night shift with one of my junior dba and he had mistakenly removed one of the system tablespace.

He didnot delete it intentionally but it was purely a human error where he pressed up arrow on the keyboard in the server terminal like below

This happens for everyone that someone might have removed old datafile due to decommission of obsolete files.

That junior dba was lazy to type commands and used up arrow to search for previously used commands and pressed enter

[oracle@orcl19x ~]$ sqlplus / as sysdba   ^
[oracle@orcl19x ~]$ . oraenv    ^
[oracle@orcl19x ~]$ ls -lrt   ^
[oracle@orcl19x ~]$ rm -rf /apps01/oradata/orcl19xsystem01.dbf  ^   <======= Enter =======> and !!!!!!!!BOOM!!!!

Understanding the situation, i tried to perform some command to verify if the instance is still intact without system tablespace

SQL> alter tablespace system add datafile '/apps01/oradata/orcl19xsystem02.dbf';
alter tablespace system add datafile '/apps01/oradata/orcl19xsystem02.dbf'
*
ERROR at line 1:
ORA-01119: error in creating database file
'/apps01/oradata/orcl19xsystem02.dbf'
ORA-17610: file '/apps01/oradata/orcl19xsystem02.dbf' does not exist and no
size specified
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

10 seconds –> After aborting the instance

[oracle@orcl19x ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Sat May 15 18:46:06 2021

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> shu abort;
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 1577058304 bytes
Fixed Size                  2924832 bytes
Variable Size            1157631712 bytes
Database Buffers          402653184 bytes
Redo Buffers               13848576 bytes
Database mounted.

30 seconds —> Checked the tablespace backup for system

RMAN> list backup of tablespace system;

using target database control file instead of recovery catalog

List of Backup Sets
===================


BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
3       Full    357.08M    DISK        00:03:18     25-FEB-21
        BP Key: 3   Status: AVAILABLE  Compressed: YES  Tag: TAG20210225T102357
        Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/03vo2glu_1_1
  List of Datafiles in backup set 3
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  1       Full 15059775   25-FEB-21 /apps01/oradata/orcl19xsystem01.dbf
  9       Full 15059775   25-FEB-21 /apps01/oradata/orcl19x/SYSTEM9302.dbf

BS Key  Type LV Size       Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ ---------------
9       Full    213.52M    DISK        00:02:23     15-MAY-21
        BP Key: 9   Status: AVAILABLE  Compressed: YES  Tag: TAG20210515T182653
        Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1
  List of Datafiles in backup set 9
  File LV Type Ckp SCN    Ckp Time  Name
  ---- -- ---- ---------- --------- ----
  1       Full 15876175   15-MAY-21 /apps01/oradata/orcl19xsystem01.dbf
  9       Full 15876175   15-MAY-21 /apps01/oradata/orcl19x/SYSTEM9302.dbf

40 seconds —-> Checked the file# for system tb

SQL> select name,file# from v$datafile where name like '%system%';

NAME             FILE#
--------------- ------
/apps01/oradata      1
/orcl19xsystem0
1.dbf

3rd minute —–> as the size of system tablespace was small, i could able to restore datafile faster from backups

[oracle@orcl19x ~]$ rman target /

Recovery Manager: Release 12.1.0.2.0 - Production on Sat May 15 18:59:16 2021

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL19X (DBID=4048647388, not open)

RMAN> restore datafile 1;

Starting restore at 15-MAY-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=21 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /apps01/oradata/orcl19xsystem01.dbf
channel ORA_DISK_1: reading from backup piece /apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1
channel ORA_DISK_1: piece handle=/apps01/product/12.1.0/dbhome_1/dbs/08vuu83e_1_1 tag=TAG20210515T182653
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:56
Finished restore at 15-MAY-21

4th minute —> Checked if datafile scn is consistent with controlfile scn

SQL> select file#,checkpoint_change#,status,recover,bytes/1073741824,tablespace_name from v$datafile_header;

     FILE# CHECKPOINT_CHANGE# STATUS  REC BYTES/1073741824 TABLESPACE_NAME
---------- ------------------ ------- --- ---------------- --------------------
         1           15876175 ONLINE  YES       .791015625 SYSTEM
         2           15876236 ONLINE  NO         .01574707 EXAMPLE
         3           15876236 ONLINE  NO          .8203125 SYSAUX
         4           15876236 ONLINE  NO        .151367188 UNDOTBS1
         5           15876236 ONLINE  NO        1.22070313 EXAMPLE

6th minute —–> Checked for the backup of archivelog from broken scn to last scn to be recovered

RMAN> list backup of archivelog scn between 15876175 and 15876236;


List of Backup Sets
===================


BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
12      12.00K     DISK        00:00:00     15-MAY-21
        BP Key: 12   Status: AVAILABLE  Compressed: YES  Tag: TAG20210515T182938
        Piece Name: /apps01/product/12.1.0/dbhome_1/dbs/0cvuu88j_1_1

  List of Archived Logs in backup set 12
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------
  1    96      15876133   15-MAY-21 15876236   15-MAY-21

7th minute —> recovered the archives until last scn

RMAN> run
2> {
3> set until scn 15876236;
4> recover datafile 1;
5> }

executing command: SET until clause

Starting recover at 15-MAY-21
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:04

Finished recover at 15-MAY-21

8th minute —> datafile is back as earlier

SQL> select file#,checkpoint_change#,status,recover,bytes/1073741824,tablespace_name from v$datafile_header;

     FILE# CHECKPOINT_CHANGE# STATUS  REC BYTES/1073741824 TABLESPACE_NAME
---------- ------------------ ------- --- ---------------- --------------------
         1           15977664 ONLINE  NO        .791015625 SYSTEM
         2           15876236 ONLINE  NO         .01574707 EXAMPLE
         3           15876236 ONLINE  NO          .8203125 SYSAUX
         4           15876236 ONLINE  NO        .151367188 UNDOTBS1
         5           15876236 ONLINE  NO        1.22070313 EXAMPLE

9th minute —> opened the database

SQL> alter database open;

Database altered.

10th minute —> client asked for RCA

And 11th minute —> My friend woke me up and i shouted to take a fresh backup as it was a bad dream 😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s