Archives October 2021

ENABLE REALTIME APPLY IN ORACLE DATAGUARD

Log apply services in dataguard plays a major role in recovery of logs from primary to standby and keep them in sync

Dataguard default mode : Redo apply (physical standby) and SQL apply (Logical standby)

As in most of cases, customers often use physical standby(PS).

Real time apply is a nice feature used in dataguard which directly apply the logs from standby redologs(SRL) to standby database and skips waiting for archival of redo part .

  • Here RFS receive the redo from primary and store in SRL.
  • MRP(physical standby) or LSP(Logical standby) directly recover the redo from SRL

Things required for Real time apply : Standby redolog files

Advantages: Fast switchover and failover due to log apply from SRL

Check the database role,open mode on both primary and standby database

kish@PRIM>select name, open_mode, database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
DBX00     READ WRITE           PRIMARY

kish@STDBY>select name, open_mode, database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
DBX00     MOUNTED              PHYSICAL STANDBY

Verify if real time apply is enabled on your primary database. If standby redologs(SRL column) is set to NO, then real time apply is not enabled

kish@PRIM>col DEST_ID for 99999
kish@PRIM>col DEST_NAME for a20
kish@PRIM>select DEST_ID,dest_name,status,type,srl,recovery_mode from v$archive_dest_status where dest_id=2;

DEST_ID DEST_NAME            STATUS    TYPE             SRL RECOVERY_MODE
------- -------------------- --------- ---------------- --- -----------------------
      2 LOG_ARCHIVE_DEST_2   VALID     PHYSICAL         NO MANAGED 

kish@STDBY>col DEST_ID for 99999
kish@STDBY>col DEST_NAME for a20
kish@STDBY>select DEST_ID,dest_name,status,type,srl,recovery_mode from v$archive_dest_status where dest_id=1;

DEST_ID DEST_NAME            STATUS    TYPE             SRL RECOVERY_MODE
------- -------------------- --------- ---------------- --- -----------------------
      1 LOG_ARCHIVE_DEST_1   VALID     LOCAL            NO  MANAGED 

By default, you will see the message from logs saying ‘RFS[1]: No standby redo logfiles createdManaged Standby Recovery not using Real Time Apply’ . This confirm that realtime apply is not active

kish@STDBY>select message from v$dataguard_status;

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
ARC1: Becoming the active heartbeat ARCH
Primary database is in MAXIMUM PERFORMANCE mode
RFS[1]: Assigned to RFS process (PID:3522)
RFS[1]: No standby redo logfiles created
RFS[2]: Assigned to RFS process (PID:3524)
RFS[3]: Assigned to RFS process (PID:3526)
RFS[4]: Assigned to RFS process (PID:3528)
RFS[5]: Assigned to RFS process (PID:3530)
Attempt to start background Managed Standby Recovery process
MRP0: Background Managed Standby Recovery process started
Managed Standby Recovery not using Real Time Apply
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_10_04/o1_mf_1_82_jonn4yj4_.arc
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_10_04/o1_mf_1_83_jonn4ynq_.arc
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_10_04/o1_mf_1_84_jonn4yoz_.arc
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_10_04/o1_mf_1_85_jonn4zl3_.arc

Check the logfile groups and their members on primary and standby

kish@PRIM>select GROUP#,count(MEMBER) from v$logfile group by GROUP# order by 1;

                                            GROUP#                                      COUNT(MEMBER)
-------------------------------------------------- --------------------------------------------------
                                                 1                                                  2
                                                 2                                                  2
                                                 3                                                  2
                                                 4                                                  2
                                                 5                                                  1
                                                 6                                                  1
                                                 7                                                  1
                                                 8                                                  1
                                                 9                                                  2
                                                10                                                  2
                                                11                                                  2
                                                12                                                  2

12 rows selected.

kish@STDBY>select GROUP#,count(MEMBER) from v$logfile group by GROUP# order by 1;

    GROUP# COUNT(MEMBER)
---------- -------------
         1             2
         2             2
         3             2
         4             2
         5             1
         6             1
         7             1
         8             1
         9             1
        10             1
        11             1
        12             1

12 rows selected.

Check if there are standby logfile added

kish@PRIM>col Redologs for a70
kish@PRIM>select ' |' ||a.group#||' | '||a.member||' | '||b.bytes/1024/1024||' | ' as Redologs from v$logfile a, v$standby_log b WHERE a.group# = b.group#;
no rows selected

kish@STDBY>col Redologs for a70
kish@STDBY>select ' |' ||a.group#||' | '||a.member||' | '||b.bytes/1024/1024||' | ' as Redologs from v$logfile a, v$standby_log b WHERE a.group# = b.group#;
no rows selected

If standby logfiles are not there, then add them both on primary and standby

kish@PRIM>alter database add standby logfile group 9 size 50M;

Database altered.

kish@PRIM>alter database add standby logfile group 10 size 50M;

Database altered.

kish@PRIM>alter database add standby logfile group 11 size 50M;

Database altered.

kish@PRIM>alter database add standby logfile group 12 size 50M;

Database altered.

kish@PRIM>select group#,bytes/1024/1024 "MB",status from v$standby_log;

                                            GROUP#                                                 MB STATUS
-------------------------------------------------- -------------------------------------------------- ----------
                                                 9                                                 50 UNASSIGNED
                                                10                                                 50 UNASSIGNED
                                                11                                                 50 UNASSIGNED
                                                12                                                 50 UNASSIGNED

On standby add standby redologs

kish@STDBY>alter database recover managed standby database cancel;

Database altered.

kish@STDBY>alter database add standby logfile group 9 size 50M;

Database altered.

kish@STDBY>alter database add standby logfile group 10 size 50M;

Database altered.

kish@STDBY>alter database add standby logfile group 11 size 50M;

Database altered.

kish@STDBY>alter database add standby logfile group 12 size 50M;

Database altered.

kish@STDBY>select group#,bytes/1024/1024 "MB",status from v$standby_log;

    GROUP#         MB STATUS
---------- ---------- ----------
         9         50 UNASSIGNED
        10         50 UNASSIGNED
        11         50 UNASSIGNED
        12         50 UNASSIGNED

Enable real time apply using below syntax

kish@STDBY>alter database recover managed standby database using current logfile disconnect from session;

Database altered.

Verify if SRL column on primary is YES. This says that Real time apply is active.

kish@PRIM>select DEST_ID,dest_name,status,type,srl,recovery_mode from v$archive_dest_status where dest_id=2;

DEST_ID DEST_NAME            STATUS    TYPE             SRL RECOVERY_MODE
------- -------------------- --------- ---------------- --- -----------------------
      2 LOG_ARCHIVE_DEST_2   VALID     PHYSICAL         YES MANAGED REAL TIME APPLY

kish@STDBY>select DEST_ID,dest_name,status,type,srl,recovery_mode from v$archive_dest_status where dest_id=1;

DEST_ID DEST_NAME            STATUS    TYPE             SRL RECOVERY_MODE
------- -------------------- --------- ---------------- --- -----------------------
      1 LOG_ARCHIVE_DEST_1   VALID     LOCAL            NO  MANAGED REAL TIME APPLY

Validate if standby redologs are used by switching the logfile on primary and check standby log status in standby database

kish@PRIM>alter system switch logfile;

System altered.

kish@STDBY>select group#,bytes/1024/1024 "MB",status,FIRST_CHANGE#,NEXT_CHANGE# from v$standby_log;

    GROUP#         MB STATUS     FIRST_CHANGE# NEXT_CHANGE#
---------- ---------- ---------- ------------- ------------
         9         50 ACTIVE           4196962
        10         50 UNASSIGNED
        11         50 UNASSIGNED
        12         50 UNASSIGNED

In logfile, you will find this message saying ‘Managed Standby Recovery starting Real Time Apply’

Attempt to start background Managed Standby Recovery process
MRP0: Background Managed Standby Recovery process started
Managed Standby Recovery starting Real Time Apply
Media Recovery Waiting for thread 1 sequence 86 (in transit)
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_10_04/o1_mf_1_86_jonn4x96_.arc
Primary database is in MAXIMUM PERFORMANCE mode
RFS[6]: Assigned to RFS process (PID:5549)
Media Recovery Waiting for thread 1 sequence 87 (in transit)
DO WE NEED TO HOUSEKEEP FLASHBACK LOGFILES MANUALLY?

Read the article until the end. There is small tip for you. Dont miss it!

Answer for the topic is no. Oracle has some internal mechanism which automatically wipes out the flashback logs when we disable flashback. Let us see how it works

I enable flashback in my database and perform some changes

kish@STDBY>alter database flashback off;

Database altered.

kish@STDBY>insert into t values('it_worked',5);

1 row created.

kish@STDBY>commit;

Commit complete.

Check the contribution of space occupation by flashback log in the below output. 3% of the space is utilized by flashback logfiles in disk

kish@STDBY>select * from v$recovery_area_usage;

FILE_TYPE               PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES     CON_ID
----------------------- ------------------ ------------------------- --------------- ----------
CONTROL FILE                           .36                         0               1          0
REDO LOG                                 8                         0               8          0
ARCHIVED LOG                         20.11                         0              60          0
BACKUP PIECE                          2.18                      1.82               6          0
IMAGE COPY                            2.89                      2.89               8          0
FLASHBACK LOG                            3                         0               3          0
FOREIGN ARCHIVED LOG                     0                         0               0          0
AUXILIARY DATAFILE COPY                  0                         0               0          0

8 rows selected.

Disable flashback

kish@STDBY>alter database flashback off;

Database altered.

The moment we disable flashback, RVWR process is stopped and all the flashback logfiles are deleted

alter database flashback off
Stopping background process RVWR
Sat Oct 02 18:47:15 2021
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_jodf2btl_.flb
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_jodf2fop_.flb
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_joj0lxh1_.flb
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_joj6tsr7_.flb
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_jojmm5j2_.flb
Deleted Oracle managed file /apps01/FRA/DBX01/flashback/o1_mf_jojpvkgk_.flb
Flashback Database Disabled
Completed: alter database flashback off

Check the FRA now. 3 turned to 0 in PERCENT_SPACE_USED

kish@STDBY>select * from v$flash_recovery_area_usage where FILE_TYPE='FLASHBACK LOG';

FILE_TYPE               PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES     CON_ID
----------------------- ------------------ ------------------------- --------------- ----------
FLASHBACK LOG                            0                         0               0          0

Thus we dont need to manually housekeep flashback logs if we switch off flashback.

If flashback is turned on, then based on retention period (default – 1440 min ~ 24 hour ~ 1 day), logs are removed.

kish@STDBY>show parameter flashback

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flashback_retention_target        integer     1440

Trick: If flashback logs need to deleted with flashback on, then reduce the db_flashback_retention_target to less time and reduce FRA size. This can be performed dynamically

kish@STDBY>alter system set db_flashback_retention_target=1;

System altered.

kish@STDBY>show parameter db_recovery_file_dest_size

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 5000M

kish@STDBY>alter system set db_recovery_file_dest_size=1000M;

System altered.

kish@STDBY>select * from v$recovery_area_usage;

FILE_TYPE               PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES     CON_ID
----------------------- ------------------ ------------------------- --------------- ----------
CONTROL FILE                          1.82                         0               1          0
REDO LOG                                60                         0              12          0
ARCHIVED LOG                         11.09                      9.67               8          0
BACKUP PIECE                          3.65                         0               2          0
IMAGE COPY                               0                         0               0          0
FLASHBACK LOG                           20                         5               4          0
FOREIGN ARCHIVED LOG                     0                         0               0          0
AUXILIARY DATAFILE COPY                  0                         0               0          0

Summary:We dont need to manually delete the flashback logfiles

ORA-38784: Cannot create restore point ‘SNAPSHOT_STANDBY_REQUIRED_10/02/2021 18:42:36’.ORA-01153: an incompatible media recovery is active

Cause:

I try to convert physical standby to snapshot standby but got the error below

kish@STDBY>alter database convert to snapshot standby;
alter database convert to snapshot standby
*
ERROR at line 1:
ORA-38784: Cannot create restore point 'SNAPSHOT_STANDBY_REQUIRED_10/02/2021 18:42:36'.
ORA-01153: an incompatible media recovery is active

Workaround:

Cancel MRP first and perform the conversion

kish@STDBY>alter database recover managed standby database cancel;

Database altered.
kish@STDBY>alter database convert to snapshot standby;

Database altered.
ORA-01126: database must be mounted in this instance and not open in any instance

Cause:

I try to convert snapshot standby to physical standby post testing and i get the error due to improper database state

kish@STDBY>alter database convert to physical standby;
alter database convert to physical standby
*
ERROR at line 1:
ORA-01126: database must be mounted in this instance and not open in any instance

Workaround:

Bounce the snap database to mount and then start conversion of physical standby. This should be performed in this way because the database should ensure consistency of data between primary and standby

kish@STDBY>shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
kish@STDBY>startup mount;
ORACLE instance started.

Total System Global Area 1509949440 bytes
Fixed Size                  2924640 bytes
Variable Size             989859744 bytes
Database Buffers          503316480 bytes
Redo Buffers               13848576 bytes
Database mounted.
kish@STDBY>alter database convert to physical standby;

Database altered.

kish@STDBY>alter database recover managed standby database using current logfile disconnect from session;

Database altered.

kish@STDBY>
FAL[server, ARC3]: Error 12541 creating remote archivelog file ‘dbx01’ – PING[ARC2]: Heartbeat failed to connect to standby ‘dbx01’. Error is 12541.

Cause:

MRP is not applying logs to standby database due to heartbeat issue

MESSAGE
--------------------------------------------------------------------------------
ARC1: Completed archiving thread 1 sequence 75 (3766426-3767787)
Error 12541 received logging on to the standby
Check whether the listener is up and running.
FAL[server, ARC3]: Error 12541 creating remote archivelog file 'dbx01'
Error 12541 received logging on to the standby
Check whether the listener is up and running.
Error 12541 received logging on to the standby
Check whether the listener is up and running.
Error 12541 for archive log file 4 to 'dbx01'
Error 12541 received logging on to the standby
Check whether the listener is up and running.

MESSAGE
--------------------------------------------------------------------------------
PING[ARC2]: Heartbeat failed to connect to standby 'dbx01'. Error is 12541.
Error 12541 received logging on to the standby
Check whether the listener is up and running.
FAL[server, ARC3]: Error 12541 creating remote archivelog file 'dbx01'
Error 12541 received logging on to the standby
Check whether the listener is up and running.
Error 12541 for archive log file 1 to 'dbx01'
ARC1: Beginning to archive thread 1 sequence 76 (3767787-3770055)
ARC1: Completed archiving thread 1 sequence 76 (3767787-3770055)
Error 12541 received logging on to the standby
Check whether the listener is up and running.

Workaround:

If tnsping is not working, then check for your tns entries correction

kish@PRIM>!tnsping dbx01

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 02-OCT-2021 12:23:30

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.197)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbx01)(UR = A)))
OK (0 msec)


-bash-4.2$ tnsping dbx00

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 02-OCT-2021 12:23:36

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/apps01/product/12.1.0/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.56.41)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbx00)))
OK (0 msec)

Check if listener is up and running

-bash-4.2$ lsnrctl start

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 02-OCT-2021 12:21:39

Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Starting /apps01/product/12.1.0/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.1.0.2.0 - Production
System parameter file is /apps01/product/12.1.0/dbhome_1/network/admin/listener.ora
Log messages written to /apps01/base/diag/tnslsnr/orcl19x/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.197)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.197)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                02-OCT-2021 12:21:40
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /apps01/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File         /apps01/base/diag/tnslsnr/orcl19x/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.56.197)(PORT=1521)))
Services Summary...
Service "dbx01" has 1 instance(s).
  Instance "dbx01", status UNKNOWN, has 1 handler(s) for this service...
Service "orcl19x1" has 1 instance(s).
  Instance "orcl19x1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
MRP0: Detected orphaned datafiles!Recovery will possibly be retried after flashback…

I got a weird message on a monday morning in the physical standby database displaying message as ‘MRP0: Detected orphaned datafiles!’. I wonder what MRP detected with datafiles

kish@STDBY>select message from v$dataguard_status;

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
WARNING: STANDBY_FILE_MANAGEMENT initialization parameter is
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
Primary database is in MAXIMUM PERFORMANCE mode
RFS[1]: Assigned to RFS process (PID:10578)
RFS[1]: No standby redo logfiles created
ARC1: Becoming the active heartbeat ARCH
Attempt to start background Managed Standby Recovery process
MRP0: Background Managed Standby Recovery process started
Managed Standby Recovery not using Real Time Apply
MRP0: Detected orphaned datafiles!
Recovery will possibly be retried after flashback...
MRP0: Background Media Recovery process shutdown

18 rows selected.

When i checked the consistency in standby database, i can see some datafiles require media recovery and one fuzzy datafile where datafile header SCN is lower than controlfile header SCN

kish@STDBY>col file# for 9999999
select
a.file#,
case
when a.CHECKPOINT_CHANGE# > b.CHECKPOINT_CHANGE# then 'Fuzzy header!'
kish@STDBY>  2    3    4    5  when a.CHECKPOINT_CHANGE# = b.CHECKPOINT_CHANGE# then 'ACID complaint!'
when a.CHECKPOINT_CHANGE# < b.CHECKPOINT_CHANGE# then 'Media recovery required!'
end Consistency_check
from
v$datafile a
inner join v$datafile_header b on (a.FILE#=b.FILE#)
order by 1;
  6    7    8    9   10   11
   FILE# CONSISTENCY_CHECK
-------- ------------------------
       1 Media recovery required!
       3 Media recovery required!
       4 Media recovery required!
       5 ACID complaint!
       6 Media recovery required!
       7 ACID complaint!
       8 ACID complaint!
       9 ACID complaint!
      10 ACID complaint!
      11 ACID complaint!
      12 ACID complaint!
      13 ACID complaint!
      14 Fuzzy header!

13 rows selected.

I checked the datafiles in standby and compared with primary for missing files

kish@STDBY>select name from v$datafile;

NAME
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/apps01/base/oradata/dbx01/datafile/system.268.1083714775
/apps01/base/oradata/dbx01/datafile/sysaux.273.1083714719
........
/apps01/base/oradata/dbx01/datafile/users.279.1083903263

MRP is not running because the incarnation in which the standby database run is an orphaned version. In simple terms , standby database SCN is higher than primary due to some inconsistent recovery or human errors

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
RFS       IDLE                                                          0
RFS       CLOSING                                                      42

6 rows selected.

I bounced the database to mount state

kish@STDBY>shu immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
kish@STDBY>startup nomount;
ORACLE instance started.

Total System Global Area                                         1509949440 bytes
Fixed Size                                                          2924640 bytes
Variable Size                                                     989859744 bytes
Database Buffers                                                  503316480 bytes
Redo Buffers        

Restored the standby controlfile using service feature in 12c and mounted the database. Thanks to oracle for introducing this “service” feature. God bless you !

RMAN> restore standby controlfile from service dbx00;

Starting restore at 26-SEP-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=22 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using network backup set from service dbx00
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:06
output file name=/apps01/FRA/DBX01/controlfile/o1_mf_jnl5gscz_.ctl
Finished restore at 26-SEP-21

RMAN> alter database mount;

Statement processed
released channel: ORA_DISK_1

Restored the database itself using service instead of specific datafiles which needs media recovery.

As we know that RMAN has the intelligence to skip the datafiles which are already consistent

RMAN> run
{
set newname for datafile '+DATAC1/DBX00/DATAFILE/system.268.1083714775' to '/apps01/base/oradata/dbx01/datafile/system.268.1083714775';
set newname for datafile '+DATAC1/DBX00/DATAFILE/sysaux.273.1083714719' to '/apps01/base/oradata/dbx01/datafile/sysaux.273.1083714719';
set newname for datafile '+DATAC1/DBX00/DATAFILE/undotbs1.266.1083714851' to '/apps01/base/oradata/dbx01/datafile/undotbs1.266.1083714851';
set newname for datafile '+DATAC1/DBX00/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/system.260.1083714939' to '/apps01/base/oradata/dbx01/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/system.260.1083714939';
set newname for datafile '+DATAC1/DBX00/DATAFILE/users.257.1083714849' to '/apps01/base/oradata/dbx01/datafile/users.257.1083714849';
set newname for datafile '+DATAC1/DBX00/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/sysaux.267.1083714939' to '/apps01/base/oradata/dbx01/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/sysaux.267.1083714939';
set newname for datafile '+DATAC1/DBX00/CC5EB83E86C4281FE053CD38A8C0DB1B/DATAFILE/system.270.1083716807' to '/apps01/base/oradata/dbx01/cc5eb83e86c4281fe053cd38a8c0db1b/datafile/system.270.1083716807';
set newname for datafile '+DATAC1/DBX00/CC5EB83E86C4281FE053CD38A8C0DB1B/DATAFILE/sysaux.277.1083716807' to '/apps01/base/oradata/dbx01/cc5eb83e86c4281fe053cd38a8c0db1b/datafile/sysaux.277.1083716807';
set newname for datafile '+DATAC1/DBX00/CC5EB83E86C4281FE053CD38A8C0DB1B/DATAFILE/users.274.1083716893' to '/apps01/base/oradata/dbx01/cc5eb83e86c4281fe053cd38a8c0db1b/datafile/users.274.1083716893';
set newname for datafile '+DATAC1/DBX00/CC5EBF37A8122A8BE053CD38A8C0EFDF/DATAFILE/system.269.1083716919' to '/apps01/base/oradata/dbx01/cc5ebf37a8122a8be053cd38a8c0efdf/datafile/system.269.1083716919';
set newname for datafile '+DATAC1/DBX00/CC5EBF37A8122A8BE053CD38A8C0EFDF/DATAFILE/sysaux.272.1083716919' to '/apps01/base/oradata/dbx01/cc5ebf37a8122a8be053cd38a8c0efdf/datafile/sysaux.272.1083716919';
set newname for datafile '+DATAC1/DBX00/CC5EBF37A8122A8BE053CD38A8C0EFDF/DATAFILE/users.278.1083717009' to '/apps01/base/oradata/dbx01/cc5ebf37a8122a8be053cd38a8c0efdf/datafile/users.278.1083717009';
set newname for datafile '+DATAC1/DBX00/DATAFILE/users.279.1083903263' to '/apps01/base/oradata/dbx01/datafile/users.279.1083903263';
set newname for datafile '+DATAC1/DBX00/TEMPFILE/temp.258.1083714929' to '/apps01/base/oradata/dbx01/tempfile/temp.258.1083714929';
set newname for datafile '+DATAC1/DBX00/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012021-09-19_11-56-29-pm.dbf' to '/apps01/base/oradata/dbx01/fd9ac20f64d244d7e043b6a9e80a2f2f/datafile/pdbseed_temp012021-09-19_11-56-29-pm.dbf';
set newname for datafile '+DATAC1/DBX00/CC5EB83E86C4281FE053CD38A8C0DB1B/TEMPFILE/temp.276.1083716869' to '/apps01/base/oradata/dbx01/cc5eb83e86c4281fe053cd38a8c0db1b/tempfile/temp.276.1083716869';
set newname for datafile '+DATAC1/DBX00/CC5EBF37A8122A8BE053CD38A8C0EFDF/TEMPFILE/temp.261.1083716969' to '/apps01/base/oradata/dbx01/cc5ebf37a8122a8be053cd38a8c0efdf/tempfile/temp.261.1083716969';
restore database from service dbx00 using compressed backupset;
switch datafile all;
}2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20> 21> 22>
executing command: SET NEWNAME
Starting restore at 26-SEP-21
Starting implicit crosscheck backup at 26-SEP-21
released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=22 device type=DISK
Crosschecked 5 objects
Finished implicit crosscheck backup at 26-SEP-21

Starting implicit crosscheck copy at 26-SEP-21
using channel ORA_DISK_1
Finished implicit crosscheck copy at 26-SEP-21

searching for all files in the recovery area
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnkzn8dc_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnkzsny1_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl09ko2_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl1tr77_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl26k15_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl3dclh_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl3wl1c_.ctl
File Name: /apps01/FRA/DBX01/controlfile/o1_mf_jnl4mn2c_.ctl
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_18_jnl4qw44_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_19_jnl4s1m3_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_20_jnl4s3fh_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_21_jnl4v0mr_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_22_jnl4vxfj_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_23_jnl4x2t7_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_21/o1_mf_1_24_jnl4xwfl_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_25_jnnd9613_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_26_jnnd96lo_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_27_jnnd972v_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_28_jnnd97f1_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_30_jnnd9b4p_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_31_jnnd9b5c_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_32_jnnd9b7k_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_33_jnnd9bns_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_34_jnnd9c6s_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_29_jnnd9d9n_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_35_jnndm8lk_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_36_jnnm5vr2_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_37_jnnm6c8p_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_38_jnnnb1kz_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_39_jnnq8nrs_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_40_jnnqd1wg_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_41_jnnqd9px_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_42_jnnshgqb_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_45_jnpwsdy2_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_43_jnpwsfrg_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_22/o1_mf_1_44_jnpwsfv2_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_23/o1_mf_1_46_jnqjml11_.arc
File Name: /apps01/FRA/DBX01/archivelog/2021_09_23/o1_mf_1_47_jnqmcoj1_.arc
File Name: /apps01/FRA/DBX01/autobackup/2021_09_22/o1_mf_s_1083818671_jnnbsr61_.bkp
File Name: /apps01/FRA/DBX01/autobackup/2021_09_22/o1_mf_s_1083890807_jnnf3l1j_.bkp
File Name: /apps01/FRA/DBX01/autobackup/2021_09_22/o1_mf_s_1083902714_jnnqk3fl_.bkp

using channel ORA_DISK_1

skipping datafile 5; already restored to SCN 1744270
skipping datafile 7; already restored to SCN 1744270
skipping datafile 8; already restored to SCN 1857203
skipping datafile 9; already restored to SCN 1857203
skipping datafile 10; already restored to SCN 1857203
skipping datafile 11; already restored to SCN 1857214
skipping datafile 12; already restored to SCN 1857214
skipping datafile 13; already restored to SCN 1857214
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using compressed network backup set from service dbx00
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /apps01/base/oradata/dbx01/datafile/system.268.1083714775
channel ORA_DISK_1: restore complete, elapsed time: 00:02:26
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using compressed network backup set from service dbx00
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00003 to /apps01/base/oradata/dbx01/datafile/sysaux.273.1083714719
channel ORA_DISK_1: restore complete, elapsed time: 00:03:06
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using compressed network backup set from service dbx00
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00004 to /apps01/base/oradata/dbx01/datafile/undotbs1.266.1083714851
channel ORA_DISK_1: restore complete, elapsed time: 00:02:39
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using compressed network backup set from service dbx00
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00006 to /apps01/base/oradata/dbx01/datafile/users.257.1083714849
channel ORA_DISK_1: restore complete, elapsed time: 00:02:26
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: using compressed network backup set from service dbx00
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00014 to /apps01/base/oradata/dbx01/datafile/users.279.1083903263
channel ORA_DISK_1: restore complete, elapsed time: 00:02:25
Finished restore at 26-SEP-21

Then compared the incarnation of the database on primary and standby

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       DBX00    2195240283       PARENT  1          07-JUL-14
2       2       DBX00    2195240283       CURRENT 1594143    19-SEP-21

Reset the incarnation from 3 to 2 in standby to previous state because of fuzziness(Primary(1594143) < Standby(2137590))

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       DBX00    2195240283       PARENT  1          07-JUL-14
2       2       DBX00    2195240283       PARENT  1594143    19-SEP-21
3       3       DBX00    2195240283       CURRENT 2137590    22-SEP-21

RMAN> reset database to incarnation 2;

database reset to incarnation 2

Now incarnation 3 become orphan and 2 become current in standby database

RMAN> list incarnation of database;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       DBX00    2195240283       PARENT  1          07-JUL-14
2       2       DBX00    2195240283       CURRENT 1594143    19-SEP-21
3       3       DBX00    2195240283       ORPHAN  2137590    22-SEP-21

Post that enable MRP in standby

kish@STDBY>alter database recover managed standby database using current logfile disconnect from session;

Database altered.

Check the status of redo apply in standby

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
RFS       IDLE                                                         67
RFS       IDLE                                                          0
RFS       IDLE                                                          0
MRP0      WAIT_FOR_LOG                                                  0

Check the message for any errors. In below output , oracle clears the logfile automatically after SCN reset

kish@STDBY>select message from v$dataguard_status;

MESSAGE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
RFS[20]: No standby redo logfiles created
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /apps01/base/oradata/dbx01/onlinelog/group_2.263.1083714913

Oracle then perform media recovery using archivelogs shipped from primary

Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /apps01/base/oradata/dbx01/onlinelog/group_3.264.1083714915
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 /apps01/base/oradata/dbx01/onlinelog/group_4.265.1083714917
Clearing online redo logfile 4 complete
Clearing online redo logfile 5 +REDO/DBX00/ONLINELOG/group_5.256.1083899629
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

/apps01/FRA/DBX01/archivelog/2021_09_30/o1_mf_1_61_joczyghx_.arc
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_09_30/o1_mf_1_62_joczygmw_.arc
Media Recovery Log /apps01/FRA/DBX01/archivelog/2021_09_30/o1_mf_1_63_joczyh8h_.arc

All good now! We can see a sync between primary and standby

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
RFS       IDLE                                                          0
RFS       IDLE                                                         69
RFS       IDLE                                                          0
RFS       IDLE                                                          0
MRP0      APPLYING_LOG                                                 65

kish@STDBY>/

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
RFS       IDLE                                                          0
RFS       IDLE                                                         69
RFS       IDLE                                                          0
RFS       IDLE                                                          0
MRP0      APPLYING_LOG                                                 66

9 rows selected.

kish@STDBY>/

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
RFS       IDLE                                                          0
RFS       IDLE                                                         69
RFS       IDLE                                                          0
RFS       IDLE                                                          0
MRP0      WAIT_FOR_LOG                                                 69

9 rows selected.

I finally checked the standby file management and it is set to manual due to some activity and the engineer forget to set the parameter to automatic post changes. Some one had added a new datafile on primary and the datafile was not residing in the original directory

kish@PRIM>show parameter standby_file

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
standby_file_management              string      MANUAL

I set it auto

kish@PRIM>alter system set standby_file_management=AUTO;

System altered.

Checked consistency of datafiles and it is sync ! Cheers !

kish@STDBY>set lines 200 pages 1000
col file# for 9999999
select
c_h.file#,
case
when c_h.CHECKPOINT_CHANGE# > d_h.CHECKPOINT_CHANGE# then 'Fuzzy header!'
when c_h.CHECKPOINT_CHANGE# = d_h.CHECKPOINT_CHANGE# then 'ACID complaint!'
when c_h.CHECKPOINT_CHANGE# < d_h.CHECKPOINT_CHANGE# then 'Media recovery required!'
end Consistency_check
from
v$datafile c_h
inner join v$datafile_header d_h on (c_h.FILE#=d_h.FILE#)
order by 1;kish@STDBY>kish@STDBY>  2    3    4    5    6    7    8    9   10   11

   FILE# CONSISTENCY_CHECK
-------- ------------------------
       1 ACID complaint!
       3 ACID complaint!
       4 ACID complaint!
       5 ACID complaint!
       6 ACID complaint!
       7 ACID complaint!
       8 ACID complaint!
       9 ACID complaint!
      10 ACID complaint!
      11 ACID complaint!
      12 ACID complaint!
      13 ACID complaint!
      14 ACID complaint!

13 rows selected.
CONVERT PHYSICAL STANDBY TO SNAPSHOT STANDBY DATABASE USING MANUAL METHOD IN ORACLE DATAGUARD

Snapshot standby database is a new feature introduced in oracle version 11g which acts as a clone mirror image of primary database operating in read write mode without log apply.

If there is a physical standby database running, we can easily convert to snapshot standby for testing and offloading purposes.

As the SNAPSHOT STANDBY, name itself denotes that taking a snapshot of the primary database for performing quick tests.

A guaranteed restore point is automatically created by oracle after conversion of snapshot standby for rollback purposes to get back to previous physical standby state in case of failures.

FIG – 1.0 – SNAPSHOT STANDBY

Check prerequisites in primary and standby database before conversion.

Birds view steps:

Here PRIM stands for primary database and STDBY stands for physical standby database

kish@PRIM>select name,open_mode,database_role,switchover_status from gv$database;
kish@STDBY>select name,open_mode,database_role,switchover_status from gv$database;
kish@STDBY>alter database recover managed standby database cancel;
kish@STDBY>shu immediate;
kish@STDBY>startup mount;
kish@STDBY>alter database flashback on;
kish@STDBY>select flashback_on from v$database;
kish@STDBY>alter database convert to snapshot standby;
kish@STDBY>select database_role from v$database;
kish@STDBY>alter database open;
kish@STDBY>select open_mode from v$database;

In primary – switchover state should be TO STANDBY

kish@PRIM>select name,open_mode,database_role,switchover_status from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS
--------- -------------------- ---------------- --------------------
DBX00     READ WRITE           PRIMARY          TO STANDBY

Check standby database status and cancel MRP first

kish@STDBY>select name,open_mode,database_role,switchover_status from gv$database;

NAME      OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS
--------- -------------------- ---------------- --------------------
DBX00     MOUNTED              PHYSICAL STANDBY NOT ALLOWED

kish@STDBY>alter database recover managed standby database cancel;

Database altered.

Bounce the standby database in mount state. You dont need guaranteed restore point because oracle automatically create for you. No worries! But for safer side enable flashback

kish@STDBY>shu immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
kish@STDBY>startup mount;
ORACLE instance started.

Total System Global Area                                         1509949440 bytes
Fixed Size                                                          2924640 bytes
Variable Size                                                     989859744 bytes
Database Buffers                                                  503316480 bytes
Redo Buffers                                                       13848576 bytes
Database mounted.
kish@STDBY>select * from v$restore_point;

no rows selected

kish@STDBY>select flashback_on from v$database;

FLASHBACK_ON
------------------
NO
kish@STDBY>alter database flashback on;

Database altered.

kish@STDBY>select flashback_on from v$database;

FLASHBACK_ON
------------------
YES

Issue the below command and check the database open mode. If the database is opened in read write mode, then congratz ! you have converted to snapshot standby

kish@STDBY>alter database convert to snapshot standby;

Database altered.

kish@STDBY>select database_role from v$database;

DATABASE_ROLE
----------------
SNAPSHOT STANDBY

When the time you issue convert command, check the logs parallely for any errors and background working of snapshot standby. For learning purpose, you can see the comments below

alter database convert to snapshot standby
++++++++++ RVWR process automatically start to read and write flashback data from flashback buffer to FRA++++++++++

Starting background process RVWR
Wed Sep 22 03:56:28 2021
RVWR started with pid=31, OS id=9470
Wed Sep 22 03:56:29 2021

++++++++++Oracle automatically created GRP and allocated buffers in shared pool for flashback generation. Oracle also kills all RFS process to freeze log shipping. Cool isnt it !?++++++++++++

Allocated 15937344 bytes in shared pool for flashback generation buffer
Created guaranteed restore point SNAPSHOT_STANDBY_REQUIRED_09/22/2021 03:56:28
Killing 2 processes (PIDS:9451,9449) (all RFS) in order to disallow current and future RFS connections. Requested by OS process 9447
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
RESETLOGS after incomplete recovery UNTIL CHANGE 2137589 time 09/22/2021 03:53:16
Resetting resetlogs activation ID 2195284314 (0x82d9615a)
Online log /apps01/FRA/DBX01/onlinelog/o1_mf_1_jnl5w6gs_.log: Thread 1 Group 1 was previously cleared
Online log /apps01/FRA/DBX01/onlinelog/o1_mf_2_jnl5wd63_.log: Thread 1 Group 2 was previously cleared
Online log /apps01/FRA/DBX01/onlinelog/o1_mf_3_jnl5wllv_.log: Thread 1 Group 3 was previously cleared
Online log /apps01/FRA/DBX01/onlinelog/o1_mf_4_jnl5ww8k_.log: Thread 1 Group 4 was previously cleared
Online log /apps01/FRA/DBX01/onlinelog/stdbylog1.log: Thread 1 Group 5 was previously cleared
+++++++++++
Oracle also notes down the SCN of standby  before the physical standby get converted to snapshot standby
Set the new incarnation 
Then convert to snapshot standby
++++++++++++

Standby became primary SCN: 2137587
Wed Sep 22 03:56:31 2021
Setting recovery target incarnation to 3
CONVERT TO SNAPSHOT STANDBY: Complete - Database mounted as snapshot standby
Completed: alter database convert to snapshot standby
alter database open
Wed Sep 22 03:56:52 2021
Ping without log force is disabled
.
Wed Sep 22 03:56:52 2021
Assigning activation ID 2195429791 (0x82db999f)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /apps01/FRA/DBX01/onlinelog/o1_mf_1_jnl5w6gs_.log
Successful open of redo thread 1
Wed Sep 22 03:56:52 2021
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Sep 22 03:56:52 2021
ARC2: Becoming the 'no SRL' ARCH

Open the database and there you go!

kish@STDBY> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

kish@STDBY>alter database open;

Database altered.

kish@STDBY>select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE

RFS will be vanished because oracle killed all RFS process to freeze log shipping

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS                                                SEQUENCE#
--------- ------------ --------------------------------------------------
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0
ARCH      CONNECTED                                                     0

Also the standby sequence is reset

kish@STDBY>select group#,thread#,sequence#,FIRST_CHANGE#,NEXT_CHANGE# from v$log;

    GROUP#    THREAD#  SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
---------- ---------- ---------- ------------- ------------
         1          1          1       3627733   2.8147E+14
         2          1          0             0            0
         3          1          0             0            0
         8          1          0             0            0
         5          1          0             0            0
         6          1          0             0            0
         7          1          0             0            0
         4          1          0             0            0

8 rows selected.

Check the GRP status which is automatically created

kish@STDBY>select NAME,GUARANTEE_FLASHBACK_DATABASE,SCN from v$restore_point;

NAME            GUA                                                SCN
--------------- --- --------------------------------------------------
SNAPSHOT_STANDB YES                                            2137588
Y_REQUIRED_09/2
2/2021 03:56:28

I switch the logfile in primary

kish@PRIM>alter system switch logfile;

System altered.

Then i was able to see RFS and LNS in standby database. Not sure why ! But i can confirm that RFS receive the redo from primary but not applying it. MRP process doesnot apply logs nor running.

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
RFS       IDLE                  0
RFS       IDLE                 73
LNS       CONNECTED             0

7 rows selected.

kish@STDBY>!ps -ef|grep 'mrp'
oracle   10045  9976  0 17:08 pts/1    00:00:00 /bin/bash -c ps -ef|grep 'mrp'
oracle   10047 10045  0 17:08 pts/1    00:00:00 grep mrp

Lets not discuss more about internal workings of snapshot standby in this post as we see that in another post and lets get into testing of snapshot standby

In Snap standby: – Advantage

Now it is going to be interesting to perform some DDL and DML changes in snapshot standby for testing some application or business data.

kish@STDBY>create table t1 as select * from t;

Table created.
kish@STDBY>select * from t;

NAME               ID
---------- ----------
abc                 1
abc                 1

kish@STDBY>alter table t modify name varchar2(30);

Table altered.

kish@STDBY>insert into t values('snapshot_work_or_not?',2);

1 row created.

kish@STDBY>insert into t values('rechechk_work_or_not?',3);

1 row created.

kish@STDBY>insert into t values('rechechk_again',4);

1 row created.

kish@STDBY>insert into t values('it_worked',5);

1 row created.

kish@STDBY>commit;

Commit complete.

kish@STDBY>select * from t;

NAME                                   ID
------------------------------ ----------
abc                                     1
abc                                     1
snapshot_work_or_not?                   2
rechechk_work_or_not?                   3
rechechk_again                          4
it_worked                               5

6 rows selected.

Snapshot standby is also a boon for offloading the backups to reduce overhead to primary database even during the business hours. Awesome !

RMAN> backup current controlfile;

Starting backup at 02-OCT-21
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=64 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 02-OCT-21
channel ORA_DISK_1: finished piece 1 at 02-OCT-21
piece handle=/apps01/FRA/DBX01/backupset/2021_10_02/o1_mf_ncnnf_TAG20211002T172044_jojkwoz7_.bkp tag=TAG20211002T172044 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04
Finished backup at 02-OCT-21

Starting Control File and SPFILE Autobackup at 02-OCT-21
piece handle=/apps01/FRA/DBX01/autobackup/2021_10_02/o1_mf_s_1084900849_jojkwtf4_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 02-OCT-21
RMAN-08591: WARNING: invalid archived log deletion policy

RMAN> select database_role from v$database;

DATABASE_ROLE
----------------
SNAPSHOT STANDBY

To get back to physical standby

Bounce the database to mount mode

kish@STDBY>shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
kish@STDBY>startup mount;
ORACLE instance started.

Total System Global Area 1509949440 bytes
Fixed Size                  2924640 bytes
Variable Size             989859744 bytes
Database Buffers          503316480 bytes
Redo Buffers               13848576 bytes
Database mounted.

Convert to physical standby

kish@STDBY>alter database convert to physical standby;

Database altered.

Refer logs to learn what is happening behind the back

alter database convert to physical standby
Sat Oct 02 17:48:17 2021
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (dbx01)
Sat Oct 02 17:48:17 2021
++++++ Oracle again kill RFS process for revert +++++++

Killing 2 processes (PIDS:10799,10797) (all RFS) in order to disallow current and future RFS connections. Requested by OS process 10795

++++++Oracle removes the archivelog files which were generated for the table t during the DML changes++++++

Sat Oct 02 17:48:18 2021
Deleted Oracle managed file /apps01/FRA/DBX01/archivelog/2021_10_02/o1_mf_1_80_jojmggvg_.arc
Sat Oct 02 17:48:19 2021

++++++ Post conversion oracle also drops the restore point which was created automatically++++++

Flashback Restore Start
Flashback: dropped tablespace #3: 'TEMP' of pdb #1 from the control file.
Flashback: created tablespace #3: 'TEMP' of pdb #1 in the controlfile.
Flashback: dropped tablespace #6: 'TEMP1' of pdb #1 from the control file.
Flashback Restore Complete
Drop guaranteed restore point
Guaranteed restore point  dropped
Clearing standby activation ID 2196472829 (0x82eb83fd)
The primary database controlfile was created using the
'MAXLOGFILES 192' clause.
There is space for up to 184 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl5.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl6.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl7.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl8.f' SIZE 52428800;
ALTER DATABASE ADD STANDBY LOGFILE 'srl9.f' SIZE 52428800;

++++++Datafiles which were marked offline are changed back to online++++++

Offline data file 5 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 7 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 8 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 9 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 10 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 11 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 12 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Offline data file 13 marked as online during convert to standby or switchover to standby.
Restore of backup may be required if the file is not physically accessible.
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
ARCH: Warning; less destinations available than specified
by LOG_ARCHIVE_MIN_SUCCEED_DEST init.ora parameter
Physical Standby Database mounted.
CONVERT TO PHYSICAL STANDBY: Complete - Database mounted as physical standby
Completed: alter database convert to physical standby

Check if MRP is started and applying logs

kish@STDBY>!ps -ef|grep 'mrp'
oracle   10834     1  0 17:49 ?        00:00:14 ora_mrp0_dbx01
oracle   11357 10699  0 18:19 pts/1    00:00:00 /bin/bash -c ps -ef|grep 'mrp'
oracle   11359 11357  0 18:19 pts/1    00:00:00 grep mrp

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
RFS       IDLE                  0
RFS       IDLE                 80
MRP0      WAIT_FOR_LOG         80

7 rows selected.

Check the role and status

kish@STDBY>select database_role,status from v$database,v$instance;

DATABASE_ROLE    STATUS
---------------- ------------
PHYSICAL STANDBY MOUNTED

Verify the MRP sync

kish@STDBY>alter database recover managed standby database using current logfile disconnect;

Database altered.

kish@STDBY>select process,status,sequence# from v$managed_standby;

PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
ARCH      CONNECTED             0
RFS       RECEIVING            73
RFS       IDLE                  0
MRP0      APPLYING_LOG         71

7 rows selected.

Restore point is dropped

kish@STDBY>select * from v$restore_point;

no rows selected

Turn flashback off if no longer needed

kish@STDBY>alter database flashback off;

Database altered.