STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Alejandro Vargas |Principal |Principal Support Support Consultant Oracle Advanced Customer Services
Index Index........................................................ ................................................................................................... ........................................... ........ ................ ................. ......... 1 Summary...................................................... ........................................................................................................ .................................................. ......... .............. ..... 2 Important Configuration Tips ................................................... ..................... ............................. .............. ...... 3 Implementation Summary.................................................... ........................................................................................... ....................................... .3 backup...................................................... ........................................................................................................ .................................................. ......... .............. ..... 3 Standby Controlfile.................................................. ........................................ ........ 3 A copy of the password file generated on the primary database.......... ................... ................. .......... .. 3 An init.ora parameter file with specific standby database parameters set for both the primary and standby role .................................................... .................................................................................... ................................4 Configure the network to provide connectivity connec tivity to both primary and standby databases................................................... ............................................ ........ ................ ........... ... 6 Configure the local_listener parameter for none standard ports ........................ .............................. ...... 7 Create a set of standby redologs on both the primary and standby databases .... ........ ......7 Enabling Recovery on the Standby Database ............................................. ........ ................ ..........8 Enabling Dataguard broker on both Databases ....................................................... ............................................................ ..... 9 Create the Broker Configuration................................................. ............................... 10 Check the Broker Configuration..................................................... .......................................................................... ........................... ...... 11 Enable the Broker Configuration...................................................... ....................................................................... ......................... ........ 13 Check the Databases Transport Mode Setting............................................. ......... .............. ..... 14 Test Manual Switchover.................................................. .......................... .................................. ................ ........ 14 Check after Manual Switchover................................................... ..............................16 16 Test Manual Switchover Back to the Original Primary....................................... ...... ......16 16 Test Failing over ov er after a Primary Database Failure ................................. ........ ................ .......... .. 18 Reinstating the Primary Database after a Failure ............................. ..................................... ................. ................. ........ 20 Recreate the crashed primary after a failed reinstate................................... ......... .............. ..... 23 Recreate the broker configuration............................................... ............................... 26 Configure Flashback Database on the Standby S tandby Database .................................... ......28 28 Configure flashback database on the primary database.................................... ......... 28 Test switchover back and forth again aga in................................................ ........ ................ ................ ........ 29 Test Failing over after a Primary Database Failure (with Flashback Database set) . .33 Reinstating the Primary Database after a Failure ............................. ..................................... ................. ................. ........ 36 Cleanup of Archived Logs on the Standby Location.......................................... ....... 46 Script rman_clean_arch.bat ...................................................................................46 46 Script rman_clean_arch.rmn....................................................... .......................................................................... ........................... ........ 47 Log File rman_clean_arch.log ....................................................... ................................................................. ................... .............. ..... 47 Tnsnames Configuration for the Applications................................................ ........ ........... ... 48
1/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Summary The test described on this document was implemented to demonstrate how to manually manage switchover and emergency failover of a standby database configured in maximum performance lgwr async noaffirm mode. The maximum performance in Async No Affirm provide high level of protection but do not compromise the availability and performance of the primary database. The main purpose of this configuration was defined as ‘Provide a quick way to restore service in case of severe outage on the primary site’. It was decided not to use fast start failover but to put on the DBA team the decisi decision on of when when to fail fail over over to the the stand standby by site site after after assess assessing ing the crash crash situation. Two failover situations were analyzed and are presented on this document: On the first first scena scenario rio the datab database ases s were were conf configu igure red d with with flash flashba back ck data databas base e enabled with a retention time of 60 minutes. On the second scenario flashback database was not configured. Flashback database makes much easier to reinstate the failed database in case of requiring a failover failover to the standby. Once the failed server server is restarted restarted the old primary only requires to be mounted and the broker will flashback it to a previous SCN and then reinstate, recover and reconfigure it to have the standby role, automatically. That reduces the time where the main database is unprotected to the minimum. When flashback database database is not enabled, after failing over to the standby the old primary database will require to be rebuilt from a backup taken from the new primary. That makes the reinstate process more complex and it takes a much longer time. The environment used for this test was based on a Windows 2003 R2 with Oracle RDBMS 10.2.0.4 This document does not contain Dataguard step by step implementation steps but provide a summary of them. For a step by step implementation document please check Dataguard Configuration for FSFO
2/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Summary The test described on this document was implemented to demonstrate how to manually manage switchover and emergency failover of a standby database configured in maximum performance lgwr async noaffirm mode. The maximum performance in Async No Affirm provide high level of protection but do not compromise the availability and performance of the primary database. The main purpose of this configuration was defined as ‘Provide a quick way to restore service in case of severe outage on the primary site’. It was decided not to use fast start failover but to put on the DBA team the decisi decision on of when when to fail fail over over to the the stand standby by site site after after assess assessing ing the crash crash situation. Two failover situations were analyzed and are presented on this document: On the first first scena scenario rio the datab database ases s were were conf configu igure red d with with flash flashba back ck data databas base e enabled with a retention time of 60 minutes. On the second scenario flashback database was not configured. Flashback database makes much easier to reinstate the failed database in case of requiring a failover failover to the standby. Once the failed server server is restarted restarted the old primary only requires to be mounted and the broker will flashback it to a previous SCN and then reinstate, recover and reconfigure it to have the standby role, automatically. That reduces the time where the main database is unprotected to the minimum. When flashback database database is not enabled, after failing over to the standby the old primary database will require to be rebuilt from a backup taken from the new primary. That makes the reinstate process more complex and it takes a much longer time. The environment used for this test was based on a Windows 2003 R2 with Oracle RDBMS 10.2.0.4 This document does not contain Dataguard step by step implementation steps but provide a summary of them. For a step by step implementation document please check Dataguard Configuration for FSFO
2/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Important Configuration Tips The implementation of a physical standby database, configured on Maximum Perfor Performan mance ce LGWR LGWR ASYN ASYNC C NO AFFI AFFIRM RM trans transpor portt mode mode does does requir require e the the following: 1) A full full backup backup of of the primary primary datab database ase 2) A standby standby controlfile controlfile for the standby standby database database generated generated on on the primary primary database 3) A copy of the password password file file generated generated on the primary primary database database 4) An init.ora init.ora parameter parameter file with with specific standby database database parameters parameters set for both the primary and standby role 5) A network network configured configured to provide connectivity connectivity to both both primary primary and standby standby databases 6) A local_liste local_listener ner parameter parameter for none standard ports 7) A set of standby redologs on both the primary primary and standby standby database databases s
Implementation Summary
backup
The full backup was made by copying over all database files, while the primary primary databas database e was shutdo shutdown, wn, to the standby standby server. server. The primary database may be copied over while backup mode is enabled. Standby Controlfile
Was created on the primary database using the command: > alter database create standby controlfile as ‘c:\standby-ctlfile.ctl’ ; The standby controlfile was copied over to the standby server, and its location set on the parameter file of the standby database. A copy of the password file generated on the primary database
The password file needs to be generated on the primary and copied over; it cannot be generated on the standby server
3/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST An init.ora parameter file with specific standby database parameters set for both the primary and standby role
First I do prepare the a pfile from the normal primary database spfile, then edit it for the primary and for the standby. This is the version used for the primary database: AVDB.__db_cache_size=2197815296 AVDB.__java_pool_size=16777216 AVDB.__large_pool_size=16777216 AVDB.__shared_pool_size=268435456 AVDB.__streams_pool_size=117440512 *.audit_file_dest='C:\oracle\pro *.audit_file_dest='C:\oracle\product\10.2.0\admi duct\10.2.0\admin\AVDB\adum n\AVDB\adum p' *.background_dump_dest='E:\ORACL *.background_dump_dest='E:\ORACLE\ADMIN\AVDB\BDU E\ADMIN\AVDB\BDUMP' MP' *.compatible='10.2.0.3' *.control_files='E:\ORACLE\ORADA *.control_files='E:\ORACLE\ORADATA\AVDB\CONTROL0 TA\AVDB\CONTROL01.CTL','G:\ 1.CTL','G:\ ORACLE\ORADATA\AVDB\CONTROL02.CT ORACLE\ORADATA\AVDB\CONTROL02.CTL','D:\ORACLE\OR L','D:\ORACLE\ORADATA\AVDB\ ADATA\AVDB\ CONTROL03.CTL' *.core_dump_dest='E:\ORACLE\ADMI *.core_dump_dest='E:\ORACLE\ADMIN\AVDB\CDUMP' N\AVDB\CDUMP' *.db_block_size=8192 *.db_domain='' *.db_file_multiblock_read_count=16 *.db_files=2048 *.job_queue_processes=60 *.nls_date_format='DD/MM/YYYY' *.open_cursors=300 *.pga_aggregate_target=314572800 *.processes=1000 *.recovery_parallelism=4 *.REMOTE_LOGIN_PASSWORDFILE='EXCLUSIVE' *.resource_manager_plan='NIGHTTIME' *.session_cached_cursors=50 *.sessions=555 *.sga_max_size=2621440000 *.sga_target=2621440000 *.streams_pool_size=104857600 *.undo_management='AUTO' *.undo_tablespace='UNDOTBS' *.user_dump_dest='E:\ORACLE\ADMI *.user_dump_dest='E:\ORACLE\ADMIN\AVDB\UDUMP' N\AVDB\UDUMP' *.utl_file_dir='F:\ORA_UTL_DIR', *.utl_file_dir='F:\ORA_UTL_DIR','E:\ORACLE\admin 'E:\ORACLE\admin\AVDB\udump' \AVDB\udump' ## ## Standby relevant parameters ## ## db_file_name_convert and log_file_name_convert state in ## pairs the conversion that ## needs to be applied to the names registered on the \ ## standby controlfile. In this case the files located on
4/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ## the remote server under 'D:\D\ORACLE\ORADATA\AVDB' will ## be located on the local server on ## 'D:\ORACLE\ORADATA\AVDB'. The convertion continue for ## the next pair of paths and so on. ## *.db_file_name_convert='D:\D\ORACLE\ORADATA\AVDB','D:\ORACL E\ORADATA\AVDB','D:\E\ORACLE\ORADATA\AVDB','E:\ORACLE\ORADA TA\AVDB','D:\F\ORACLE\ORADATA\AVDB','F:\ORACLE\ORADATA\AVDB ','D:\G\ORACLE\ORADATA\AVDB','G:\ORACLE\ORADATA\AVDB' *.log_file_name_convert='D:\D\ORACLE\ORADATA\AVDB','D:\ORAC LE\ORADATA\AVDB' ## ## Db_name is the same for all databases on the ## configuration ## db_unique_name is specific for each instance ## *.db_name='AVDB' *.db_unique_name='AVDB' *.instance_name='AVDB' ## ## fal server will be always the remote database and client ## the local. These parameters will be active only when the ## database is on the standby role ## *.fal_client='AVDB' *.fal_server='AVDBSTD' ## ## log_archive_config list the names of the instances ## *.LOG_ARCHIVE_CONFIG='DG_CONFIG=(AVDB,AVDBSTD)' ## ## log_archive_dest_ is defined here for having the ## local database as primary and the destination 2 for the ## remote database as standby ## *.LOG_ARCHIVE_DEST_1='LOCATION=L:\ORACLE\oradata\AVDB\archi ve VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=AVDB' *.LOG_ARCHIVE_DEST_2='SERVICE=AVDBSTD LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) *.LOG_ARCHIVE_DEST_STATE_1='ENABLE' *.LOG_ARCHIVE_DEST_STATE_2='ENABLE' *.log_archive_format='AVDB_%T_%s_%R.arc' *.LOG_ARCHIVE_MAX_PROCESSES=4 ## ## standby_file_management auto instruct the server to ## replicate all ddl’s like create tablespace or add ## datafiles ## *.standby_file_management='auto'
5/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Configure the network to provide connectivity to both primary and standby databases
The listener and tnsnames need to have this specific configuration when using dataguard: # LISTENER.ORA SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = AVDB_DGMGRL) (SID_NAME = AVDB) (ORACLE_HOME = C:\oracle\product\10.2.0\db_1) ) (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = C:\oracle\product\10.2.0\db_1) (PROGRAM = extproc) ) ) INBOUND_CONNECTION_TIMEOUT_LISTENER = 0 LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER01)(PORT = 2115)) ) )
# TNSNAMES.ORA AVDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER01)(PORT = 2115)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = AVDB_DGMGRL) ) ) AVDBSTD = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER02)(PORT = 2115)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = AVDBSTD_DGMGRL) ) )
6/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ## The listeners need to be registered on tnsnames.ora also LISTENER_AVDB= (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER01)(PORT = 2115))) LISTENER_AVDBSTD= (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER02)(PORT = 2115)))
Configure the local_listener parameter for none standard ports
The local_listener parameter needs to be configured to the listener registered on the tnsnames.ora in this case on the primary database: > alter system set local_listener=’LISTENER_AVDB’ scope=both; On the standby database: > alter system set local_listener=’LISTENER_AVDBSTD’ scope=both; Check that the listener respond to tnsping: C:\Documents and Settings\db411>tnsping listener_AVDB TNS Ping Utility for 64-bit Windows: Version 10.2.0.4.0 Production on 19-JUL-2010 11:06:20 Copyright (c) 1997,
2007, Oracle.
All rights reserved.
Used parameter files: C:\oracle\product\10.2.0\db_1\network\admin\sqlnet.ora
Used TNSNAMES adapter to resolve the alias Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = SERVER01)(PORT = 2115))) OK (20 msec)
Create a set of standby redologs on both the primary and standby databases
In order to enable LGWR ASYNC transport mode we need to create standby redologs on both the primary and standby databases: The standby redologs need to be the same size as the online redologs and have the same number of online redo log groups +1 or more. In this case we have 8 online groups so we created 10 standby redologs on each database:
7/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
alter database add standby logfile group 08 'D:\ORACLE\ORADATA\AVDB\REDO08.LOG' SIZE 209715200; alter database add standby logfile group 09 'D:\ORACLE\ORADATA\AVDB\REDO09.LOG' SIZE 209715200; alter database add standby logfile group 10 'D:\ORACLE\ORADATA\AVDB\REDO10.LOG' SIZE 209715200; alter database add standby logfile group 11 'D:\ORACLE\ORADATA\AVDB\REDO11.LOG' SIZE 209715200; alter database add standby logfile group 12 'D:\ORACLE\ORADATA\AVDB\REDO12.LOG' SIZE 209715200; alter database add standby logfile group 13 'D:\ORACLE\ORADATA\AVDB\REDO13.LOG' SIZE 209715200; alter database add standby logfile group 14 'D:\ORACLE\ORADATA\AVDB\REDO14.LOG' SIZE 209715200; alter database add standby logfile group 15 'D:\ORACLE\ORADATA\AVDB\REDO15.LOG' SIZE 209715200; alter database add standby logfile group 16 'D:\ORACLE\ORADATA\AVDB\REDO16.LOG' SIZE 209715200; alter database add standby logfile group 17 'D:\ORACLE\ORADATA\AVDB\REDO17.LOG' SIZE 209715200; alter database add standby logfile group 18 'D:\ORACLE\ORADATA\AVDB\REDO18.LOG' SIZE 209715200;
Enabling Recovery on the Standby Database Once all this configuration steps are ready we can check enabling recovery, for this we startup mount the standby database and execute the recover database command: alter database add standby logfile group 10 'D:\ORACLE\ORADATA\AVDB\REDO10.LOG' SIZE 209715200; Then we can do switch logs on the primary and check that the changes are applied to the standby The Standby:
Database log mode Archive Mode Automatic archival Enabled Archive destination D:\L\ORACLE\oradata\AVDB\archive Oldest online log sequence 0 Next log sequence to archive 0 Current log sequence 1031
8/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDBSTD MOUNTED PHYSICAL STANDBY
The Primary:
Database log mode Automatic archival Archive destination L:\ORACLE\oradata\AVDB\archive Oldest online log sequence Next log sequence to archive Current log sequence
Archive Mode Enabled
1026 1031 1031
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDB READ WRITE PRIMARY
Enabling Dataguard broker on both Databases Once the standby is receiving transactions and is kept in sync we can move over to the dataguard broker the responsibility of managing the environment, by setting both on the primary and standby databases the following parameter:
> alter system set dg_broker_start=true This parameter will start the broker background process and will register the XPT service with the local listener. This service will be used for communications between the broker and the databases. It is important to assure that this service was attached to the listener before configuring the borker, to do this use the lsnrctl status command: C:\Documents and Settings\db411>lsnrctl status LSNRCTL for 64-bit Windows: Version 10.2.0.4.0 - Production on 19-JUL-2010 11:11:03 Copyright (c) 1991, 2007, Oracle.
All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER01) (PORT=2115))) STATUS of the LISTENER -----------------------Alias LISTENER
9/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Version TNSLSNR for 64-bit Windows: Version 10.2.0.4.0 - Production Start Date 19-JUL-2010 10:25:32 Uptime 0 days 0 hr. 45 min. 36 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File C:\oracle\product\10.2.0\db_1\network\admin\listener.ora Listener Log File C:\oracle\product\10.2.0\db_1\network\log\listener.log Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=SERVER01) (PORT=2115))) Services Summary... Service "PLSExtProc" has 1 instance(s). Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service... Service "AVDB" has 1 instance(s). Instance "AVDB", status READY, has 1 handler(s) for this service... Service "AVDB_DGMGRL" has 1 instance(s). Instance "AVDB", status UNKNOWN, has 1 handler(s) for this service... Service "AVDB_XPT" has 1 instance(s). Instance "AVDB", status READY, has 1 handler(s) for this service... The command completed successfully
Create the Broker Configuration Using the dataguard broker greatly simplify the administration of the standby configuration, it automates most of the administrative tasks.
C:\Documents and Settings\db411>dgmgrl sys/AVDBPW DGMGRL for 64-bit Windows: Version 10.2.0.4.0 - 64bit Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> create configuration dgAVDB as primary database is AVDB connect identifier is AVDB; Configuration "dgAVDB" created with primary database "AVDB"
10/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
DGMGRL> add database AVDBSTD as connect identifier is AVDBSTD maintained as physical; Database "AVDBSTD" added
Check the Broker Configuration DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: NO Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": DISABLED DGMGRL> show database verbose AVDB; Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PRIMARY NO OFFLINE
Properties: InitialConnectIdentifier ObserverConnectIdentifier LogXptMode Dependency DelayMins Binding MaxFailure MaxConnections ReopenSecs NetTimeout LogShipping PreferredApplyInstance ApplyInstanceTimeout ApplyParallel StandbyFileManagement ArchiveLagTarget LogArchiveMaxProcesses LogArchiveMinSucceedDest
11/49
= = = = = = = = = = = = = = = = = =
'AVDB' '' 'ASYNC' '' '0' 'OPTIONAL' '0' '1' '300' '180' 'ON' '' '0' 'AUTO' 'auto' '0' '4' '1'
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST DbFileNameConvert = 'D:\D\ORACLE\ORADATA\AVDB, D:\ORACLE\ORADATA\AVDB, D:\E\ORACLE\ORADATA\AVDB, E:\OR ACLE\ORADATA\AVDB, D:\F\ORACLE\ORADATA\AVDB, F:\ORACLE\ORADATA\AVDB, D:\G\ORACLE\ORADATA\AVDB, G:\ORACLE\ORADATA\AVDB' LogFileNameConvert = 'D:\D\ORACLE\ORADATA\AVDB, D:\ORACLE\ORADATA\AVDB' FastStartFailoverTarget = '' StatusReport = '(monitor)' InconsistentProperties = '(monitor)' InconsistentLogXptProps = '(monitor)' SendQEntries = '(monitor)' LogXptStatus = '(monitor)' RecvQEntries = '(monitor)' HostName = 'SERVER01' SidName = 'AVDB' LocalListenerAddress = '(ADDRESS=(PROTOCOL=TCP) (HOST=SERVER01)(PORT=2115))' StandbyArchiveLocation = 'L:\ORACLE\oradata\AVDB\archive' AlternateLocation = '' LogArchiveTrace = '0' LogArchiveFormat = 'AVDB_%T_%s_%R.arc' LatestLog = '(monitor)' TopWaitEvents = '(monitor)' Current status for "AVDB": DISABLED DGMGRL> show database verbose AVDBSTD; Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PHYSICAL STANDBY NO OFFLINE
Properties: InitialConnectIdentifier ObserverConnectIdentifier LogXptMode Dependency DelayMins Binding MaxFailure MaxConnections ReopenSecs NetTimeout LogShipping PreferredApplyInstance ApplyInstanceTimeout ApplyParallel StandbyFileManagement ArchiveLagTarget LogArchiveMaxProcesses
12/49
= = = = = = = = = = = = = = = = =
'AVDBSTD' '' 'ASYNC' '' '0' 'OPTIONAL' '0' '1' '300' '180' 'ON' '' '0' 'AUTO' 'AUTO' '0' '4'
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST LogArchiveMinSucceedDest = '1' DbFileNameConvert = 'D:\ORACLE\ORADATA\AVDB, D:\D\ORACLE\ORADATA\AVDB, E:\ORACLE\ORADATA\AVDB, D:\E\OR ACLE\ORADATA\AVDB, F:\ORACLE\ORADATA\AVDB, D:\F\ORACLE\ORADATA\AVDB, G:\ORACLE\ORADATA\AVDB, D:\G\ORACLE\ORADATA\AVDB' LogFileNameConvert = 'D:\ORACLE\ORADATA\AVDB, D:\D\ORACLE\ORADATA\AVDB' FastStartFailoverTarget = '' StatusReport = '(monitor)' InconsistentProperties = '(monitor)' InconsistentLogXptProps = '(monitor)' SendQEntries = '(monitor)' LogXptStatus = '(monitor)' RecvQEntries = '(monitor)' HostName = 'SERVER02' SidName = 'AVDBSTD' LocalListenerAddress = '(ADDRESS=(PROTOCOL=TCP) (HOST=SERVER02)(PORT=2115))' StandbyArchiveLocation = 'D:\L\ORACLE\oradata\AVDB\archive' AlternateLocation = '' LogArchiveTrace = '0' LogArchiveFormat = 'AVDB_%T_%s_%R.arc' LatestLog = '(monitor)' TopWaitEvents = '(monitor)' Current status for "AVDBSTD": DISABLED
Enable the Broker Configuration DGMGRL> enable configuration; Enabled. DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": SUCCESS DGMGRL> enable database AVDBSTD; Enabled. DGMGRL> show configuration; Configuration Name: Enabled:
13/49
dgAVDB YES
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PRIMARY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PHYSICAL STANDBY YES ONLINE
Current status for "AVDBSTD": SUCCESS
Check the Databases Transport Mode Setting DGMGRL> show LogXptMode DGMGRL> show LogXptMode
database AVDB logxptmode = 'ASYNC' database AVDBSTD logxptmode = 'ASYNC'
Test Manual Switchover Before doing a manual switchover is always recommended to check that the configuration is healthy: DGMGRL> show configuration; Configuration
14/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": SUCCESS
Execute the switchover:
DGMGRL> switchover to AVDBSTD; Performing switchover NOW, please wait... Operation requires shutdown of instance "AVDB" on database "AVDB" Shutting down instance "AVDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires shutdown of instance "AVDBSTD" on database "AVDBSTD" Shutting down instance "AVDBSTD"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "AVDB" on database "AVDB" Starting instance "AVDB"... ORACLE instance started. Database mounted. Operation requires startup of instance "AVDBSTD" on database "AVDBSTD" Starting instance "AVDBSTD"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "AVDBSTD" Check after the switchover succeeded:
DGMGRL> show configuration Configuration Name: Enabled: 15/49
dgAVDB YES
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Physical standby database AVDBSTD - Primary database Current status for "dgAVDB": SUCCESS
Check after Manual Switchover
On the new primary database AVDBSTD we can see that the broker automatically reconfigured the log_archive_dest_2 to match the transport mode for the standby. SQL> show parameters log_archive_dest_2 log_archive_dest_2 got reconfigured, note the LGWR ASYNC NOAFFIRM property NAME TYPE VALUE ------------------------------------ ----------- -----------------------------log_archive_dest_2 string service="(DESCRIPTION=(ADDRESS _LIST=(ADDRESS=(PROTOCOL=TCP)( HOST= SERVER02 )(PORT=2115)))(CONNECT_D ATA=(SERVICE_NAME=AVDB_XPT)(IN STANCE_NAME=AVDB)(SERVER=dedic ated)))", LGWR ASYNC NOAFFI RM delay=0 OPTIONAL max_failur e=0 max_connections=1 reopen =300 db_unique_name="AVDB" reg ister net_timeout=180 valid_f or=(online_logfile,primary_rol e)
On the new standby log_archive_dest_2 got cleared: SQL> show parameters log_archive_dest_2 NAME TYPE VALUE ------------------------------------ ----------- -----------------------------log_archive_dest_2 string
Test Manual Switchover Back to the Original Primary DGMGRL> switchover to AVDB Performing switchover NOW, please wait... Operation requires shutdown of instance "AVDBSTD" on database "AVDBSTD"
16/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Shutting down instance "AVDBSTD"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires shutdown of instance "AVDB" on database "AVDB" Shutting down instance "AVDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "AVDBSTD" on database "AVDBSTD" Starting instance "AVDBSTD"... ORACLE instance started. Database mounted. Operation requires startup of instance "AVDB" on database "AVDB" Starting instance "AVDB"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "AVDB" DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
17/49
AVDB PRIMARY YES ONLINE
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PHYSICAL STANDBY YES ONLINE
Current status for "AVDBSTD": SUCCESS DGMGRL>
Test Failing over after a Primary Database Failure In this case we will check the scenario this configuration is planned to be used on. The primary database crash and cannot be recovered in a short period of time, then we need to fail over to the standby database. Before starting we will check status of both databases: The Standby:
Database log mode Archive Mode Automatic archival Enabled Archive destination D:\L\ORACLE\oradata\AVDB\archive Oldest online log sequence 0 Next log sequence to archive 0 Current log sequence 1031 NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDBSTD MOUNTED PHYSICAL STANDBY
The Primary:
Database log mode
18/49
Archive Mode
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Automatic archival Archive destination L:\ORACLE\oradata\AVDB\archive Oldest online log sequence Next log sequence to archive Current log sequence
Enabled
1026 1031 1031
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDB READ WRITE PRIMARY
Abort the primary database to simulate a failure: SQL> shutdown abort;
From alert log: Mon Jul 19 13:24:52 2010 LNS: Standby redo logfile selected for thread 1 sequence 1031 for destination LOG_ARCHIVE_DEST_2 Mon Jul 19 13:25:16 2010 Shutting down instance (abort) License high water mark = 5 Instance terminated by USER, pid = 2204 Mon Jul 19 13:29:39 2010 WARNING: inbound connection timed out (ORA-3136) Connect to DGMGRL and to the standby database:
DGMGRL> connect sys/AVDBPW@AVDBSTD Connected. DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Primary database AVDBSTD - Physical standby database Current status for "dgAVDB": Error: ORA-16625: cannot reach the database Execute the failover:
19/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
DGMGRL> failover to AVDBSTD; Performing failover NOW, please wait... Failover succeeded, new primary is "AVDBSTD" DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Physical standby database (disabled) AVDBSTD - Primary database Current status for "dgAVDB": SUCCESS Check the new primary Note that the sequence was reset to be #1
Database log mode Archive Mode Automatic archival Enabled Archive destination D:\L\ORACLE\oradata\AVDB\archive Oldest online log sequence 1 Next log sequence to archive 1 Current log sequence 1 NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDBSTD READ WRITE PRIMARY
Reinstating the Primary Database after a Failure The next step is to reinstate the failed Primary database, here we may find one of two scenarios: 1) the primary can be reinstated, this is most feasible when there were no changes on the new primary yet, or 2) the primary needs to be recreated. We need to check the feedback the broker will provide and act accordingly. To avoid being required to rebuild the primary it is convenient to enable flashback database that will permit to the broker to flashback the database to an SCN prior to the crash and then reinstate it.
20/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Physical standby database (disabled) AVDBSTD - Primary database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PRIMARY YES ONLINE
Current status for "AVDBSTD": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY NO ONLINE
Current status for "AVDB": Error: ORA-16661: the standby database needs to be reinstated The message ORA-16661 means that the broker will be able to reinstate the database; otherwise we will need to recreate the old primary. To reinstate we need to mount the failed database
21/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
C:\Documents and Settings\db411\Desktop\scripts>sqlplus sys/AVDBPW@AVDB as sysdba SQL*Plus: Release 10.2.0.4.0 - Production on Mon Jul 19 13:39:12 2010 Copyright (c) 1982, 2007, Oracle.
All Rights Reserved.
Connected to an idle instance. SQL> startup mount; ORACLE instance started. Total System Global Area 2634022912 bytes Fixed Size 2068072 bytes Variable Size 419430808 bytes Database Buffers 2197815296 bytes Redo Buffers 14708736 bytes Database mounted. Then connect to the DGMGRL
C:\Documents and Settings\db411>dgmgrl sys/AVDBPW@AVDBSTD DGMGRL for 64-bit Windows: Version 10.2.0.4.0 - 64bit Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDB - Physical standby database (disabled) AVDBSTD - Primary database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB
22/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY NO ONLINE
Current status for "AVDB": Error: ORA-16661: the standby database needs to be reinstated
DGMGRL> reinstate database AVDB; Reinstating database "AVDB", please wait... Error: ORA-16653: failed to reinstate database Failed. Reinstatement of database "AVDB" failed DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY NO ONLINE
Current status for "AVDB": Error: ORA-16795: database resource guard detects that database re-creation is required
Recreate the crashed primary after a failed reinstate If reinstatement fail then we need to recreate the failed primary database by following the exact same procedure we used to create the original standby Once the old primary is ready as a standby database we will remove it from the broker and re-add it to the configuration, in addition we will configure flashback database to test again reinstantiation after a critical failure. Once the backup finished I have edited the new pfile for the standby and create from it a new spfile 23/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
C:\Documents and Settings\db411>sqlplus SYS/AVDBPW@AVDB AS SYSDBA Connected to an idle instance. SQL> CREATE SPFILE FROM PFILE='C:\oracle\product\10.2.0\db_1\database\init4standby. ora'; File created. SQL> STARTUP MOUNT; ORACLE instance started. Total System Global Area 2634022912 bytes Fixed Size 2068072 bytes Variable Size 419430808 bytes Database Buffers 2197815296 bytes Redo Buffers 14708736 bytes Database mounted. On the primary database enable destination 2 and check the standby is receiving the changes:
SQL> show parameters log_archive_dest_2 NAME TYPE VALUE ------------------------------------ ----------- -----------------------------log_archive_dest_2 string service="(DESCRIPTION=(ADDRESS _LIST=(ADDRESS=(PROTOCOL=TCP)( HOST=SERVER01 )(PORT=2115)))(CONNECT_D ATA=(SERVICE_NAME=AVDB_XPT)(IN STANCE_NAME=AVDB)(SERVER=dedic ated)))", LGWR ASYNC NOAFFI RM delay=0 OPTIONAL max_failur e=0 max_connections=1 reopen =300 db_unique_name="AVDB" reg ister net_timeout=180 valid_f or=(online_logfile,primary_rol e)
Check that Primary and Standby are in Sync Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence NAME
24/49
DB_UNIQUE_NAME
Archive Mode Enabled D:\L\ORACLE\oradata\AVDB\archive 3 10 10 OPEN_MODE
DATABASE_ROLE
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST --------- ------------------------------ ---------- ---------------AVDB AVDBSTD READ WRITE PRIMARY Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence
Archive Mode Enabled L:\ORACLE\oradata\AVDB\archive 9 0 10
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDB MOUNTED PHYSICAL STANDBY
After that I need to enable the broker configure the local listener and check that the XPT service was registered with the listener
SQL> alter system set dg_broker_start=true scope=both; System altered. SQL> alter system set local_listener='LISTENER_AVDB' scope=both; C:\Documents and Settings\db411\Desktop\scripts>lsnrctl status LSNRCTL for 64-bit Windows: Version 10.2.0.4.0 - Production on 19-JUL2010 17:34:27 Copyright (c) 1991, 2007, Oracle.
All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER01) (PORT=2115))) STATUS of the LISTENER -----------------------Alias LISTENER Version TNSLSNR for 64-bit Windows: Version 10.2.0.4.0 - Production Start Date 19-JUL-2010 10:25:32 Uptime 0 days 7 hr. 9 min. 0 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File C:\oracle\product\10.2.0\db_1\network\admin\listener.ora Listener Log File C:\oracle\product\10.2.0\db_1\network\log\listener.log Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=SERVER01)(PORT=2115))) Services Summary... Service "PLSExtProc" has 1 instance(s).
25/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service... Service "AVDB" has 1 instance(s). Instance "AVDB", status READY, has 1 handler(s) for this service... Service "AVDB_DGMGRL" has 1 instance(s). Instance "AVDB", status UNKNOWN, has 1 handler(s) for this service... Service "AVDB_XPT" has 1 instance(s). Instance "AVDB", status READY, has 1 handler(s) for this service... The command completed successfully
Recreate the broker configuration C:\Documents and Settings\db411>dgmgrl sys/AVDBPW@AVDBSTD DGMGRL for 64-bit Windows: Version 10.2.0.4.0 - 64bit Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> create configuration dgAVDB > as primary database is AVDBSTD > connect identifier is AVDBSTD; Configuration "dgAVDB" created with primary database "AVDBSTD" DGMGRL> add database AVDB as > connect identifier is AVDB > maintained as physical; Database "AVDB" added DGMGRL> show configuration; Configuration Name: dgAVDB Enabled: NO Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Primary database AVDB - Physical standby database Current status for "dgAVDB": DISABLED DGMGRL> enable configuration; Enabled. DGMGRL> enable database AVDB; Enabled. DGMGRL> show configuration Configuration Name: Enabled: Protection Mode: Fast-Start Failover: Databases:
26/49
dgAVDB YES MaxPerformance DISABLED
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST AVDBSTD - Primary database AVDB - Physical standby database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PRIMARY YES ONLINE
Current status for "AVDBSTD": SUCCESS
27/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Configure Flashback Database on the Standby Database First we will configure flashback on the standby database SQL> select NAME,FLASHBACK_ON from v$database; NAME FLASHBACK_ON --------- -----------------AVDB NO SQL> alter system set db_recovery_file_dest_size=10000M scope=both; System altered. SQL> alter system set db_recovery_file_dest='L:\ORACLE\oradata\AVDB\flashback' scope=both; System altered. SQL> alter system set db_flashback_retention_target=60 scope=both; System altered. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE cancel; Database altered. SQL> ALTER DATABASE FLASHBACK ON; Database altered. SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE SWITCHOVER DISCONNECT NODELAY;
THROUGH ALL
Database altered.
Configure flashback database on the primary database SQL> alter system set db_recovery_file_dest_size=10000M scope=both; System altered. SQL> alter system set db_recovery_file_dest='D:\L\ORACLE\oradata\AVDB\flashback' scope=both;
28/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST System altered. SQL> alter system set db_flashback_retention_target=60 scope=both; System altered. SQL> SHUTDOWN IMMEDIATE; Database closed. Database dismounted. ORACLE instance shut down. SQL> EXIT Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options C:\Documents and Settings\db411\Desktop\scripts>sqlplus sys/AVDBPW@AVDBSTD as sysdba SQL*Plus: Release 10.2.0.4.0 - Production on Mon Jul 19 18:52:24 2010 Copyright (c) 1982, 2007, Oracle.
All Rights Reserved.
Connected to an idle instance. SQL> STARTUP MOUNT; ORACLE instance started. Total System Global Area 2634022912 bytes Fixed Size 2068072 bytes Variable Size 654311832 bytes Database Buffers 1962934272 bytes Redo Buffers 14708736 bytes Database mounted. SQL> ALTER DATABASE FLASHBACK ON; Database altered. SQL> ALTER DATABASE OPEN; Database altered.
Test switchover back and forth again Check that the environment is healthy before the switchover DGMGRL> connect sys/AVDBPW Connected. DGMGRL> show configuration
29/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Physical standby database AVDB - Primary database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PRIMARY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PHYSICAL STANDBY YES ONLINE
Current status for "AVDBSTD": SUCCESS
Switchover DGMGRL> switchover to AVDBSTD Performing switchover NOW, please wait... Operation requires shutdown of instance "AVDB" on database "AVDB" Shutting down instance "AVDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires shutdown of instance "AVDBSTD" on database "AVDBSTD" Shutting down instance "AVDBSTD"... ORA-01109: database not open Database dismounted. ORACLE instance shut down.
30/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Operation requires startup of instance "AVDB" on database "AVDB" Starting instance "AVDB"... ORACLE instance started. Database mounted. Operation requires startup of instance "AVDBSTD" on database "AVDBSTD" Starting instance "AVDBSTD"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "AVDBSTD"
Switchover back DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Primary database AVDB - Physical standby database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PRIMARY YES ONLINE
Current status for "AVDBSTD": SUCCESS
DGMGRL> switchover to AVDB Performing switchover NOW, please wait...
31/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Operation requires shutdown of instance "AVDBSTD" on database "AVDBSTD" Shutting down instance "AVDBSTD"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires shutdown of instance "AVDB" on database "AVDB" Shutting down instance "AVDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "AVDBSTD" on database "AVDBSTD" Starting instance "AVDBSTD"... ORACLE instance started. Database mounted. Operation requires startup of instance "AVDB" on database "AVDB" Starting instance "AVDB"... ORACLE instance started. Database mounted. Switchover succeeded, new primary is "AVDB" DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Physical standby database AVDB - Primary database Current status for "dgAVDB": SUCCESS DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PRIMARY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled:
32/49
AVDBSTD PHYSICAL STANDBY YES
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Intended State: Instance(s): AVDBSTD
ONLINE
Current status for "AVDBSTD": SUCCESS
Test Failing over after a Primary Database Failure (with Flashback Database set) Databases are in synch Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence
Archive Mode Enabled L:\ORACLE\oradata\AVDB\archive 56 61 61
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- -------------------- ---------- ---------------AVDB AVDB READ WRITE PRIMARY Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence
Archive Mode Enabled dgsby_AVDBSTD 0 0 61
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- --------------------- ---------- ---------------AVDB AVDBSTD MOUNTED PHYSICAL STANDBY
The broker configuration is healthy DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Physical standby database AVDB - Primary database Current status for "dgAVDB": SUCCESS
33/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PRIMARY YES ONLINE
Current status for "AVDB": SUCCESS DGMGRL> show database AVDBSTD Database Name: Role: Enabled: Intended State: Instance(s): AVDBSTD
AVDBSTD PHYSICAL STANDBY YES ONLINE
Current status for "AVDBSTD": SUCCESS
Check That Flashback Database is Configured On Primary and Standby On AVDB SQL> select name,db_unique_name,flashback_on from v$database; NAME DB_UNIQUE_NAME FLASHBACK_ON --------- ------------------------------ -----------------AVDB AVDB YES select estimated_flashback_size, retention_target, flashback_size from v$flashback_database_log ; ESTIMATED_FLASHBACK_SIZE RETENTION_TARGET FLASHBACK_SIZE ------------------------ ---------------- -------------9856000 60 79708160 select oldest_flashback_scn, oldest_flashback_time from v$flashback_database_log ; OLDEST_FLASHBACK_SCN OLDEST_FL -------------------- ---------
34/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST 12261282579 25-JUL-10
On AVDBSTD SQL> select name,db_unique_name,flashback_on from v$database; NAME DB_UNIQUE_NAME FLASHBACK_ON --------- ------------------------------ -----------------AVDB AVDBSTD YES select estimated_flashback_size,retention_target, flashback_size from v$flashback_database_log ; ESTIMATED_FLASHBACK_SIZE RETENTION_TARGET FLASHBACK_SIZE ------------------------ ---------------- -------------11328512 60 79708160 select oldest_flashback_scn,oldest_flashback_time from v$flashback_database_log ; OLDEST_FLASHBACK_SCN OLDEST_FL -------------------- --------12261282297 25-JUL-10
The primary database crashes Sun Jul 25 09:00:12 2010 LNS: Standby redo logfile selected for thread 1 sequence 61 for destination LOG_ARCHIVE_DEST_2 Sun Jul 25 09:23:32 2010 Shutting down instance (abort) License high water mark = 5 Instance terminated by USER, pid = 1044
Connect to DGMGRL and to the standby database:
C:\Documents and Settings\db411>dgmgrl sys/AVDBPW@AVDBSTD DGMGRL for 64-bit Windows: Version 10.2.0.4.0 - 64bit Production Copyright (c) 2000, 2005, Oracle. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected. DGMGRL> show configuration
35/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Physical standby database AVDB - Primary database Current status for "dgAVDB": Error: ORA-16625: cannot reach the database Execute the failover:
DGMGRL> failover to AVDBSTD; Performing failover NOW, please wait... Failover succeeded, new primary is "AVDBSTD" DGMGRL> show configuration Configuration Name: dgAVDB Enabled: YES Protection Mode: MaxPerformance Fast-Start Failover: DISABLED Databases: AVDBSTD - Primary database AVDB - Physical standby database (disabled) Current status for "dgAVDB": SUCCESS Check the new primary Note that the sequence was reset to be #1 Database log mode Automatic archival Archive destination Oldest online log sequence Next log sequence to archive Current log sequence
Archive Mode Enabled D:\L\ORACLE\oradata\AVDB\archive 1 1 1
NAME DB_UNIQUE_NAME OPEN_MODE DATABASE_ROLE --------- ------------------------------ ---------- ---------------AVDB AVDBSTD READ WRITE PRIMARY
Reinstating the Primary Database after a Failure
36/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST
The next step is to reinstate the failed Primary database, here we may find one of two scenarios: 1) the primary can be reinstated, this is most feasible when there were no changes on the new primary yet, or 2) the primary needs to be recreated. We need to check the feedback the broker will provide and act accordingly. To avoid being required to rebuild the primary it is convenient to enable flashback database that will permit to the broker to flashback the database to an SCN prior to the crash and then reinstate it.
DGMGRL> show database AVDB Database Name: Role: Enabled: Intended State: Instance(s): AVDB
AVDB PHYSICAL STANDBY NO ONLINE
Current status for "AVDB": Error: ORA-16661: the standby database needs to be reinstated The message ORA-16661 means that the broker will be able to reinstate the database; otherwise we will need to recreate the old primary. To reinstate we need to mount the failed database C:\Documents and Settings\db411\Desktop\scripts>sqlplus sys/AVDBPW@AVDB as sysdba SQL*Plus: Release 10.2.0.4.0 - Production on Sun Jul 25 09:32:24 2010 Copyright (c) 1982, 2007, Oracle.
All Rights Reserved.
Connected to an idle instance. SQL> startup mount ORACLE instance started. Total System Global Area 2634022912 bytes Fixed Size 2068072 bytes Variable Size 436208024 bytes Database Buffers 2181038080 bytes Redo Buffers 14708736 bytes
37/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Database mounted.
From within DGMGRL reinstate the failed database
DGMGRL> reinstate database AVDB; Reinstating database "AVDB", please wait... Operation requires shutdown of instance "AVDB" on database "AVDB" Shutting down instance "AVDB"... ORA-01109: database not open Database dismounted. ORACLE instance shut down. Operation requires startup of instance "AVDB" on database "AVDB" Starting instance "AVDB"... ORACLE instance started. Database mounted. Continuing to reinstate database "AVDB" ... Reinstatement of database "AVDB" succeeded Check the reinstated database alert log
All the stages of the reinstate can be followed on the alert log of the reinstated database: 1) database is mounted Database mounted in Exclusive Mode Completed: ALTER DATABASE MOUNT Sun Jul 25 09:32:41 2010
2) the broker process is started and it will drive the reinstatement Starting Data Guard Broker (DMON) NSV0 started with pid=16, OS id=3868 INSV started with pid=17, OS id=4064 Sun Jul 25 09:32:48 2010
3) the old primary log_archive_dest_2 and log_archive_dest_state_2 are reset ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH; NSV0 started with pid=18, OS id=3676 RSM0 started with pid=19, OS id=2900 Sun Jul 25 09:34:56 2010 ALTER SYSTEM SET log_archive_dest_2='' SCOPE=BOTH; Sun Jul 25 09:34:56 2010
38/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH; Sun Jul 25 09:34:56 2010
4) database is flashed back to an SCN so that it can be resynchronized with the new primary FLASHBACK DATABASE TO SCN 12261313471 Flashback Restore Start Sun Jul 25 09:35:11 2010 Flashback Restore Complete Flashback Media Recovery Start parallel recovery started with 7 processes Sun Jul 25 09:35:19 2010 Recovery of Online Redo Log: Thread 1 Group 3 Seq 58 0 Mem# 0: D:\ORACLE\ORADATA\AVDB\REDO3.LOG Sun Jul 25 09:35:19 2010 Recovery of Online Redo Log: Thread 1 Group 4 Seq 59 0 Mem# 0: D:\ORACLE\ORADATA\AVDB\REDO4.LOG Sun Jul 25 09:35:20 2010 Recovery of Online Redo Log: Thread 1 Group 5 Seq 60 0 Mem# 0: D:\ORACLE\ORADATA\AVDB\REDO5.LOG Sun Jul 25 09:35:20 2010 Recovery of Online Redo Log: Thread 1 Group 6 Seq 61 0 Mem# 0: D:\ORACLE\ORADATA\AVDB\REDO6.LOG Sun Jul 25 09:35:21 2010 Incomplete Recovery applied until change 12261313472 Flashback Media Recovery Complete Completed: FLASHBACK DATABASE TO SCN 12261313471
Reading mem
Reading mem
Reading mem
Reading mem
5) The database is converted to a physical standby Sun Jul 25 09:35:27 2010 alter database convert to physical standby Clearing standby activation ID 1046488175 (0x3e60246f) The primary database controlfile was created using the 'MAXLOGFILES 42' clause. There is space for up to 34 standby redo logfiles Use the following SQL commands on the standby database to create standby redo logfiles that match the primary database: ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl5.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl6.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl7.f' SIZE 209715200;
39/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ALTER DATABASE ADD STANDBY LOGFILE 'srl8.f' SIZE 209715200; ALTER DATABASE ADD STANDBY LOGFILE 'srl9.f' SIZE 209715200; Completed: alter database convert to physical standby
6) The database is shutdown Sun Jul 25 09:35:30 2010 Shutting down instance: further logons disabled Sun Jul 25 09:35:30 2010 Stopping background process CJQ0 Sun Jul 25 09:35:30 2010 Stopping background process MMNL Sun Jul 25 09:35:31 2010 Stopping background process MMON Sun Jul 25 09:35:32 2010 Shutting down instance (immediate) License high water mark = 9 Sun Jul 25 09:35:32 2010 Stopping Job queue slave processes, flags = 7 Sun Jul 25 09:35:32 2010 Job queue slave processes stopped Sun Jul 25 09:35:32 2010 alter database CLOSE NORMAL ORA-1109 signalled during: alter database CLOSE NORMAL... Sun Jul 25 09:35:32 2010 alter database DISMOUNT Completed: alter database DISMOUNT ARCH: Archival disabled due to shutdown: 1089 Shutting down archive processes Archiving is disabled Archive process shutdown avoided: 0 active Shutting down Data Guard Broker processes Sun Jul 25 09:35:34 2010 Completed: Data Guard Broker shutdown Sun Jul 25 09:35:36 2010 ARCH: Archival disabled due to shutdown: 1089 Shutting down archive processes Archiving is disabled Archive process shutdown avoided: 0 active Sun Jul 25 09:35:37 2010
7) The database is mounted Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Picked latch-free SCN scheme 3 Autotune of undo retention is turned on. IMODE=BR ILAT =121
40/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST LICENSE_MAX_USERS = 0 SYS auditing is disabled ksdpec: called for event 13740 prior to event group initialization Starting up ORACLE RDBMS Version: 10.2.0.4.0. System parameters with non-default values: processes = 1000 sessions = 1105 sga_max_size = 2634022912 __shared_pool_size = 285212672 __large_pool_size = 16777216 __java_pool_size = 16777216 __streams_pool_size = 117440512 streams_pool_size = 117440512 nls_date_format = DD/MM/YYYY resource_manager_plan = NIGHTTIME sga_target = 2634022912 control_files = G:\ORACLE\ORADATA\AVDB\AVDB_STANDBY_CTL.CTL db_file_name_convert = D:\D\ORACLE\ORADATA\AVDB, D:\ORACLE\ORADATA\AVDB, D:\E\ORACLE\ORADATA\AVDB, E:\ORACLE\ORADATA\AVDB, D:\F\ORACLE\ORADATA\AVDB, F:\ORACLE\ORADATA\AVDB, D:\G\ORACLE\ORADATA\AVDB, G:\ORACLE\ORADATA\AVDB log_file_name_convert = D:\D\ORACLE\ORADATA\AVDB, D:\ORACLE\ORADATA\AVDB db_block_size = 8192 __db_cache_size = 2181038080 compatible = 10.2.0.3 log_archive_config = dg_config=(AVDBSTD) log_archive_dest_1 = location="L:\ORACLE\oradata\AVDB\archive", valid_for=(ONLINE_LOGFILE,ALL_ROLES) log_archive_dest_2 = log_archive_dest_state_1 = ENABLE log_archive_dest_state_2 = ENABLE log_archive_max_processes= 4 log_archive_min_succeed_dest= 1 standby_archive_dest = log_archive_trace = 0 log_archive_format = AVDB_%T_%s_%R.arc fal_client = (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER01) (PORT=2115)))(CONNECT_DATA=(SERVICE_NAME=AVDB_XPT) (INSTANCE_NAME=AVDB)(SERVER=dedicated))) fal_server = (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER02) (PORT=2115)))(CONNECT_DATA=(SERVICE_NAME=AVDBSTD_XPT) (SERVER=dedicated))) archive_lag_target = 0
41/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST db_files = 2048 db_file_multiblock_read_count= 16 db_recovery_file_dest = L:\ORACLE\oradata\AVDB\flashback db_recovery_file_dest_size= 10485760000 standby_file_management = auto recovery_parallelism = 4 db_flashback_retention_target= 60 undo_management = AUTO undo_tablespace = UNDOTBS remote_login_passwordfile= EXCLUSIVE db_domain = instance_name = AVDB local_listener = LISTENER_AVDB session_cached_cursors = 50 utl_file_dir = F:\ORA_UTL_DIR, E:\ORACLE\admin\AVDB\udump job_queue_processes = 60 audit_file_dest = C:\ORACLE\PRODUCT\10.2.0\ADMIN\AVDB\ADUMP background_dump_dest = E:\ORACLE\ADMIN\AVDB\BDUMP user_dump_dest = E:\ORACLE\ADMIN\AVDB\UDUMP db_name = AVDB db_unique_name = AVDB open_cursors = 300 pga_aggregate_target = 314572800 dg_broker_start = TRUE PMON started with pid=2, OS id=2532 PSP0 started with pid=3, OS id=2404 MMAN started with pid=4, OS id=496 DBW0 started with pid=5, OS id=3088 LGWR started with pid=6, OS id=4072 CKPT started with pid=7, OS id=1072 SMON started with pid=8, OS id=3644 RECO started with pid=9, OS id=1996 CJQ0 started with pid=10, OS id=3752 MMON started with pid=11, OS id=1752 MMNL started with pid=12, OS id=2136 DMON started with pid=13, OS id=4076 Sun Jul 25 09:35:38 2010 alter database mount Sun Jul 25 09:35:42 2010 Setting recovery target incarnation to 6 ARCH: STARTING ARCH PROCESSES ARC0 started with pid=15, OS id=3900 ARC1 started with pid=16, OS id=3916 ARC2 started with pid=17, OS id=3660 Sun Jul 25 09:35:42 2010 ARC0: Archival started ARC1: Archival started ARC2: Archival started
42/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ARC3: Archival started ARC3 started with pid=18, OS id=1560 Sun Jul 25 09:35:43 2010 ARCH: STARTING ARCH PROCESSES COMPLETE Sun Jul 25 09:35:43 2010 ARC0: Becoming the 'no FAL' ARCH ARC0: Becoming the 'no SRL' ARCH ARC0: Thread not mounted Sun Jul 25 09:35:43 2010 ARC1: Becoming the heartbeat ARCH ARC1: Thread not mounted Sun Jul 25 09:35:43 2010 ARC2: Thread not mounted Sun Jul 25 09:35:43 2010 Successful mount of redo thread 1, with mount id 1046492586 Sun Jul 25 09:35:43 2010 Allocated 15937344 bytes in shared pool for flashback generation buffer Starting background process RVWR RVWR started with pid=19, OS id=2464 Sun Jul 25 09:35:43 2010 Physical Standby Database mounted. Completed: alter database mount Sun Jul 25 09:35:44 2010 ARC3: Thread not mounted Sun Jul 25 09:35:46 2010 Starting Data Guard Broker (DMON) INSV started with pid=20, OS id=1068 NSV0 started with pid=21, OS id=3340 RSM0 started with pid=22, OS id=2896 Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_dest_1='location="L:\ORACLE\oradata\AVDB\archive"',' valid_for=(ALL_LOGFILES,ALL_ROLES)' SCOPE=BOTH SID='AVDB'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_dest_state_1='ENABLE' SCOPE=BOTH SID='AVDB'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET standby_archive_dest='L:\ORACLE\oradata\AVDB\archive' SCOPE=BOTH SID='AVDB'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='AVDB'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_format='AVDB_%T_%s_%R.arc' SCOPE=SPFILE SID='AVDB'; Sun Jul 25 09:35:58 2010
43/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST ALTER SYSTEM SET standby_file_management='auto' SCOPE=BOTH SID='*'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*'; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET db_file_name_convert='D:\D\ORACLE\ORADATA\AVDB','D:\ORACLE\ORADAT A\AVDB','D:\E\ORACLE\ORADATA\AVDB','E:\ORACLE\ORADATA\AVDB','D:\F \ORACLE\ORADATA\AVDB','F:\ORACLE\ORADATA\AVDB','D:\G\ORACLE\ORADA TA\AVDB','G:\ORACLE\ORADATA\AVDB' SCOPE=SPFILE; Sun Jul 25 09:35:58 2010 ALTER SYSTEM SET log_file_name_convert='D:\D\ORACLE\ORADATA\AVDB','D:\ORACLE\ORADA TA\AVDB' SCOPE=SPFILE; Sun Jul 25 09:35:59 2010 ALTER SYSTEM SET fal_server='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP) (HOST=SERVER02)(PORT=2115))) (CONNECT_DATA=(SERVICE_NAME=AVDBSTD_XPT)(SERVER=dedicated)))' SCOPE=BOTH; Sun Jul 25 09:35:59 2010 ALTER SYSTEM SET fal_client='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP) (HOST=SERVER01)(PORT=2115)))(CONNECT_DATA=(SERVICE_NAME=AVDB_XPT) (INSTANCE_NAME=AVDB)(SERVER=dedicated)))' SCOPE=BOTH; Sun Jul 25 09:35:59 2010
8) The database is ready to start receiving transactions from the primary ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE MRP0 started with pid=23, OS id=2332 Managed Standby Recovery starting Real Time Apply parallel recovery started with 7 processes Sun Jul 25 09:36:10 2010 Waiting for all non-current ORLs to be archived... Clearing online redo logfile 1 D:\ORACLE\ORADATA\AVDB\REDO1.LOG Clearing online log 1 of thread 1 sequence number 56 Sun Jul 25 09:36:11 2010 Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE Sun Jul 25 09:36:11 2010 Clearing online redo logfile 1 complete Clearing online redo logfile 2 D:\ORACLE\ORADATA\AVDB\REDO2.LOG Clearing online log 2 of thread 1 sequence number 57
44/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Clearing online redo logfile 2 complete Clearing online redo logfile 3 D:\ORACLE\ORADATA\AVDB\REDO3.LOG Clearing online log 3 of thread 1 sequence number 58 Sun Jul 25 09:36:14 2010 Redo Shipping Client Connected as PUBLIC -- Connected User is Valid RFS[1]: Assigned to RFS process 2296 RFS[1]: Identified database type as 'physical standby' Primary database is in MAXIMUM PERFORMANCE mode Sun Jul 25 09:36:14 2010 RFS LogMiner: Client disabled from further notification Primary database is in MAXIMUM PERFORMANCE mode Sun Jul 25 09:36:14 2010 Redo Shipping Client Connected as PUBLIC -- Connected User is Valid RFS[2]: Assigned to RFS process 3848 RFS[2]: Identified database type as 'physical standby' Sun Jul 25 09:36:14 2010 RFS[1]: Successfully opened standby log 9: 'D:\ORACLE\ORADATA\AVDB\STDREDO09.LOG' Sun Jul 25 09:36:14 2010 Clearing online redo logfile 3 complete Clearing online redo logfile 4 D:\ORACLE\ORADATA\AVDB\REDO4.LOG Clearing online log 4 of thread 1 sequence number 59 Sun Jul 25 09:36:15 2010 RFS[2]: Successfully opened standby log 10: 'D:\ORACLE\ORADATA\AVDB\STDREDO10.LOG' RFS[2]: Detected missing archivals for Branch(resetlogs_id): 724771929 RFS[2]: Last archived SCN: 2:-623590551 Last change SCN: 2:623588413 RFS[2]: New Archival REDO Branch(resetlogs_id): 725275791 Prior: 724771929 RFS[2]: Archival Activation ID: 0x3e60fcc1 Current: 0x0 RFS[2]: Effect of primary database OPEN RESETLOGS RFS[2]: Managed Standby Recovery process is active New incarnation branch detected in ArchiveLog, filename D:\ORACLE\ORADATA\AVDB\STDREDO10.LOG Inspection of file changed rdi from 6 to 7 Setting recovery target incarnation to 7 Sun Jul 25 09:36:16 2010 Setting recovery target incarnation to 7 Sun Jul 25 09:36:16 2010 Clearing online redo logfile 4 complete Clearing online redo logfile 5 D:\ORACLE\ORADATA\AVDB\REDO5.LOG Clearing online log 5 of thread 1 sequence number 60 Clearing online redo logfile 5 complete Clearing online redo logfile 6 D:\ORACLE\ORADATA\AVDB\REDO6.LOG Clearing online log 6 of thread 1 sequence number 61 Clearing online redo logfile 6 complete
45/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST Media Recovery Waiting for thread 1 sequence 61 MRP0: Incarnation has changed! Retry recovery... Sun Jul 25 09:36:25 2010 Errors in file e:\oracle\admin\AVDB\bdump\AVDB_mrp0_2332.trc: ORA-19906: recovery target incarnation changed during recovery Managed Standby Recovery not using Real Time Apply Recovery interrupted! Sun Jul 25 09:36:26 2010 Errors in file e:\oracle\admin\AVDB\bdump\AVDB_mrp0_2332.trc: ORA-19906: recovery target incarnation changed during recovery Sun Jul 25 09:36:46 2010 Managed Standby Recovery starting Real Time Apply parallel recovery started with 7 processes Media Recovery start incarnation depth : 1, target inc# : 7, irscn : 12261313475 Sun Jul 25 09:36:53 2010 Waiting for all non-current ORLs to be archived... Media Recovery Waiting for thread 1 sequence 61 branch(resetlogs_id) 724771929 Fetching gap sequence in thread 1 branch(resetlogs_id) 724771929, gap seq 61-61 Sun Jul 25 09:36:54 2010 Redo Shipping Client Connected as PUBLIC -- Connected User is Valid RFS[3]: Assigned to RFS process 3124 RFS[3]: Identified database type as 'physical standby' RFS[3]: Archived Log: 'L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_61_0724771929.ARC' Sun Jul 25 09:37:02 2010 Redo Shipping Client Connected as PUBLIC -- Connected User is Valid RFS[4]: Assigned to RFS process 3012 RFS[4]: Identified database type as 'physical standby'
Cleanup of Archived Logs on the Standby Location On this configuration all backups are configured and running on the primary database, so that we needed a job to cleanup the archived logs generated on the standby database, for this purpose we setup the following batch script: Script rman_clean_arch.bat # This script is a batch file that invoke an rman script that # cleanup archived logs older than 3 hours # rman_clean_arch.bat set echo on
46/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST set oracle_sid=AVDBSTD set oracle_home=C:\oracle\product\10.2.0\db_1 cd L:\ORACLE\DG-Scripts rman target / log=rman_clean_arch.log @rman_clean_arch.rmn # eof script rman_clean_arch.bat
Script rman_clean_arch.rmn # This script remove all archived logs that are older than 2 hours run { allocate channel d1 type disk; delete noprompt force archivelog until time 'sysdate -120/1440'; release channel d1; } Exit # eof rman_clean_arch.rmn
Log File rman_clean_arch.log
This is the log file produced by the rman_clean_arch.rmn script: Recovery Manager: Release 10.2.0.4.0 - Production on Sun Aug 1 16:20:00 2010 Copyright (c) 1982, 2007, Oracle.
All rights reserved.
connected to target database: AVDB (DBID=803764395, not open) RMAN> run { allocate channel d1 type disk; 2> delete noprompt force archivelog until time 'sysdate -120/1440'; 3> release channel d1; 4> } 5> exit using target database control file instead of recovery catalog allocated channel: d1 channel d1: sid=1091 devtype=DISK deleted archive log archive log filename=L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_64015_0702125008 .ARC recid=889 stamp=725897700 deleted archive log archive log filename=L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_64016_0702125008 .ARC recid=890 stamp=725897709 deleted archive log
47/49
STANDBY DATABASE CONFIGURATION FOR MANUAL SWITCHOVER END TO END TEST archive log filename=L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_64017_0702125008 .ARC recid=891 stamp=725897725 deleted archive log archive log filename=L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_64018_0702125008 .ARC recid=892 stamp=725897737 deleted archive log archive log filename=L:\ORACLE\ORADATA\AVDB\ARCHIVE\AVDB_001_64019_0702125008 .ARC recid=893 stamp=725898425 Deleted 5 objects
released channel: d1 Recovery Manager complete.
Tnsnames Configuration for the Applications The tnsnames for the applications is configured to use a generic service that is started using an “on database open” trigger AVSDR = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER01)(PORT = 2115)) (ADDRESS = (PROTOCOL = TCP)(HOST = SERVER02)(PORT = 2115)) ) (CONNECT_DATA= (SERVICE_NAME=AVSDR) ) )
The service is created using the following code exec DBMS_SERVICE.CREATE_SERVICE ( service_name => 'AVSDR', network_name => 'AVSDR', failover_method => 'BASIC', failover_type => 'SELECT', failover_retries => 180, failover_delay => 1);
The trigger is created using the following code: CREATE OR REPLACE TRIGGER manage_dgservice
48/49