International Turnkey Systems SDG Telco (Service Delivery Group)
User Acceptance For 10g Upgrade at BGL Project/Application: CCBS
Prepared For: BGL
Version Number: 1.0
Page 1 of 21
Document Control Change Record Date
Author
Version
Change Reference
20/11/2008
Umashankar Barua
1.0
BGL10gUpgrade
Reviewers Name
Department / Position
Nazir Akhtar
Team Lead
Review Date 28-Nov-2008
Distribution List
Copy No.
Name
Location
References
Document Version Number
Customer / Author Name
Document Type
1.0
BGL
Rollout/Rollback
Document Name
Date
BGL10gUpgrade
20/11/2008
Contact Persons S No.
Person
Email
1
Umashankar Barua
[email protected] [email protected]
2
Nazir Akhter
[email protected]
Telephone/Ext
Page 2 of 21
TABLE OF CONTENTS
INTRODUCTION ………………………………………………………………………………………………………………………………………… 3 OVERVIEW …………….……………………..…………………………………………………………………………………………………………… 4 Existing Environment ………………………………………………………………………………………………………………………….. 4 Target Environment ………………………………………………………………………………………………………………………….. 5 Database Upgrade Methods ………………………………………………………………………………………………………………… 6 HARDWARE AND SOFTWARE REQUIREMENTS ……………………………………………………………………………………….. 7 ADOPTED UPGRADE METHODS ………………………………………………………………………………………………………………… 8 BACKUP STRATEGIES ……………………………………………………………………………………………………………………………….. 9 ROLLBACK PLANS ……………………………………………………………………………………………………………………………………. 10 UPGRADING DATABASES.………………………………….……………………………………………………..……………………………..12 TABS …………………………………………………………………………………………………………………………………………………. 12 TABSMED …………………………………………………………………………………………………………………………………………….14 CONFIGURING THE APPLICATION SERVERS …………………………………………………………………………………………..16 CONFIGURING WEBUTIL ………………………………………………………………………………………………………………………….18 CUTOVER PLAN …………………………………………………………………………………………………………………………………………19
Page 3 of 21
INTRODUCTION This document intended to cover the aspects of the upgrade principals for Banglalink Billing environment. In this document following topics will be covered— 1. 2. 3. 4. 5. 6. 7. 8. 9.
Upgrade methodology for each type of Database Reconfiguration of application servers Hardware and Software requirements for upgrade. Test cases for Database and application server related components Risk assessment Backup strategy for the upgrade Fallback planning and methods in case of abandoning the upgrade. Common pitfalls for the operation Step by step guide for upgrade
Page 4 of 21
OVERVIEW EXISTING ENVIRONMENT Databases Database Name: Database Type: Instances: Database Version: Cluster Ware: Operating System: Machine: TABS version:
TABS Oracle 9i RAC (Real application Cluster) 2 (TABS1, TABS2) 9.2.0.6 MC/SeviceGuard (A 11.16) Hp-ux 11i (PA-RISC) HP 9000 Series servers 6.5.3
Database Name: Database Type: Instances: Database Version: Operating System: Machine: TABS version:
TABSMED Single Instance (non-clustered) 1 (TABSMED) 9.2.0.8 Hp-ux 11i (PA-RISC) HP 9000 Series servers 6.5.3
Application Servers Application Server Version: Installation Type: TABS version: Operating System: Machine:
Oracle 10g R1 (9.0.4) Mixed (2 with FRServices, 2 with Infrastructure) 6.5.3 Windows 2003 server HP R4440
Page 5 of 21
OVERVIEW TARGET ENVIRONMENT Databases Database Name: Database Type: Instances: Database Version: CRS version: Cluster Ware: Operating System: Machine: TABS version: Server Role:
TABS Oracle 10g R2 RAC (Real application Cluster) 2 (TABS1, TABS2) 10.2.0.4 10.2.0.4 MC/SeviceGuard (A 11.16) Hp-ux 11i (PA-RISC) HP 9000 Series servers 6.5.3 Cater frontend application and post-paid mediation, rating.
Database Name: Database Type: Instances: Database Version: Operating System: Machine: TABS version: Server Role:
TABSMED Single Instance (non-clustered) 1 (TABSMED) 10.2.0.4 Hp-ux 11i (PA-RISC) HP 9000 Series servers 6.5.3 Cater pre-paid loading
Application Servers Application Server Version: Installation Type: TABS version: Operating System: Machine:
Oracle 10g R2 (10.1.2) Mixed (All with Infrastructure) 6.5.3 Windows 2003 server HP R4440
Page 6 of 21
OVERVIEW DATABASE UPGRADE METHODS Oracle Database 10g supports the following tools and methods for upgrading a database to the new Oracle Database 10g release:
Database Upgrade Assistant (DBUA) Provides a graphical user interface (GUI) that guides you through the upgrade of a database. The DBUA can be launched during installation with the Oracle Universal Installer, or you can launch the DBUA as a standalone tool at any time in the future.
Manual upgrade using SQL scripts and utilities
Export and Import utilities
CREATE TABLE AS SQL statement
All the method’s have their own advantages and disadvantages. Oracle recommends to use DBUA, as its automate almost all the aspects of the Upgrade process. However, by using DBUA, there is very little control over the overall upgrade process. In manual upgrade, Many steps can be controlled by the DBA or system administrators. And in Export/Import method we can have a fresh database, specially a brand new data dictionary. But comparatively export/import method can take a lot of time to accomplish depending on the size of the database. We have chosen mixed type of upgrade method for different database, according to the business needs.
Page 7 of 21
HARDWARE AND SOFTWARE REQUIREMENTS Hardware No additional Hardware required for the upgrade activity. Storage For TABS database the activity will require exactly double amount of space that allocated to the existing environment. For TABSMED database Enough space should be provided to keep a full cold backup of database. Software Operating system Following patches need to be installed Quality Pack Bundle: HP-UX 11i Quality Pack (GOLDQPK11i) HP-UX 11.11 Patches: PHNE_31097: PHSS_31221: PHSS_32508: PHSS_32509: PHSS_32510: Serviceguard and OS PHSS_32731: PHSS_32658: PHSS_32732:
ONC/NFS general release/performance patch HP aC++ -AA runtime libraries (aCC A.03.60) HP aC++ Compiler (A.03.63) ANSI C compiler B.11.11.12 cumulative patch +O4/PBO Compiler B.11.11.12 cumulative match Patches: Serviceguard A.11.16.00 (replaces PHSS_31075) Serviceguard Extension for RAC A.11.16.00 (replaces PHSS_31079) COM B.03.00.00 (replaces PHSS_31077)
Page 8 of 21
Patches for JDK on HP-UX 11.11: PHSS_30970: ld (1) and linker tools cumulative patch Database Oracle 10g Release 2 (10.2.0.1) Oracle 10g Patchset 4 (10.2.0.4) Application server Oracle Application server 10g Release 2 (10.1.2) with infrastructure for windows. Webutil 106 Jacob 1.8
ADOPTED UPGRADE METHOD For TABS database The upgrade method chosen for TABS database is Export/Import Considerations taken to choose this method
To have a fresh data dictionary To have an standby instance to fallback with minimum efforts and downtime To redistribute the loads on storage
Risks associated
The Export procedure can take long time due the data dictionary problem with current installation Import can take longer then expected time due to performance issue on newly allocated storage
For TABSMED database The upgrade method chosen for TABSMED database is DBUA Consideration taken to choose this method
Reliability of DBUA Lesser business criticality No points to have a fresh data dictionary Upgrade timing
Risks associated
Page 9 of 21
DBUA may fail to accomplish task, which will require redoing everything as DBUA provides very less control over upgrade process In case of fallback it might take longer time then expected to restore the backup
For APPLICATION SERVERS The application servers will be re-installed and reconfigured Consideration taken to choose this method
different file structures Different combinations of software’s Problem with report servers
Risks associated
Client side browser and jinitiator version mismatch can cause troubles
BACKUP STRATEGIES For TABS database We will keep the original database untouched throughout the Upgrade process time window. And we may keep the database for several days, in case, if decision comes to revert back to previous version of database. To keep this with original setting, we will use different voting disk and different OCR for the Oracle 10g CRS installation. So we will take following measure prior to install CRS.
Stop GSD services using gsdctl copy the contents of 9i OCR into the 10g OCR device using the dd command $ whoami oracle $ dd if=<9i_ocr_file> of=
bs=1024
Where ocr file name found in /var/opt/oracle/srvConfig.loc
Edit the srvConfig.loc file to point to the file
For TABSMED database For TABSMED we are going use DBUA for upgrading the database. So for this we need to have a valid backup to be restored if any emergency.The preferred method is to take a full database cold backup to disk. After Shutting down the database
Page 10 of 21
rman "target / nocatalog"
startup mount;
RUN { ALLOCATE CHANNEL c1 TYPE DISK; BACKUP as copy DATABASE FORMAT 'some_backup_directory%U' TAG before_upgrade; BACKUP as copy CURRENT CONTROLFILE format 'backup_location'; }
p.s Allocate as much as the number of cpu in machine
For APPLICATION SERVERS For application servers we will re-configure the servers. So we will keep a copy of the server to reinstate when necessary. For each server, before reconfiguration
Take a system state backup
Take a copy backup of full file system after shutting down all oracle services
ROLLBACK PLANS For TABS database As the existing database is not touched during the upgrade, we need to re-point to the previous configuration files and bring up the GSD components and run 9i database on it. To bring up the 9i database we need to perform following steps
stop 10g database with srvctl srvctl stop database -d TABS
stop CRS with crsctl crsctl stop crs
Edit the srvConfig.loc file to point to the <9i_ocr_file>
Start GSD with gsdctl gsdctl start
Start the database using srvctl srvctl start database -d TABS
Page 11 of 21
If in case, CRS installation need to be removed, following steps needs to considered
Run following scripts from $ORA_CRS_HOME/install rootdelete.sh rootdeinstall.sh if any problem to run these scripts or if it fails then proceed to next step to manually remove CRS configuration
Stop the Nodeapps on all nodes srvctl stop nodeapps -n
Prevent CRS from starting when the node boots rm /sbin/init.d/init.cssd rm /sbin/init.d/init.crs rm /sbin/init.d/init.crsd rm /sbin/init.d/init.evmd rm /sbin/rc2.d/K960init.crs rm /sbin/rc2.d/K001init.crs rm /sbin/rc3.d/K960init.crs rm /sbin/rc3.d/S960init.crs rm -Rf /var/opt/oracle/scls_scr rm -Rf /var/opt/oracle/oprocd rm /etc/inittab.crs cp /etc/inittab.orig /etc/inittab
Reboot the nodes
Remove the ocr.loc
De-install the CRS home in the Oracle Universal Installer
Remove the CRS install location
Clean out the OCR and Voting Files with dd commands
For TABSMED database In case of abandoning the upgrade process the backup taken before upgrade needs to be restored. To bring up the database, following steps needs to be done
rman "target / nocatalog"
STARTUP NOMOUNT
RUN { REPLICATE CONTROLFILE FROM 'backup_location'; ALTER DATABASE MOUNT; RESTORE DATABASE FROM TAG before_upgrade; ALTER DATABASE OPEN RESETLOGS;
Page 12 of 21
}
For APPLICATION SERVERS To restore and reinstate the previous application servers we need to restore
The files and folders from backup
Restore the system state backup
UPGRADING DATABASES For TABS database PREUPGRADE TASKS 1. Purging all data according to the policy 2. Dropping all temporary tables 3. Generate scripts from production for following objects TABLESPACES ROLES PROFILES USERS DATABASE LINK DIRECTORIES VIEWS PROCEDURES FUNCTIONS
Page 13 of 21
PACKAGES TYPES TRIGGERS JOBS SEQUENCES GRANTS SYNONYMS 4. Customize the tablespace script to reflect the new locations of the datafiles 5. Taking Backup of oracle binaries and orainventory 6. Taking optimizer statistics for the TABS schema UPGRADE TASKS Current Version/Release: 9.2.0.6 Destination Version/Release: 10.2.0.4 Upgrade Method: Export/Import ORACLE_HOME: /orabin/appl/oracle/product/10.2.0 1. Export TABS, HIST schema and also any other schema used by in-house built applications. 2. Add entries for VIP in /etc/hosts files 3. Edit network configuration to add alias for the VIP with port 801 4. Shutdown the 9i database 5. shutdown GSD Daemons 6. Apply Operating system patches 7. Adjusting kernel parameters 8. Create another pair of voting disk and OCR to preserve previous ones. 9. Modify srvConfig.loc to point new copied OCR 10. check /var/opt/oracle with ll 11. Install Oracle 10g CRS 12. Change ownership of all the raw devices to oracle. 13. Run root.sh in both nodes 14. check /var/opt/oracle with ll and compare with pre-CRS check 15. Check for CRS processes 16. Upgrade CRS and apply patchset 4 17. Install oracle 10g database binaries 18. Apply database patchset 4 19. Create a new database 20. Create the tablespaces and users 21. Import TABS schema 22. Adjusting TABS schema with updated script 23. Recreate all synonyms 24. Recreate all database links 25. Recreate all sequences POSTUPGRADE TASKS 1. Take full database backup 2. Change all environment variables accordingly (tabsenv, TABS-Profile.sh). Following parameters should be changed/added ORACLE_HOME PATH ORA_NLS10 LD_LIBRARY_PATH
Page 14 of 21
3. Set THRESHOLD values for tablespace alerts if necessary 4. Upgrade the catalog database Known Issues and workarounds Problem: VIPCA fails at the end of CRS installation Reason: VIPCA treats 172 series ip as private ip Solution: We need to add the nodeapps manually with srvctl command. For example, srvctl add nodeapps -n machinename -o -A 1.2.3.4/255.255.255.0 (interface)
UPGRADING DATABASES For TABSMED database PRE-UPGRADE TASKS 1. Purging all data according to the policy 2. Dropping all temporary tables 3. Generate scripts from production for following objects TABLESPACES DATABASE LINK DIRECTORIES
Page 15 of 21
JOBS SEQUENCES SYNONYMS 4. 5. 6. 7.
Taking Backup of oracle binaries and orainventory Keep minimum possible partitions in historical data Move tables to tablespaces with standard block sizes from non standard block sizes. Adjust Buffer cache parameter
UPGRADE TASKS Current Version/Release: 9.2.0.8 Destination Version/Release: 10.2.0.4 Upgrade Method: DBUA ORACLE_HOME: /orabin/appl/oracle/product/10.2.0 1. Running the Pre-upgrade script utlu102i.sql 2. Adjust the database according to the output of pre-upgrade script. 3. shutdown database 4. Take a full cold database backup as copy using RMAN. 5. Apply operating system Patches 6. Adjusting kernel parameters 7. Install Oracle binaries 8. Install Patch level 4 9. Install additional components for Oracle 10g companion media 10. Take a full cold database backup to disk using RMAN 11. startup open the database 12. Run netca from 10g database home to configure listener and tnsnames.ora 13. Run DBUA and select the database 14. Create sysaux tablespace with 3G size when prompted 15. Check the DBUA upgrade log for any discrepancies. 16. Recreate all database links POSTUPGRADE TASKS 1. Take full database backup 2. Change all environment variables accordingly (tabsenv, TABS-Profile.sh). Following parameters should be changed/added ORACLE_HOME PATH ORA_NLS10 LD_LIBRARY_PATH 3.
Set THRESHOLD values for tablespace alerts if necessary
Page 16 of 21
APPLICATION SERVER CONFIGURATION Operating System: - Windows 2003 server Application Server: - 10 g Release 2 with Infrastructure (10.1.2) Webutil: - Version 106 with Jacob 18
1. Install 10G release 2
Page 17 of 21
Installation type=Oracle Forms and Business intelligence For Infrustructure Oracle home= Infrahome ORACLE_HOME=d:\oracle10gR2INFRA ORACLE_SID=TABSINF For AS Middle tier Oracle home= AShome ORACLE_HOME= d:\oracle10gR2AS
2. Create $bin Location (For Front End executables) Create folder c:\$BIN 3. Copy jar files Copy jar files to $ORACLE_HOME/forms/java directory Namely JGo, JGoSVG ,ITSBeans ,FTPBean ,Sftp ,Hst65 ,Icons ,Browser, Browser2 4. Customize Application server environment files Copy default.env to tabs.env in $ORACLE_HOME/forms/server Add/edit following parameters FORMS_PATH=c:\$bin REPORTS_PATH=c:\$bin NLS_LANG=AMERICAN_AMERICA.AR8ISO8859P6
5. Customize formweb.cfg File Go to following location d:\oracle10gR2AS\forms\server Take a Backup of formsweb.cfg file before modifying. Add config section tabs in the formsweb.cfg file tabs config in formsweb.cfg will look like, [tabs] envfile=tabs.env HTMLbeforeForm=<script>var windowHandle;function openWindowURL(URLString){var sptest = URLString.split("/");var mstring = sptest[sptest.length-1];var fstring = mstring.split("_");var furl = sptest[0]+"//"+sptest[2]+"/"+sptest[3]+"/"+fstring[0].substring(2,fstring[0].length) +"/"+mstring;a = "100%";mw_content = "Bill_View<\/title><\/head><iframe width='"+a+"' height='"+a+"' frameborder='0' src='"+furl+"'><\/iframe><\/body><\/html>";windowHandle =
Page 18 of 21
window.open("javascript:opener.mw_content","Bill","location=false,scrollbars,resizab le");windowHandle.defaultStatus="Bill";windowHandle.focus();} otherparams=CONNECTSTRING=tabs FRUN=sommenu.fmx webUtilArchive=/forms/webutil/frmwebutil.jar,/forms/webutil/jacob.jar,/forms/webutil /JScript.jar,/forms/webutil/CtcPjcBean.jar,/forms/webutil/medTcp.jar,/forms/webutil/ ocxTcp.jar,/forms/webutil/rolloverbutton.jar baseHTMLJInitiator=webutiljini.htm baseHTMLjpi=webutiljpi.htm baseHTML=webutilbase.htm imageBase=Codebase archive_jini=frmall_jinit.jar,icons.jar,colorpicker.jar,xalan.jar,xercesImpl.jar,xml apis.jar,xmlcomp.jar,xmlparserv2.jar,FormsGraph.jar,modcursor.jar,TCPIP.jar,browser. jar,xerces.jar,browser2.jar,frmall.jar,JScriptWindowPJC.jar,jdev-rt.jar form=SOMPASS.fmx separateFrame=true width=1100 height=760 splashScreen=false Restart the Application server Middle tier d:\oracle10gR2AS\opmn\bin\opmnctl stopall d:\oracle10gR2AS\opmn\bin\opmnctl startall
6. ADD tns entry of the backend database in tnsnames.ora file Add tns entry in following files d:\oracle10gR2AS\network\admin\tnsnames.ora d:\oracle10gR2INFRA\network\admin\tnsnames.ora For example TABSRND = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 172.16.10.14)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TABSRND) ) )
7. overwrite key resource file namely fmrweb_utf8.res fmrweb.res Location (windows): d:\oracle10gR2AS/forms/ Take backup of Previous files and then place the files. CONFIGURING WEBUTIL
webutil version: 106 Jacob version: 18
Page 19 of 21
Create a Webutil folder inside $ORACLE_HOME/forms. Copy webutil folder in $ORACLE_HOME/forms we will refer this as webutil home. Create a directory lib inside webutil folder Edit $ORACLE_HOME/forms/server/forms.conf Comment out #AliasMatch ^/forms/webutil/(..*) "/appl/oracle/ias10/forms/webutil/$1"
And add
AliasMatch ^/forms/webutil/(..*) "/appl/oracle/ias10/forms/webutil/lib/$1"
Modify entry of of WEBUTIL_CONF parameter in application server env file WEBUTIL_CONFIG=$ORACLE_HOME/forms/webutil/server/webutil.cfg
Modify $ORACLE_HOME/forms/server/formsweb.cfg to add following webUtilArchive=/forms/webutil/frmwebutil.jar,/forms/webutil/jacob.jar baseHTMLjinitiator=\server\webutiljini.htm baseHTMLjpi= \server\webutiljpi.htm baseHTML=\ server\webutilbase.htm
Sign the jar files Set JDK_HOME=$ORACLE_HOME/jdk Sign_webutil.bat Jacob.jar Sign_webutil.bat frmwebutil.jar
Install WEBUTIL_DB package in webutil user following paths in CLASSPATH of your application server env file frmwebutil.jar,Jacob.jar,rt.jar
Put forms_base_ie.js and forms_ie.js in $ORACLE_HOME/forms/java folder
CUTOVER PLAN
Page 20 of 21
The Upgrade process consists of following components 1. 2. 3. 4.
Upgrading the RAC environment for TABS database. Re-creating TABS database with Export/import Upgrading the TABSMED database Reconfiguring the Application server
The tasks are listed down with approximate timings. Some of the activities will be performed in parallel. For the Upgrade activities and Pre go-live testing we will require 48 hours downtime (approx.) DB/AS TABS
TABSMED
Application server
Task Export TABS, HIST schema and also any other schema used by in-house built applications.
Duration (Approx.) 6 hrs
Shutdown the 9i RAC
20 min
Apply Operating system patches and adjusting kernel parameters Create another pair of voting disk and OCR to preserve previous ones. Install Oracle 10g CRS and patching CRS Install oracle 10g database binaries and apply patches Create a new database Create the tablespaces and users Import TABS schema Adjusting TABS schema with updated script Checking for invalid objects and recreating all necessary objects Updating and adjusting the environment for TABS applications Deploying applications
3 hrs
Running the Pre-upgrade script utlu102i.sql and adjust the database Shutdown the database and Take a full cold database backup as copy using RMAN Apply operating system Patches and adjusting kernel parameters Install Oracle binaries and apply the patches Run DBUA followed by DBUA to upgrade the database Apply synchronization scripts
1 hrs
For each application server to commission
6 hrs
20 min 1 hrs 3 hrs 1 hrs 1 hrs 15 hrs 1 hrs 1 hrs 1 hrs 20 min
3 hrs 3 hrs 3 hrs 3 hrs 2 hrs
Page 21 of 21