HA SAP on Oracle DataGuard

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________

Proof of concept documentation

SAP High Availability system with Oracle DataGuard on Linux Cluster

Including Physical and Logical standby database and Transparent Application Failover (TAF)

covering SAP Netweaver 7.0 including EHP2 on Oracle 10.2 / 11.2

Written by Kjell Erik Furnes version 1.00, Sep 2011

Written and tested by Kjell Erik Furnes [email protected]

Page 1 of 202


Contents Disclaimer .......................................................... ........................................................... ..................................... 6 Credits ...................................................... ............................................................ .............................................. 6 Writers comments............................................................................................................................................... 7 What does not work work from SAP standard to this configuration .......................................................... ............... 10 3 – 2 – 1 disaster disaster .......................................................... ........................................................... ......................... 11 Crash course in SAP system design.................................................................................................................. 12

System design ..................................................... ........................................................... ......................... 13 SAP Netweaver distributed installation ...................................................... ............................................ 15 SAP Netweaver High Availiability installation .................................................... .................................. 16 SAP Netweaver Netweaver High Availiability Availiability installation with Oracle DataGuard ................................................ 17 Table design.......................... ............................................................ ...................................................... 19 SAP Transport system .................................................. ............................................................................................................. ........................................................... ............... 19 The curse of SAP .......................................................... ........................................................... ......................... 21

SAP – Oracle license ......................................................... ........................................................... ............... 23 Hardware mirroring mirroring vs. Oracle Oracle DataGuard ........................................................ ............................................ 26

Oracle DataGuard vs. RAC ......................................................... ........................................................... ..... 29 Disaster / Inconvenience:............................................................. ........................................................... ..... 30 Return to Normal Time Objective (RNTO) .......................................................... .................................. 30 SAP High Availability Solution ........................................................ ........................................................... ..... 31 SAP Directory Structure................................................................................................................................... 31

SAP <-> Oracle communication channles ............................................................ .................................. 33 Connection channels and Oracle DataGuard ........................................................ .................................. 34 JDBC ........................................................ ............................................................ .................................. 35 Configuring VMware guest servers.................................................................................................................. 37

Test environment ..................................................... ........................................................... ......................... 37 VMWare server Configuration ......................................................... ...................................................... 37 Time synchronization ................................................... .............................................................................................................. ........................................................... ............... 37 Installation ..................................................... ........................................................... ................................... 40 Installing the Linux servers servers (OS and configuration) configuration) ...................................................... .................................. 40

Disk, Volume and File systems ................................................... .............................................................................................................. ........................................................... ..... 40 Display Physical Disk availiable on your system ..................................................... ......................... 40 Create a Physical Volume ............................................................ ...................................................... 41 Create a new Volume Group.................................................................. Group...... ............................................................ ............................................ 41 Resize Volume Group ........................................................ ........................................................... ..... 41 Display Volume Group information ...................................................... ............................................ 41 Create a new Logical Volume................................................................ Volume.... ............................................................ ............................................ 41 Display Logical Volume information ......................................................... ............................................ 42 Logical Volumes ...................................................... ........................................................... ............... 42 Resize logical volumes....................................................... ........................................................... ..... 42 File System ......................................................... ........................................................... ......................... 42 Create ext3 filesystem........................................................ filesystem ........................................................ ........................................................... ..... 42 Resizing ext3 file file systems systems based on LVM ....................................................... .................................. 43 Create gfs filesystem.......................................................... filesystem .......................................................... ........................................................... ..... 43 Resizing gfs file file systems systems based on LVM ........................................................ ................................... 43 Adding disk in a vmware vmware environment ....................................................... ............................................ 43 ISCSI targets....................................................... targets ....................................................... ........................................................... ......................... 44 Creating block devices devices (ISCSI Target device)........................ ........................................................... ..... 44 Presenting the ISCSI block devices to the ISCSI clients .................................................... ............... 44 Display ISCSI target information............................. information........................................................................................ ........................................................... ............... 45 Configuration of the ISCSI initiator ........................................................... ............................................ 46 Persistent ISCSI mount ........................................................... ........................................................... ..... 47


Page 2 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Resizing iscsi targets........................................... ........................................................... ......................... 48 Firewall........................................................................................ Firewall............................. ........................................................... ........................................................... ..... 49 ISCSI iptable rules ........................................................ ........................................................... ............... 51 ntp iptable rules................................................... ........................................................... ......................... 51 Cluster iptable rules ...................................................... ........................................................... ............... 51 NFS iptable rules .......................................................... ........................................................... ............... 51 ORACLE iptable rules...................................................................... rules.......... ............................................................ ...................................................... 52 SAP iptable rules .......................................................... ........................................................... ............... 52 ds8000 ISCSI Target server..................................... server................................................................................................ ........................................................... ......................... 52 Disk config................................................ ........................................................... ................................... 52 Physical Volume ........................................................... ........................................................... ............... 52 Volume Groups............................... Groups........................................................................................... ............................................................ ............................................ 52 Logical Volumes............................. Volumes......................................................................................... ............................................................ ............................................ 53 ISCSI Target config ...................................................... ........................................................... ............... 53 Network configuration:......... ............................................................ ...................................................... 53 Firewall ..................................................... ........................................................... ................................... 53 Required Java version.................................... version................................................................................................ ............................................................ .................................. 55 Hostname restrictions ........................................................ ........................................................... ............... 55 SAPLocales ................................................... ............................................................................................................... ............................................................ .................................. 55 User limits ..................................................... ............................................................ .................................. 55 SAP Netweaver 7.0 including including EHP2 ABAP+Java High Available Solution Solution .................................................... 56

VMware Shared disk...................................... disk................................................................................................. ........................................................... ......................... 56 Preparation........................................... ............................................................ ............................................ 57 Linux Cluster ........................................................... ........................................................... ......................... 57 pocnfsc cluster.................................................................................................................................................. 58

(NFS server cluster) ...................................................... ........................................................... ............... 58 pnfscn01 ........................................................ ............................................................ .................................. 59 Network configuration: ...................................................... ........................................................... ..... 59 Service configuration ......................................................... ........................................................... ..... 59 pnfscn02 ........................................................ ............................................................ .................................. 59 Network configuration: ...................................................... ........................................................... ..... 59 Cluster Resources pocnfscIP Address............................................................................ ......................... 60 Global file systems (GFS) ............................................................ ...................................................... 60 NFS Exports............................................................. Exports.. ........................................................... ........................................................... ............... 60 NFS Client ..................................................... ........................................................... ......................... 60 pocsapcl cluster................................................................................................................................................ 61

(SAP Central services Cluster) .................................................... ........................................................... ..... 61 psapcn01........................................................ psapcn01 ........................................................ ............................................................ .................................. 61 Network configuration:......... ............................................................ ...................................................... 61 Service configuration.................................................... configuration .................................................... ........................................................... ............... 62 psapcn02........................................................ psapcn02 ........................................................ ............................................................ .................................. 62 Network configuration:......... ............................................................ ...................................................... 62 Cluster resources posapcl.............................................. ........................................................... ............... 62 IP Address........................................................... ........................................................... ......................... 62 SAP Instance............................................. ........................................................... ................................... 62 poradg00 Oracle Primary server .................................................... ........................................................... ..... 64

Disk config................................................ ........................................................... ................................... 64 Physical Volume ........................................................... ........................................................... ............... 64 Volume Groups............................... Groups........................................................................................... ............................................................ ............................................ 64 Logical Volumes............................. Volumes......................................................................................... ............................................................ ............................................ 64 Network configuration:......... ............................................................ ...................................................... 64 Firewall ..................................................... ........................................................... ................................... 65 Mount........................................................ ........................................................... ................................... 65 poradg01 Oracle Standby server...................................................................................................................... 65

Disk config................................................ ........................................................... ................................... 65 Physical Volume ........................................................... ........................................................... ............... 65 Volume Groups............................... Groups........................................................................................... ............................................................ ............................................ 66


Page 3 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Logical Volumes............................. Volumes......................................................................................... ............................................................ ............................................ 66 Network configuration:......... ............................................................ ...................................................... 66 Firewall ..................................................... ........................................................... ................................... 66 Mount........................................................ ........................................................... ................................... 67 psapap01 (SAP Central Instance Instance ) ................................................... ........................................................... ..... 67

Disk config................................................ ........................................................... ................................... 67 Physical Volume ........................................................... ........................................................... ............... 67 Volume Groups ....................................................... ........................................................... ......................... 67 Logical Volumes ..................................................... ........................................................... ......................... 67 Network configuration:...................................................... ........................................................... ............... 68 Firewall........................................................................................ Firewall............................. ........................................................... ........................................................... ..... 68 Mount .................................................. .............................................................................................................. ............................................................ ............................................ 68 psapap02 (SAP Dialog instance)...................................................................................................................... 69

Disk config................................................ ........................................................... ................................... 69 Physical Volume ........................................................... ........................................................... ............... 69 Volume Groups............................... Groups........................................................................................... ............................................................ ............................................ 69 Logical Volumes............................. Volumes......................................................................................... ............................................................ ............................................ 69 Network configuration:......... ............................................................ ...................................................... 69 Firewall ..................................................... ........................................................... ................................... 69 Mount .................................................. .............................................................................................................. ............................................................ ............................................ 70 ORACLE........................................................................................................................................................... 71

Housekeeping................................................. ........................................................... ......................... 74 RMAN (Recovery Manager) ...................................................... ........................................................... ..... 74 Creating a RMAN RMAN Catalog Catalog database:......................................................................... .................................. 75 Database Backup .......................................................... ........................................................... ............... 76 ARCHIVELOG backup .......................................................... ........................................................... ..... 77 Misc RMAN commands ......................................................... ........................................................... ..... 77 Relocating backup when using Oracle DataGuard. ....................................................... ......................... 77 Oracle Maintained Files (OMF) and SAP ....................................................... ............................................ 78 Software installation ........................................................................................................................................ 78

NFS Mount options for Oracle Oracle filesystems on Linux X86-64 ........................................................... ..... 78 listener.ora .......................................................... ........................................................... ......................... 78 SAP Installation................................................................................................................................................ 80

SAP Central instance .................................................... ........................................................... ............... 80 Installing SAP ASCS........................... ............................................................ ............................................ 81 Installing SAP SCS........................................ SCS................................................................................................... ........................................................... ................................... 84 Installing SAP ERS ........................................................... ........................................................... ............... 87 Cluster node 1 ..................................................... ........................................................... ......................... 87 Cluster node 2 ..................................................... ........................................................... ......................... 89 Modify Linux cluster configuration............................................................ configuration ............................................................ ............................................ 91 Installing SAP Database instance .......................................................... ...................................................... 92 SAP Oracle database installation............................. ........................................................... ....................... 104 Oracle Patching 10.2.0.1 -> 10.2.0.4.0 ............................................................ .......................................... 105 Continue with the SAP Database Instance Instance Installation.............................................................................. 107 Installing SAP Central Central Instance Instance (Primary Application Server) .................................................... ............. 108 Installing Application server....................................................................... server........... ............................................................ .......................................... 120 Oracle database database modification in preparation to handle Oracle DataGuard and TAF ................................ 128 Oracle Transparent Transparent Application Failover (TAF) .......................................................... ................................. 130

Java application failover abilities............................................ ........................................................... ... 131 Oracle DataGuard Broker .................................................. ............................................................................................................. ........................................................... ............. 133 Setting up the DataGuard Broker configuration ........................................................ ................................ 134 Oracle DataGuard Broker Observer................................................................ Observer..... ........................................................... ................................. 136 Perform a switchover to test the configuration ......................................................... ....................... 137 Perform a failover to test the configuration .................................................... ................................. 137 Oracle administration .................................................. ........................................................... ....................... 137


Page 4 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Oracle DataGuard ......................................................................................................................................... 139

Physical standby vs. Logical Standby ................................................... ....................................................................................................... .................................................... 140 RedoApply vs. SQLApply................... ............................................................ .......................................... 140 Oracle DataGuard DataGuard solution with Physical Physical standby database..................................... database...................................................................... ................................. 140 Create a physical standby standby database (Oracle 10)....................................................................... ............. 142 On primary server: ................................................... .............................................................................................................. ........................................................... ............. 142 On your standby server: ..................................................... ........................................................... ... 142 on standby server ..................................................... ........................................................... ............. 144 on primary server ..................................................... ........................................................... ............. 145 Rolling upgrade using a transient logical standby......................................................................................... 147

Prerequisite for this procedure: ......................................................... .................................................... 148 DataGuard Broker Configuration files store in separate filesystems......................................................... 148 Rolling upgrade upgrade using transient logical standby standby database database .................................................. ......................................................................... ....................... 149 Rolling upgrade upgrade using transient logical standby standby database database .................................................. ......................................................................... ....................... 152 Part I: 10.2.0.4 -> 10.2.0.5.................. ............................................................ .......................................... 152 Preparation........................................... ............................................................ .......................................... 153 Post upgrade steps......................................................... ........................................................... ............. 160 Rolling upgrade upgrade using transient logical standby standby database database .................................................. ......................................................................... ....................... 162 Part II: 10.2.0.5 -> 11.2.0.2 ........................................................ ........................................................... ... 162 Post upgrade steps......................................................... ........................................................... ............. 167 Updating the compatible parameter................................................................. parameter..... ............................................................ .......................................... 168 Rolling upgrade upgrade using transient logical standby standby database database .................................................. ......................................................................... ....................... 169 Part III: 11.2.0.2 -> 11.2.0.2 (Patch apply)..................................................... apply) ..................................................... .......................................... 169 Post upgrade steps......................................................... ........................................................... ............. 171 Oracle DataGuard solution with Logical standby database.......................................................................... 171 Using Oracle DataGuard partial Logical Standby for for maintaining legacy (non-SAP) reporting systems systems .... 174

Creation process: ..................................................... ........................................................... ....................... 174 Prerequisite: ................................................... .............................................................................................................. ........................................................... ....................... 174 Configure the subset Logical standby database. ....................................................... ....................... 174 SKIP procedure......................................... ........................................................... ................................. 175 UNSKIP Procedure......................... Procedure..................................................................................... ............................................................ .......................................... 175 INSTANTIATE procedure ..................................................... ........................................................... ... 177 Housekeeping on the logical standby database...................................... database.......................................................................................... .................................................... 179 Oracle DataGuard DataGuard in a SAP upgrade scenario scenario........................................................... ........................................................... ................................ 180 Oracle DataGuard Broker .................................................. ............................................................................................................. ........................................................... ............. 180 Observer ........................................................ ............................................................ ................................ 181 Location of the Oracle DataGuard DataGuard Observer...................... Observer...................... ........................................................... ... 182 Oracle Transparent Transparent Application Failover (TAF) .......................................................... ................................. 183

Java application failover abilities............................................ ........................................................... ... 185 Log apply delay .................................................. ............................................................................................................. ........................................................... ....................... 186 Monitor log file generation ..................................................... ........................................................... ... 187 Transparent Application Failover (TAF) .................................................... .......................................... 189 Appendix 1 NFS Cluster Config.................................................. ........................................................... ... 191 Appendix 2 tnsnames.ora .................................................. ............................................................................................................. ........................................................... ............. 192 Appendix 3 Automated RMAN backup scripts ........................................................ ................................ 193 Archivelog backup........................................................ backup ........................................................ ........................................................... ............. 194 Rolling incremental backup .................................................... ........................................................... ... 197 Weekly Full backup ...................................................... ........................................................... ............. 199 Daily check routine for your RMAN backup.............................................. .......................................... 200


Page 5 of 202


Disclaimer This document was written as a proof of concept to show the possibilities of a certain configuration. The writer or anyone associated with the writer cannot be made liable for any damage to your systems nor can they be made liable for any loss of data if you choose to follow the concept shown in this document. This document is not meant to replace the official installation documentation or configuration guides provided by SAP, Red Hat, CentOS or Oracle. If you chose to follow this document you do so at your own risk. The comments and statements made in this document are solely those of the writer and does not reflect the views or opinion of the writers employer or anyone associated with the writer.

Credits Linux is a registered trademark of Linux Torvalds SAP is a registered trademark of SAP AG in Germany and in several other countries ®

®

Oracle and Java are registered trademark of Oracle and/or its affiliates Red Hat is a registered trademark of Red Hat Inc Windows is a registered trademark of Microsoft Corporation in the United States and other countries VMware is a registered trademark of Vmware in the United States and other countries

The installation and test done in this document is done under temporary and experimental licensing from SAP and Oracle. For commercial use of SAP and Oracle you will need to purchase licensing to cover your specific needs. Much of the initial server installation described in this document is based on several modified kickstart files using the kickstart file used by the EAL4(+) Common Criteria certificate installation guides for Red Hat Enterprise Linux 5 and or 5.3 by HP, IBM and DELL as a base. NOTE 1. The installations described in this document are not Common Criteria EAL4 certified. NOTE 2. The installation described in this document does not follow the recommendations by SAP in regards to several requirements for hardware sizing and choice of operating system.


Page 6 of 202


Writers comments Most of this document and the underlying installation/testing was done in the hours between 23 (11pm) and 05 (05am) when the house was quiet and everyone else was sleeping. Therefore the progress of the document has taken long and I have been forced to take the testing and documentation in very small portions. This document is a proof-of-concept document, Therefor in order to show the entire process the document contains more screendumps and command-line commands and results that is usual in documents like this. I am a little more focused on the prosess of getting the system to work than I am of the techical description. If you need more technical background information on the different subjects there are many excelent documents and books availiable on the market. The downside of running proof-of-concept installations and tests in private like I have done here is that it takes a long time, and some of the errors I found in my earliest attempts of the document have been corrected in later versions of the documentations from SAP. For instance I started out my tests using SAP Netweaver 7.0 SR3 which had some configuration specifications I disagreed with. So the sections I had written on why the file system configuration from SAP was wrong for the High Availability setup had to be thrown out when the new updated documentation from SAP came with the SAP Netweaver 7.0 including EHP2 came out. I do however take some comfort that I had arrived at the same conclusions as SAP has. With SAP Netweaver 7.0 including EHP2 SAP has taken a gigant leap in concern to the installation of SAP on High Availability platform on other platforms than Microsoft Windows. The inclusion of a fully automated Enqueue Replication Server installation with sapinst makes a High Availability solution on Linux a household configuration, and in my opinion should become the new standard of SAP installations. In this document I will try to take you out of the comfort zone in regards to how you manage your SAP installation in regards to how the database is managed. I will challenge you to change how you perform your system backups. I will challenge you on how your SAP system is configured. I will do this because I believe SAP is an application which has a database interface like very many other applications. SAP is a huge application but yet still only an application.


Page 7 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ SAP has many recommendations on how you should run your applications, and SAP also provides you with procedures on how to perform the management of your servers. What very many SAP administrators do not realise is that the recommendations from SAP are not written on stone tablets. You are allowed to use other and often better methods for managing your system without breaking your contract with SAP. You will however not be able to run to SAP to get support on how to perform the tasks that you have in place which are not SAP standard. In this document I will show how you can configure your SAP system to have a disaster recovery solution in place. I will show you how you can set up your system in a High Availability configuration I will show you how you can use Oracle DataGuard to offload your backup from your production system so that you can reduce the load on your primary system. I will show you how to use Oracle RMAN to perform incremental backup of your SAP system which can reduce your backup time drastically. I will show you how you can use Oracle DataGuard to perform a rolling database upgrade with minimum downtime to your SAP system. I will show how Oracle Active DataGuard (Oracle 11) will help against hardware block corruption. I will show you how you can use Oracle DataGuard to create a separated subset of your production database to use for legacy (non-SAP) reporting systems. When I first started to plan this proof of concept installation I did so because I wanted to see if it was possible for me to set up a SAP system to use Oracle DataGuard. Initially I saw two major issues from a DBA point of view. 1.

The default installation of SAP ABAP uses the SID parameter to communicate with the database. To be able to use TAF I had to check if it was possible to have SAP use the SERVICE parameter instead of the SID parameter. SAP use a number of programs to connect to the database so this needed to be checked for all types of programs ( R3Trans, R3Up, etc)

2.

The default installation of SAP Java use oracle thin jdbc to connect to the database. According to SAP documentation you will need to use JDBC VII or OPEN SQL to be able to have a HA solution in place for Oracle. To be able to use TAF according to this documentation I had to check if it was possible to have SAP use either OPEN SQL connection or Oracle OCI connection (JDBC VII) instead of the oracle thin JDBC witch uses a hostname specific connection parameter.

Note: The documentation from SAP concerning the limitations in HA capabilities for the Oracle database using JDBC VI connections are true only if you limit the JDBC thin connection string to the options used in default SAP installation. Written and tested by Kjell Erik Furnes [email protected]

Page 8 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ In this document I will point out some statements from SAP which I believe are either wrong or incomplete. This is not to say that I think SAP or the documentation from SAP is poor. My intention is only to point out that I disagree with that particular statement. I find SAP to be an excellent product, and the documentation base from SAP surpasses most other products on the market. When I comment on a specific statement my comments are limited to this statement only and should not be taken out of context. When starting out I was pretty confident that setting up SAP for Oracle DataGuard and TAF was not too big of an issue. So to make the task a little more interesting and challenging to me I decided to do this with a scenario where I would set up the system to simulate a High Availability solution with multiple clusters over multiple datacenters including SAP Enqueue Replication Servers.

SAP Java ERS

SAP Java ERS

SAP ABAP ERS

SAP ABAP ERS

SAP Java SCS ABAP SCS

r e t s u l C A H


SAP Primary Primary APP Serv Serv SAP APP Serv SAP APP Serv

Database

SAP APP Serv Optional Optional Application Application servers servers

d r e a l c u a r G a t O a D

Database


SAP APP Serv

Database

Drawing 1: SAP High Available system with 3 node Oracle DataGuard

This installation will simulate an SAP installation over three different datacenters where two datacenters participate in a Linux cluster configuration + Oracle DataGuard, and the third datacenter is a pure Disaster Recovery site included in Oracle DataGuard only. The Oracle DataGuard implementation and configuration is the same for both the SAP Distributed and High Available installation scenarios. I am fundamentally for splitting application and database for just about any type of application. By having a “pure” database server I know that there will be no conflicts in needed software packages, and I am able to limit the access to the servers to only those who needs database access.


Page 9 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ I have made this installation to show the possibilities of a specific configuration. The configurations shown here are not geared for performance. To tune your system for performance please see the SAP documentation for your solution.

What does not work from SAP standard to this configuration During my testing of a dataguard configuration there is two distinct routines that does not work as default SAP standard when using a Oracle DataGuard configuration with an Active DataGuard configuration. 1. SAP directory structure. Default when you create a new file with the SAPDBA the suggested setup is to create a new directory under which there is only one file. This configuration is a historically challenged configuration which stems from a limitation in the BRTools package used for backup. The newer versions of the BRTools package does not have these limitations, but SAP is still suggest this. If you create a new file by using the default suggestion from SAPDBA the directory structure of the primary and the standby is no longer similar (the standby does not have the new directory). So the standby will now create the new file under the location specified by the profile parameter db_create_file_dest if this is set. If not set the new file will be created under $ORACLE_HOME/dbs. 2.

SAP transaction DB13. In this transaction you schedule BRTools tasks.

NOTE: Thsi is only a problem if you rename the database instance to accheive a switchover envirinment. This transaction is actually a simple API to execute operating system commands which are executed in the adm environment on the OS. If you set up an active DataGuard you will change the SID of the database for the ora user which will control the database while the adm user uses the SID as a connection parameter to talk to the SERVICE_NAME of the database. However, the BRTools is a SUID process which means that it is executed as the ora user with the adm environment variables. In my configuration this means that the BRTOOLS commands executed by the adm /t06adm) will try to execute commands on the T06 SID. You will then get an error saying that the T06 database is down. You can however scedule all BRTools activities directelly on the OS level as cron jobs under the ora user. Tasks which are scheduled on the OS level will apear in DB13 as completed jobs so that you can view the result directelly in SAP, but you will be unable to change the schedule from within SAP.


Page 10 of 202


3 – 2 – 1 disaster At 03:23 (AM) on the night before the year-end closure there is an electrical fire and your cooling system is knocked out. At 03:36 the temperature in your datacenter has reached critical temperature and your data systems are now starting to fail due to overheating. Your datacenter is now effectively down for the count. Fixing the cooling system will probably take only a few hours, but there are internal damages to several of your data systems because they were overheated. You are now looking at disk systems where you need to change multiple disks. Possibly you might be compromising the redundancy of the data, and you might have several servers which need extensive surgery just to be able to start. Effectively you have days worth of manual work to have all systems back up and running as normal if you are at all lucky enough to be able to salvage all your production data. This easy to imagine scenario is the type of scenario where you would need to have a HA or Disaster Recovery solution in place. It does not have to be a terrorist attack, fire, flood etc. Your systems can be rendered unavailable by something as insignificant as a blown fuse, and you should be prepared knowing that this can happen. Closing your eyes hoping that it will not happen to you is not a very good solution.


Page 11 of 202


Crash course in SAP system design SAP system design can be done in many ways. First of all you need to know that referring to an SAP Netweaver system is to general to make any real impression. To use the system you also need to know what type of system you are talking about, because you can have a multitude of different types of SAP installation. You can have an ERP/ECC system which is the most common first SAP installation. This where you have the Enterprise Resource Planning (ERP).The ECC part is the ERP Central Component (economics, material management etc). It is not uncommon that a system like this has around 70 000 tables and as many indexes. Then you have the BI (Business intelligence), PI (Process Integration), SRM (Supplier Relationship Management), CRM (Customer Relationship Management) and many other types of installations. The next thing you need to know is that there are three different types of installation. Knowledge on this subject is important from a database administration point of view because knowing this tiny fragment of information enables you to configure how to communicate with your SAP database and where the database is located.

Single-stack ABAP installation is where you have one SAP user in

the database, and you only use the proprietary SAP ABAP programming language. This type of installation uses the traditional tnsnames.ora/listener.ora as the only communication channel to the database.

SAP ABAP stack

Database

Single-stack Java installation is where you also have only one SAP

SAP

user in the database, and you only use Java. This type of installation use the Oracle JDBC Thin driver as the only communication channel to the database.

Java stack

There is one exception to this and that is the SAP backup software, BRTools, which always uses the traditional tnsnames.ora/listener.ora

Database


Page 12 of 202


Dual-stack where you have a combined ABAP and Java installation

where you have two SAP users in the database. This type of installation uses both the traditional tnsnames.ora/listener.ora communication channel and the Oracle JDBC Thin communication channel to the database.

SAP Java stack ABAP stack

Database

System design Knowing the stack type of installation is core knowledge, but this knowledge has no meaning if your system is installaed on one single server. So there is another piece to the puzzle. There are three types of system designs which you can choose from when you do your installation

SAP Netweaver Standard installation.

Optional Optional Application Application servers servers

Unfortunately this is the most SAP APP Serv SAP APP Serv common installation. In this type of SAP installation you install everything on Java stack SAP APP Serv SAP APP Serv one server (SAP Central instance + ABAP stack database). This can be a single-stack or a dual-stack installation. You can Database also install additional application servers which can be placed on separate servers, but these are optional and does not add any redundancy for the central parts of the SAP system. The problem with this type of system design is that there is no business continuance ability built into the SAP system design. I refer to this type of installation as an “install and run” installation, and as a system administrator you should never set up a critical production system like this. The system does work, and it is easy to install and maintain which is probably why it is so widespread. But there is no fail-tolerance in this type of system except the fail tolerance built into the hardware it is running on. My bed-side alarm-clock runs on power-grid only. There is no battery backup to cover for power-failures, and there used to be only one clock (single installation). This design is inexpensive and it has 100% up-time most of the time. Except if there is a power-failure, a system failure or my two-year old boy engage in system administration activities. He does not know exactly what happens when he pushes the buttons (but find it fun to see the blinking lights change). He is now a junior system administrator with the ability to change the system, but without the proper knowledge he can implement un-tested changes which can render the system useless. Since Human-Errors like this is the most common cause of system failures Written and tested by Kjell Erik Furnes [email protected]

Page 13 of 202

Proof of concept documentation for SAP High Availability system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ (80%), I have implemented a second battery powered alarm-clock which is placed out of reach of my two-year old boy to avoid human errors. This clock is set to go off about five minutes after the first clock so if there is a failure on my first clock I will loose only five minutes. Can you afford to have a SAP system design with a system design which is has less redundancy than my bed-side alarm-clock ?


Page 14 of 202


SAP Netweaver distributed installation This type of installation is not very different than a standard installation except that the central components of the SAP system is distributed on different servers ( SAP Central instance on one server and Database installation on a different server). You can also install additional application servers which can be placed on separate servers, but these are optional and does not add any redundancy for the central parts of the SAP system.

SAP Java stack ABAP stack

SAP APP Serv


SAP APP Serv

SAP APP Serv

Database

The great thing about this system design is that you have now split the database from the SAP installation. There are a couple of reasons why this is a better design then a standard installation. First I am going to give you another piece of information which is often overlooked: In a normal ERP installation it is not uncommon that up to 85% of the transaction responsetime lies in the database. Having that information it is easy to see that your system performance is directly connected with the system performance of your database. So anything that you can do to improve the performance of your database will give you a better SAP transaction responsetime. This includes separating the database from the SAP system so that the database does not have to compete for the system resources. In addition to this you need to remember what I said in the section of the different types of SAP system installations, namely the communication channels used to communicate with the database. Having split the SAP system from the database now enables you to replace the database server to increase the system performance without having to do anything with the SAP installation. In the current configuration this operation would mean system downtime, but we will get to that later in this document. Another great thing about this system design is that you now have a perfect set-up to implement Oracle DataGuard to provide redundancy and security for your SAP database.


Page 15 of 202


SAP Netweaver High Availiability installation This type of installation can be done in couple of different ways depending on if you want to include the database in the SAP HA cluster together with the SAP Central Services, or if you want to have a separate HA cluster for the database, or if you place the database completely outside of the HA cluster.

SAP Java ERS

SAP Java ERS

SAP ABAP ERS

SAP ABAP ERS


r e t s u l C A H


SAP Primary Primary APP Serv SAP APP Serv


SAP APP Serv

SAP APP Serv

Database

It is important to note that in this drawing you have all the benefits of the distributed installation, but there is only HA capabilities for the SAP Central services. I have left the database unprotected because this is the type of system where I want to set up Oracle DataGuard for. In a configuration like this it is easy to point at the SAP Primary Application server and assume that this has now become the new single point of failure. If you assumed that. You would be right. If you perform a default SAP High Availiability installation with a default Central Instance (Primary application server) and a number of default SAP Additional application servers the Primary Application server would now be a single point of failure. The reason for that is that in the default installation this is the only place where you have UPDATE and SPOOL worker processes. If you loose your Primary Application server your system will still be availiable to your users, but your system will soon grind to a halt because all your worker processes will be waiting for available update processes (which is only defined on your central instance, which is down). This is easy to fix because as instances goes the primary instance is little different than the additional instances you install later. The only thing you need to do after installing the additional Application servers is that you need to define theses missing processes in the profile of your additional dialog instances. This is of course just a small task in the post installation configuration that you need to do after installing the SAP Dialog Instance, but it would be nice if SAP would not omit this information in the documentation. After doing the post-installation configuration where you define the missing process types on your dialog instances there is in fact in a HA installation no significant difference between the Central Instance and the Additional Dialog Instances. As long as you have one of these Dialog/Central instances up and running you are able to log and work on to your SAP system.


Page 16 of 202


SAP Netweaver High Availiability installation with Oracle DataGuard

SAP Java ERS

SAP Java ERS

SAP ABAP ERS

SAP ABAP ERS

SAP

This type of system design will give you all the benefits of a true HA capable system design.

Java SCS ABAP SCS

r e t s u l C A H


SAP Primary Primary APP Serv Serv

This design will be the core on SAP APP Serv which I base the rest of this document upon. When SAP APP Serv performing the tests later in the document I will also add another database in the DataGuard configuration to add not only HighAvailiability to the system but also a Disaster recovery solution.

SAP APP Serv Optional Optional Applicatio Application n servers servers SAP APP Serv

Database


Database

It is true that this system design is more complex than a standard installation, and it does require more knowledge from the system administrators, and even regular tests to verify that the configuration works as you intend. But with this configuration you can loose 50% of your hardware resources and still be up and running.


Page 17 of 202


By deliberately placing your system components in different locations you are even able to keep your systems up and running after a complete DataCenter Failure.

DataCenter1

DataCenter2 Vitrual host

SA P W ebDisp at ch er

SAP WebDispa tcher

Load distribution on information information Cluste rNod e1

Vitrual host

ERS

Cl usterNode2

ERS

Active Active /Passive /Passive Cluster (A)SCS

There are a number of benefits coupled with this design and during the course of this document I will try to point out some of them.

ApplicationServer

ApplicationServer

(A)SCS

ApplicationServer

Active SAP applicationservers included in logon groups

ApplicationServer

Oracle DataGuard

DG0

DG1

Drawing 2: SAP Distributed system with Oracle DataGuard


Page 18 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________

Table design From a database administration point of view it makes no real different which type of SAP system is installed (ECC,BI,CRM, etc) . They all use the database in much the same way and it is all about relational tables like any other RDBMS based system. The only system you would think would be different is the BI system which alternates between being named Business Intelligence and Business Warehouse. At least the latter gives the impression that this would be a DataWarehouse where the schema design would be different. Instead of having a relational table design you would have a star/snowball-schema design (William Inmon/Ralph Kimball). Instead SAP claims to have invented a new schema design The Enhanced Star-Schema design. This is one of the parts where I disagree with SAP. The schema design in BI in my point of view is a pure Snowflake design bordering dangerously close to being a traditional relational table design. One thing which is impressive with the design is the reuse of the attribute tables, but this is also the reason why the SAP BI system is unable to compete with a traditional DataWarehouse when it comes to massive data processing.

SAP Transport system An SAP installation is very seldom a single installation. There is usually at least a Development system, and a Production system. All development is usually done in one system (DEV) and then via a transport mechanism the changes done in DEV are propagated to the other systems in the system environment. The DEV system is now the “owner” of all program and customizing changes in your system. The receiving systems only hold the latest version of the program which was transported in to the system while the transactional data are done in the other systems (QAS/PRD).

DEV

QAS

PRD

This often leads to a situation where the developers complain about data quality in the development system, and to avoid a situation where your developers start developing directly in production you need to make production data availiable in a non production system by copying downwards. In SAP this process is called System Copy. You should however never copy your production down to overwrite you development system because that is where all your development history lies. This is the reason for why you often need a QAS system that you are able to refresh from production to be able to test new developments/ upgrades with production-like data before they are put into production. D EV


QAS

PRD

Page 19 of 202


ECC

BI

If you have only one SAP system (ECC) you are in the clear. A system copy from Production to Quality Assurance (QAS) system is a fairly easy and well documented task. But if you also have BI, SRM, PI, Portal, CRM and others then all of these systems will need to be kept in sync because the data in one system is dependent on the other systems.

SCM

So if you need a production line of DEV, QAS, and PRD for your ECC system you should also maintain DEV, QAS and PRD for your BW system (and CRM, SRM, Portal, PI and other legacy (non SAP) systems)

CRM

This often leads to the creation of silos of systems containing a separate consistent entity of systems.

PI

LEGACY

A configuration like this puts an increased strain on your system administration and developer teams because of the share volume of systems and tasks involved in a system copy process. Especially the process involving project systems which often exist on different SAP versions makes this complicated. A scenraio like this means that the developers need to multiply or propagate their development into the project systems.

LEGACY


Page 20 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ The basic production line contains DEV, QAS, PRD, but of course you can implement more systems into the production line, Project, Education, SandBox.

SBX

DEV The “worst” implementation I personally have worked on was an environment having 8 different system entities, and all of the “extra systems” (outside of the basic production line) had different maturity levels with their own transport and refresh needs.

PR1

QAS

PRD

EDU

PR2

PR3 This meant that if there was a need for a system copy of the project systems for PR1 we needed to perform system copies of the ECC, BI, SRM, PI and CRM systems to keep the inter-system data integrity in sync after the system copy.

The curse of SAP SAP as we know it today is not created for any specific database. It can be run on top of a number of databases, and it uses the database mostly as a dumb container for its data. The SAP system does not utilize any database specific functions or procedures. For instance a 20 TB large SAP ECC system based on Oracle does not contain any PL/SQL code except the standard Oracle owned (SYS/SYSTEM). SAP contains a proprietary compiler which uses its own programming language (ABAP) where the programming code is stored in separate tables within the database. So when a SAP program (ABAP) is executed the program is first executed within the memory segments of SAP. Then the program is fed through a number of library files where segments within the ABAP program is converted to database specific SQL statements for the needed database access. What SAP has managed by this approach is that the SAP system is not reliant on any specific database vendor. The only significant change is the database specific library files needed for the database access. Another thing SAP has managed by not committing to any specific database platform is that this opens up for the customer to fairly easily migrate from one database platform to another. When SAP has done this they have also created their own tools for each of the different database platform for performing a number of database maintenance tasks needed for the dayto-day operations. This allows the system administrators to perform many of the maintenance tasks for the databases without having to know what database it is running on. And Written and tested by Kjell Erik Furnes [email protected]

Page 21 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ unfortunately I am certain that somewhere you will find well paid SAP system administrators who does not even know what database the system is running on. The downside of this approach is that all the tools and methods created by the database vendors are disregarded. So you miss out on a lot of the brilliant tools the vendors of a specific database platform have made for their specific database. This means that even though SAP does provide a method for performing the needed maintenance tasks, it might not be the best way to perform these tasks for your specific database platform. This is where the database teams need to apply their expertise on the specific database platform and think outside of the SAP confinements to see if there are ways of performing your tasks without using the SAP recommended way but still not break your SAP contract.


Page 22 of 202


SAP – Oracle license An Oracle DataGuard configuration is covered by your standard SAP license according to SAP Note 740897. You are however not immediately allowed to set up a DataGuard solution on your basic SAP license. One of the reasons for this is that when you purchase SAP your Oracle license is bundled with your SAP purchase (between 20 and 25% (currently 21%)) of your SAP license is a kickback fee to Oracle). This means that the Oracle installation covers installation of one and only one Oracle database server. The SAP note is therefore not incorrect in regards to the DataGuard option. You do have the ability to configure the SAP installation with Oracle DataGuard according to your SAP purchased license if you chose to do so on the same server as your primary server. You can however with the blessing of SAP buy your Oracle licenses directly from Oracle or their partners and configure Oracle DataGuard based on these licenses. See SAP Note 740897 – Info about the Oracle license scope; Required Oracle options The the The the

'SAP License' column indicates whether the option is contained in Oracle database license offered by SAP. 'SAP Support' column indicates whether or under which conditions option is supported by SAP.

Option

SAP License

SAP Support

Partitioning Advanced Security Adv. Compression Std. Mgmt Pack DBA Mgmt Pack Diagnostic Pack Tuning Pack Change Mgmt Pack Provisioning Pack Intermedia OLAP Oracle DataMining Spatial RAC separate Database Vault Audit Vault Label Security Content Database Records Database Real appl.testing appl.testing Oracle Failsafe Oracle Data Guard

yes yes yes yes yes yes yes yes yes yes no yes (3) no (5) separate (5) no no no no no yes yes

yes (1) yes (7) yes (2) (2) (2) (2) (2) (2) no no (4) no (6) yes no no no no no yes yes

If you plan to purchase your Oracle license directly from Oracle make sure that you include at least those options indicated as included in the SAP License. You should also make sure that SAP and Oracle are in sync about your Oracle license agreements so that if you do purchase your Oracle licenses directly from Oracle you do not end up paying double.


Page 23 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ In addition you will find that the parameters you need to set to get an Oracle DataGuard configuration to work are not included in the optimal recommended parameter settings from SAP Notes 830576 – Parameter recommendations for Oracle 10g 1431798: Oracle 11.2.0: Database parameter settings

Which specifies ... You should not set any parameter that are not explicitly mentioned mentioned in this note. ...

There is however a get-out-of-jail statement in both of these two SAP Notes “ ... Exceptions: •

the parameter is required for implementing an individual configuration ( for example, multiple archiver destinations, check functions, special settings

... “

(...) denotes that the note contains more information information and the text here is only an excerpt from from the full text.

This information is included here to show you that the SAP reccomendations are just reccomendations, and you are not breaking your agreement to SAP by setting the needed parameters to set up the Oracle DataGuard configuration. Unfortunately many SAP administrators reads the reccomendations from SAP as regulations and are more likely to grind the system to a halt with the SAP recomendations in place than to implement the reccomendations from their database or OS vendors. Although this aproach might be comfortable for the administrators because they can allways point their finger at SAP saying they are to blame it really is not in their best interest.


Page 24 of 202


The environment I am setting up for this proof of concept installation is Server name

Ipadress (eth0)

esxiserv esxiosl2 esxivexj reposerv netgear duo ds8000 psapcn01 psapcn02 pocsapcl

10.47.253.6 10.47.253.7 88.89.80.35 10.47.253.50 10.47.253.5 10.47.253.4 10.47.253.201 10.47.253.202 10.47.253.200

pnfscn01 pnfscn02 pocnfscl

Ipadress (eth1)

Comment

172.20.16.201 172.20.16.202 172.20.16.200

VMware Server 2 VMware Server 2 VMware Server 2 Repository Server NAS Storage ISCSI Server for shared disk in cluster SAP Cluster Node 1 SAP Cluster Node 2 SAP Central Services cluster IP

10.47.253.51 10.47.253.52

172.20.16.51 172.20.16.52 172.20.16.100

NFS Cluster Node 1 NFS Cluster Node 2 SAP NFS Cluster IP

poradg00

10.47.253.80

172.20.16.80

poradg01 poradg02 pocrcat

10.47.253.81 88.89.80.35 10.47.253.75

172.20.16.81 172.20.16.75

Oracle DataGuard Node (Primary DataBase) Oracle DataGuard Node(Physical Standby) Oracle DataGuard Node(Physical Standby) RMAN server for backup

psapap01 psapap02

10.47.253.211 10.47.253.212

172.20.16.211 172.20.16.212

SAP Central Instance SAP Dialog Instance

These servers are all vmware guests based on three VMware server 2.0 running Centos 5.5 (esxiserv, esxiosl2 + esxivexj)


Page 25 of 202


Hardware mirroring vs. Oracle DataGuard if all you have is a hammer, everything looks like a nail. (Abraham Maslow, The Psychology of Science 1966)

When discussing how to set up a HA/disaster solution you will soon find that hardware people want to do this by hardware mirroring, Oracle people want to do this by DataGuard, programmers want to do this by developing their own programs, and integration people want to set up extractors and pushers between databases to ensure recoverability. It is here important to differentiate between what a High Availability solution and a Disaster Recovery solution should provide. In my view a HA solution is a solution where you have zero or close to zero downtime in case of a system failure of some sort, and you can run the environment at close to 100% even if parts of your environment has failed. While a Disaster Recovery solution as I see it is your last line of defence. This is the type of solution where you accept downtime for a longer timeperiod. The solution might mean a reduction in performance, and additional work or application installation is required to get back up and running, but all your data is secured and intact. There are two ways of replicating data by using hardware replication. Synchronous where you have two storage systems acting as one, and a I/O write on one of the storage systems must be verified by the other storage before the write operation is complete. This configuration is limited by distance and bandwidth and should not be chosen in environments where you risk frequent network outages. If you do have a network outage between two locations in a synchronous hardware replication you may encounter a hang situation because the storage system is waiting for verification from the remote storage system. The other type of hardware replication is asynchronous replication where the I/O operations are queued and sent over the network as normal network traffic without any delay in I/O complete notification to the user or application. While a synchronous hardware replication gives a very good HA solution on the disk-level and you are guaranteed database consistency for a same-site or close proximity data enters this type of configuration is seldom used as a disaster recovery solution system. Disaster solution based on asynchronous data replication on hardware level is a little more dangerous to use for databases because asynchronous hardware replication is usually based on identifying block changes, and transferring the changed blocks to the disaster site. However, a transaction in Oracle is a logical Unit of Work (LUW) which may consist of several steps and as such the database might be in an in-consistent state after a disaster recovery purely based on asynchronous hardware replication. Worst case scenario is that you are left unable to start your database at all due to inconsistency on block level.


Page 26 of 202


If you take a look at a database change you will se that a DML (insert into a table) This will be an insert into the table (block change), a record in the redo files (block change) and the undo rollback segments,, an update of x numbers of indexes (block changes). When the record is committed the redo block will be flagged as committed (block change), and when the redo file is archived the redo record is paged out of the redo files (block change), and the redo file is archived (block change)

Table Index Undo tablespace Redolog Archivelog

Hardware replication X X X X X

Oracle DataGuard

X

In terms of data traffic Oracle DataGuard uncompressed only transfer about 20-25% of the data compared to a hardware replication solution. In addition you can with Oracle 11 implement compression of the log files which reduces the amount of data transported by another 60 – 70%. This means that compared to uncompressed hardware mirroring you are now down to 10 – 15 % of the network load compared to traditional hardware mirroring. The drawback of using logfile compression is that this will put some additional load on your CPU. So if you are experiencing a CPU bottleneck in your DataGuard solution compressing the logfiles will not help you. In all fairness I should also mention that hardware replication does have some benefits compared with Oracle DataGuard, and most providers of hardware replication do have their own way of minimizing the network load. One benefit of running hardware replication instead of Oracle DataGuard is that when using hardware replication the system is independent on version or vendor of software running the database. Another benefit is that when using hardware replication you can also include non-oracle database files in the replication to achieve a specific point-in-time (snapshot) for all files in the solution. This is especially true if you are using a standard installation of your SAP system where both the application and the database reside on the same server. A definite drawback of using hardware replication instead of Oracle DataGuard in my opinion is that in a solution using hardware replication of data the standby systems must be cold systems whereas when using Oracle DataGuard the standby database is already up and running and need only a state change which is completed in seconds to start operating as the primary database. Therefore activating the standby or disaster solution either during a disaster scenario or in a training exercise for the database is far quicker by using Oracle DataGuard compared to any other form of replication.


Page 27 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Before you implement any form of replication solution you should use some time to identify your needs and possible bottlenecks. You need to be clear on which scenarios of failure or disaster you want to have a solution for. In this document I describe one possible solution, but then I am an Oracle DBA so I guess I am just a little bit biased as to which solution I would prefer. You can also as a disaster recovery solution implement some form of point-in-time solution, but this is to me more a quick-backup/quick-recover solution based on some form of splitmirror or pointer-table backup than it is an online replication solution. A point-in-time solution should not be used as a replacement of a Disaster recovery solution, but could easily be used as a supplement to for instance Oracle DataGuard. NOTE: If you use a point-in-time solution together with a DataGuard in a restore scenario on your primary site you need to take care to handle your DataGuard solution correct. Before you restore at your primary site you need to disable any fast-start failover and DataGuard Broker configuration. After restore at your primary site you need to find the current SCN status of your primary, and then flashback all your standby databases to the same SCN in order to keep the synchronization between the primary and standby systems.


Page 28 of 202


Oracle DataGuard vs. RAC Normally if you are discussing a same-site database solution with some form of hardware failtolerance Oracle will point to Oracle Real Application Cluster (RAC), and I agree there are some benefits to use Oracle RAC for same-site or even close proximity datacenter solutions. But since I also wanted to bring in disaster recovery to the equation I needed to see if it was possible to do this with an Oracle DataGuard solution. I addition there are no complete documentation available from SAP on setting up the system with Oracle Dataguard. There is one document from Oracle concerning implementation of Oracle DataGuard for SAP, but this document in my opinion is incomplete because it only describes DataGuard for a single stack ABAP system. If you do want to implement Oracle RAC it is also possible to combine Oracle RAC and Oracle DataGuard. You could for instance set up RAC at your primary site, and use a single instance on your secondary site. I have chosen to focus on the Oracle DataGuard scenarion in this configuration in this document, and will not cover RAC. The configurations changes you need to perform for SAP are the same for implementing Oracle RAC as it is for Oracle DataGuard. (Oracle database connection configuration changes)


Page 29 of 202


Disaster / Inconvenience: What is a disaster, and who can initiate a disaster solution ? To most users any form of disruption in dataflow is a disaster. However, a blown fuse in some remote users fuse box rarely initiates the implementation of a central disaster recovery solution. It should also be noted that there are some distinct differences between a High Availability configuration and a Disaster Recovery configuration. In a High Availability solution you often have same-site or close proximity sites participating in cluster configuration where switchover-time is reduced to a minimum whereas a Disaster Recovery solution is often located at a completely different location. Oracle DataGuard can provide both High Availability and Disaster Recovery and more. There are some questions that you need to ask before you start implementing any form of disaster recovery or High Availability solution, and the answers to these questions should be part of a Service Level Agreement (SLA) so that everyone involved when a situation occurs knows what is expected and who needs to take the decisions and who needs to do the actions..

Return to Normal Time Objective (RNTO) In a complex environment on of the most difficult things to handle is not the disaster solution. This is quite easy to hadle in a lot of different scenarios, and while in a real life disaster scenarion you are under battle-command where orders and actions are taken quickly. However the point in time where you need o return from a disaster situation and back to your normal configuration can be a huge challenge because for most companies the multitude of data systems are often so integrated that they should really be viewed as a single entity, and many of them should be part of the same disaster solution. This statement is especially important when considering your data-loss policy and your return to normal procedures If integrated systems are not part of the same disaster solution you may risk that after returning to normal your systems are out of sync and would need to be built / reloaded from square one. D a t a l o s s

Recove Recoverr time time Site recov recovere ered d

n c y s o f t u s o m e s t e S y

e r t e s s a D i Drawing 3: Return to normal data loss


Page 30 of 202


SAP High Availability Solution SAP Directory Structure Clustered

Clustered NFS

Global filesystem

Local

usr

sapmnt

sap < SID > < SID >

trans

SYS global

global

profile

profile

exe

exe

(nu (nuc / uc uc ) < platfor platform m> < win32 > < hpux64 >

DAA ASCS ERS

(nuc (n uc / uc uc )

log data work exe

SCS log data work

log data work exe

exe

sapcpe

Drawing 4: HA SAP Central instance services directory structure

The SAP documentation prior to the installation Guide for SAP Netweaver 7.0 EHP2 specified that everything below /usr/sap should be a separate file system. In a normal configuration, and even in a simple HA configuration this was ok, but if you are looking to implement the SAP Enqueue Replication Servers (ERS) this requires a subdirectory of /usr/sap/ to be a local file system to hold the ERS instances. So if you were to follow the SAP documentation you would now be mounting a local file system inside a switchover file system. Additionally if you had more than one SAP system in your cluster or standalone system on your cluster hosts this configuration was wrong and you would be unable to switchover your SAP system. I believe the configuration is a wrong approach since you will then need to be very careful with the order of mounting and unmounting your file systems. I would advise against the recommended SAP procedure unless you have a complete control of the startup, shutdown and switchover process. Since I do not want my file systems mounted in this way I have instead decided to create separate file systems for each SAP instance (DAA, ASCS, SCS, ERS (ABAP) and ERS (java))


Page 31 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ In some SAP documentation you will also find that ASCS and SCS must be part of the same file system. I also believe this too to be wrong. I would agree if the statement would be that ASCS and SCS file systems must be part of the same switchover package.


Page 32 of 202


SAP <-> Oracle communication channles All communication made by any application, even SAP, to the database is done by JDBC Type I, or SQLNet. The JDBC Type I (Thin), is bypassing the tnsnames.ora file, and does not require an Oracle Client installation.

Application/User/Java

) n i h t ( I V C B D J e l c a r O

C B D J e v i t a N

L Q S e l b a t r o P

J2EB

Open Open SQL /SQL-J

JDO

Open Open SQL /JDBC

Open SQL Engine Engine SQL Pre-Processor DB Meta structures structures Statement Statement pooling

DB Access Layer

SQL Trace

O r a c l e J D B C V I I ( O C I )

O r a c l e S Q L N e t

tnsnames.ora

All other types of communication to the listener database use SQLNet in Oracle SQL engine some form ( ODBC/JDBC II/OCI/SQLPlus) which uses the configuration parameter combinations from the tnsnames.ora to communicate with the Listener. Armed with that piece of information you can see that could easily split any application from your database (distributed system), and place the database any place where you can have sufficient network connectivity..


Page 33 of 202


Connection channels and Oracle DataGuard In order to have an application to be able to communicate to a DataGuard solution we need to make some changes to the default tnsnames.ora file, and we need to use the SERVICE connection parameter instead of the SID connection parameter. The reason for this is that the SID connection parameter is database specific while the SERVICE connection parameter can point to several database installations. Application/User/Java

J2EB

OpenSQL /SQL-J

) n i h t ( I V C B D J e l c a r O

C B D J e v i t a N

JDO

OpenSQL /JDBC

Open SQL Engine Engine SQL Pre-Processor DB Meta structures structures Statement pooling

DB Access Layer

SQL Trace

L Q S e l b a t r o P

O r a c l e J D B C V I I ( O

O r a c l e S Q L N e t

C I )

tnsnames.ora

listener

listener

Oracle SQL engine

Oracle SQL engine

A tnsnames.ora file for a service could look like this XX1.WORLD (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = ON) (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.210)(PORT=1527)) (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.81)(PORT=1527)) (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.82)(PORT=1527)) ) ) (CONNECT_DATA = (SERVICE_NAME = XX1) ) )

Since we are running a DataGuard solution and not a RAC solution we need however to make sure that this service run on only one of our systems. We can do this by creating a simple trigger which is activated in the role-change sequence where the database take over the role of becoming the new primary or when the database resigns from the role of being the primary database. There is of course one tiny caveat to this. When an oracle database is started there is one service that is always created by the PMON service, and that is a service with the same name as the database name, i.e. a database named PRD will automatically have a service named PRD. When prepearing your SAP system to use Oracle DataGuard for switchover capabilities


Page 34 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ you will need to make a few easy changes to your system so that you can use a re-locatable SERVICE. I will get to that at the end of the SAP system installation section. The changes to the tnsnames.ora file would cover the traditional connection channels, but it would not cover the JDBC version I connections.

JDBC The Oracle JDBC thin driver was originally created by Oracle as a means for small java applications and applets to connect to a database without having an Oracle client installation. For larger Java applications there is the JDBC VII driver (OCI) driver. For a enterprise system like SAP where you do have the Oracle client installed on every server it should be natural to use the OCI driver which has more possibilities, and uses the “normal” oracle connection by utilizing the tnsnames.ora for connection details, but SAP have still opted to use the thin driver in the default JDBC DataSource connection. The documentation from SAP specifies that there is no HA capabilities for the Oracle database connection using JDBC version I. According to SAP documentation you will need to use JDBC VII or OPEN SQL to be able to have a HA solution in place for Oracle. The process of changing the default SAP JDBC DataSource connection to use a different driver is not something that is straight forward. So according to the SAP documentation you are now stuck. You are unable to use Oracle DataGuard for any SAP system with a java-stack. In the default SAP installation the JDBC connection is set up using a specific connection string: jdbc:oracle:thin:@:: Example: jdbc:oracle:thin:@poradg00:1527:XX1

And in this configuration it is true. You can only point to one single Oracle instance on one single host. The secret here is that the statement in the SAP documentation is only true if you limit the JDBC thin connection string to the options used in the default SAP installation. In the specifications of the JDBC Thin driver you will see that there are other very easy ways of defining the connectionstring to include multiple databases and the SERVICE connection parameter. You will only need to change the connection URL to a format like the tnsnames.ora file to point to the other listeners. There are ways can alter the connection definition of the default SAP JDBC connection and still use the thin driver. jdbc:oracle:thin@(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.210)(PORT=1527)) (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.81)(PORT=1527)) (ADDRESS=(PROTOCOL=TCP)(HOST=10.47.253.82)(PORT=1527))))(CONNECT_DATA=(SERVIC E_NAME=XX1)))

Note: No linefeeds or whitespace in this string


Page 35 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ In contradiction to the SAP documentation you can by changing the connection URL establish HA capabilities for your Oracle database in Java and still using the jdbc oracle.thin driver. In order to have a SAP dual stack system (ABAP + JAVA) communicate with an Oracle DataGuard solution we will need to make changes to both the tnsnames.ora file for the ABAP instance and the jdbc thin driver for the Java instance. I will return to the specifics of setting up SAP for DataGuard at the end of the SAP Database instance installation.


Page 36 of 202


Configuring VMware guest servers Test environment

VMWare server Configuration The two main VMWare servers are two Packard Bell iXtreme 5140 Servers with 6 Core AMD Athlon CPU and 8 GB RAM. The VMware guest servers I will be defining will have a basic standard. 2 CPU 3 Hard drives HDD 1 : HDD 2 : HDD 3:

7GB 2GB 2GB

System swap swap

NIC 1 : NIC 2:

IP ADDR 10.47.253.x 172.20.16.x

2 NIC NETMASK 10.47.253.0/24 172.20.16.0/24

GATEWAY 10.47.253.1

DNS 10.47.253.1

To make sure that the servers are installed identically I will be using a kickstart file with many of the tasks to be performed automated in the initial script.

Time synchronization SAP is a distributed system over several servers. You will need to synchronize time between all these servers. In my environment I have set up the two VMware servers as ntp servers. All other servers in this environment are synchronizing with these to servers. The configuration consists of: Two Linux clusters 1. SAP central services (ABAP +JAVA) 2. NFS server One ISCSI server simulating SAN One SAP Central Instance Two SAP Application servers Four Oracle Database servers making up the Oracle DataGuard (3 Physical and One Logical) One Oracle Database server for the RMAN Catalog Service


Page 37 of 202


Setting up a Linux cluster is something that will require work in both the configuration gui and as text editing of the configuration file. The easiest approach is to stat with the GUI to get a basic configuration file, and then do the specifics by hand as text editing afterwards. The clusters will start off with a very basic configuration name="pocnfsc"> post_join_delay="3"/> quorum_dev_poll="400000"/> " vmlogin="kfurnes" vmpasswd="" vmware_type="server2"/> vmware_type="server2"/> " vmlogin="kfurnes" vmpasswd="" vmware_type="server2"/> vmware_type="server2"/> syslog_facility="local7"> syslog_facility="local7"/> syslog_facility="local7"/>

There are a few things to be made note of on this standard cluster configuration. 1. I am using vmware-fencing which means there is a needed installation of the vmware agent software and some text editing to get in place. 2.


Page 38 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ a. This statement is needed because the disk configuration I am using for this PoC system is very slow, and to avoid the dreaded Quorum Dissolved error I have increased the timeout value for the quorum device. 3.

a. The totem parameters are usually not needed unless you have really large clusters where you have synchronization issues. But again, since my configuration is very limited on resources I experienced some timeout errors due to synchronization problems.


Page 39 of 202


Installation

Installing the Linux servers (OS and configuration) See SAP Note 171356. SAP software on Linux: Essential information

In my configuration I have a ISCSI server providing shared disk between the two different vmware host servers. In a normal configuration the disk solution would be provided by a SAN solution, but since my test environment is far from enterprise grade I have set up this disk provider server. This server have a quite large disk pool on which I create block devices which I present as ISCSI target devices to the other servers in my configuration. Creating block devices like this means that the disk devices on the servers will be very slow. I have accepted this since my test case here is not based on performance, but rather on functionality of a certain configuration.

Disk, Volume and File systems My configuration is based on the use of Logical Volumes because this makes it easy to increase the filesystem size if there is a need for that. To create a logical volume you first need one or more unused Physical Disk or partition.

Display Physical Disk availiable on your system The command fdisk –l will show you the disks which area availiable on your system [root@ds8000 target0]# fdisk -l Disk /dev/sda: 6442 MB, 6442450944 bytes 255 heads, 63 sectors/track, sectors/track, 783 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/sda1 * /dev/sda2

Start 1 14

End 13 783

Blocks 104391 6185025

Id 83 8e

System Linux Linux LVM

Disk /dev/sdb: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdb doesn't contain a valid partition table Disk /dev/sdc: 2147 MB, 2147483648 bytes 255 heads, 63 sectors/track, sectors/track, 261 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdc doesn't contain a valid partition table


Page 40 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Disk /dev/sdd: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdd doesn't contain a valid partition table

Create a Physical Volume pvcreate /dev/sd

To display the information on the Physical volumes in your system enter the command pvscan [root@ds8000 ~]# PV /dev/sdd PV /dev/sde PV /dev/sdf

pvscan VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3

lvm2 [10.00 GB / 0 lvm2 [10.00 GB / 0 lvm2 [10.00 GB / 0

free] free] free]

If you need more information about the Physical Volumes defined on your system enter command pvdisplay sample output [root@ds8000 orafra]# pvdisplay /dev/sdd --- Physical volume --PV Name /dev/sdd VG Name vg_target0_ext3 vg_target0_ext3 PV Size 10.00 GB / not usable 4.00 MB Allocatable yes PE Size (KByte) 4096 Total PE 2559 Free PE 0 Allocated PE 2559 PV UUID 6XvVpw-3sJe-VjwH-03LD-nUI 6XvVpw-3sJe- VjwH-03LD-nUI v-8gCd-rtGkzZ v-8gCd-rtGkz Z

Create a new Volume Group vgcreate –cn vg_oradata_ext3 /dev/sd /dev/sd

The variables and are availiable disks where you have created physical volumes.

The parameter –c specifies if this volume group should be cluster availiable or not (y|n)

Resize Volume Group Over time you might find that you need to resize a volumegroup vgextend vg_oradata_ext3 /dev/sd

The variable is the identifier of the new availiable disks where you you have created physical volume.

Display Volume Group information [root@ds8000 ~]# vgscan Found volume group "vg_oradata_ext3" using metadata type lvm2 Found volume group "vg_oradiag_ext3" using metadata type lvm2

Create a new Logical Volume lvcreate –l 90%FREE –n lv_oradata_ext3 lv_oradata_ext3 vg_oradata_ext3


Page 41 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ The –l 90%FREE is used to specify how much of the volumegroup you want to allocate to the new logical volume. The value of the parameter is oposite of what you would expect. The value of 90%FREE means that 90% of the volume group is allocated. A specification of 100%FREE means use all of the availiable size. You can also specify the size of the new Logical Volume by specifying the size in actual size. lvcreate –L 10G –n lv_oradata_ext3 vg_oradata_ext3 vg_oradata_ext3

Display Logical Volume information Logical Volumes [root@ds8000 ~]# lvscan ACTIVE ACTIVE

'/dev/vg_orad ata_ext3/lv_oradata_ext3' ata_ext3/lv_ oradata_ext3' [35.98 GB] inherit '/dev/vg_orad iag_ext3/lv_oradiag_ext3' iag_ext3/lv_ oradiag_ext3' [39.98 GB] inherit

Resize logical volumes lvresize –L+20G /dev/vg_oradata_ext3/lv_oradata_ext3

or lvextend –l 90%FREE /dev/vg_oradata_ext3/lv /dev/vg_oradata_ext3/lv_oradata_ex _oradata_ext3 t3

File System In my configuration I am using the filesystems (Extended Filesystem version 3) ext3 and (Global Filesystem) gfs. Ext3 is more or less the standard filesystem on Linux systems, and is a journaling filesystem. You use this filesystem for filesystems where there is single instance mount. Gfs is a cluster filesystem which you can use where there are multiple servers mounting the filesystem. In my cluster configuration I will be using gfs for filesystems which are included in switchover packages. Since my clusters are Active-Passive clusters I could have used ext3 for the switchover packages because there will only be one active host writing to the disk, but that would mean that during a switchover/failover the ext3 filesystem would need to be unmounted on the failing server and then remounted on the new server. By using gfs your switchover package filesystems are allready mounted on the other server so you reduce the switchover time considderably.

Create ext3 filesystem mkfs.ext3 –L ORADATA –b 4096 /dev/vg_oradata_ext3/lv_oradata_ext3

To avoid having to use the entire path of the filesystem when mounting the filesystem I am creating a LABEL by using the –L parameter. The block size in the filesystem is defined by using the –b parameter (1024, 2048 or 4096). Written and tested by Kjell Erik Furnes [email protected]

Page 42 of 202


Resizing ext3 file systems based on LVM After you have resized the Volume Groups and Logical Volumes you need to resize your filesystem to set the new size availiable. If you need to increase the size of your filesystem, you can do this while the filesystem is mounted. resize2fs –p /dev/vg_oradata_ext3/lv_oradata_ext3

If you need to reduce the size of your filesystem you need to unmount the filesystem before you can run the resize2fs command.

Create gfs filesystem mkfs.gfs –D –O –t pocnfsc:gfs_global –p lock_dlm –j 3 /dev/vg_global_gfs/lv_global_gfs

-D -O -t -p -j

enable debug information skip filesystems creation confirmation locktable_nam e (clustername:filesystemna (clustername :filesystemna me) name of lock table protocol journaling number. The number of servers that can mount this filesystem ( I use cluster nodes +1)

Resizing gfs file systems based on LVM In order to resize the gfs filesystem you run the command gfs_grow only on one of the nodes in the cluster. The filesystem must be mounted, and all nodes in the cluster must be up and running. To test the command before you run the real grow command enter the command gfs_grow –T -vv /dev/VolGroup01/LogVol01 /dev/VolGroup01/LogVol01

-T -v

Test, No changes are made to your filesystem verbose

To run the actual resize of the file system enter command gfs_grow /dev/VolGroup01/LogVol01

Adding disk in a vmware environment In a vmware environment you are able to add disk while your guest system is up and running, but the disks will not automatically be visible in your guest OS. To be able to see the new disk in the guest OS you will need to kick-start a new scan of the scsi modules. After you add a new disk for the guest OS in your VMware host system you must run a command in your VMware guest OS. As root execute the command echo “- - -“ > /sys/class/scsi_host/host0/scan


Page 43 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ This command will force the SCSI module to rescan the SCSI controller, and in our case find a new disk without taking down the system.

ISCSI targets In my environment I am creating block devices to act as pseudo devices which is presented as ISCSI targets. From the OS level of the ISCSI initiator (the receiving client) this will apear to be physical disks, but the performance of these block devices which are presented accross the network is really slow. Since this is a test system performance is not important, but remember that this is not in any way, shape or form something that you should use in a critical system environment.

Note in this solution I will be using a centralized disk solution based on ISCSI. Due to an error in the configuration in the default shutdown scripts the system will hang and eventually panic because the system will try to run disk synchronization over the network. The reason why this fail is that at this point the network is already switched off. Workaround for this error is to upgrade your initscripts package to at least 8.45.33-1.el5

Creating block devices (ISCSI Target device) To create the block devices use the low-level byte duplicator utility dd. The dd command is one of the oldest UNIX commands which has survived til today. Some of the reasons why it has survived is that this command has many uses (backup, disk copy etc). When we come around to automated block recovery in Oracle 11 at the end of this document we are actually going to use the dd command to create a corrupted datafile to enable the demonstration. Since the dd command operates on a low-level it is very powerfull and there is no fail-safe built into the command. So if you mix-up the input (if) with the output (of) you are able to damage your sytem severely. The original name of the command was Data Descriptor, but it is also referred to as “Data Destroyer” and “Disk Delete”. It is old and distinguished so use it with respect. To create a new block device the command is: dd if=/dev/zero of=/opt/iscsi/target2/LUN3 bs=4096 count=800000 if

input file By using /dev/zero you are creating a file filled with null characters

of Output file bs block size count The number of blocks of bs to output

Presenting the ISCSI block devices to the t he ISCSI clients


Page 44 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ In order to present the blockdevices to the ISCSI devises the tgtd service must be running on the server. Ths tgtd service is provided in the scsi-target-utils package. There are some confision about the usage of names connected to the iscsi configuration. There are two names which creates this confusion. The target and the initiator. In a standard server – client context the target is the server which provides the disk. The initiator is the client which receives the disk information from the target (server). On the target (server) there is actually only one file which you need to care about. /etc/tgt/targets.conf This file is quite basic and it has a XML like content get0> > removable 0 device-type disk lun 1 > removable 0 device-type disk lun 2 # Global parameters vendor_id SLYNGA49 product_id iscsi write-cache on initiator-address initiator-address 172.20.16.51 172.20.16.51 initiator-address initiator-address 172.20.16.52 172.20.16.52 incominguser sapusr somepassword

The initiator-adress is the ip-adress of the initiators (clients) which are allowed to conenct to this configuration. The incomminguser is the username and passwords which is used by the initiators to identify to the target. This password is transmitted in clear-text over the network so this password should be different than the password of any priviledged system user. If the tgtd service is running you can run the command service tgtd reload

to force the tgtd service to re-read the configuration. If the service is not running you start the service by running service tgtd start

Display ISCSI target information


Page 45 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ [root@ds8000 ~]# tgtadm --lld iscsi --op show --mode target Target 1: iqn.2011-02.net.slynga49:ds8000.target0 System information: Driver: iscsi State: ready I_T nexus information: information: I_T nexus: 23 I_T nexus: 60 Initiator: iqn.2011-02.net.slynga49 iqn.2011-02.net.slynga49:pnfscn02 :pnfscn02 Connection: 0 IP Address: 172.20.16.52 I_T nexus: 61 Initiator: iqn.2011-02.net.slynga49 iqn.2011-02.net.slynga49:pnfscn01 :pnfscn01 Connection: 0 IP Address: 172.20.16.51 LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB Online: Yes Removable media: No Backing store type: rdwr Backing store path: None LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 42 MB Online: Yes Removable media: No Backing store type: rdwr Backing store path: /opt/iscsi/target0/LUN1 LUN: 2 Type: disk SCSI ID: IET 00010002 SCSI SN: beaf12 Size: 8844 MB Online: Yes Removable media: No Backing store type: rdwr Backing store path: /opt/iscsi/target0/LUN2

In the show command here you can see that there are two initiator clients which are connected to the systems.

Configuration of the ISCSI initiator On the initiator side there is some more configuration that you need to handle. The services iscsi and iscsid needs to be running on the initiator. These services are included in the iscsi-initiator-utils package. For the basic configuration there are only two files which you need to edit.


Page 46 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ /etc/iscsi/initiatorname.iscsi

Content of this file is only the name of the server InitiatorName=iqn.2011InitiatorName=iqn.2011-02.net.slyng 02.net.slynga49:pnfscn01 a49:pnfscn01 /etc/iscsi/iscsid.conf

This file has some more lines that needs configuration node.startup = automtic node.session.auth.authm node.session.auth.authmethod ethod = CHAP node.session.auth.usern node.session.auth.username ame = sapuser node.session.auth.passw node.session.auth.password ord = somepassword

To initiate the communication between the initiator client and the target server enter the comamnd iscsiadm –m discovery –t sendtargets --portal 172.20.16.4 172.20.16.4 –login

To see that you now have the new ISCSI disks availiable on your systems you can take a look in the /dev/disk/by-path directory [root@pnfscn01 by-path]# ls ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-1 ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-2 ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-3 ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-4 ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-5 ip-172.20.16.4:3260-iscsi-iqn.2011-02.net.slynga49:ds8000.target0-lun-6

pci-0000:00:07.1-ide-0:0 pci-0000:00:10.0-scsi-0:0:0:0 pci-0000:00:10.0-scsi-0:0:0:0-part1 pci-0000:00:10.0-scsi-0:0:0:0-part2 pci-0000:00:10.0-scsi-0:0:1:0 pci-0000:00:10.0-scsi-0:0:2:0

This information is however a little difficult to use for anything other than to identify that there are new disks availiable. To see your new disk you can enter the command fdisk –l Your new disks will now be entered as /dev/sd, and is avialiable for you to create your new LVM and filesystem configuration on. There is however one big caveat in this basic configuration. Since these new disks are presented over the network and receives their identification sd on a first come – first served basis there is no guarantee that /dev/sdb will be /dev/sdb after your next reboot. To avoid a situation where your filesystems are messed up after your next reboot you need to set up your system to identify the disks persistently across re-boots.

Persistent ISCSI mount To get a persistent ISCSI mount across re-boots of your ISCIS initiator clients you need to collect some information from your target server. On your target server run the command tgtadm --lld iscsi --op show --mode target


Page 47 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ The lines you are looking for are the lines for each of the LUN. SCSI ID: IET

00010001

You can use this information to write some rules which are used by the udev program which is responsible for the mount process. Create a new file in the /etc/udev/rules.d directory. Remember that the initial number (here 61) needs to be a higher number that the xx-net.rules file.

[root@pnfscn01 [root@pnfscn01 rules.d]# cat 61-iscsi.rules 61-iscsi.rules # /etc/udev/rules.d/55-openiscsi.rules # KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/bin/iscsi-device-mapping.sh %b",SYMLINK+="iscsi/%c/part%n" # udev QUORUM iSCSI volume KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p",RESULT=="1IET_00010001", SYMLINK+="QUORUM%n" # udev SAPMNT iSCSI volume KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="1IET_00010002", SYMLINK+="SAPMNT%n" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="1IET_00010003", SYMLINK+="SAPTRANS%n" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="1IET_00010006", SYMLINK+="GLOBAL%n"

In this file you see that you use the result from the tgtadm command in the RESULT parameter to identify and create an alias for the disk. With this rule in place you need to restart the iscsi service or perform a re-boot. After the restart you will see that you have now some new symbolic links in the /dev directory [root@pnfscn01 [root@pnfsc n01 dev]# ls -l |grep lrwxrwxrwx 1 root root 3 lrwxrwxrwx 1 root root 3 lrwxrwxrwx 1 root root 3 lrwxrwxrwx 1 root root 3 lrwxrwxrwx 1 root root 3 lrwxrwxrwx 1 root root 3

^l|grep sd Sep 2 23:57 Sep 2 23:57 Sep 2 23:57 Sep 2 23:57 Sep 2 23:57 Sep 2 23:57

GLOBAL -> sdi ORADIAG -> sdg ORAFRA -> sdh QUORUM -> sdd SAPMNT -> sde SAPTRANS -> sdf

These symbolic links are dynamic across the re-boots so the /dev/sd identifier is no longer important. IMPORTANTe: Remember that when defining the new LVM configuration you need to use the symbolic link name to create the LVM Example pvcreate /dev/ORADIAG vgcreate –cn vg_oradiag_ext3 /dev/ORADIAG

Resizing iscsi targets The iscsi environment consists of a target (the iscsi server) and an initiator (the iscsi client). The way that I have configured this system is a poor-man’s solution. I am defining block devices inside vmware disks inside vmware guests. This must be the slowest types of disk available today, but it does get the testing done within my budget frame (no-cost), and I am working over several vmware servers so for a test environment it’s ok. You will see that this is really slow disks by the extra parameters we need to set to avoid the QUORUM timeout and QUORUM dissolved because scanning the quorum takes too long. Written and tested by Kjell Erik Furnes [email protected]

Page 48 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ However, because of restrictions in the iscsi tools it is not possible to resize an iscsi target where there are initiators (clients) still logged on. You can change the configuration, and reload the target service all you want, but no changes will be visible neither on the target server nor on the initiator clients. This is a good thing, and it prevents you from doing terminal damage to your data. The way that you can add more data to you iscsi initiators is to add a new target configuration and let this propagate out to your initiator clients by doing a service tgtd reload

on your initiator clients you can now run iscsiadm –m session –rescan

Firewall You will find SAP documentation telling you to stop all firewalls in your environment to ensure that all traffic goes un-hindered. For being one of the worlds largest software distributor of enterprise critical information I am not sure turning off the security measures would be the advise I would be giving nor is it an advise I would like to get. I will therefore be setting up iptables firewalls to ensure that there is some level of security within the environment.

Setting up a basic firewall with iptables is quite easy To view the ports that you machine is listening on use the command: netstat –an |grep LISTEN |grep –e tcp –e udp

To identify what process is using a particular port enter the command: fuser –v –n [root@psapcn01 [root@psapcn01 ~]# fuser -v -n tcp 8110

8110/tcp:

USER xx1adm

PID ACCESS COMMAND 6817 F.... ms.sapXX1_SCS10 ms.sapXX1_SCS10

To identify the exact process enter the command Ps –ef |grep [root@psapcn01 ~]# ps -ef |grep 6817 xx1adm 6817 6804 0 Sep14 ? 00:00:03 ms.sapXX1_SCS10 ms.sapXX1_S CS10 pf=/usr/sap/XX1/SYS/profile/XX1_SCS10_pocsapcl

To see the current status of your firewall enter service iptables status [root@psapcn01 ~]# service iptables status Table: filter Chain INPUT (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all -- 0.0.0.0/0


destination 0.0.0.0/0

Page 49 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Chain FORWARD (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all --

0.0.0.0/0


Chain OUTPUT (policy ACCEPT) num target prot opt source

destination

Chain RH-Firewall-1-INPUT (2 references) num target prot opt source 1 ACCEPT all -- 0.0.0.0/0 2 ACCEPT icmp -- 0.0.0.0/0 3 ACCEPT esp -- 0.0.0.0/0 4 ACCEPT ah -- 0.0.0.0/0 5 ACCEPT udp -- 0.0.0.0/0 6 ACCEPT tcp -- 0.0.0.0/0 7 ACCEPT all -- 0.0.0.0/0 8 ACCEPT tcp -- 0.0.0.0/0 9 ACCEPT tcp -- 0.0.0.0/0 10 ACCEPT tcp -- 0.0.0.0/0 11 ACCEPT tcp -- 0.0.0.0/0 12 ACCEPT tcp -- 0.0.0.0/0 13 ACCEPT tcp -- 0.0.0.0/0 14 ACCEPT tcp -- 0.0.0.0/0 15 ACCEPT tcp -- 0.0.0.0/0 16 ACCEPT tcp -- 0.0.0.0/0 17 ACCEPT tcp -- 0.0.0.0/0 18 ACCEPT tcp -- 0.0.0.0/0 19 ACCEPT tcp -- 0.0.0.0/0 20 ACCEPT tcp -- 0.0.0.0/0 21 ACCEPT tcp -- 0.0.0.0/0 22 ACCEPT udp -- 0.0.0.0/0 23 ACCEPT udp -- 0.0.0.0/0 24 ACCEPT tcp -- 0.0.0.0/0

destination 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0

icmp type 255

udp dpt:631 tcp dpt:631 state RELATED,ESTABLISHED state NEW tcp dpt:8084 state NEW tcp dpt:22 state NEW tcp dpt:11111 state NEW tcp dpt:14567 state NEW tcp dpt:16851 state NEW tcp dpt:21064 state NEW tcp dpt:41966 state NEW tcp dpt:41967 state NEW tcp dpt:41968 state NEW tcp dpt:41969 state NEW tcp dpt:50006 state NEW tcp dpt:50007 state NEW tcp dpt:50008 state NEW tcp dpt:50009 state NEW udp dpt:5404 state NEW udp dpt:5405 state NEW tcp dpt:111

To save the iptables rules as a text file enter the command iptables-save iptables-save > /tmp/iptables-rules.txt /tmp/iptables-rules.txt

edit the file to your liking # Generated by iptables-save v1.3.5 on Fri Sep 16 00:19:05 2011 *filter :INPUT ACCEPT [10123:715944] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [3052839:780112867] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport -A RH-Firewall-1-INPUT -i eth1 -p udp -m state --state NEW -m udp --dport -A RH-Firewall-1-INPUT -i eth1 -p udp -m state --state NEW -m udp --dport -A RH-Firewall-1-INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport COMMIT # Completed on Fri Sep 16 00:19:05 2011

8084 -j ACCEPT 22 -j ACCEPT 11111 -j ACCEPT 14567 -j ACCEPT 16851 -j ACCEPT 21064 -j ACCEPT 41966 -j ACCEPT 41967 -j ACCEPT 41968 -j ACCEPT 41969 -j ACCEPT 50006 -j ACCEPT 50007 -j ACCEPT 50008 -j ACCEPT 50009 -j ACCEPT 5404 -j ACCEPT 5405 -j ACCEPT 111 -j ACCEPT

Feed the new file into the iptables iptables-restore /tmp/iptables-rule.txt

Save the new configuration service iptables save


Page 50 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Restart he iptables service to load the new configuration service iptables restart

To see that you new iptables configuration is in place, enter the command service iptables status

To allow Red Hat Cluster nodes to communicate with each other, you must enable the IP ports assigned to certain Red Hat Cluster components. At each cluster node, enable IP ports according to Table IP Port Number

Protocol

Component

8084 5404, 5405 11111 14567 16851 21064 41966, 41967, 41968, 41969 50006, 50008, 50009 50007

TCP UDP TCP TCP TCP TCP TCP TCP UDP

luci cman (Cluster Manager) ricci (part of Conga remote agent) gnbd (Global Network Block Device) modclusterd (part of Conga remote agent) dlm (Distributed Lock Manager) rgmanager (high-availabil ity service management) ccsd (Cluster Configuration System daemon) ccsd (Cluster Configuration System daemon)

ISCSI iptable rules -A RH-Firewall-1-INPUT -i eth1 –s 172.20.16.0/24 -p udp -m state --state NEW -m udp --dport 3260 -j ACCEPT

ntp iptable rules -A RH-Firewall-1-INPUT -i eth1 –s 172.20.16.0/24 -p udp -m state --state NEW -m udp --dport 123 -j ACCEPT

Cluster iptable rules -A RH-Firewall-1-INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 8084 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 11111 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 14567 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 16851 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 21064 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 41966 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 41967 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 41968 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 41969 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 50006 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m udp --dport 50007 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 50008 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p tcp -m state --state NEW -m tcp --dport 50009 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p udp -m state --state NEW -m udp --dport 5404 -j ACCEPT -A RH-Firewall-1-INPUT -i eth1 -p udp -m state --state NEW -m udp --dport 5405 -j ACCEPT

NFS iptable rules -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 111 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 2049 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 2049 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 32769 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 32769 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 32803 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 32803 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 662 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 662 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 892 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 892 -j ACCEPT


Page 51 of 202


ORACLE iptable rules -A RH-Firewall-1-INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 1527 -j ACCEPT

SAP iptable rules IP Port Number

Protocol

Component

1128 (hostcontrol agent) 32 33 36 39 81 500 504 507 508 513 516

TCP

SAP Landscape Host integration service

TCP TCP TCP TCP TCP TCP TCP TCP TCP TCP TCP

SAP Enqueue server port SAP Gateway port /Enqueue Replication port Central Instance Message server port Message server Message server HTTP port SAP HTTP port SAP P4 port SAP IIOP SAP Telnet Start SAP service Enqueue replication port

* SAP P4 = Provider servcie used for communication Java <-> <-> Java * SAP IIOP = Internet Inter-ORM Protocol used for communication CORBA <–> Java / Java <-> Java

ds8000 ISCSI Target server

Disk config Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk

/dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: /dev/sdf: /dev/sdg: /dev/sdh: /dev/sdk: /dev/sdl: /dev/sdm:

10.7 10.7 10.7 10.7 10.7 10.7 10.7 10.7 10.7 10.7

GB GB GB GB GB GB GB GB GB GB

Physical Volume [root@ds8000 ~]# PV /dev/sdd PV /dev/sde PV /dev/sdf PV /dev/sdg PV /dev/sdh PV /dev/sdb PV /dev/sdc PV /dev/sdk PV /dev/sdl PV /dev/sdm PV /dev/sda2

pvscan VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3 VG vg_target1_ext3 vg_target1_e xt3 VG vg_target0_ext3 vg_target0_e xt3 VG vg_target0_ext3 vg_target0_e xt3 VG vg_target0_ext3 vg_target0_e xt3 VG vg_target0_ext3 vg_target0_e xt3 VG vg_target0_ext3 vg_target0_e xt3 VG VolGroup00

lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2

[10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 5.00 GB free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [5.88 GB / 0 free]

Volume Groups [root@ds8000 ~]# vgscan


Page 52 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Reading all physical volumes. This may take a while... Found volume group "vg_target1_ext3" using metadata type lvm2 Found volume group "vg_target0_ext3" using metadata type lvm2 Found volume group "VolGroup00" using metadata type lvm2

Logical Volumes [root@ds8000 ~]# lvscan ACTIVE '/dev/vg_target1_ext3/lv_target1_ext3' [44.98 GB] inherit ACTIVE '/dev/vg_target0_ext3/lv_target0_ext3' [49.98 GB] inherit ACTIVE '/dev/VolGroup00/LvRoot' [2.78 GB] inherit ACTIVE '/dev/VolGroup00/LvLogs' [288.00 MB] inherit ACTIVE '/dev/VolGroup00/LvHome' [32.00 MB] inherit ACTIVE '/dev/VolGroup00/LvSwap' [2.78 GB] inherit

ISCSI Target config [root@ds8000 total 40G drwx------ 2 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1

target0]# ls -lh root root root root root root root root root root root

root root root root root root root

16K 40M 8.3G 8.3G 3.9G 16G 3.9G

Mar Sep Sep Sep Sep Sep Sep Sep Sep Sep Sep

[root@ds8000 target1]# ls -lh total 13G drwx------ 2 root root 16K Mar -rw-r--r-- 1 root root 40M Sep -rw-r--r-- 1 root root root 6.0G Sep Sep -rw-r--r-- 1 root root root 6.0G Sep Sep

9 9 9 9 9 9 9

2011 07:44 07:44 07:44 07:35 07:35 07:35 07:35 07:44 07:39 07:39

lost+found LUN1 LUN2 LUN3 LUN4 LUN5 LUN6

9 2011 lost+found 9 07:44 LUN1 9 07:44 07:44 LUN2 9 07:44 07:44 LUN3

[root@ds8000 target1]# mount /dev/mapper/VolGroup00-LvRoot on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/VolGroup00-LvLogs on /var/log type ext3 (rw) /dev/mapper/VolGroup00-LvHome on /home type ext3 (rw) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/vg_target0_ext3-lv_target0_ext3 on /opt/iscsi/target0 type ext3 (rw) /dev/mapper/vg_target1_ext3-lv_target1_ext3 on /opt/iscsi/target1 type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

Network configuration: eth0 ip address netmask gateway

: : :

10.47.253.4 10.47.253. 4 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.4 255.255.255.0

eth1

Firewall [root@ds8000 [root@ds8000 ~]# iptables-save > /tmp/iptables.txt /tmp/iptables.txt


Page 53 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ [root@ds8000 [root@ds8000 ~]# cat /tmp/iptables.txt /tmp/iptables.txt # Generated by iptables-save v1.3.5 on Fri Sep 16 00:29:05 2011 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [7954121:5968571088] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3260 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Fri Sep 16 00:29:05 2011


Page 54 of 202


Required Java version SAP Note 1172419: Linux Supported Java versions on the x86_64 platform The file you are looking for is IBMJava2-AMD64-142-SDK-1.4.2-13.6.x86_64.rpm

Hostname restrictions According to SAP the hostname of any server in a SAP landscape is restricted to a hostname of 8 characters, and on UNIX/Linux platforms this check is often done by the hostname command. Therefore the hostname command must not return the fully qualified hostname (FQDN), but instead only return the short name of the server. On Linux this is done by only entering the short name of the server in the configuration file /etc/sysconfig/network with the parameter HOSTNAME Content of file /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=poradg00 GATEWAY=10.47.253.1

in addition the /etc/hosts file must be ordered ip-adress short name FQDN Content of file /etc/hosts # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost localhost.localdomain ::1 localhost6 localhost6.localdomain6 localhost6.l ocaldomain6 10.47.253.80 10.47.253.8 0 poradg00 poradg00.slynga49.net poradg00.sly nga49.net 10.47.253.81 10.47.253.8 1 poradg01 poradg01.slynga49.net poradg01.sly nga49.net 10.47.253.82 10.47.253.8 2 dporadg02 poradg02.slynga49.net poradg02.sly nga49.net

SAP Note 1567511 Oracle Linux SAP Note 1048303 Red Hat Enterprise Linux 5.x: Installation and Upgrade

SAPLocales Since we are setting up UNICODE system only you can ignore the additional SAPLocales installation (According to SAP Note 171356)

User limits SAP Note 722273 Red Hat Enterprise Linux 3.x, 4.x: Installation and Upgrade


Page 55 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ /etc/security/limits.conf @sapsys @sapsys @dba @dba

hard soft hard soft

nofile nofile nofile nofile

32800 32800 32800 32800

The @ denotes that this setting is for a group. In addition you should set oraxx1 oraxx1 oraxx1 oraxx1

soft hard soft hard

nproc nproc nofile nofile

2047 16384 4096 63536

SAP Netweaver 7.0 including EHP2 ABAP+Java High Available Solution This solution will simulate a multi-datacenter configuration over two datacenters. DataCenter 1 (oslo1) and DataCenter 2 (oslo2) will be two separate VMware servers with shared storage. In a production environment the shared storage will usually be provided by your friendly HW teams, but since my test environment is not exactly enterprise grade I will be using a ISCSI server sharing ISCSI disks between the two datacenters to be able to establish the HA clusters. It is possible in Linux to set up a HA cluster using Distributed Replicated Block Device (DRBD / GlusterFS) to avoid having the ISCSI server as a single point of failure, but this is outside of the scope of what I wanted to test. Therefore I have accepted that my ISCSI server is single-point-of-failure. Instead I will close my eyes and imagine that there is a HighAvailability Disk solution in place.

VMware Shared disk If you are going to set up a cluster on one single VMware server you need only to manually modify the *.vmx file on all servers that will share the disk. scsi1.present scsi1.present = "TRUE" scsi1.sharedBus scsi1.sharedBus = "virtual" scsi1.virtualDev = "lsilogic" disk.locking="false" scsi1:0.present scsi1:0.present = "TRUE" scsi1:0.fileName = "/opt/vm/shared/QOURUM_55_x86_64. vmdk"


Page 56 of 202


Preparation In the initial installation I am going to perform in this document both SAP 7.0 and Oracle 10 have some requirements to the OS on which the installation is possible. The OS which I am using Red Hat Enterprise Linux Server 5.3 is not supported. Therefore I am going to be like David Copperfield, and create an illusion for the installation programs. The illusion I am making is to make the installation programs think the OS is Red Hat Enterprise Linux 4. This can be done by modifying a single file (/etc/redhat-release) For Red Hat 5.3 the contents of this file is Red Hat Enterprise Linux Server release 5.3 (Tikanga) We are going to change the contents of this file to be consistent for Red Hat Enterprise Linux 4 Red Hat Enterprise Linux Server ES release 4 (Nathant Update 6)

Linux Cluster One of the most heard arguments used against setting up a HA cluster is that by using a cluster you are introducing a complexity to your configuration which in it self result in a more unstable environment. This argument is true. A clustered environment is more complex than a stand-alone system, and as a more complex environment it demands more knowledge, planning and structure by the system administrators to make it a stable environment. You can easily create a unstable environment by introducing clustering to your configuration if you disregard the new demands this configuration has to your hardware and to your administration teams knowledge. When setting up a Linux cluster like the ones I describe in this document you can include up to 250 nodes in the cluster. So for the purpose set in this document I could easily have set up one single cluster with x number of nodes to handle the different services, but I wanted to reduce the services in the SAP cluster to make this cluster as fast as possible in a switchover/failover situation. Note: There are some issues with NFS in a cluster where you will get STALE file systems if you try to export from all nodes in the cluster by using the /etc/exports method instead of using the cluster service.


Page 57 of 202


pocnfsc cluster (NFS server cluster) This cluster consist of to nodes DataCenter1

DataCenter2 Vitrual Vitrual host

SAP WebDispatcher

SAP WebDispatcher

Load distribution distribution information ClusterNode1

Vitrual Vitrual host

ERS

ClusterNode2

ERS Active Active /Passive /Passive Cluster

(A)SCS /sapmnt /usr/sap/trans

ApplicationServer

ClusterNode1

(A)SCS


NFS

ApplicationServer

DG0

ClusterNode2 NFS

ApplicationServer

DG1 Oracle DataGuard

Active SAP applicationservers applicationservers included in logon groups

ApplicationServer

DG2

Offloading backup from production system

pnfscn01 pnfscn02

NOTE: For Oracle and SAP configurations we are required to use NFS version 3.Therefor we

need to disable NFS4 support on the servers. On both servers edit the file /etc/sysconfig/nfs Turn off nfs version 1 and 2. Turn on nfs version 3 MOUNTD_NFS_V1=”no” MOUNTD_NFS_V2=”no” MOUNTD_NFS_V3=”yes”

Turn off nfs version 4 RPCNFSDARGS=”-N RPCNFSDARGS=”-N 4”


Page 58 of 202


pnfscn01 Network configuration: eth0 ip address netmask gateway

: : :

10.47.253.51 10.47.253. 51 255.255.255.0 10.47.253.1

: :

172.20.16.51 172.20.16. 51 255.255.255.0

eth1 ip address netmask

Service configuration chkconfig chkconfig chkconfig chkconfig chkconfig chkconfig chkconfig chkconfig

--level --level --level --level --level --level --level --level

3 ricci on 3 luci on 3 rgmanager on 23 gfs on 3 clvmd on 3 iscsid on 3 cman on 3 nfs on

service ricci start service luci stop luci_admin init service luci restart service rgmanager start service cman start service clvmd start

Create the quorum disk mkqdisk -l QUORUM -c /dev/sdd

pnfscn02 Network configuration: eth0 ip address netmask gateway

: : :

10.47.253.52 10.47.253.5 2 255.255.255.0 10.47.253.1

: :

172.20.16.52 255.255.255.0

eth1 ip address netmask


Page 59 of 202


Cluster Resources pocnfscIP Address IP address Monitor link

172.20.16.100 172.20.16.1 00

Global file systems (GFS) Name sapmnt Mount point /sapmnt/XX1 Device /dev/vg_sapmnt_gfs/lv_sapmnt_gfs File system type GFS Options acl Filesystem ID Force unmount Reboot node if unmount fails Name saptrans Mount point /usr/sap/trans /usr/sap/trans Device /dev/vg_saptrans_gfs/lv_saptrans_gfs File system type GFS Options acl Filesystem ID Force unmount Reboot node if unmount fails Name oradiag Mount point /oradiag Device /dev/vg_oradiag_gfs/lv_oradiag_gfs File system type GFS Options acl Filesystem ID Force unmount Reboot node if unmount fails Name orafra Mount point /orafra Device /dev/vg_orafra_gfs/lv_orafra_gfs File system type GFS Options acl Filesystem ID Force unmount Reboot node if unmount fails

NFS Exports name name name name

nfs_sapmnt nfs_saptrans nfs_oradiag nfs_orafra

NFS Client Name Target Options Allow Recover

nfsc_172net 172.20.16.0/24 rw,no_root_squash 1


Page 60 of 202


pocsapcl cluster (SAP Central services Cluster) This cluster consist of to nodes DataCenter1

DataCenter2 Vitrual Vitrual host

SAP WebDispatcher

SAP W ebDispatcher

Load distributi distribution on information information ClusterNode1


ERS

ClusterNode2

ERS Active Active /Passive /Passive Cluster

(A)SCS /sapmnt /usr/sap/trans

ClusterNode1

(A)SCS


NFS

ApplicationServer

ClusterNode2 NFS

ApplicationServer

ApplicationServer

DG0

DG1 Oracle DataGuard

Active SAP applicationservers included in logon groups

ApplicationServer

DG2

Offloading backup from production system

psapcn01 psapcn02 Installed SAP Services Service ASCS SCS ERS ERS

SNO 00 10 50 60

psapcn01


: : :


10.47.253.201 10.47.253. 201 255.255.255.0 10.47.253.1 Page 61 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ eth1 ip address netmask

: :

172.20.16.201 172.20.16. 201 255.255.255.0

Service configuration chkconfig chkconfig chkconfig chkconfig chkconfig chkconfig chkconfig

--level --level --level --level --level --level --level

3 ricci on 3 luci on 3 rgmanager on 23 gfs on 3 clvmd on 3 iscsid on 3 cman on

service ricci start service luci stop luci_admin init service luci restart service rgmanager start service cman start service clvmd start

Create the quorum disk mkqdisk -l QUORUM -c /dev/sdd

psapcn02


: : :

10.47.253.202 10.47.253.2 02 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.202 172.20.16.202 255.255.255.0

eth1

Cluster resources posapcl IP Address IP address Monitor link

10.47.253.200

SAP Instance InstanceName StartProfile Executable Dir StartupWaittime

XX1_SCS10_pocsapcl /usr/sap/XX1/SYS/profile/START_SCS10_pocsapcl /sapmnt/XX1/exe /sapmnt/XX1 /exe 30


Page 62 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ InstanceName StartProfile Executable Dir StartupWaittime

XX1_ASCS00_pocsapcl /usr/sap/XX1/SYS/profile/START_ASCS00_pocsapcl /sapmnt/XX1/exe /sapmnt/XX1 /exe 30


Page 63 of 202


poradg00 Oracle Primary server Disk config Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk

/dev/sda: /dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: /dev/sdf: /dev/sdg: /dev/sdh: /dev/sdi: /dev/sdj:

6442 2147 2147 10.7 10.7 10.7 10.7 10.7 40.9 8192

MB MB MB GB GB GB GB GB GB MB

Physical Volume [root@poradg00 ~]# pvscan PV /dev/ORAINV VG vg_orainv_ext3 PV /dev/sdi VG vg_orabin_ext3 PV /dev/sdd VG vg_oradata_ext3 PV /dev/sde VG vg_oradata_ext3 PV /dev/sdf VG vg_oradata_ext3 PV /dev/sdg VG vg_oradata_ext3 PV /dev/sdh VG vg_oradata_ext3 PV /dev/sda2 VG VolGroup00

lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2 lvm2

[7.63 GB / 784.00 MB free] [38.14 GB / 3.82 GB free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 0 free] [10.00 GB / 5.00 GB free] [5.88 GB / 0 free]

Volume Groups [root@poradg00 ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "vg_orainv_ext3" using metadata type lvm2 Found volume group "vg_orabin_ext3" using metadata type lvm2 Found volume group "vg_oradata_ext3" using metadata type lvm2 Found volume group "VolGroup00" using metadata type lvm2

Logical Volumes [root@poradg00 ~]# lvscan ACTIVE '/dev/vg_orainv_ext3/lv_orainv_ext3' [6.86 GB] inherit ACTIVE '/dev/vg_orabin_ext3/lv_orabin_ext3' [34.33 GB] inherit ACTIVE '/dev/vg_oradata_ext3/lv_oradata_ext3' [44.98 GB] inherit ACTIVE '/dev/VolGroup00/LvRoot' [4.81 GB] inherit ACTIVE '/dev/VolGroup00/LvHome' [128.00 MB] inherit ACTIVE '/dev/VolGroup00/LvLogs' [448.00 MB] inherit ACTIVE '/dev/VolGroup00/LvSwap' [512.00 MB] inherit


: : :

10.47.253.80 10.47.253.8 0 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.80 255.255.255.0

eth1


Page 64 of 202


Firewall [root@poradg00 ~]# iptables-save > /tmp/iptables.txt [root@poradg00 ~]# cat /tmp/iptables.txt # Generated by iptables-save v1.3.5 on Fri Sep 16 00:40:41 2011 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [7317243:3668775226] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 1527 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Fri Sep 16 00:40:41 2011

Mount [root@poradg00 ~]# mount /dev/mapper/VolGroup00-LvRoot on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/VolGroup00-LvHome on /home type ext3 (rw) /dev/mapper/VolGroup00-LvLogs on /var/log type ext3 (rw) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/vg_oradata_ext3-lv_oradata_ext3 /dev/mapper/vg_oradata_ext3-lv_oradat a_ext3 on /oradata type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 172.20.16.100:/sapmnt/XX1 on /sapmnt/XX1 type nfs (rw,addr=172.20.16.100) 172.20.16.100:/orafra on /orafra type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=1600,addr=172.20.16.100) /dev/mapper/vg_orabin_ext3-lv_orabin_ext3 /dev/mapper/vg_orabin_ext3-lv_orabin_ ext3 on /orabin type ext3 (rw) /dev/mapper/vg_orainv_ext3-lv_orainv_ext3 /dev/mapper/vg_orainv_ext3-lv_orainv_ ext3 on /orainv type ext3 (rw) 10.47.253.5:/backup on /mnt/NAS type nfs (rw,addr=10.47.253.5) 172.20.16.100:/oradiag on /oradiag type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=1600,addr=172.20.16.100)

poradg01 Oracle Standby server Disk config Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk

/dev/sda: /dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: /dev/sdf: /dev/sdg: /dev/sdh: /dev/sdi: /dev/sdj:

6442 2147 2147 10.7 10.7 10.7 10.7 10.7 40.9 8192

MB MB MB GB GB GB GB GB GB MB

Physical Volume [root@poradg01 ~]# pvscan PV PV PV PV PV

/dev/sdj /dev/sdf /dev/sdg /dev/sdh /dev/sdi

VG VG VG VG VG

vg_oraInv_ext3 vg_oraInv_ext3 vg_oradata_ext3 vg_oradata_e xt3 vg_oradata_ext3 vg_oradata_e xt3 vg_oradata_ext3 vg_oradata_e xt3 vg_oradata_ext3 vg_oradata_e xt3


lvm2 lvm2 lvm2 lvm2 lvm2

[3.00 GB / 0 [10.00 GB / 0 [10.00 GB / 0 [10.00 GB / 0 [10.00 GB / 0

free] free] free] free] free]

Page 65 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ PV /dev/sdd VG vg_orabin_ext3 vg_orabin_ext3 lvm2 [8.00 GB / 0 PV /dev/sde VG vg_orabin_ext3 vg_orabin_ext3 lvm2 [8.00 GB / 0 PV /dev/sda2 VG VolGroup00 lvm2 [5.88 GB / 0 Total: 8 [64.85 GB] / in use: 8 [64.85 GB] / in no VG: 0

free] free] free] [0 ]

Volume Groups [root@poradg01 ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "vg_oraInv_ext3" using metadata type lvm2 Found volume group "vg_oradata_ext3" using metadata type lvm2 Found volume group "vg_orabin_ext3" using metadata type lvm2 Found volume group "VolGroup00" using metadata type lvm2

Logical Volumes [root@poradg01 ~]# lvscan ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE

'/dev/vg_oraInv_ext3/lv_oraInv_ext3' [3.00 GB] inherit '/dev/vg_oradata_ext3/lv_oradata_ext3' [39.98 GB] inherit '/dev/vg_orabin_ext3/lv_orabin_ext3' [15.99 GB] inherit '/dev/VolGroup00/LvRoot' [4.72 GB] inherit '/dev/VolGroup00/LvLogs' [576.00 MB] inherit '/dev/VolGroup00/LvHome' [96.00 MB] inherit '/dev/VolGroup00/LvSwap' [512.00 MB] inherit


: : :

10.47.253.81 10.47.253.8 1 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.81 255.255.255.0

eth1

Firewall [root@poradg01 ~]# iptables-save > /tmp/iptables.txt [root@poradg01 ~]# cat /tmp/iptables.txt # Generated by iptables-save v1.3.5 on Fri Sep 16 00:50:32 2011 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [7317243:3668775226] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 1527 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Fri Sep 16 00:50:32 2011


Page 66 of 202


Mount [root@poradg01 ~]# mount /dev/mapper/VolGroup00-LvRoot on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/VolGroup00-LvLogs on /var/log type ext3 (rw) /dev/mapper/VolGroup00-LvHome on /home type ext3 (rw) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/vg_orabin_ext3-lv_orabin_ext3 /dev/mapper/vg_orabin_ext3-lv_orabin_ ext3 on /orabin type ext3 (rw) /dev/mapper/vg_oradata_ext3-lv_oradata_ext3 /dev/mapper/vg_oradata_ext3-lv_oradat a_ext3 on /oradata type ext3 (rw) /dev/mapper/vg_oraInv_ext3-lv_oraInv_ext3 /dev/mapper/vg_oraInv_ext3-lv_oraInv_ ext3 on /oraInv type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 172.20.16.100:/sapmnt/XX1 on /sapmnt/XX1 type nfs (rw,addr=172.20.16.100) 172.20.16.100:/orafra on /orafra type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=1600,addr=172.20.16.100) 172.20.16.100:/oradiag on /oradiag type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=1600,addr=172.20.16.100)

psapap01 (SAP Central Instance ) Installed SAP Services Service ABAP DAA

SNO 00 97

Disk config Disk Disk Disk Disk Disk Disk Disk

/dev/sda: /dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: /dev/sdf: /dev/sdg:

6442 2147 2147 8589 4294 4294 4294

MB MB MB MB MB MB MB

Physical Volume [root@psapap01 ~]# pvscan PV /dev/sdg VG vg_sapsmd_ext3 vg_sapsmd_ext3 lvm2 [4.00 GB / 0 free] PV /dev/sdf VG vg_saphost_ext3 vg_saphost_ext3 lvm2 [4.00 GB / 0 free] PV /dev/sde VG vg_sapdaa_ext3 vg_sapdaa_ext3 lvm2 [4.00 GB / 0 free] PV /dev/sdd VG vg_sapsid_ext3 vg_sapsid_ext3 lvm2 [8.00 GB / 820.00 MB free] PV /dev/sda2 VG VolGroup00 lvm2 [5.88 GB / 0 free] Total: 5 [25.86 GB] / in use: 5 [25.86 GB] / in no VG: 0 [0 ]

Volume Groups [root@psapap01 ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "vg_sapsmd_ext3" using metadata type lvm2 Found volume group "vg_saphost_ext3" using metadata type lvm2 Found volume group "vg_sapdaa_ext3" using metadata type lvm2 Found volume group "vg_sapsid_ext3" using metadata type lvm2 Found volume group "VolGroup00" using metadata type lvm2

Logical Volumes ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE

'/dev/vg_sapsmd_ext3/lv_sapsmd '/dev/vg_sapsm d_ext3/lv_sapsmd _ext3' [4.00 GB] inherit '/dev/vg_saphost_ext3/lv_sapho '/dev/vg_sapho st_ext3/lv_sapho st_ext3' [4.00 GB] inherit '/dev/vg_sapdaa_ext3/lv_sapdaa '/dev/vg_sapda a_ext3/lv_sapdaa _ext3' [4.00 GB] inherit '/dev/vg_sapsid_ext3/lv_sapsid '/dev/vg_sapsi d_ext3/lv_sapsid _ext3' [7.20 GB] inherit '/dev/VolGroup00/LvRoot' '/dev/VolGroup 00/LvRoot' [2.81 GB] inherit


Page 67 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ ACTIVE ACTIVE ACTIVE

'/dev/VolGroup00/LvLogs' '/dev/VolGroup 00/LvLogs' [288.00 MB] inherit '/dev/VolGroup00/LvHome' '/dev/VolGroup 00/LvHome' [32.00 MB] inherit '/dev/VolGroup00/LvSwap' '/dev/VolGroup 00/LvSwap' [2.75 GB] inherit


: : :

10.47.253.211 10.47.253.2 11 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.211 172.20.16.211 255.255.255.0

eth1

Firewall [root@psapap01 XX1]# cat /tmp/iptables_20110914.txt # Generated by iptables-save v1.3.5 on Wed Sep 14 14:19:23 2011 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [290644:53123525] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8000 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3200 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 40000 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 40001 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 40002 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 3300 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 64999 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 65000 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 1128 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 41805 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 111 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 40080 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 38578 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 50013 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 59704 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Wed Sep 14 14:19:23 2011

Mount [root@psapap01 ~]# mount /dev/mapper/VolGroup00-LvRoot on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/VolGroup00-LvLogs on /var/log type ext3 (rw) /dev/mapper/VolGroup00-LvHome on /home type ext3 (rw) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/vg_sapsid_ext3-lv_sapsid_ext3 on /usr/sap/XX1 type ext3 (rw) /dev/mapper/vg_sapdaa_ext3-lv_sapdaa_ext3 on /usr/sap/DAA type ext3 (rw) /dev/mapper/vg_sapsmd_ext3-lv_sapsmd_ext3 on /usr/sap/SMD type ext3 (rw) /dev/mapper/vg_saphost_ext3-lv_saphost_ext3 on /usr/sap/hostctrl type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 172.20.16.100:/sapmnt/XX1 on /sapmnt/XX1 type nfs (rw,addr=172.20.16.100) 172.20.16.100:/orafra on /orafra type nfs (rw,addr=172.20.16.100)


Page 68 of 202


psapap02 (SAP Dialog instance) Installed SAP Services Service ABAP DAA

SNO 00 97

Disk config Disk Disk Disk Disk Disk

/dev/sda: /dev/sdb: /dev/sdc: /dev/sdd: /dev/sde:

6442 2147 2147 8589 5368

MB MB MB MB MB

Physical Volume [root@psapap02 ~]# pvscan PV /dev/sde VG vg_sapdaa_ext3 vg_sapdaa_ex t3 lvm2 [5.00 GB / PV /dev/sdd VG vg_sapsid_ext3 vg_sapsid_ex t3 lvm2 [8.00 GB / PV /dev/sda2 VG VolGroup00 lvm2 [5.88 GB / Total: 3 [18.87 GB] / in use: 3 [18.87 GB] / in no

512.00 MB free] 820.00 MB free] 0 free] VG: 0 [0 ]

Volume Groups [root@psapap02 ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "vg_sapdaa_ext3" using metadata type lvm2 Found volume group "vg_sapsid_ext3" using metadata type lvm2 Found volume group "VolGroup00" using metadata type lvm2

Logical Volumes [root@psapap02 ~]# lvscan ACTIVE inherit ACTIVE inherit ACTIVE ACTIVE ACTIVE ACTIVE

'/dev/vg_sapdaa_ext3/lv_ '/dev/vg_sap daa_ext3/lv_sapdaa_ext3' sapdaa_ext3' [4.50 GB] '/dev/vg_sapsid_ext3/lv_ '/dev/vg_sap sid_ext3/lv_sapsid_ext3' sapsid_ext3' [7.20 GB] '/dev/VolGroup00/LvRoot' '/dev/VolGro up00/LvRoot' '/dev/VolGroup00/LvLogs' '/dev/VolGro up00/LvLogs' '/dev/VolGroup00/LvHome '/dev/VolGroup00/LvHome' ' '/dev/VolGroup00/LvSwap' '/dev/VolGro up00/LvSwap'

[2.75 GB] inherit [288.00 MB] inherit [32.00 MB] inherit [2.81 GB] inherit


: : :

10.47.253.212 10.47.253.2 12 255.255.255.0 10.47.253.1

ip address netmask

: :

172.20.16.212 172.20.16.212 255.255.255.0

eth1

Firewall [root@psapap02 tmp]# service iptables status


Page 69 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Table: filter Chain INPUT (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all --

0.0.0.0/0


Chain FORWARD (policy ACCEPT) num target prot opt source 1 RH-Firewall-1-INPUT all --

0.0.0.0/0


Chain OUTPUT (policy ACCEPT) num target prot opt source

destination

Chain RH-Firewall-1-INPUT (2 references) num target prot opt source 1 ACCEPT all -- 0.0.0.0/0 2 ACCEPT icmp -- 0.0.0.0/0 3 ACCEPT esp -- 0.0.0.0/0 4 ACCEPT ah -- 0.0.0.0/0 5 ACCEPT udp -- 0.0.0.0/0 6 ACCEPT udp -- 0.0.0.0/0 7 ACCEPT tcp -- 0.0.0.0/0 8 ACCEPT all -- 0.0.0.0/0 9 ACCEPT tcp -- 0.0.0.0/0 10 ACCEPT tcp -- 0.0.0.0/0 11 ACCEPT tcp -- 0.0.0.0/0 12 ACCEPT tcp -- 0.0.0.0/0 13 ACCEPT tcp -- 0.0.0.0/0 14 ACCEPT tcp -- 0.0.0.0/0 15 ACCEPT tcp -- 0.0.0.0/0 16 ACCEPT tcp -- 0.0.0.0/0 17 ACCEPT tcp -- 0.0.0.0/0 18 ACCEPT tcp -- 0.0.0.0/0 19 ACCEPT tcp -- 0.0.0.0/0 20 REJECT all -- 0.0.0.0/0

destination 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 224.0.0.251 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0 0.0.0.0/0

icmp type 255

udp dpt:5353 udp dpt:631 tcp dpt:631 state RELATED,ESTABLISHED state NEW tcp dpt:22 state NEW tcp dpt:8000 state NEW tcp dpt:3200 state NEW tcp dpt:40000 state NEW tcp dpt:59713 state NEW tcp dpt:3300 state NEW tcp dpt:64999 state NEW tcp dpt:65000 state NEW tcp dpt:1128 state NEW tcp dpt:50013 state NEW tcp dpt:59704 reject-with icmp-host-prohibited

Mount [root@psapap02 ~]# mount /dev/mapper/VolGroup00-LvRoot on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/VolGroup00-LvLogs on /var/log type ext3 (rw) /dev/mapper/VolGroup00-LvHome on /home type ext3 (rw) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/vg_sapsid_ext3-lv_sapsid_ext3 on /usr/sap/XX1 type ext3 (rw) /dev/mapper/vg_sapdaa_ext3-lv_sapdaa_ext3 on /usr/sap/DAA type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 172.20.16.100:/sapmnt/XX1 on /sapmnt/XX1 type nfs (rw,addr=172.20.16.100)


Page 70 of 202


ORACLE 30 years and still only a single user system. Whatever do I mean by a statement like that? Well how many books and documents have you read where the installation user is “oracle”? An installation like that is ok if there is only one database on this server. But what if there is more than one database? In my day-job I am responsible for several database installations. In fact a three digit number of installations. Should we use the same “oracle” user? Should we use the same binary installation? What if the upgrade schedule is different? So now we need a different ORACLE_HOME. This is feasible enough through the .dbenv script. But I don’t want the “other” administrator to be able to log in to my database. Answer: use a different server.

What if we adopt SAP’s method? We have a different oracle user owning the different oracle binary installations. This works very well. We have different security levels and different access rules allowing only the administrators that are meant to administer this specific database. But what about the oraInventory. We cannot use the common oraInventory because there is more than one user. Ok, so let’s have a separate oraInventory for each user. No problem there. But what if we want to connect the Enterprise Manager? With the Enterprise Manager you install an agent on the server (host), but this agent will only be able to see one oraInventory. The other oracle installations will have an error state in the Enterprise Manager because the oraInventory is not accessible. Answer: use a different server.

And how about starting and stopping the database. There we have the /etc/oratab owned by root, so this should be ok. but it is not. If you try to shutdown the database with the root user you will need to set all sorts of environment variables and also give the right group accesses. But worse is that if you try and actually succeed to start your database with the root user. Your database will now run with the access levels of root, and you really don’t want that. So what if we just start dbshut with the first ora user on the server. This would then read the /etc/oratab file and traverse through the file and shut down all our databases, right? Wrong. If there is more than one database in your oratab you will be able to shutdown the databases that you own, but you will receive a permission denied for all the other databases. To this there are some easy solutions: Modify the dbshut and dbstart scripts for your solutions to adopt a separate oratab per installation. Around line 75 you will find the lines ORATAB=/etc/oratab if [ ! $ORATAB ] ; then echo "$ORATAB not found" exit 1; fi


Page 71 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Change that into (for instance) ORATAB=/orabin/XX1/oratab if [ ! $ORATAB ] ; then echo "$ORATAB not found" exit 1; fi

(My favourite) Use a common dbstart/dbshut script (they are compatible back to oracle 6), and modify a few lines Around line 75 you will find the lines ORATAB=/etc/oratab if [ ! $ORATAB ] ; then echo "$ORATAB not found" exit 1; fi Change that into ORATAB=/etc/oratab if [ ! $ORATAB ] ; then echo "$ORATAB not found" exit 1; else grep -e $ORACLE_SID $ORATAB > /tmp/oratab ORATAB=/tmp/oratab fi

Then you need to set up a script to be excuted at you server startup #!/bin/sh ################################################# # Starting all databases on this server # Starting all LISTENER processes on this server # PREREC: Oracle user is named ora ################################################# # # set -x ORATAB_FILE=/etc/oratab GLOBAL_DIR=/global/ora_scripts echo "This is the ORATAB FILE :" $ORATAB_FILE for d in `cat ${ORATAB_FILE} |grep -v "^#"|grep ":Y$"` do col1=`grep -i "${d}" $ORATAB_FILE |awk -F ':' '{print $1}'` col2=`grep -i "${d}" $ORATAB_FILE |awk -F ':' '{print $2}'` LOWSID=ècho $col1 |awk '{print tolower($1)}'` if [ -f /tmp/oratab ]; then rm -rf /tmp/oratab fi ORACLE_USER=ora${LOWSID} print ">>>>> Starting $col1 <<<<<" ORAPARM=`grep -e $d /etc/oratab` col2=ècho ${ORAPARM} | awk -F ':' '{print $2}'` export ORACLE_HOME=${col2} echo $ORACLE_HOME su - ${ORACLE_USER} -c "$GLOBAL_DIR/bin/dbstart_112 $ORACLE_HOME" rm -rf /tmp/oratab fi done print "########################################################"


Page 72 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ print print print print print

"#" "#" "# SCRIPT COMPLETE" "#" "########################################################"

You need a similar script for the shutdown #!/bin/sh ################################################# # Stoping all databases on this server # Stoping al LISTENER’s on this server ################################################# # set -x ORATAB_FILE=/etc/oratab GLOBAL_DIR=/global/ora_scripts

echo "This is the ORATAB FILE :" $ORATAB_FILE for d in `cat ${ORATAB_FILE} ${ORATAB_FILE} |grep -v "^#"|grep ":Y$"` do col1=`grep -i "${d}" $ORATAB_FILE |awk -F ':' '{print $1}'` col2=`grep -i "${d}" $ORATAB_FILE |awk -F ':' '{print $2}'` LOWSID=ècho $col1 |awk '{print tolower($1)}'` if [ -f /tmp/oratab ]; then rm -rf /tmp/oratab fi ORACLE_USER=ora${LOWSID} print ">>>>>>>> Stoping $col1 <<<<<<<" ORAPARM=`grep -e $d /etc/oratab` col2=ècho ${ORAPARM} ${ORAPARM} | awk -F ':' '{print $2}'` export ORACLE_HOME=${col2} export ORACLE_SID=${col1} su - ${ORACLE_USER } -c "$GLOBAL_DIR/bin/dbshut_1 "$GLOBAL_DIR /bin/dbshut_1 12 $ORACLE_HOME" rm -rf /tmp/oratab fi done print "###################### "################################## ####################### ##############" ###" print "#" print "#" print "# SCRIPT COMPLETE" print "#" print "###################### "################################## ####################### ##############" ###"

Or, you could use a different server.

My point here is that the Oracle RDBMS database which I am very fond of does have some quirks and shortcomings, and to me it seems that the capacity of the hardware servers has either outgrown the administration tools or managment has put some restraints on development. In my point of view anyway oracle database as it is today is not fully compatible with big-iron, multi-user, multi installation hardware. It is still after 30 years a single user (single installation) system. In fact this ties right into the statements made by Larry Ellison in his keynotes at Oracle OpenWorld in 2010 where he predicted the death of the big-iron servers. It also ties right into the increased focus on pushing small size Oracle RAC servers.


Page 73 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ We are going to install five stand-alone servers which will act as the three physical nodes + two logical nodes in our Oracle DataGuard environment

Housekeeping if [ -f $ORACLE_HOME/bin/adrci ]; then adrci exec="set homepath $ADR_BASE;purge -age 60 -type TRACE" fi

RMAN (Recovery Manager) The Oracle RMAN utility is what you should be using for taking backup of your database if you are using the parameter combination db_recovery_file_dest (directory) and db_recovery_file_dest_size (quota). It is not a requirenment to use RMAN, but when using these parameters Oracle is keeping an internal control over db recovery files in the directories under the destination. If you are using a third-party backup solution, or even if you are using the old-type begin backup/end backup solution this is not communicating with the Oracle file management. So over time as you perform your archivelog backup and delete the archivelogs as you go along the filesize of the archivelogs are not retracted from the quota-usage and your database will grind to a halt because you have used all the space availiable in the quota even if there are no files in the destination directory. The RMAN is a very powerful yet simple utility. It has a limited command language which is geared for the task of taking backup, and provifding the ability to restore your database. a simple backup command for RMAN looks like: RMAN>backup database;

It cannot get much simpler than that. When you talk about RMAN there is the question about if you should use a RMAN Catalog or not. Wether or not you are using a RMAN Catalog or not does not have any impact on your ability to perform a backup. In its simplest form the RMAN Catalog is only a log repository keeping tabs on backups that has been run successfully. The data traffic from your database to the backup location does not go to or via the RMAN Catalog server. If you are not using an RMAN Catalog the logs from your backup is written to the controlfile of the database beeing backed up. The limitations you run into by not using a RMAN catalog is that by using only the controlfile you cannot get a central database for all the logs for all the databases you are taking backups of. There is however a more vital limitation, the timelimit of for records in the controlfile. The controlfile has a maximum keep time of 30 days. What this imposes as a limitation for your backup is that you are unable to restore any backup which is older than 30 days simply Written and tested by Kjell Erik Furnes [email protected]

Page 74 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ because there is no record of these backups in your controlfile. So if you need a backup retention of more than 30 days, you need to use a RMAN Catalog.

Creating a RMAN Catalog database: After you have created a suitable pfile/spfile #> sqlplus /nolog SQL>connect / as sysdba SQL>startup nomount SQL>CREATE DATABASE RCAT USER SYS IDENTIFIED BY USER SYSTEM IDENTIFIED BY MAXLOGFILES 16 MAXLOGMEMBERS 3 MAXDATAFILES 100 MAXINSTANCES 8 MAXLOGHISTORY 292 ARCHIVELOG CHARACTER SET UTF8 NATIONAL CHARACTER SET UTF8 UNDO TABLESPACE UNDOTBS DATAFILE AUTOEXTEND ON MAXSIZE 8G DEFAULT TABLESPACE USERTBS DATAFILE AUTOEXTEND ON MAXSIZE 8G DEFAULT TEMPORARY TABLESPACE TEMP TEMPFILE AUTOEXTEND ON MAXSIZE 8G;

SQL>CREATE TABLESPACE RMAN_CATALOG_PRD AUTOEXTEND ON MAXSIZE 8G; SQL>CREATE USER RMAN_OWNER_PRD DEFAULT TABLESPACE RMAN_CATALOG_PRD TEMPORARY TABLESPACE TEMP IDENTIFIED BY ;

#> orapwd file=orapwRCAT password= entries=10 SQL>grant recovery_catalog_owner to RMAN_OWNER_PRD

You will also need to maintain your listener and tnsnames files and have a running listener which you are able to reach from all the databases you are going to take backup of. Tip: On each of the databases you are going to take backups from. Create a user ex: “RMAN_USER_PRD”. Do not grant this user any other privileges except sysdba. This user will not be able to connect to the database (no CREATE SESSION Privilege), and cannot be used for anything except taking backups. #> rman RMAN> connect target rman_user_prd/@XX1_DG1 RMAN> connect catalog rman_owner_prd/@RCAT RMAN> create catalog; RMAN> register database;

That is it, you now have an RMAN Catalog database with one database registered in it. You are no able to take backups of your database (in this case XX1_DG1) and store the logs in the RMAN Catalog database. #> rman


Page 75 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ RMAN> connect target rman_user_prd/@XX1_DG1 RMAN> connect catalog rman_owner_prd/@RCAT

To keep the backup commands as a block you need to specify the commands within a run block run {

--- COLLECTION OF RMAN COMMANDS --}

Database Backup When taking a backup you are able to compress the backupset with the following command RMAN> backup as compressed backupset database;

This will create an entry in your catalog server with a name which is difficult to identify as to what the backup contains. Therefor you can specifuy a TAG for the backup. RMAN> backup as compressed backupset tag ‘DB_BACKUP_20110914’ ‘DB_BACKUP_20110914’ database;

Expanding this a little more you can also specify how long you want to retain the backup in your system RMAN> run { backup as compressed backupset tag ' DB_BACKUP_20110914' DB_BACKUP_20110914' keep until time 'sysdate +35' logs database; }

This command specifies that your backup has a retention of 35 days, and that any archivelogs needed to roll forward from this backup until current time withing these 35 days will not be tagged as obsolete.


Page 76 of 202


ARCHIVELOG backup To backup your archivelogs run { SQL 'alter system archive log current'; SQL 'alter system switch logfile'; backup as compressed backupset tag 'ARCHIVELOGS' 'ARCHIVELOGS' archivelog all delete input; }

This comamnd will take a backup of your archivelogs, and when the backup is complete will delete those archivelogs which has been backed up successfully. Exception to this is if there are other limitations set up ( do no delete until applied on standby / Do not delete until backed up with two generations)

Misc RMAN commands RMAN> RMAN> RMAN> RMAN> RMAN> RMAN> RMAN> RMAN> RMAN> RMAN>

list backup summary list backup completed after 'sysdate -2'; list backup of archivelog all completed after 'sysdate -2'; report obsolete; report obsolete device type disk; report obsolete device type tape; delete obsolete; delete noprompt obsolete; restore archivelog from logseq 79798 until logseq 80000; restore archivelog logseq 79540;

Relocating backup when using Oracle DataGuard. When thinking about how to relocate the backup by using Oracle DataGuard there is acctually only one concept that you must understand. “In a DataGuard with a Physical standby database the DBID of the Primary and the Standby database(s) is the same”

SQL> select dbid, name, db_unique_name, database_role from v$database; DBID NAME DB_UNIQUE_NAME DB_UNIQUE_NAME DATABASE_ROLE DATABASE_ROLE ---------- --------- ---------------------------------------------------------- ------------------------------4193169376 XX1 XX1_DG0 PRIMARY

SQL> select dbid, name, db_unique_name, database_role from v$database; DBID NAME DB_UNIQUE_NAME DATABASE_ROLE ---------- --------- ------------------------------ ---------------4193169376 XX1 XX1_DG1 PHYSICAL STANDBY


Page 77 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ What this means is that for the controlfiles on both databases each of the files on primary and standby databases have the same unique identification. The consequence of this is that you are now able to run any type of backup of your datafiles on your standby database, and use this backup to restore any part of your primary database. Your Primary database can now run 100% production without having any backup window or additional I/O load during the backup.

Oracle Maintained Files (OMF) and SAP Warning: Do not use OMF with SAP. Doing so will create errors in later upgrades and/or system copies and/or migrations when using SAP internal tools. The reason why you will get errors is that with OMF the files gets very specific names, and these names are registered in the control file as OMF names. When you later use R3Up or other tools for upgrading the database and SAP tries to import the dump files it will try to recreate the OMF files. You will then get an error ORA-01276 Cannot add file OMF error. Workaround for this error if you do choose to use OMF is to pre-create the different tablespaces containing OMF filenames.

Software installation installation NFS Mount options for Oracle filesystems on Linux X86-64 For Binaries

rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp For Oracle Data files

rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp,actimeo=0

See Metalink Doc ID: Note: 359515.1 for more details.

listener.ora 1. LISTENER = 2. (DESCRIPTION_LIST = 3. (DESCRIPTION = 4. (ADDRESS_LIST = 5. (ADDRESS = (PROTOCOL = tcp)(HOST = 172.20.16.80)(port = 1527)) 6. ) 7. ) 8. ) 9. 10. SID_LIST_LISTENER = 11. (SID_LIST = 12. (SID_DESC = 13. (SID_NAME = XX1_DG0)


Page 78 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ 14. 15. 16.

17.

(GLOBAL_DBNAME = XX1_DG0) (ORACLE_HOME = /orabin/XX1/102_64) ) )

If you are going to maintain several Oracle databases I would recommend that you have one central tnsnames.ora file where you maintain everything, and then an automated job that replicates the central file to a location local to each of your database servers. You should then use symbolic links from your default tnsnames.ora location to the local copy of your central tnsnames.ora


Page 79 of 202


SAP Installation This section will consist mostly of the screendumps from the installation process. I have done this because the document is meant to be a proof-of-concept document, and I needed to show that the SAP installation is quite easy, and more or less default. All the screndumps of all the steps in the installation process is included here. For those screens where the input data is default the screendump is included in small-size. The only section where I make some deliberate deviations from the SAP recommendations is the installation of the database instance where I prepare the system for a DataGuard configuration during the installation. If you are familiar with the SAP installation procedure you can skip all these screendumps, but you should take a look at the database installation sections.

SAP Central instance According to the SAP central instance in a HA environment is no longer a central unit. After setting up the HA environment the Central Instance services ((A) SCS) is now the SPOF’s. Along with a number of other statements from SAP this statement does not apply to a default installation of SAP HA. The Central Instance is now the Primary Instance meaning that this is the instance you install first of the “worker” instances, but as instances goes it is little different than the additional instances you install later. This comment is only partly true. In a default SAP HA installation you will have the central services ASCS and SCS. In addition you will have the Central Instance with the worker processes DIALOG, BATCH, UPDATE, UPDATE2, SPOOL, ICM and J2EE. On your Dialog Instances you will have DIALOG, BATCH, ICM and J2EE, but there will be neither UPDATE nor SPOOL processes defined in the Dialog Instance installation. You will need to define theses missing processes on your additional dialog instances. If you do not define these missing processes on your dialog instances then your system will grind to a halt if you loose your central instance because all your worker processes will be waiting for available update processes (which is only defined on your central instance, which is down) This is of course just a small task in the post installation configuration that you need to do after installing the SAP Dialog Instance, but it would be nice if SAP would not omit this information in the documentation. After doing the postinstallation configuration where you define the missing process types on your dialog instances there is in fact in a HA installation no significant difference between the Central Instance and the Dialog Instances. As long as you have one of these Dialog/Central instances up and running you are able to log and work on to your SAP system.


Page 80 of 202


Installing SAP ASCS

Screendump 1: Select SAP ASCS ASCS installation

Screendump 2: General Parameters

Screendump 3: Enter the SAP system ID

Screendump 4: Enter the master password

Screendump 5: Location of required software package for UC Kernel NW 7.20


Page 81 of 202


Screendump 6: Enter System number for ABAP ASCS

Screendump 7: Enter message server port numbers

Screendump 8: Enter path to SAP Cryptographic Library

Screendump 9: Choose cryptographic library to install

Screendump 10: Select archives to unpack


Page 82 of 202


Screendump 11: System installation parameter summary

Screendump 12: Installation progress

Screendump 13: ASCS Installation successful


Page 83 of 202


Installing SAP SCS

Screendump 14: Select SCS instance

Screendump 15: Enter SAP system ID

Screendump 16: Enter master password

Screendump 17: Enter location of software package UC Kernel NW 7.20


Page 84 of 202


Screendump 18: Filebrowser

Screendump 19: Enter SCS Instance number

Screendump 20: Enter SCS Message Messa ge Server port

Screendump 21: Installation parameter summary



Page 85 of 202


Screendump 23: Installation of SCS finished successfully


Page 86 of 202


Installing SAP ERS

Cluster node 1

Screendump 24: Select Enqueue Replication Server instance

Screendump 25: Enter profile directory

Screendump 26: Select Select Central service instance for ERS

Screendump 27: Enter location of required software package UC Kernel NW 7.20


Page 87 of 202


Screendump 28:File Browser

Screendump 29: Restart ASCS instance to reflect changes

Screendump 30: Enter the ERS system number for the SCS service

Screendump 31: Restart the ASCS to reflect the changes



Page 88 of 202


Screendump 33: Installation of ERS finished successfully

Cluster node 2 After completing the installation of the ERS instance on node 1 you need to switch over the cluster so that your SAP system is now running on the other node. Your cluster configuration will now only contain the IP adress

Screendump 34: Select Enqueue Replication server instance

Screendump 35: Enter path of the profile directory

Screendump 36: Enter central service instances for the ERS


Page 89 of 202


Screendump 37: Enter location of the required software packages UC kernel NW 7.20

Screendump 38: File browser

Screendump 39: Enter the system number of the ERS instance for ASCS

Screendump 40: Enter the system number of the ERS instance for SCS



Page 90 of 202



Screendump 43: Installation of ERS completed successfully

Modify Linux cluster configuration

Screendump 44: LInux cluster configuration for SAP Central services


Page 91 of 202


Installing SAP Database instance

Screendump 45: Seleect Database instance

Screendump 46: Enter location of required software packages Java Component NW 702

Screendump 47: File Browser

Screendump 48: Enter JDk directory path


Page 92 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Screendump 49: File Browser

Screendump 50: Enter profile directory

Screendump 51: Enter master password

Screendump 52: Enter Database ID

Screendump 53: Enter location of required software packages UC Kernel NW 7.20


Page 93 of 202



Screendump 55: Enter location of requires softwar packages Installation Export NW 702


Screendump 57: Enter parameters and password for the Database system


Page 94 of 202


Screendump 58: Enter database version and parameters for the installation

This is the first time during the entire installation where I am starting to make some changes. I am marking the entry Advanced DB Configuration. Note. At the time of testing no SAP released database installation CD’s for Oracle 11.2 exist event though there is a selection dropdown where you can select Oracle version 11.2. Therefor I continue with the installation of Oracle 10.2.

Screendump 59: Enter passwords for the standard oracle database users


Page 95 of 202


Screendump 60: Enter listener name and port

Screendump 61: Enter the inputscreens for database configuration changes


Page 96 of 202


Screendump 62: Enter the database installation directories

Here I have changed the directory of the $ORACLE_HOME and the SAPDATA home directories. I have done this to prepare the directory structure for a Oracle DataGuard which I will set up after the SAP installation is complete.

Screendump 63: Enter database instance file systems


Page 97 of 202

Proof of concept documentation for SAP High Available system with Oracle DataGuard and TAF on Linux Cluster ___________________________________________________________________________ Here I have removed all other filesystems besides SAPDATA1. Note aso that I have directory structure is prepared for the Oracle DataGuard configuration, and I have also adopted the filestructure of the Oracle Managed Files (OMF) with the /datafile directory structure. This is not neccessary to make the Oracle DataGuard work, but I am now implementing a standard which I can use on database systems besides the SAP system.

Screendump 64: Enter database instance specific information


Page 98 of 202


Screendump 65. Set all SAP directory aliases to SAPDATA1

Screendump 66: All SAP directory aliases set to SAPDATA1


Page 99 of 202


Screendump 67: Enter the location of the controlfiles

Here I have again adopted the OMF directory structure by putting the one instance of the controlfile under //controlfile and another instance of the controlfile under //controlfile. The prefix1 directory is the filesystem under which the datafiles are created. in a OMF configuration this prefix is named in the parameter db_create_file_dest. The prefix2 is the filesystem under which the flash recovery is located if you set this up by the parameter db_recovery_file_dest

Screendump 68: Enter the tablespace storage parameters


Page 100 of 202


Screendump 69: Enter the number of parallel import jobs for the ABAP dumps

Screendump 70: Enter the key phrase for the Java SecureStore

Screendump 71: Specify if you want the database statistics to be generated during the installation prosess.

Since this is only a test system I skip the generation of the database statistics.


Page 101 of 202


Screendump 72: Enter location fo the required software package, Oracle Client


Screendump 74: Unpack Archives

Unfortunaltely someone at SAP have desided to hardcode the directory structure used by the client (/oracle). This is not according to my design where I want all oracle files installed under /orabin. Therefor I create a symbolic link on /orabin to /oracle ln –s /orabin /oracle


Page 102 of 202


Screendump 75: Parameter summary


Screendump 77: SAPinst halt installation for database installation


Page 103 of 202


SAP Oracle database installation

Screendump 78: Specify Inventory and credentials

Screendump 79: Available Product Components

Screendump 80: Product – Specific Prerequisite Checks

Screendump 81: Warning Failed prerequisite checks

Click the Yes button to continue.


Page 104 of 202


Screendump 82: Installation summary


Screendump 84: Execute Configuration Scripts

Screendump 85: End of Installation

Screendump 86: Exit confirmation

Oracle Patching 10.2.0.1 -> 10.2.0.4.0

Screendump 87: Welcome screen


Page 105 of 202


Screendump 88: Specify Home Details

Screendump 89: Product-Specific Prerequisite Checks

Screendump 90: Oracle Configuration Manager Registration

Screendump 91: Installation summary


Screendump 93: Execute Configuration Scripts


Page 106 of 202


Screendump 94: End of installation

Screendump 95: Exit confirmation

Continue with the SAP Database Instance Installation

Screendump 96: SAPinst resume installation

Screendump 97: SAP Database Instance installation complete


Page 107 of 202


Installing SAP Central Instance (Primary Application Server)

Screendump 98: Select Central Instance

Screendump 99: Enter location of required software packages, Java Component NW 702

Screendump 100: Filebrowser

Screendump 101: Enter path to the JDK directory


Page 108 of 202



Screendump 103: Enter the Profile directory

Screendump 104: Error in prerequisite checks

The prereuisite for the central installation is about 5GB of RAM. Since my test system is resource limited I am accepting that I don’t have the recomended memory.


Page 109 of 202


Screendump 105: SAPinst prerequisite failed

Press [Cancel] to continue with the installation and bypass the failed checks.

Screendump 106: Enter listener configuration

Screendump 107: Enter the Central Instance system number


Page 110 of 202


Screendump 108: Enter the names of the java users

If you have several SAP Java systems in your environment you should change the name of the Administrator and Guest Java users to reflect the SID of your system. Otherwise you can easily run into problems where several systems are running with the same user which can often lead to password and account locking problems where it is dificult to identify the root cause. Note: The name limit is 12 characters.

Screendump 109: Enter the password of the user management engine (UME)


Page 111 of 202


Screendump 110: Enter the password of the DDIC user or keep the default default

Screendump 111: Enter the required required software packages: Oracle Oracle Client


Screendump 113: Enter the path of the SAPCryptographic Library


Page 112 of 202



Screendump 115: Choose the cryptographic library

Screendump 116: Select SAP archives to unpack


Page 113 of 202


Screendump 117: Enter SAP Diagnostics agent sytem ID

Screendump 118. Specify use of SAP JVM


Page 114 of 202


Screendump 119: Enter the path to the JCE policy file

Note. Even though the required java package for this platform according to the installation notes is IBMJava2-AMD64-142 which normally would need the JCE file unrestricted.zip from IBM you are now running with the SAP JVM which requires the jce-policy-6.zip file



Page 115 of 202


Screendump 121: Enter the system number for the SAP Diagnostics agent

Screendump 122: Select archives to unpack

Screendump 123: Prepare the system for NWDI integration

Screendump 124: System Landscape Directory (SLD) destinatiosn

Screendump 125. Enter the parameters for a lo cal SLD configuration


Page 116 of 202

HA SAP on Oracle DataGuard

Recommend Documents