‘0 oleate 1 Fae FA 0 Whe Wine 808 An Oracle White Paper December 2011 SAP with Oracle Real Application Clusters 11g Release 2 (RAC) on Microsoft Windows 2008 Release 2 DATABASE‘0 oleate 1 Fae FA 0 Whe Wine 808 Introduction Naming Genventions and Terms Installation and Migration Roacmaps... New Installation Migrating from Oracle 11.2 Single Instance Migrating from Oracle RAG 10.2 Planning Oracle ASM Diskgroups. ‘OCR and Voting Oracle ACFS Database Files Prerequisites for running Oracle Grid infrastructure 11g Release 2 and Oracle Database 11g Release 2 18 Hardware. 18 Software 19 Gommon Setup and installation Tasks 20 Preparing Microsoft Windows 2008 for Oracle Grid Infrastructure 20 ‘Oracle Grid Infrastructure Installation ........ 1 25 ROBMS Home: Installation Patching Oracle Grid Home and Orace RDBMS Home PAC Enabling. ‘SAP-specific database changes Checking the Seup Reusing Oracle ASM Disks Working with Oracle ACFS Checking Oracle ACFS. Backup and Restore. Configuring BR*Tools for Oracle ASM. Example initesid> sap fies: SESERSS5h5 EES‘0 oleate 1 Fae FA 0 Whe Wine 808 SAP Standalone Gateway Testing. Failures Hardware Failures. ‘Oracle Interconnect Failures. Storage Failures. Software Failures ‘SAP Application Workload Additional Performance Recommenda: fons. QSSaRRERSIntroduction Ouele Real Application Clustes 1g Release 2 (Ciracle RAC) and Oracle Aurtomatic Storage Management 11g Release Z (Qtacle ASM) technologies provide unexcelled smuilability, scalability and performance for yout SAP applications. ‘This document guides you through the most impostant steps sequized, to install sucha cluster and will also discuss the recommended path to anigrate an Ciracle Database 10g Release 2 based system (single instance and RA) to-Cieacle RAC 11.2. ‘The pre-installation steps for setting up the underlying cluster hardwrane and the operating system are also discussed within this document However, due to the fact thathardware plotforms are evelving quickly, the paper can just give some guidelines on whst hardorare is required and how it should be configured In addition, as Oracle Global Naming Seevice (GINS) is cecommended for bigger clusters where nodes ace dynamically jowing. and leaving, this lind of cluster configurations not discussed in detail, Instead the paper focuses on cluster configurations mnaing without GNS. Cin Microsoft Windows it may ‘be impessitle to integrate GNS into some customer's existing network environments aa ih requites s DHCP server and DNS server on the same network, While with Oracle 91 and Oracle 10.2 the Oracle Cluster Filesystem fir Windows (OCFS) vores the peofrred way to share data amonga aumber of elustor nodes OCFS is not supported anymoce with SAP and Oracle RAC 11.2 oa the Windows platfonn. Oracle ASM and Ceacle ASM Cluster Filesystem (Ocacle ACFS) is now the prefucced and recommended storage platform and therefore installations with OCFS are not discussed here, Basic cenfigusation steps an Oracle ASM will be desosled tut moce sophisticated Crracle ASM configunttions like configusations with move than ane storage system orstorage based system copies are not covesed within this document These topics are discussed in another whitepaper' SAP pith Orat Real Application Chasers 11g Release 2 and Oracle Auttomatic Storage Management Vg Rekaw 2 - Adoancrd Comfgenations © Tretia: An additional, complementary whitepaper ‘Maing jomr CAP slaisbare Onact Audomatic Sioeage Manageme 3g Reiease 2~ A Dest Pravtées Gui? is wwailable om SAP Developer Norerosk and on Oeacle TechDet Tt discusses two diffacant ways of moving an existing chtabase from filesyetem to Oracle ASM. SAP specific configuration hints and recommendations will be provided in this document as well as mggestions about basic test scenarios,Finally the document will provide information about SAP specific configumation steps in order te properly run BR*Tools and database tasks scheduled fiom SAP application servers. Naming Conventions and Terms Nameor | Description / Purpose Synonym Oracle ACPS | Oracle ASM Clinics Pile Sytem isa duster Ole apacm that uses Oracle ASM as the underlying volume manager inftaetucture. Grade ASM | Ocasle Automatic Storage Management iv bocislly a volume manages Fae all ‘Oracle Databate selatod Blea and Oracle ACFS. Grace ASM | An Oracle ASM disk is uaualipa physieal dick wilh a loge pariigan on ic Om Dice ‘Windows the legieal patton is crested ae an extended partion with a Lagieal ‘volume init Gracle ASM | K number of Grade ASW dks form an Oracle ASM didk group. An Oracle Dik Group ASM disk group has | or moe ‘failure groupe’, An extended partition can epan at of adisk-or arshole dak, Logical wohumes ace created inside an exten partition. FalliseGrap | A named group to which an Oracle AGM disk belongs to. Oacle ASM Qplatantees 19 never siote the same data rwice within a Gulure group. Redundancy ie achieved by diteibuting the same data to multiple flute geoupe. GER ‘Oracle Custer Repository HaMan ‘Gaacle Recowacy Manager=DBNAME> ‘Specifies the name of the databace (eg C1), sDENAME > is derived from the Gracle initialization parameter'db, name\. IF ‘db_unique_name! ic oet thea iz derived foom 'db_unique_name! =DESID= ‘Specifies the name OF the datakase Inetanca, For single inatance databases “DBSID® usually equale 7DBNAME> og, Olt, ‘With RAC the number of the instance ie appended ¢.g.C111, C112, ¢te. In some examples below the abbreviation will be used where equals the number of the AC instance. ‘The environment variable ORACLE. SID isuaually set to o SASMSID>. This the depende on the environment of the C18 usec ‘Specifies the name oF the SAP aystem id tis used for OS user names e.g, clladm ae-well 38 for authentication of the 08 user with Oracle eg OPSICLLADM. SIHRDEME RDBMS Safiwace Inetallation Home, The shared installation Ocack Home: izectory where the Oacle RDBMS Software is installed. This is a shased disestory on Oracle ACP, , Therefore it is recommended to set Oracle Base to ‘C:\oracle\ BASE! during the Grid Softee installation.=OHRDEMS> RDBMS Runtime Geacle Home. The munis path of the Oracle Home where the RDBMS Software wazinztalied. On Microsoft Windoure this ie the sam ae - \11202,'C:\ oracle 411203", Sean “The Single Client AncessDlame i the name of a vieual haat In DNS the name should cecalve to 3 diffesent virtual IP addzecces. NODE-P The IP address of the node on the public network The NODE-IP cannot Falover and therefore identifies s physical cluetee node, ScAN-VIP TFSCAN was configused to cesolve to 3 wvietual IP accesses the SCAN-VIP ia fone of thove, Please refer to ‘29069 pufon Cucle TechNet for farther information about SCAN. NODE-VIP The vistual IP addzere of a cluster node on the public network The NODE VIP can switch fEom one node to another if the ouming nade Bile ox iz taken effineoe “The Oracle Cluster Interconnect This fe the pelvate network of the cloter Installation and Migration Roadmaps ‘This chapteroutlines the recommended installation path for new installations and upgrade pith fir upemdes from older Oracle single anstance os clustered environments te Oracle RAC 11.2 with Grele ASM. We recommend to always start with new hanikvare as this will leave yourold system untouched and you will be able to abort the migration if unexpected problems srise. Steps that are common between the different installation ‘paths are deccbed in the chapter 'Conmon Sigh ond Installation Tack. Note that migsations from Oracle 10.2 single instance te Ouaele RAC 11.2 / Oracle ASM 11.2 will not be discussed here Esch installation/upgrade path is based. on the approach that sn existing database is being. "RAC enabled’, moved to Oracle ASM and then pat under the control of Oracle Glusterware. Therefore itis not recommended to upgrade Oracle Clusterware 10.2 ter Ousele Clustecwase 11.2. Instead the database will be upgraded to Ocacle 11.2, enabled for RAC ifnecessary and finally arided to the GOR of a feesh installed Oracle Custerwace nz. Referte the following SAP notes about Oracle 11,2 anc upgrading to Oracle 11,2, #ias1800 Ghadle 11.20; Cental Technical Note #1178 ‘Oracle 11.20; Database Parameter Settings FIWsI ‘Oracle 11.20; Patches J Patch collections for 11.202 #1631951 Sracde 11.20: Farches / Patch collections for 1120.3New Installation (On Microsoft Windows Omcle ACFS is not supperted for SAP iretallatiens. This means that ts not possible to install SAP software on Chmcle ACFS nor is it possible to install az Cacle RAC 11.2 systen dleectly, If you would install SAP on ACFS SAPINST would fail when it detects that the Oracle RDBMS Home is on Oracle ACES, Although new versions of SAPINST allow creating databases on Oracle ASM they do not allow Cracle AGFS for the Oracle RDBMS Home. ‘When using a version of SAPINST that dees mot support Oracle ASM we recommend to temnpocaaily cxzate local NTFS volumes, perfoan a complete standacd database secres installation (Distributed Installation’) with local Oracle RDBMS Home and initial database on NTFS and switch to Oracle ASM/ACFS later when the installation is complete. ‘When using « vession of SAPTNST that alseady supports Oracle ASM we recommend to create a local NTFS volume for the tenporary Cxacle RDBMS Home and to coeate the cnew database on Oracle ASM. After the installation you will just have to remove the temporasy Ciracle RDBMS Home and use the Coaele RDEMS Home from your Cnacle ACES filesystem. Note that SAPINST creates new databases as single instance databases. Every fresh ‘installed database is configured as a single instance database and requires'RAC Enabling which is discussed in a later chapter, “The following steps ace saquiced: 4 Ifyou do not use new harchrare remove any old operating systems and everything, that is riot tended Getall the latest divers fer your hardware that-might: be sequised for Micsosoft Windows 2008 2. ‘© Install and prepare the new opesating system. ‘* Installs local — temporary - Oracle RDBMS Home on NTS, Jastall an SAP datobase on NITPS if your SAPINST dace act suppect Ozacle ASM, Cthersrse install the new SAF database on Oncle ASM, Jostall and configuee Cracle Guid Inftastractase Home. ‘Conate all required Oeacle ASM disk peoups Jastall and canfiguee Crmele RDBMS Home(s) on Oracle ACFS, If yor installed the database in NTFS move your database be Oracle ASM manually by copying the datatiles using the ‘asaacend! utility. This cequises creating anew contiolfile on Gincle ASM with the new names of che datafles. fees* Altematively st ie possible to use 'RMAN DUPLICATE FROM ACTIVE DATABASE! as described in the whitepaper ‘Moving your SAP clatab are o Oracle Automatic Storage Managerent ig Release 2A Bact Pracives Gaidt'. This method is basically meant for movinga database from old server hardware to new server hardware, but i can also be used for Local database copies, "RAC Enabling’. Install database server specific SAP Softerare (eg. SAP BR*Tocls}. Migratng from Oracle 11.2 Single Instanea Make suze you have a full backup of your database. On new hardware eee Install and prepare the new operating system, Install and configure Oracle Grid Infrastractore Hore. (Create all required Oracle ASM disk groups Jastall end canfiguse Cracle RDBMS Home(s) on Oracle ACES, Bnsuse that your source system is on the same database release and patch level than you target system, Eig 11.2,0.2 + latest recommended Windows bundle patch, ‘Move your datshare to Oracle ASM as descuibed in the whitepaper Moreig yor SAP database so Onack Antrratic Storage Managenent 11g Rekeaw 2 - A Brest Prosticer Gnide If you are-using the MAN DUPLICATE! method you can crate & vemporary database instance from the Oracle RDBMS Home(s) installed "RAC Enabling’. Install databace server specific SAP Softerare (e.g. SAP BR*Tosls} On existing hardware sees Ifyou do not use new 'plain’ hardwase remove any old opesating systems and everthing thatis not needed but keep your NTFS volumes where yoine database and your eld Omele RDBMS Home sosides. Ger the latest daivers for your hhacdwase that might be requised for Miccosoft Windows 2008 R2 Jnstall and peepace the new opemting system, Jastall and canfigues Cimele Grid Infrastractam Home. ‘Create all required Oracle ASM disk groups Install snd configure Cnarle ROBMS Home(s) om Oracle ACKS, Ensure that your sours system is on the same database release and patch level than your target system. Eg. 11.2.0.2 + latest secommended Winderss bandle patch. ‘Move your database to Oracle ASM using the ‘asmemd utility e follow the procedure dhseribed in the whitepaper ‘Moving your LAP dlatahaw to Cari tomate Storage Management 11g Reitaw 2 -A Beer Proctves Guide” Lf you ace using the MAN DUPLICATE’ method you can emate a temporary database instance ftom the ‘Oracle RDBMS Home(s) installed© "RAC Enabling’ © Install databace server specific SAP Software (e.g. SAP BR"Tosls), Migrating from Orade RAC 10.2 Assuming your database is Located on OGFS it is strongly recommended to provide new ‘hacdwase oe at least a server whose you can exclusively monst all the OCFS-volunes and ‘pesfonn the database upgmde to Ciacle 11.2. Make suse you have a full backup of your database! Migrating to new hardware © On your new hardware: © Install and propase the new opemting cystom. Install and configure Oracle Guid Infeastracture Home. © Create all cequised Oracle ASM disk proups Os pour old haccoase: © Shutdown cach database instance using'sevet' and stop and disable your ‘OracleService without the thiead auenber at the end eg PRD not PRDI. Set ORACLE HOME to the Gacle 11.2 home directacy. Create a temporary OracleService*DBSID™ Service in Windows using ‘oradimene! frem your Qucle 1L2 RDBMS Heme, © Frepare a database parameter file (pile) for this database using the SAP secommended irstora parameters for Oracle 11.2 RDBMS and your RAC specific patameters but oath parameter ‘cluster_database! set to FALSE, o Stamup your Cracle Database 10.2 in upgade mede using the newly installed Chacle L1.2 RDBMS Home and-with the newly cazated pile to ensure the database opens successfully. Eg. sqlplus startup upgrade pile=myspgradeplileora © the database instance starts, shutdoon the database again and start a manual database upgrade as described in the Oracle documentation and the according SAP notes. 4 Mour your upgraded database tes Oracle ASM as described in the whitepaper ‘Movig ponr SAP databace so Cnack Ansmatic Stomage Maragement i1g Resear 2 —A Best Praise Caide’ Ifyou are using the 'RMAN DUPLICATE? method you can create 2 tempomury database instance from one of the installed Oracle 112 RDBMS Home()."RAC Baabling’ - As the database was already RAC enabled not all steps ace secqiced. Install chtabace server specific SAP Software (eg. SAP BR*Took}. On existing hardware * teens Shutdown each database instance using'sevct!' and stop and disable your OracleServiee without the thread number at the end eg. PRD not PRDI. Set ORACLE HOME tothe 11.2 home directory. (Cosatea temporary CacleService?SID> Service in Windows using 'oradim exe! foom your newly installed Oracle 11.2 RDBMS Home. Prepare a pfile (¢.g. mpupgeadepfile om) fos this database using the SAP secommented initora parameters for Chaole 11.2 and your RAC specific [parameters but with parameter chister_databace’ set to RALSE, Startup your Oracle 10,2 database in upgrade mode using the newly installed Ohacle 11.2 RDBMS software and with the newly created pile to ensure the Satabase opens successfully. Eug, sqlplus startup upgade pfilemyupgadepfileom If the database instance starts, shutdown the database again and start a marmal upgide @lways sin osadim faem the path where itis installed! 0 Remove the Oracle 11.2 CxaeleSericee SID service with omdim, © Setup your enriranment for Chacle 102, eseate a Chacle 10.2 ‘OracleSermice service with oradim, © Create an Oracle 11.2 OnaclService?1D% service with oradim and pediorm a manual upgrade. Dion't forget to ma SAP specific upgsad: tasks ane scripts, Backup the database, Remeve your ald Micraaoft Windewa installations from the chsster nades Install and prepare the new operating system, Install and configure Gracle Grid Infrastractum Home. ‘Cosabe all required Oracle ASM disk gooups Restore your database to a file system if sequized, If you have a copy of your database on a file system or netwede deive you can continue with moving the database to Omele ASM.© Mave your datubase to Oracle ASM manually by copying the datadiles using the ‘asmemd! tool. This cequines creating a new controlfile on Oracle ASM with the new-names ef the datafiles, Alternatively itis possible do use 'RMAN DUPLICATE FROM ACTIVE DATABASE! desedbed in the whitepaper Mowing your SAP databace Oracls Automatic Storage Management 11g Reikawe 2 A Bart ‘Practives Gnide', This methashis basically meant for moving a database from old server hardware te new server hardorare but it can be weed for lecal database copies too. Planning Oracle ASM Diskgroups Shared disks are often located on an intelligent stomge system. Although Oracle ASM is able and designed to work with thumb and cheap JBOD systems we strongly recommend using storage systems that already have built-in fault tolerance, If your storage slieady provides fault tolerance we recommench creating all Omcle ASM ick peoups with ‘extsmal cedundancy’ with the exception af the Oaacle ASM disk peoup "OCR! for which we recommend noanal sedunchney. Some mom advanced stomge systems support additional capabilitics lke offlining and online dicks or creating image copier of dicks and therefine alloer copying complete dirk groups within the storage system which provides amasingly fast SAP system copies and disk image baced backups. Both sre discussed in detail in the whitepaper ' SAP wish Orah Rea? Application Chucrs 11g Reteare2 ond Crack Ausamansc Ssvrage Management 11g Ralkase 2~ Adsanied Configurations eb Tevisigues If yous storage does net prowide any fault tolesince we secommend nomnal or high redundancy far all Oracle ASM disk groups. For Oracle ASM disk group 'QGR! we always recommend high redundancy. ‘When caleulating space requirements and eeqniced number of disks it is always necessary to take the sdundancy level ofa disk geoup into account. GR and Voting Ovacle Geid Infrastenctuce equises an Chacle ASM disk group for OCR and Voting Files. ‘We strongly recommend creating an Omcle ASM disk group exclusively for OCR and Voting Files. For stocage systems with builtin fault tolecance we scommend using 3 shareel disks wile JBOD systems should use 5-shared disks. All disks should be of the same dive and at least AGB in sive,‘The recommended name for this Oracle ASM disk group is 'HOCR’ but you are free to choose ansther name, Oracle ACFS ‘While Qaacle Geid Infrasteucmuce is installed on Local disks of each of the elusternedes Oracle RDBMS Software is installed on Oracle ACFS, Oracle ASM disk groups can contain multiple Oracle ACES file systems the same way a standard disk can have multiple ‘pattitions each with its oor file system. ‘We recommend creating one Oracle ASM disk group for all Oracle ACFS file systems and one Oacle ACFS file system per Quacle RDBMS Home, Keep in enind that with Caale 112 patch sete ame being installed out-of place which means that you wil need to recenve enough space fora second or third Cracle RDBMS Home. ‘The recommended size for each Oracle ACFS file system is at least 20GB fer Oracle RDBMS Heme, database logs and trace files from each database instance, BR*Tools logs, ‘patch set installation, ‘The secommended minimum size of the Gmele ASM disk group that holds all the Oeacle ACES file systems therefore is numberof systems> * . ‘The socemmensded name foc this Oracle ASM disk geoup is '‘+ACES', Eg. for 3 SAP systems we recommend 3 Oracle ACFS file systems (each 20GB ormoce) onan Cracle ASM disk group with at least S0GB of free space. Filesystemname ‘Mounipoint Size [GBI cu GAGRAGLEYGIL 20 c1 CLORACLE\CIZ 20 cs CAORACLE\CIS 20Database Files. For an SAP installation on Oracle with Omele ASM we recommend to choose one out of thee variants on how to name and configure your Oracle ASM disk groups for database specific fles, Which vacant you choose depends on yous backup steatsgy erhich in mas usualy depends on the size of your database and required availability. Variant 1 ~ small to medium data volumes, restore time not critioal Customer has small oc medium sized databases where backup and/or restore + racomery of all databases can be accomplished by RMAN in a timely fashion. Example: [fa complete Omcle ASM dick group with data files from one or more databases is beoken and all data files from all databases need ta be sestaced to a newly created Oracle ASM disk group using RMAN. Owele ASM disk group secommendaticns Cease ASM Disk Group. Name Heres +DATA ‘All data Biles ~ Allternp flee = Contzal file (Beet copy) ~ Ghline rede logs (Best copy) ARCH ~ Control file (econd spy) ~Achived redo lage +RECO = Cont file (thied 2op) = Cline sede logs (second copy = RMAN backups (epticnal Fast recovery area(options!) Variant 2 medium to large data volumes, restore time important Customer has databases of mnediuun oc lagge slae whece backup and/or-settore + excowery of all databases cannot be accomplished in a timely fishica. Usually customers will use RMAN and Splst-Mirror techniques in combination, Major differences to Variant 1 ates © Separate Orade ASM disk group foreach database * “SDBNAMB2_'is the prefix for the name of the AAT disk gooup.© Optionally a mumberis appended to the name of the DATA disk geoup-if the database Spans over move than one DATA disk geoup, © Online redo logy are not stored in the 'DAT.A' dick groups. Instead online sedologs ave stoned in an acektional disk geoup thatis used forthe fist manberof each online cede lag group. Important: Please consider that ory a maximum of 63 Oracle ASM disk groups can be configured Owele ASM disk group secommendation: (Oracle ASM Disk Geoup Name ‘Stores +=DBNAME?_DATA[#] se ERP DATA, CRM_DATA, ERF_DATAL (eg fer tempor ical spuecaqatemersOuingSa® pate: ce tepcais) ‘All data Glex All eg files Control file (Best copy) +OLOS Ghline cada Jog (First copy) +ARCH ‘Control file (eecond copy} ‘Acchived redo loge +RECO - CConirel file (tied G29) Galine cede logs (eecend egy) RMAN backups aor eocvery area Variants - very lange data and-data change volumes, restore time crucial Customer has very Isage databases where backup and/or restore + recovery of a single dhtabases cannot be accomplished in a timely Fashion, “Usually customes: will use RMAN and Split-Miceos techniques in combination, Inthis variant each eelevant database file type is separated to different Oracle ASM disk groupe, Important: Please consider that orily a maximum of 3 Oracle ASM disk groups can Tes configured.Oracle ASM disk group secommendation: Oracle ASM Disk Group Name Biores +_DATAI#] ce ERPDATA, CRM_DATA, ERP_DATAL (eg fertempon dco spiorempeaments ng AD opal angen) ‘All daa flex All temp files Control file (Best 2epy) +=DBNAME>_ARCH Control file Gecond copy eg ERF_ARCH, Archived reds loge CRM_ARCH + _RECO ‘Control file (thied copy) eg ERPLRECO, = RMAN backupe CRM_RECO ~ Fast resoregy area +2 DBNAME®_OLOG eg ERFLOLOG, CRM_OLOG Gnline seda loge (Fist copy) +e DBNAMES LOG se ERPMLOG, CRM_MLOG Galine vada logs (second spy Important Always place all files from one databasc inte the according Oracle ASM disk ‘poougs of that ditabase, Neverplace a file from one database in diskecoups of another chtabase.Prerequisites for running Oracle Grid Infrastructure 11g Flelease 2 and Oracle Database 11g Release 2 ‘When stacting with Omale gRAC we mcommond ma amdy the hasd- and software requizemeats discussed in the according Oracle documentation. We recommend using, snewr chistec hacchwase as this allows leaming your old system untouched until the configuration of yournew system is complete and your system is working poopedty, Hardware Make suse your chuster is build from certified and well tested components. We recommend thateach of the cluster nodes has the same-number of CPUs and network interfaces as well as the same amowntof memory and local disk space, As Oracle Guid Infiastnictare will be installed on a local volume make sure you have enough local disk space for installation and patching. Note that patch sets for Cele Gaid Infrastructuse and Oracle Database are installed out= of-place while patch setupdates (or mini patches) ae in-place, ForOmele Gaid Infiastructare we recommend at least ISGB of fiee local disk space, (Onscle Grid Infrastructure requires twra network interfaces — ane for the chsster intexconmectandl one for the public netwers. We recommend configuring 4 network intezfaces into pairs (teaming). “] | ~] oo0°0 Public Network SoritchShared storage should be attached wia fiber channel adaptors. For performance and availability reasons multiple fiber channel links and fiber channel switches between closter nodes andstomge system are recommended. This can be achieved by products lke BMC Powerpath for example. - a00 Feechnnd Sitch 1 Software (On each cluster node you need to be install the same operating system version and patthes, We recommend using Microsoft Windows 2008 R2 Serres s4Bit as this is the latest and most up-to-date Microsoft operating eystam. Windows 2003 Seaver 64Bitic also supported butdeprecated and will not be discussed within this document.Common Setup and Installation Tasks Preparing Microsoft Windows 2008 for Orade Grid Infrastructure Before starting with Omcle Geid Infastructum installation some mandatory pre- installation steps ave requized in oxder to be able to install Oracle Grid Infrastmctuce successfully, Windows Registry Changes Windows Time Synchronization Encure that Windows never automatically comects system time backsurds by setting MaxMogPhoseComection = 0 under 'HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\ Services| W227 ime\Condig "in the Registry. See Microsoft Knowledge Base #864776, Wagons Interacve Desktop Hesosize “The default sise of an interactive desktop heap under Windows 2006 R2 G4Bitis 20MB. As all Windows Services that mn under the 'LocalSystem’ account shase the same desktop ‘heap it is cecommended to increase the defiult value of 20MB to 40MB. If you experience 'TNS-12531: TNS:cannot allocate memory* messages ‘in your listener log file after several days or weeks uptime it is very likely that you have hit that problem, In addition to the TNS ecrocit is vecy likely that you will experience a crash ofene of the DB or Oracle ASM instances, Change the second parameter of the key SharedSection’ under 'HEGY_LOCAL_MACHINE\system\ currentcontrelsst\centrel\session wanager\subsystems\windows' to meet your requirements Note that the value is in kilobytes, suse zefrt the flowing Linke ox cask Missaft Kanomsge Bass for none‘Windows User Account Control and Automatic Elevation * Tum OFF Windows UAC: centrel Panel-»syatem and security->change User Account central settings + Change automatic elevation settings UAC: Adnin approval mode = Disabled uact Behavior of the elevation prompt for administrators = Elevate without prompting uAc: Run all administrators in Admin approval node = Disabled Network Infrastructure etwon Inertaces © Rename the network intesfices and choose speaking names fer them. E.g. PUB, ole. Don tuse the name'PUBLIC' as this will cause an error dicing the installation. © Semup IP adeleesses, subnet mask and DNS of the netwode interfaces: 4 Tumoff IP ¥6 on all interfaces.© Setthe same bind osder of all netwodke intecfices, The public interface ‘PUB! should be first, Oracle Interconnect 'O1C second and then the other netorack. interiaces (if ang} * Disable Windows Firewall for all neterock interfaces on all cluster nodes. * Disable DHCP Media-Sensing (Microsoft Knowledge Base #238924) © Tum OFF dynamic resizing of TCP/IP receive srindoor (Microroft Knowledge Base #934450) by issuing the following command from the command line prompt netsh interface tcp set global autotuningedisabled + Query state byissuing: netah interface top show global. © Reboot and se-cherk everything Also check if each NODE-IP and OIC-IP can be pinged. as Each cluster nade sequires 2 IP addsesses and accosding hoetmames oa the public network: The node [P adkress and a node viral IP address (NODB-¥IP). In sedition ‘sre recommend reserving 3 SCAN virtual [P adedcesses (SCAN-VIP), Create state ertaie sin your DNS server fe forward and reverse lonkcup.eatin) weozzm wa eta this oe esta bass pase estat teaair oe eta maze ae eat tage fae ett teaaz pad ‘ati teaais oe Heat teas me met tena pad ot teaay esta tease Reveree-Lookup: Greaens aes 9m) octane se eae Gimonzae ewes Om) cecotcuckinnd sa 2a Cieasas ae Or) ceciteehieim se, tae Fjmenaie fee) Roncsscese snp. Bae Groen ‘Powter OTR) cea oracle ne stat 5 Bl Reward zone [tena Per) cecosoandevnd.s ae SS Ealscersoraa [02 Perea) cmonip ented sa For Cole Cluster Intesconnset (O1C}:no enteias ae required in DMS but we recommend adding them to the ‘hosts! file together with an OIC specific hostname. Eg 192168111 arsed l-oie 192.168.1.12 oraex30d-cie 192.168.1.13 oracx3y3-cic Windows Domain and OS Users With Qracle RAG 112 a new nocr has been introduced that plays several rolna: 4 Qumer of all Oracle: softxare components (Crace Grid Infiastmicture home and. ‘Oracle RDEMS home), © Adminiotator of Oracle Grid Infiastruchare, Oranle ASM and databascs ‘The wecommended name of this user is ‘oracle’ “Although the user'ackn! still exists and is still sxquised it is only used fos mnning SAP software installed an the cluster nodes. E.g. forexccuting BR*Tools intemetively or for moning database related jobs from transaction [B13 via SAP Gateway Serves,© Each cluster node should he member of your Windaws Demain © Create user ‘oracle’ and user ‘aem' in your Windows demain, © Add user ‘oracle! and 'Ssapsid™adm’ to each cluster nock's local 'Administraters! Oracle Administrator Groups Create the following local groups on each cluster node and add your'oradle! user as well as ‘pour ' aden’ usex( “You can sither create group CRA_DBA and ORA_OPER (recommended}.or you can cweate group: ORA DEA and ORA_ _OPER foreach instance, Preparing Oracle ASM Disks ‘When new disks age attached for the fist time they ane displayed as ‘offline! and ‘uninitistized! Set them ‘online! in the Windows disk administrator and initialize them, Before new disks can be used with Oracle ASM the disks must be prepsred by creating an extended partition and a logical partition on it. 'This should be done with the ‘diskpart’ toa Ay select disk <4 2) create partition extended 3) create partiticn logical Important: When a nes logical partition is created Micmsoft Windows will ask ter format: the disk for use writh a file system. Do not cceate a file systam on those disks, Make suse that new state of the disk is vieible to all nodes bay initiating a scan ef the SCSI bus or by rebooting the cluster nedes and semave all daveletters ftom those disks yeu plan to be Orucle ASM disks. Make suse that nobody formats an Ciracle ASM disk with a file system ‘by documenting which disk is used for which purpose. If you already hase comnected all chared dicks you plan to use fir your cluster prepare all the other disks in the same way. E.g. After propasing 3 shased disks for Cmacle Gad Infmsteuctuse installation they should Jock like show in the pictuse below.—_ EEE ore [eset See RR Gracie Grd Infrastructure Installation As desceibed above all installation and configusstion steps regaeting Cracle Softorare ate vperfomnedd as ‘omcle' user Make eure you are logged on as this uses, Setting environment variables as needed ‘The 'omcle' useris used on different clusternodes with differant Oxacle Homes (OHGRID and one or more OHRDBMS)), Because of this we recommend to create small shell scripts to setthe erriranment variables ORAGLE_HOMB and ORACLE_SID to the values needed, If you need to surtch between the mrming instances you just need te mun the according script Eg (Oe cluster node orace3ri: C:\Users\oraclestype env_asm-cmd set ORACLE_HOMB=C: \ORACLESGRID\ 11202 set ORACLE _SID-+ASM1 C:\Users\oracle>type env_cl1-cmd set ORACLE_HOMI SORACLESC11\ 11202 set ORACLE SID-C111 (On cluster node orace2r2: Cr \Users\oracle>type env_aam.cmd 2 \ORACLESGRID\ 11202C:\Users\oraclestype eny_cll.cmd set ORACLE HOME=Cs \ORACLESC11\ 11202 set ORACLE SID=¢112 On cluster nede ocace3r3: C:\Users\oraclestype env_asm.cmd set ORACLE_HOME=C: \ORACLE\GRID\ 11202 set. ORACLE_STD=+ASM3 C:\Users\oraclestype eny_cll.cmd set ORACLE HOME-C: \ORACLESC1I\11202 set ORACLE_STD=¢113 ‘These scripts will be used to set the enviromnent of the 'aracle! uses within the whale document, Initial instalation Extmact the instalation media, open a command line prompt, move to that location and stut'setup.exe’ with the following command line: setup —J-D-user.language=en Screen 1: Ghosse 'Skip software updates! anc click . Screen 4: Select 'Bnglish! as product language and click .Screen 9: Enter'OCR’ as the Osacle ASM disk group name, select the redundancy level of the disk. group and click ‘Stamp Disk’ Screen 9a: In the popup dialog select 'Add or change label and click , You can select multiple lines by holding down the or key. By. a oe Ca] cot Steen 9c: This popup dlog chovwe what label will be essigned to which disks Click et Ci] cont Screen Od: Click . Screen 11; Choose ‘Do notuse Intelligent Platform Management Interface (IPMI}'and click to start the installatien.After the installation and configuration of the Oracle Grid [nftastracture is finished check that Oracle Clustermare is running on all the clusternodes, ersctl status resource -t ‘Upgrade the compatibility attributes ‘COMPATIBLE ASM, 'COMPATIBLERDBMS' aed ‘COMPATIBLE.ADYM of dick proup ‘OCR ta '11.20.2.0' Alse choose a DISK_REFAIR_TIME that matches to your type of storage system. For storage systems ‘with built-in fault tolerance you should select very high values eg 2401 hours}, For storage systems without buildin fault tolerance you may want te keep the default value or even choose a smaller value. Ee. Ci\tserstoraclaseny_aom-ond Ci\tsersteraclascet ORACLE OME=C: \ORACLENGRID\L 1202, C:\Usersloraclesset ORACLE STD+ASML c:\Users\oracte> sqipius SL-Plus: Release 11.2.0.2.0 Precuction on Thu aug 11 11:35:55 2012 Capyraght (c) 1962, 2010, Gracie. 11 rights reserved. Enter user-name: J as sysasn connected to: Qacle Databace lig Enterprise Eaitien Release 11,2.0.2,0 - Sabir production With the Real applicstion clusters and auteuatic storage Management options SOLs alter diskgroup OCR set attribute ‘compatible.asm’='11.2.0.2-0"; Diskgreup altered, SQL> alter diskgroup OCR set attribute ‘compatible. rabms'="11.2.0.2.0"; Diskgreup altered sQis alter diskgreup OCR set attribute ‘compatible. advm'="11.2-0.2.0"7 Diskarup altered. Sole alter diskgreup OCR seb attribute ‘dish repair_bime'-"240h Diskgreup alteredCreating Oracle ACES File Systems. An Oracle ACFS tile system is created inan Quacle ASM volume. An Oracle ASM ‘volume is a specific Oracle ASM file type (like datafiles from a database) and is distsibuted over all Ouusle ASM disks belonging te the same failure gzoup. If the undedping Oracle ASM diskgronpis configused for nozmal or high redundancy Cracle ASM Dynamic Yolume Manager (ADWM) ~ which sits betorcen Oracle ACFS and ASM ~ ensures that the required number of redundant ecpies of s piece of dats are written te Oracle ASM disks in the other failure groups. To manage Cracle ASM disk groups and Qcacle ACPS file systeans we use the tol Cresting a new "Gracle AGFS' Sekgroup Open a new command prompt and set the enviconment to work with Oracle Grid Infiasteuctuce and Oracle ASM: cav_asen.cmd (created in an eacliee chaptes) Screen 1: Shows a list of existing Omcle ASM disk groups. Click . Seren 2: Click , Screen 3: Choose ‘dd or change Isbel’ and click < Next®. Sereon 4: Chosse the disks you want bo stamp with on Oracle ASM link and choose @ descaiptive peofix for the labels eg. 'Ocacle ACPS! Click «Next. Screen 5: This screen allows you the werify the name of the Czacle ASM link thar will be ‘written to each disk you selected, Click if everything is corsect Seren G Click Screen 3: Now you sre back on screen 2. Enter a name for the nesr diskgroup (e.g. ‘ACFS}, select the redundancy level and select the disks you want to add to the Oracle ASM diskgroup, Click on'Showr Advanced Options and enter '11,2,0,2.0' (without quotes) for ASM, Database and ADVM compatibility, Finally click to create the new Oracle ASM diskgcoup.ea canine [ars —__] earn wacom sara ie tara ae po Pee ee I om ete Orme Crema Eas ese Ben aie Oto AE ‘Genter aps reese engi er hrs on lca en ee yrs Sarpy tH 2 ta (ecole Pan sats ameeue Beanery ey (Goeatmerer rl reoee] ca immense 3) to enamel (ABH Compal (izaz ] ees aes taste Sonnet Aa GaSe fr as CoM a (esos ors] [a] [or] [re] “Use this procedure to create all your Grace ASM disk groups. -Alvrays set the compatibility attabutes to '11.2.0.2.0' oc higher as this will enable seme fonctionalities youmsy want to use later,ABU tgs | ier Ae ‘Yyeanstenmtocinte sor wou tess were Toomer arses yeh 2Bteomoe sere fara ae pare MOUNTED 9 ets mala baer HoUnED (re) (seoa (sosea] (ex Crecting the Oracle ACS volume Although it is possible to create just one Oracle ACES volume and file system for all your Ossele RDBMS Homes we cecommend creating one Oracle ACFS volume and file system for each Oracle RDBMS Home for maintenance reasens, Screen 1: Click the 'Volumes' tsb, Scesen 2: This screen shows all existing Oracle ACFS volumes. Click Serson 2: Choose the ACFS velume where you want to create the ACFE on, selest "Database Home File System’, enter the mountpoint and elick “OK. Ee ‘resna te ASM Custer Fle Sytem cresie he cr-doksrucue. The ie eyenneed oe mcured before Ml seoaneie er wen vous (ery summer 110-2400 a "ie Choose an ith vhane ice of cae ape vim by croosmng Creat Vote nthe opdown, (© Database Hone le System you crete wy ASM Cher ie Sten or rating Oracle dsb home Weis be eter th Gr Infradracure asa manage resource Ged iraducire uses hsrescurce io vise Gatsbace, And CFS evo doperdencis OabaseHone Maui Fort [CGRACLENCHT Browne (CGerert Pune Fl Syoem Agar ASM Cte Fe Syst Mur Prt wll alow Pe ees tobe adomaicay marten sree Mot Pont fogater wort Pace © (a cane]Screen 1: The newly created ACES should:newr be listed. Check that all cluster nodes have mounted the new file system. Organise = Inchadeinlieary = Share with = New folder e&-0@ al anaes Baers syiyont eae fe fer Bvstetound fyyao11 306M Pe fer RDBMS Home Installation ‘The installation of the Oxacle RDBMS Home(s) is very similar to installations in a none clustered environment The major difference is that during the installation in a cluster each node's cepisteyis updated andl a central inzentogy is created on each node. Extract the installation media to a staging directory and epen a command prompk Mave to the staging dicectory and cun the follovring commanc: "setup -J-Duser.language-en* Scceen 1: Uncheck 'T wish te receive secusty updates via My Oracle Support and click Next, You reccive the waming that you did not specify your email address. Confirm with “Yes, Seceen 2: Choose ‘Ship software updates’ and click Scecen 4: Choose 'Cracle Real Application Clusters database installation! and male suse each cluster node is checked in the listbox belew, Click Screen 5: Make sure ‘English is the only selected language int the right listbox and click Next Screen 6: Choose 'Enterptise Edition! and click ‘Select Options! Sceeen fa: Uncheck all components you don’t need or yon don’thave a license for and click Screen 7: Enter 'Cracle Buse! and ‘Software Location! and click . Nate thatin this dialog Oracle Base is different from Oracle Base used during the installation of the Oracle Grid Infrastructure, Both directories have ta be on Oracle ACFS, jmntne uci i) secey wont sy Crich aot te spas en em cr Se (toe ae cyt tpons rseryatacrsn uae tere Cie Cane il neta) Croer) Comead } Screen & Verify the information and stast theinstallation process by clicking , Repeat the installation procedure for each additional Orarle RDBMS Home yeu sant te snstallPatching Oracle Grid Home and Cracle RDBMS Homes “Hort to install patehos is well documented in the according patch nates and rzadme “Although, bused on cur experience, we recommend some additional steps to avoid pesblans ike open files that cannot be overwaitten. We recommend the following Stop all duster resources, 'crsctl stop cluster -all! On all clusternodes: Set all Oracle services that are configured to start automatically to start mode ‘manual’ (OracleObjectService, ‘OnaclelMT SRecoveryService, Oracle HEerviee), © Reboot all the chister nodes © Stop secvice ‘Microsoft Distributed Transaction Concdinatee * net stop omafenceservice At this point every Oracle specific fle that may be overwitten dating patch operation should be closed Chece if you need be install a new wersion of opatch into exch Grarle Home before you statt patching. To patch your Gad Home open anew command pmmpt and set ORACLE_HOME to the Grid Home fgg, C:\CRACLE\GRID) 11202). If you have created shell sesipts for switching environments just run the script to set your ‘GRID/ASM' environment, Enter the patch directory and stat patching as described in the patch readme, Follow the instmctions in the patch race in order to patch your Oracle RDBMS, ‘Home(s)RAC Enabling Database specitic changes All database specific changes can be pecformed from one duster node, Note that with Oracle ASM filenames are genemted by ‘Oracle Managed Files for ASM’. ‘The pacameters 'DB_CREATE_FILE_DEST, ‘DE_CREATE_ONLINE LOG DEST_a,'DB_RECOVERY_FILE_DEST' specify the Omcle ASM diskgroup where Cacle will create the files by default This means that there is no need to specify the filenames for datafiles -OMF -will generate a name based, on the tablespace name, The names for the controlfiles ofthe databese must be specified as Oracle ASM sbases in the spfile. In the case you need to restore controlliles there is no cneed to change the names in the spfile. Sometimes it makes sense to specify a full filename on Oracle ASM. E.g if you have smarmally copied a database from NTFS to Oeacle ASM using the asmemd utility you. "may brant to use aliases for the destination filename in onder to creste a new controlfile aftereach database fle has been moves to Oracle ASM. Undo Teblespaces Ensure that each Orele RDBMS instance has its enn dedicated undo tablespace of the apprepate size, If you have upgraded from Oracle RAG 10.2 and the number of custernedes die net change, you should already have undo tblespares for all instances and you can skip this step because the undo tablespaces will be available after moving the database to mele ASM. Eg create undo tablespace PSAPUNDOL datafile size 22000M autoextend ony great undo tablespace PSAPUNDOZ datafile size 22000M sutoextend ony On e Fedoiogs We recommend using 'DB_CREATB_ONLINB_LOG_DEST_n' to set the default Oracle ASM diskgroup fox online redologs and their duplexed mirror, E.g. Ifyou decided to create dedicated Omcle ASM disk gooups for online redologs “ELIOLOG (OLOG fos oniglog) and '+C1IMLOG' (MLOG for edplog) you should set to* .DB_CREATE_ONLINE_LOG_DBST_1='+C110L9G" * .DE_CREATE_ONLINE_LOG_DEST_2='+C11MLoc" and create new online cedolog groups after mowing your database to Oracle ASM, Each RAC instance nses a specific thoead for wsiting redo logs. You therefore need to create naw sedolegs, Eg alter database add logfile thread alter database add logfile thread alter database add logfile thread alter database add logfile thread alter database add logfile thread alter database add logfile thread group 10 size 250m; group 11 size 250m; group 12 size 280m; group 20 size 250m; group 21 size 250M; group 22 size 250M; “You should add the same rmmber of redo log geoups as in a single node envisonment. ‘The redo log sizes should alse be the same. Tf you have plenty of disk space on yous Oracle ASM diskgeoup, it’s a ood idea to use this additional disk space for more online redologs. This wall redace the chance that the system requires archived redologs duzing ctash recovery, For all new cedolog groups that have been added, the carcesponding theead has to be enabled alter database enable thread 2; SPFILE Is cosommenced and SAF standard to place the spiile that belongs to-a database on, Oracle ASM. SAP BR"Tools vill autcmatically create backups from the spfile on Oracle ASM snelallasr to moxify iritislisalion paramcters cisectly. ‘There may be reasons not to place the spfile on Oracle ASM, Note that placing the spfile teaditionally in 'CHRDBMS\ dbs’ is alse suppocted Assuming that you have a pfile with all the initialisation parameters create a spfile on Ouele ASM foun it Eg create spile="+DATA/CH/spfileam' from pfile='CA\temp\finalpfile.ora’sThis command creates an Onacle ASM aliss that points to the active spl on Ora ASM, The alias is required because Oracle ASM will use an OMF name for the newly create spile and we wantto:make sure that the spf is slways found under the same rname. Thi is important when we add the databsee and instances tothe cluster repository. RAC specific parameters are: 4 cluster_database=TRUE * sundo_management-aute 4sid?¢inatancenumber>-instansenumber- -thread=cthreadnumber> seidecingtancenumbers unde tablespace=‘ cnanesfundotablespaces! cinatancenumbers remote_listener='" * DB_CREATE_FILE_DEST='<#01SKGROUP > *og_archive_deat_1=!locat jon=c4DISKGROUPS / /ORAARCH valid for-{ALL _LOGPILES,ALL_ ROLES) ab unique nane-:“ports /sermicename™.If you still want ta use insnames.or please see the example below tnsnanes C1L.WORLD= (DESCRIPTION = (aDDRESS_LIsT (ADDRESS = (comumrsy = SAP .woRLD) (PROTOSGL - TCP) (HOST = scan.oracledev.wdf.sap-corp) (PORT - 1521) ’ a } (CONNECT_DATA = (SERVICE_NAME = cict1) : ) Oracle Instance Creation When creating or mmowing an Oracle instance service (OrscleService eg G\ORACLE\CIT\I1202 + cd MORACLE_HOMEM@\bin + omdim —new id ~startmodemarual To remove sn instance service: 4) opena nes commanc! prompt * sctORACLE HOME= eg C\ORACLENCI\11202 © = ed®ORACLE HOME \ bin + omdim —delete —sid “cbsid HinstancenoEug, for a database ‘C11’ that should have instances on 3 clusternedes you would create 3 ney instance services: Create the Oracle Clusterware Resources Setup your environment to wok with Oracle Grid Infiastractase by calling yous shell scdipt thot sets up the environment accondingly. Bug. ‘env asm.cmd. ‘Add the database. E.g. srvetl add database -d C11 -o C;\ORACLE‘C11\11202 -p 4C1L1_DATA/G11/SPFILECL1 -ORA Add instances that belong to the database. Eg. ervetl add instanse -@ Cii -i clit —n oraexSui srvetl add instance -€ CLl -4 C112 -n cracx3v2 srvetl add instance -d C11 -i ¢113 -n oracx3v3 Add database services that should be managed by Oracle Clusterware, Each Orcle RAC instance must have a service ramed like to instance itself and the service must be configured ta mun on this instance only, Tes sequiced to cazate one database service foryour SAP central instance and foe each SAP application serves you require additional databace services (.e. foen SAP application servers you need.n database services where every SAP spplication server is connected to- ‘its cum database service) — not counting the instanee specific services. Never configure a datbase service to nim on more than one datsbase instance at the same time. This would create connections from ane application server to mose than en cotabate instance, which may cause sedous pecfosmance and availablity peobleans. Eg srvctL add sorvies a C11 =s C117 =r C111" srvetl add services -d C11 -3 C112 -r "C112" srvetl add service -d Cll -3 C113 -r *ciL3* srvetl add service -4 C11 -3 CLICE -r "C111" -3 *c112,c113" arvetl add service -d C41 -3 CL1AS1 -r "C112" -a “c113,c111" srvetl add servies -d C11 -s C1IAS2 -r "C112" -a *C111,c112" Stattthe database servicesseuct! start service —d C11 {will try to stast all services defined for database 'CL1') SAP-specific database changes DDLOG Sequence SAP is using a datubase sequence ealled “DDLOG” to penemte timestamps forvadeus ‘purposes, Therefore SAP needs consecutive numbers returned from DDLOG. To ‘guarantee this ina RAC envisorunentthe sequence has to be excceated in “ocdesed* ‘mode. Please refer to SAP note #743555 for a description on how to da this. Checking the Setup In ordkr to check the status of the cluster’s Loral resources and cluster resoumces setup the appropdste envionment in your command line and run ‘crsctl status resowzce—t te print a complete list, Note that every resource except'ara ged! and 'era.ac4j' should have the status shown as 'TARGET GNLINE STATUS ONLINE. © Shotdown and power off each clusternode and then reboot them to see if every node come: up peopedy and all the cluster sernizes are correctly restarted Note that it can take ~15 minutes until all cluster esousce are started © Use tsping to verify that all listeners ane reachable and the accosding services are sepistered poopecly * Try to connect to each database sevice, © Check if database instances can be started and stoppedas expected using 'srvcil vutiity. ©) Check the alert loge fox error. Reusing Oracle ASM Disks Ifa disk is added to an Chace ASM diskpooup for the fist time the disk gets flagged asin ‘use’, This ancans that even if you cemove this Crucle ASM disk from the Oacle ASM diskgeoup you may not be able to cause it in the same oranother Oracle ASM diskgroup, This is also the case if you decide bo install Oracle Grid Infiastructire using cisks for COCR that have-been used ester, ‘The easiest way to solve this problem is te format the disk with NTFS and then use the ‘dokpast ublity to eemove all partitions and seceeate thom, shfter this 'asmea’ can be used to puta new label on the disk.Working with Oracle ACFS Checking Oracle ACFS Oaele Grid Infrastructure 11.2 includes a utility to perform offline file system checks. Te unmeunt an mounted Oracle AGFS volume all processes with open handles cn that file system must be stopped. ‘This includes the services '‘OracleService' and ‘Oracle VES Writer Secvice! and all other progtams (command prompts, tail, eo=) that may have open handles on that file system on all eluster node. © Shotdown your database that is oun out ofthe Orde 11.2RDEMS Home you ‘wantta check Eg arvetl stop database -d 4) Retrieve the virtual device name of the Osack ASM volume housing the Cracle AGES file system. Egacisutil info fs orthe names of the virtual Omele ASM volumes you winte down during theinstallation. + Btop the file system anelwait for unmountto complete. Eg srvctl stop file system -d \\.\asm-c11-210 Tfan essorie thrown the reason can be one of the following: 1) Unmounting the file system was too slow. Run ‘srvctl stop file system." command again, 2) You stil have come files open. Check whether a Windows service is still sunning ox whether another progam is still accessing the fle systorn. Fags & command promot where you dida'cd! to a diesetocy on Oacle ACPS cequisss a handle, So eithes'ed! cut of the dicectocy o: close the command prompt Ee chusers\eraclessrvct1 stop database -d cil c:\Users\oraciesacfsutii info f= C1 \ORAELE\ C11 ACES Verein: 11.2.0.2.0 flags: MeuntPeint, Available mount time: Wed aug 17 15133238 2011 wolumest 1 total size: 25760802776 total free: 21599958736 Primary velume: asm-cl1-219 abel flags: Primary, Avatlanle ADM en-disk version: 39.0 allocation unit: 4096size: 25769803776 frees 21699988736 AIM diskgreup ACES AIVM resize increment: 268428456 ADYM redundancy: unprotected ADWM stripe columns: 4 ADUM stripe width. 31072 number ef snapshote: 0 snapshot space usage: 0 cr\users\oractercrvoti step fie system a \\.\aem-o11-210 PRCR-1065 + Failed to stop recoures ara.aets.cll-aets SRE—5010: Agent "c:\GRACLE\GRID\11202 \bin \orarsotagent exe" timed eut starting process Set \ORACLE\GRID\11202\Bin \act ssangLefamount bat* for actien Mgtop"! details at "(:ecswoog09:)* in "et \ORACLE\GRID\11202\1og ora cx3 vi \agent \eesd\ou arectagent\orar eotagent. leg" CRS-S017! The resource sction ‘ora.scfs.cll.acts stop" encountered the folloxing error: {:CLSWOODDS:}UtLLs:exectnd aborted ORS-2675: Stop of ‘ora.acts.cll.acfs’ on ‘oracs3vl' failed eRS-2678: ‘ora.acfe.cll.acfs’ on ‘oracxivl" has experienced an unrecoverable failure trying again... ci\tsers\eraclessrvetl step file system -d \\.\asm-cl1-210 PECC-1O16 ¢ acfa-cll was alraady stopped PECR-1008 : Resource ora.acts-cll.acts is already stapped Users\aracle> If the file system was unmounted successfully check it with the ‘acfschledst' utility. Calling acfichiedsk without any pammeters displays a list of options. ci\Users\eraciesaefsenkask \\.\asm—cl1-210 version 1L.2.0-2.0 Geeesevers pace: y ceeressers Oracle ASM Cluster Pile system (Agro) on-Disk structure Werstent 39.0 ACES file system created at: Thu Aug 11 13133105 2011 checking primary file system Files checked in primary file system: 858 Files checked in primary file system: 1008 Checking if any files are orphaned. O crphans foundChecker completed with no errors + Restart the file syrtom. Cu\Ueers\ovaclerarvetl start file system -d \\.\aem-ell-210 © Wait for 34 minotes and check if each cluster node did mount the file system in the Windows Explorer before you restart your database instances, Backup and Restore Orcle ACFS does not support backup tools that rely on Microsoft Windows Volume Shadow Copy Secvices. Other backup tools that do not require VSS should wee fine with Ouuele ACFS. We recommend to ereate a full backup off un Oracle ACFS volume (which basically just holds Oracle 11.2 RDBMS Homes, Oracle ADR specific files andl logs seca by BRrTett) afteciaitial installation and confipusation and after each leg, after patch, Before you statt patching Oracle software located on Orarle ACFS you must shutdown the database that is un from that Ciacle RDBMS Home and to unmount and check the file systesn, If the dile system has eccoss — whieh is very unlileely -we recommend te secesate it and te restore the latest backup. Configuring BR*Tools for Oracle ASM SAP has extended the functionality of BR*T'ools to support Oracle ASM, Place the ‘nite sid> sap configucation files for archivelog backup and database backup in the ‘sappeof disectory. Note that you need specific configusstion files for each cluster nade on Which you oun backupe as the initsid>.cap configuration files need dffesent values for parameter 'aem_oma_sid' on cach node Example init §" uncompreas_cnd = "uncompress -c $8 > $* compress dir = c:\oracle\ci1 \sapreorg arshive_funstion — save_delete archive copy_dir = c:\oracle\C11\sapbackup archive_stags_dir - cf\oracle\C1l\sapbackup tape_copy_omd — cpio disk_copy_emd = copy stage_copy_oud - rep pipe_copy_emd = rsh epic_flags = -ovs spic_in_flaas - -iuvB cpio_disk_flags = -pdeu rman_channels = 4 rman_filesperset — 1 _rman_seas_stmt = ("alter session set optimizer _mode=RULE"} zman_send = { "channel sbt_1 ‘NSE ENV=(NSR_SERVER=xxbackapp2— s¢,USR_CLIENT=oracxSvi, NSK_DATA VOLUME POOL=INES Oraclei 4 Woaks, NSR_GROUP-2200xxbackupOZanl oral] *", “channel sbt_? ‘NSR_PNV=(NSR_SPRVER=xxbackapp02-se, NSR_CLTENT= oraex1,NSR_DATR_VOLUME_POOL-IDES Oracloz 4 Weaks, NSR_GRGUP-2200xxbackupZanl oraz] *", "Channel sbt_3 'NSE_SNW=(NSR_SERVER=ccbackappO2-se, NSR_CLIENT= oraex2¥L,NSR_DATA VOLUME POOL-IDES Oracle? @ Weeks, NSR_GROUP=2200xebackupo2sn0 Lora3)"", "Channel sbt_d ‘NSe_stW=(NSR_SERVER-cxcbackappO2-se, NSR_CLIENT= oragxd1,NSR_OATR_VOLUME_POOL-IDRS Oracle4 4 Weeks, NSR_GRGUP=2200xxbackup02snlllorad] **1 rewind = "mt -£ $ rewind” @- "mt -£ § offline” tape_po: mt -£ § Esf $* tape_size = 1006 exe parallel = 0 tape_eddress = fdew/rmt/Omn tape_address_rew = fdev/xmt/om volume_archive = (SIDAO1, SIDAO2, SIDAQ3, SIDA0d, SIDAO5, SIDAO6, SIDAOT, STDADS, SIDAOS, SIDA1O, SIDAIL, STDAL2, SIDA13, SIDA14, SIDAIS, SIDAI6, SIDA17, SIDA1@, SIDAIS, SIDA20, SIDAZ1, SIDA22, SIDAZ3, SIDAZ4, SIDA25, SIDAZ6, SIDAZ7, STDAZ8, STDAZS, STDR30) volume backup = (SIDBO1, SIDBO2, SIDBO2, SIDBO4, SIDBOS, SIMBOG, SIDBOT, SIDBOS, STDBOS, SIDB10, SIDB11, STMB12, SIDB13, sippid, srpsi8, SIDEié, s1pBi?, SIDB1a, s1DBi9, srpB30, SrDR2i, SIDB22, SIDB23, SIDB24, SIDE2S, SIDBZé, SIDB2?, SIDEZ2, SIDE29, STDB30) expir_poriod - 14 tape_oae_count = 100 parallel_instances = { Cilire:\oracle\cii\ii2028c1iL, c12ie:\oracle\el1 112024112, C113:e:\Oracle\CLN 1120280113} aam_ora gid = +ASML asm_ora_howe = C:\oracle\GRID\11202 agm_root_dir = ASM SAP Standalone Gateway Invercke to be able to ma automated database and archivelog backups fram transaction "DB1G install a SAP Standalone Gateway on each of the cluster nades and extract the atest BRY Tools bundle, Please refer te the Fallewing SAP notes ferrorn information: Eater DBA Gorkpit: Planning calendsr and remote Orade databases #853610 ‘Configuiing transaction DB13C for Oracle HioBca Ourrview of DBA Cockpit for OracleTesting Failures One of the most important things before going into production with a new cluster environmentis testing. Only by testing the behaviour of the cluster by simulating chardsure and software failures ensures that it will resct as expected when a real failure occurs The following tera sections descsibe some essential tests [tis highly recommended :not to go inte production if those tests do-not show the expected result, “The list is not complete and customess may think about additional tests that have to be done fortheis specific cluster enmiccewnent and HA sequicements Hardware Failures Power Leos ox Complete Failure of Cluster Node. Action One of the mostlikely failures that may oocuris 8 crash of s cluster node, Simulate a crash ‘by issuing a “hard power-off for one of the cluster nodes Expected Behaviour One of the remaining nedes should doa crash secovesy fos the cashed node. on the service definitions and theic thlover policy the secvices may be activated on one of the remaining nodes, The whole failorer process may take some minutes. To vetify whether the failover process is finished nun 'crsctl status resource -t!, Check that one of the remaining nodes finished the cmsh recovery by viewing the slezt-logs, Finally the YIP of the failed chister node and all services should be available as defined Action Power on the shutdown cluster node Expected Behaviowr ‘The VIP of the started cluster node that was serviced by one of the remaining cluster nodes should fall back to the rebooted cluster node. All defined Osacle instances (if more than one database instance was cunning on the node befuse) sheuld stast automatically All services that fell over to other cluster nodes should stay there. They have to be moved Manually to their cocresponding nodes using ‘srvetl celecate service... Gracie Interconnect Failures Cluster rade has leet its connection to the exitch,Action Disconnect one clusternode from the switch by unplugging all network cables of the (Oracle Interconnect. If you have NIC teaming active (redundant interconnect) also check what happens when you unplug only one network cable, Expected Behaviour Ina two neds RAC cluster ene of beth nodes should crush, [na these oranoce node RAC cluster the node where you did unplug the netwodk: cables should crash. In both situations the VIP should move to one of the remaining nodes, One of the remaining instances should do the crash recovery for the crashed nade. Services should mave to the remaining codes as defined (Multiple) Switch Failures with all nodes losing their connection Action Tum off of disable the network striteh of the Ozele Interonnect Expected Behaviour Regardless of the number of chister nades all nades except one should crash. The remaining node should take over the VIP's of the emshed nedes, perfoom a emsh recovery and provide the database services as defined Storage Failures (One cluster node loses its stomge connection, Action Unplug one ef the duster nedes from the storage, IF you have redundant paths te the storage also check what happens shen you unplug only one (fberchannel-} cable Expected Behaviour ‘The node should ceash without automatic ssboot, Automatic wbeot may be a configuration eption in the BICS of the node. All cluster nodes Tose their storage connection, Action ‘Unplug all cluster nedes from the storage orm off all switches to the storage. Expocted Behaviour All cluster nodes should crash, Afber reconnecting the nodes to the storage and rebooting all he nodes the datsbese should be available as expected.Software Failures: Shutdown abort, Action Shutdown an Oracle instance with “shutdown aboxt”, Expected Behaviour “The cluster node chould stay up and sunning One of the semaining Omucle instances should do the ccash secovecy. All services of the stopped instance should anove to the previously defined remaining instances, 'crecil status resource -! wtlity should display “OFFLINE” as the target state for the stepped instance. Kill sraclevexe: Action Kull scacleexe with the Windows Tusk Manages, Expected Behaviour ‘The cluster node should stay up snd sunning One of the remaining Oracle instances should do the crash recovery. Al services of the killed instance should move to the -provigusly defined remaining instances, CRS should start the killed instance automaticall « ‘All secvices that moved to other instances should stay there. They have two be relocated smarmally. SAP Application Wortdoad ‘The most impostant tests to can are workload tests where customers should try to mun their SAP applications under simulated business conditions. Testing the perfomance of com tansactions and batch jobs will halp to identify and ceselve bettlenscks and configuration issues befor going into production with your new cluster environment ‘These tests can be combined with some of the failure test scenarios descnbed cartier. Check if local update dispatching (descsibed in the next chapter) is active and wodking as expected with “dpmonene” ocSAP Management Consele, Both tools show the number of times an update proses was active Check if database service relacation works as scpected and every SAP application server can commect to its gervice,Additional Performance Recommendations “Using ASSM tablespaces is mandatozy. Ifyou still have Non-ASSM tablespaces you must wnigatte youctables to ASSM tablespaces first Use local update dispatching for SAP update-processes Local Update Dispatching, ‘The SAP application performs updates on the database asynchronously through 'update- ‘prncesses' A dislng wark process inserts an update job inte tables VBHDR, VBDATA, YVBMOD and posts the dispatcher process on the central instance to select an-update- -psccess to actually pedfoon the update job. This update-psocess may be connected to- asothes RAC instance, simalacly te the dialog process, If this is the case, the update -psccess would read the data just wsitten by the dlalog-process. To satisfy this seadl request, all sequised database blocks have to be shipped over the interconnect from the instance where the dats was incerted te the instance where the data has to be read, This approach ‘produces unneressary additional data load and can be avoided. In ordee to avoid these ‘non-local updates’, itis eecemmended that you © have a mumber of upeate pincesses on each SAP instance * tum off update-dispatching and use the local update-precesses ‘To tum off update dispatching, set risp/vb_dispatching to 0 as described in SAP note #02203. Alse, set caispy/vb_ name to the name of the local instance (e.g. edisp/vb_eame = app!_Cl1_d0},ORACLE ean crataFua Appusten cure tg Fokaoe on Mr Wine Sine Pabne 2 Danse eas eu oretg sui ate Beh, an ere mem orpenten eon one st oneefaoe usa reeounsese: OB | tee cretn Copptgr 201 ocele eau angus Agr eon The cea eons oa pap O°) aha Contant we ata stage we Tis sneer warasto bean fr coca oh ‘eerie sete, nha aegeeederd re nm Hoi pets a mae Yt feet ttt nares Pangaea arbor eyo pc He aia oe ena a ty toyz cunt Te ciara oo pce eed a6 as, aad of meceaL Re arypaeca, eo tbe DME, (au anda atest ars as cee soni ete Ome ep be Italic retarted ana hd ome. SPARE tars a cad nr ean see aruaicor inal nina SPAR MD Cee ND panne ND yee Incararce mcert Wnit ten tan vce UR Angin Fae mer ang ua cert rdvara and Software, Engineered t Work Toyatber