Oracle Database 10g: Real Application Clusters

Volume 2 • Student Guide

D17276GC10 Edition 1.0 December 2004 D40140

Authors James Womack Jean-Francois Verrier Technical Contributors and Reviewers Troy Anthony Harald van Breederode Bill Bridge Michael Cebulla Carol Colrain Jonathan Creighton Joel Goodman Yunrui Li Vijay Lunawat Paul Manning John McHugh Erik Peterson Javier Seen Nitin Vengurlekar Publisher Joseph Fernandez

Copyright © 2004, Oracle. All rights reserved. This documentation contains proprietary information of Oracle Corporation. It is provided under a license agreement containing restrictions on use and disclosure and is also protected by copyright law. Reverse engineering of the software is prohibited. If this documentation is delivered to a U.S. Government Agency of the Department of Defense, then it is delivered with Restricted Rights and the following legend is applicable: Restricted Rights Legend Use, duplication or disclosure by the Government is subject to restrictions for commercial computer software and shall be deemed to be Restricted Rights software under Federal law, as set forth in subparagraph (c)(1)(ii) of DFARS 252.227-7013, Rights in Technical Data and Computer Software (October 1988). This material or any portion of it may not be copied in any form or by any means without the express prior written permission of Oracle Corporation. Any other copying is a violation of copyright law and may result in civil and/or criminal penalties. If this documentation is delivered to a U.S. Government Agency not within the Department of Defense, then it is delivered with “Restricted Rights,” as defined in FAR 52.227-14, Rights in Data-General, including Alternate III (June 1987). The information in this document is subject to change without notice. If you find any problems in the documentation, please report them in writing to Education Products, Oracle Corporation, 500 Oracle Parkway, Box SB-6, Redwood Shores, CA 94065. Oracle Corporation does not warrant that this document is error-free. All references to Oracle and Oracle products are trademarks or registered trademarks of Oracle Corporation. All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.

Contents
I Introduction Overview I-2 What Is a Cluster? I-3 What Is Oracle Real Application Clusters? I-4 Why Use RAC? I-5 Clusters and Scalability I-6 Levels of Scalability I-7 Scaleup and Speedup I-8 Speedup/Scaleup and Workloads I-9 A History of Innovation I-10 Course Objectives I-11 Typical Schedule I-12 Architecture and Concepts Objectives 1-2 Complete Integrated Cluster Ware 1-3 RAC Software Principles 1-4 RAC Software Storage Principles 1-5 OCR Architecture 1-6 RAC Database Storage Principles 1-7 RAC and Shared Storage Technologies 1-8 Oracle Cluster File System 1-10 Automatic Storage Management 1-11 Raw or CFS? 1-12 Typical Cluster Stack with RAC 1-13 RAC Certification Matrix 1-14 The Necessity of Global Resources 1-15 Global Resources Coordination 1-16 Global Cache Coordination: Example 1-17 Write to Disk Coordination: Example 1-18 RAC and Instance/Crash Recovery 1-19 Instance Recovery and Database Availability 1-21 Efficient Inter-Node Row-Level Locking 1-22 Additional Memory Requirement for RAC 1-23 Parallel Execution with RAC 1-24 Global Dynamic Performance Views 1-25 RAC and Services 1-26 Virtual IP Addresses and RAC 1-27 Database Control and RAC 1-28 Summary 1-29

1

iii

2

RAC Installation and Configuration (Part I) Objectives 2-2 Oracle Database 10g RAC Installation: New Features 2-3 Oracle Database 10g RAC Installation: Outline 2-5 Preinstallation Tasks 2-6 Hardware Requirements 2-7 Network Requirements 2-8 RAC Network Software Requirements 2-9 Package Requirements 2-10 hangcheck-timer Module Configuration 2-11 Required UNIX Groups and Users 2-12 The oracle User Environment 2-13 User Shell Limits 2-14 Configuring User Equivalency 2-15 Required Directories for the Oracle Database Software 2-17 Linux Kernel Parameters 2-19 Cluster Setup Tasks 2-21 Obtaining OCFS 2-22 Installing the OCFS RPM Packages 2-23 Starting ocfstool 2-24 Generating the ocfs.conf File 2-25 Preparing the Disks 2-26 Loading OCFS at Startup 2-27 Mounting OCFS on Startup 2-28 Using Raw Partitions 2-30 Binding the Partitions 2-31 Raw Device Mapping File 2-33 Installing Cluster Ready Services 2-35 Specifying the Inventory Directory 2-36 File Locations and Language Selection 2-37 Cluster Configuration 2-38 Private Interconnect Enforcement 2-39 Oracle Cluster Registry File 2-40 Voting Disk File 2-41 Summary and Install 2-42 Running the root.sh Script on All Nodes 2-43 Verifying the CRS Installation 2-44 Summary 2-46 Practice 2: Overview 2-47

iv

3

RAC Installation and Configuration (Part II) Objectives 3-2 OUI Database Configuration Options 3-3 Install the Database Software 3-4 Specify File Locations 3-5 Specify Cluster Installation 3-6 Select Installation Type 3-7 Products Prerequisite Check 3-8 Select Database Configuration 3-9 Check Summary 3-10 The root.sh Script 3-11 Launching the VIPCA with root.sh 3-12 VIPCA Network Interface Discovery 3-13 VIP Configuration Data and Summary 3-14 Installation Progress 3-15 End of Installation 3-16 Database Preinstallation Tasks 3-17 Creating the Cluster Database 3-19 Node Selection 3-20 Select Database Type 3-21 Database Identification 3-22 Cluster Database Management Method 3-23 Passwords for Database Schema Owners 3-24 Storage Options for Database Files 3-25 Database File Locations 3-27 Flash Recovery Area 3-28 Database Components 3-29 Database Services 3-30 Initialization Parameters 3-31 Database Storage Options 3-32 Create the Database 3-33 Monitor Progress 3-34 Manage Default Accounts 3-35 Postinstallation Tasks 3-36 Patches and the RAC Environment 3-37 Inventory List Locks 3-38 Summary 3-39 Practice 3: Overview 3-40

v

4

RAC Database Instances Administration Objectives 4-2 The EM Cluster Database Home Page 4-3 Cluster Database Instance Home Page 4-5 Cluster Home Page 4-6 The Configuration Section 4-7 Operating System Details Page 4-8 Performance and Targets Pages 4-9 Starting and Stopping RAC Instances 4-10 Starting and Stopping RAC Instances with EM 4-11 Starting and Stopping RAC Instances with SQL*Plus 4-12 Starting and Stopping RAC Instances with SRVCTL 4-13 RAC Initialization Parameter Files 4-14 SPFILE Parameter Values and RAC 4-15 EM and SPFILE Parameter Values 4-16 RAC Initialization Parameters 4-18 Parameters Requiring Identical Settings 4-20 Parameters Requiring Unique Settings 4-21 Adding a Node to a Cluster 4-22 Adding a Node to an Existing Cluster 4-23 Adding the RAC Software to the New Node 4-25 Reconfigure the Listeners 4-27 Add an Instance by Using DBCA 4-28 Deleting Instances from a RAC Database 4-29 Node Addition and Deletion and the SYSAUX Tablespace 4-31 Quiescing RAC Databases 4-32 How SQL*Plus Commands Affect Instances 4-33 Administering Alerts with Enterprise Manager 4-34 Viewing Alerts 4-35 Blackouts and Scheduled Maintenance 4-37 Summary 4-39 Administering Storage in RAC Part I Objectives 5-2 What Is Automatic Storage Management? 5-3 ASM: Key Features and Benefits 5-4 ASM: New Concepts 5-5 ASM: General Architecture 5-6 ASM Instance and Crash Recovery in RAC 5-8 ASMLibs 5-9 Oracle Linux ASMLib Installation: Overview 5-10
vi

5

Oracle Linux ASMLib Installation 5-11 ASM Library Disk Creation 5-13 ASM Administration 5-15 ASM Instance Functionalities 5-16 ASM Instance Creation 5-17 ASM Instance Initialization Parameters 5-18 RAC and ASM Instances Creation 5-19 ASM Instance Initialization Parameters and RAC 5-20 Discovering New ASM Instances with EM 5-21 Accessing an ASM Instance 5-22 Dynamic Performance View Additions 5-23 ASM Home Page 5-24 ASM Performance Page 5-25 ASM Configuration Page 5-26 Starting Up an ASM Instance 5-27 Shutting Down an ASM Instance 5-28 ASM Administration 5-29 ASM Disk Group 5-30 Failure Group 5-31 Disk Group Mirroring 5-32 Disk Group Dynamic Rebalancing 5-33 ASM Administration Page 5-34 Create Disk Group Page 5-35 ASM Disk Groups with EM in RAC 5-36 Disk Group Performance Page and RAC 5-37 Create or Delete Disk Groups 5-38 Adding Disks to Disk Groups 5-39 Miscellaneous Alter Commands 5-40 Monitoring Long-Running Operations Using V$ASM_OPERATION 5-42 ASM Administration 5-43 ASM Files 5-44 ASM File Names 5-45 ASM File Name Syntax 5-46 ASM File Name Mapping 5-48 ASM File Templates 5-49 Template and Alias: Examples 5-50 Retrieving Aliases 5-51 SQL Commands and File Naming 5-52 DBCA and Storage Options 5-53 Database Instance Parameter Changes 5-54

vii

Summary 5-56 Practice 5 Overview 5-57 6 Administering Storage in RAC Part II Objectives 6-2 ASM and SRVCTL with RAC 6-3 ASM and SRVCTL with RAC: Examples 6-4 Migrating to ASM: Overview 6-5 Migration with Extra Space: Overview 6-6 Migration with Extra Space: Example 6-7 Tablespace Migration: Example 6-11 Migrate an SPFILE to ASM 6-12 ASM Disk Metadata Requirements 6-13 ASM and Transportable Tablespaces 6-14 ASM and Storage Arrays 6-15 ASM Scalability 6-16 Redo Log Files and RAC 6-17 Automatic Undo Management and RAC 6-18 Summary 6-19 Practice 6 Overview 6-20 Services Objectives 7-2 Traditional Workload Dispatching 7-3 Grid Workload Dispatching 7-4 What Is a Service? 7-5 High Availability of Services in RAC 7-6 Possible Service Configuration with RAC 7-7 Service Attributes 7-8 Service Types 7-9 Creating Services 7-10 Creating Services with DBCA 7-11 Creating Services with SRVCTL 7-13 Preferred and Available Instances 7-14 Everything Switches to Services 7-15 Using Services with Client Applications 7-16 Using Services with Resource Manager 7-17 Services and Resource Manager with EM 7-18 Services and Resource Manager: Example 7-19 Using Services with Scheduler 7-20

7

viii

Services and Scheduler with EM 7-21 Services and Scheduler: Example 7-23 Using Services with Parallel Operations 7-24 Using Services with Metric Thresholds 7-25 Changing Service Thresholds Using EM 7-26 Services and Metric Thresholds: Example 7-27 Service Aggregation and Tracing 7-28 Cluster Database: Top Services 7-29 Service Aggregation Configuration 7-30 Service Aggregation: Example 7-31 The trcsess Utility 7-32 Service Performance Views 7-33 Managing Services 7-34 Managing Services with EM 7-36 Managing Services: Example 7-38 Summary 7-39 Practice 7 Overview 7-40 8 High Availability of Connections Objectives 8-2 Types of Workload Distribution 8-3 Client Side Connect-Time Load Balancing 8-4 Client Side Connect-Time Failover 8-5 Server Side Connect-Time Load Balancing 8-6 Fast Application Notification: Overview 8-7 Fast Application Notification Benefits 8-9 FAN-Supported Event Types 8-10 FAN Event Status 8-11 FAN Event Reasons 8-12 FAN Event Format 8-13 Server-Side Callouts Implementation 8-14 Server-Side Callout Parse: Example 8-15 Server-Side Callout Filter: Example 8-16 Configuring the Server-Side ONS 8-17 Configuring the Client-Side ONS 8-18 JDBC Fast Connection Failover: Overview 8-19 JDBC Fast Connection Failover Benefits 8-20 Transparent Application Failover: Overview 8-21 TAF Basic Configuration: Example 8-22 TAF Preconnect Configuration: Example 8-23

ix

TAF Verification 8-24 FAN Connection Pools and TAF Considerations 8-25 Restricted Session and Services 8-26 Summary 8-27 Practice 8 Overview 8-28 9 Managing Backup and Recovery in RAC Objectives 9-2 Protecting Against Media Failure 9-3 Configure RAC Recovery Settings with EM 9-4 Configure RAC Backup Settings with EM 9-5 Log History Records 9-6 Initiate Archiving 9-7 Archived Log File Configurations 9-8 RAC and the Flash Recovery Area 9-9 Oracle Recovery Manager 9-10 Configuring RMAN 9-11 RMAN Default Autolocation 9-12 User-Managed Backup Methods 9-13 Offline User-Managed Backup 9-14 Online User-Managed Backup 9-15 Channel Connections to Cluster Instances 9-16 Distribution of Backups 9-17 One Local Drive CFS Backup Scheme 9-18 Multiple Drives CFS Backup Scheme 9-19 Non-CFS Backup Scheme 9-20 RAC Backup and Recovery using EM 9-21 Restoring and Recovering Redo Log Files 9-22 Parallel Recovery in Real Application Clusters 9-23 Fast-Start Parallel Rollback in Real Application Clusters 9-25 Managing OCR: Overview 9-26 Recovering the OCR 9-27 Recovering the Voting Disk 9-28 Summary 9-29 Practice 9: Overview 9-30

10 RAC Performance Tuning Objectives 10-2 CPU and Wait Time Tuning Dimensions 10-3 RAC-Specific Tuning 10-4 Analyzing Cache Fusion Impact in RAC 10-5

x

Typical Latencies for RAC Operations 10-6 Wait Events for RAC 10-7 Wait Event Views 10-8 Global Cache Wait Events: Overview 10-9 2-way Block Request: Example 10-11 3-way Block Request: Example 10-12 2-way Grant: Example 10-13 Considered “Lost” Blocks: Example 10-14 Global Enqueue Waits: Overview 10-15 Session and System Statistics 10-16 Most Common RAC Tuning Tips 10-17 Index Block Contention Considerations 10-19 Oracle Sequences and Index Contention 10-20 Undo Block Considerations 10-21 High-Water Mark Considerations 10-22 Cluster Database Performance Page 10-23 Cluster Cache Coherency Page 10-25 Database Locks Page 10-26 Automatic Workload Repository: Overview 10-27 AWR Tables 10-28 AWR Snapshots in RAC 10-29 Generating and Viewing AWR Reports 10-30 AWR Reports and RAC: Overview 10-31 Statspack and AWR 10-33 Automatic Database Diagnostic Monitor 10-34 ADDM Problem Classification 10-35 RAC-Specific ADDM Findings 10-36 ADDM Analysis: Results 10-37 ADDM Recommendations 10-38 Summary 10-39 Practice 10: Overview 10-40 11 Design for High Availability Objectives 11-2 Causes of Unplanned Down Time 11-3 Causes of Planned Down Time 11-4 Oracle’s Solution to Down Time 11-5 RAC and Data Guard Complementarity 11-6 Maximum Availability Architecture 11-7 RAC and Data Guard Topologies 11-8 RAC and Data Guard Architecture 11-9
xi

Data Guard Broker (DGB) and CRS Integration 11-11 Data Guard Broker Configuration Files 11-12 Hardware Assisted Resilient Data 11-13 Rolling Patch Upgrade Using RAC 11-14 Rolling Release Upgrade Using SQL Apply 11-15 Database High Availability Best Practices 11-16 Extended RAC: Overview 11-17 Extended RAC Connectivity 11-18 Extended RAC Disk Mirroring 11-19 Additional Data Guard Benefits 11-20 Using Distributed Transactions with RAC 11-21 Using a Test Environment 11-22 Summary 11-23 Appendix A: Practices Appendix B: Solutions Appendix C: RAC on Windows Installation Appendix D: Add Node Appendix E: Remove Node

xii

________________

Appendix A Practices
________________

Conventions

For most of these practice exercises, you will be using various Oracle tools (OUI, DBCA, EMCA, and so on), and telnet sessions. Your instructor will give you details about your nodes and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. As indicated by your instructor, you will be paired with another student, and both of you will access the same cluster. We hope that you will coordinate the lab work with your coworker, and that you will form a great team. Each cluster is composed of two machines with names similar to the following: ed-toraclin1a, and ed-toraclin1b. In the above example, 1a represents the first node (a) of the first cluster (1), and 1b represents the second node (b) of the first cluster (1). To ease the connection to your cluster during certain labs, you will also be able to use a simplified version of the machine names: rac1a, rac1b, ... We highly recommend that you do not hesitate to look at the proposed solutions before you start a particular lab. When possible, we also provide a set of lab and solution scripts that can help you resolve a lab. If you get stuck on some of the questions, and you want to use a script, make sure you execute it on the right node of your cluster, and connected as the right user. The lab and solution scripts are located either in the $HOME/labs directory or $HOME/solutions directory of each node, and each solution script’s name as the following form: lab | sol_<lesson number>_<lab number>_<step number><sequence letter>_<n1 | n2>_<w1 | w2 | …>_<DB name>, where: • n1 represents the first node, and n2 represents the second node. • wn represents the telnet window n. For example, sol_08_02_04a_n1_w1_RDBA.sh is a shell script that should be executed first on the first node of the first cluster during the foorth step of the second lab of the eight’s lesson. In addition, whenever this guide references a database name like RDBA, you should replace the last letter with your corresponding letter. Note: If you want to execute the proposed solution scripts, make sure that you execute them from the $HOME/solutions directory of the oracle user, or from the $HOME/labs directory of the oracle user.

Oracle Database 10g: Real Application Clusters A-2

Practice 02-1: Prerequisites to the Installation Process For most of these practice exercises, you will be using various Oracle graphical tools (OUI, DBCA, EMCA, and so on), and telnet sessions. Your instructor will give you details about your node names and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Using a telnet session, connect as user root to your first RAC node. Make sure that you have enough memory, swap space, and temporary space to install your CRS and RAC software on both nodes. 2. Using the redhat-config-network command, check your network configuration. Make sure that you have at least two correctly configured network adaptors. Repeat the check on both nodes. 3. Make sure that you can reach each node from the other one. Do this for both nodes. 4. Make sure that the required OS packages are loaded with a sufficient compatibility version. Check for this on both nodes. 5. Make sure that the hangcheck timer is started and properly configured on each node. 6. Make sure that the oracle and nobody users and the oinstall and dba groups exist. Check for this on both nodes. 7. Edit the.bash_profile file of the oracle user, and set the environment variables. Do this on both nodes. 8. As the root user, make sure that you have the following lines in your /etc/security/limits.conf file: • * soft nproc 2047 • * hard nproc 16384 • * soft nofile 1024 • * hard nofile 65536 In addition, check to see that you have the following entry in your /etc/pam.d/login file: session required /lib/security/pam_limits.so. Check the values above on the second node also.

Oracle Database 10g: Real Application Clusters A-3

9. Make sure that the /etc/hosts.equiv file contains the public node names for each node in your cluster. Repeat this step on the second node. 10. Connect as user oracle to the first node. Check that you can use the rsh command to connect to the other node of your cluster without being prompted for a password. Do this from the second node also. 11. Connect as user root on the first node. Change directory to /u01, and create the app directory. Make sure that it is owned by oracle and the group is oinstall. Change the effective user to oracle, and create the /u01/app/oracle directory. Do this on both nodes. 12. Connect as user root on the first node. Check that the Linux kernel parameters are set to at least the following values: • semmsl 250 • semmns 32000 • semopm 100 • semmni 128 • shmall 2097152 • shmmax Half the size of physical memory • shmmni 4096 • file-max 65536 • ip_local_port_name 1024-65000 When finished, do the same for the second node.

Oracle Database 10g: Real Application Clusters A-4

Practice 02-2: OCFS Installation Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Connect as user root on the first node, and change your directory to the /stage directory. Make sure that you have a subdirectory called ocfs. Then change your directory to the ocfs subdirectory, and see that you have the following .rpm files. These files are downloaded from the OTN Web site, and you may need different ones depending on your machine. Check the OCFS Web pages on the OTN Web site for more information: • ocfs-EL • ocfs-support • ocfs-tools When finished, do the same for the second node. 2. Use the rpm command to install the following OCFS packages from the /stage/ocfs directory: • ocfs-support • ocfs-n-EL (single processor module) or ocfs-n-EL-smp (multiprocessor module) • ocfs-tools Install the ocfs-support rpm first, the OCFS kernel module rpm second, and the ocfstools rpm third. Note in the example, the single processor OCFS kernel module is used. When finished, repeat the steps on the second node.
3. Launch the ocfstool utility as the root user. Use it to generate the OCFS configuration file, /etc/ocfs.conf. Then exit the utility, and check the ocfs.conf file that is generated.

When done, perform the same steps on the second node. 4. Ensure that OCFS is automatically started by the system at the next reboot. Repeat this step on the second node. 5. In order that the two nodes do not need to be rebooted, manually execute the load_ocfs command as the root user to load the OCFS module. Note that you may get warnings depending on the kernel version used by your Linux system. If the kernel version you are using is greater than the corresponding OCFS rpms, a forced load is done automatically. However, the reverse is not true. Repeat this step on the second node. 6. Using the fdisk command, ensure that a partition called sda1 exists on your shared disk, /dev/sda. The fdisk command must be run as the root user. Note: There is no need to do this on the second node because it is the shared disk! 7. As the root user, create a new directory called /ocfs, and make sure that it is owned by the oracle user and the associated group is dba. Adjust the permissions appropriately. Repeat these actions on the second node.

Oracle Database 10g: Real Application Clusters A-5

8. Next, the /dev/sda1 slice must be OCFS formatted by using the mkfs.ocfs command. This step must be performed from the first node only! 9. The OCFS volume must be mounted on both nodes. Again, you can use ocfstool to do this. This step must be performed on both nodes. 10. On both nodes, make sure that the /ocfs directory is mounted automatically during the next reboot of the system. On both nodes, edit the /etc/fstab file to mount the OCFS volume whenever the system boots. When you have finished this step on the first node, go to your second node and repeat step 10. 11. Check that /dev/sda1 is mounted on /ocfs correctly. When finished, perform the same for the second node.

Oracle Database 10g: Real Application Clusters A-6

Practice 02-3: CRS Installation Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate. In this practice, you will install Oracle Cluster Ready Services on both nodes. The CRS software is located in the /stage directory under the crs10g_10.1.0.2.0 subdirectory. The CRS software is installed by using the Oracle Universal Installer (OUI). This is done on the first node only. 1. Before the OUI can be started, prepare the system to enable nonroot users to redirect the display of graphical applications. 2. Go to the CRS software staging location, and start OUI as oracle.

Oracle Database 10g: Real Application Clusters A-7

Practice 03-1: Oracle RAC Database Software Installation
In this practice, you use the Oracle Universal Installer (OUI) to install the Oracle Database 10g RAC software on both nodes in your cluster. 1. Go to the database software staging location, and start the OUI. 2. Use the OUI to install the database software on both nodes in your cluster. 3. The VIP Configuration Assistant (VIPCA) is launched from the root.sh script on the first node. 4. After the VIP configuration finishes on the first node, run the root.sh script on the second node. 5. Before continuing with database creation, make sure that all the nodeapps are running.

Oracle Database 10g: Real Application Clusters A-8

Practice 03-2: Database Creation
In this practice, you create a RAC database by using the Database Configuration Assistant (DBCA).
1. Start the DBCA to begin the creation of the RAC database.

2. If DBCA reports an EM “Null argument” configuration error, you will have to correct this by running emca manually. 3. Check that all necessary nodeapps and services are online.

Oracle Database 10g: Real Application Clusters A-9

Practice 03-3: Identifying Components of Real Application Clusters For most of these practice exercises, you will be using Enterprise Manager Database Control. Your instructor will give you details about your nodes and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Connect as user oracle to the first node by using a telnet session. Determine the following: • List of processes associated with the instance on the first node. (For example, RDBA1, RDBB1, etc.) • List of processes running CRS • Notification Server processes When finished, perform the same for the second node.

2. Using the same telnet session, execute the ocrdump command. When executed, look at the generated file to determine the list of components that are maintained in the OCR.

3. On the first node, execute the following command: evmwatch -A -t "@timestamp @@". Create a second telnet session connected as user oracle on the first node. In the second session, kill the SMON process on node 1 (1 in this example) when finished, what do you observe in the first session? After a few moments, determine the list of processes running 1. What is your conclusion? Note: Once done, type [CTRL] [C] in the first session to stop the evmwatch command.

4. Create two SQL*Plus sessions connected as SYSDBA to each instance. Determine the following for each instance: • The database name • The instance name • The instance number • The thread • The instance status • The list of redo log groups and their corresponding thread and members • The undo tablespaces What do you observe when looking at the status of each instance?

Oracle Database 10g: Real Application Clusters A-10

Practice 04-1: RAC Database Instances Administration
1. In this practice, you will use the srvctl utility to: • Stop the cluster database • Start the first instance only, then bring up the second instance • Start the first instance only in MOUNT mode • Start the second instance and open the database

2. In this practice you will use EM Dbconsole to stop and start your database.

Oracle Database 10g: Real Application Clusters A-11

Practice 05-1: ASMLib Installation
Unless specified otherwise, you must log in as root on a terminal session through VNC on the first node of your cluster (rac1). Before starting this lab, execute the setup_asm_partitions.sh script located in your labs directory.
1. Change your current directory to /stage/asmlib. Make sure that the following RPMs are in

that directory: • oracleasm-support-1.0.2-1.i386.rpm • oracleasm-2.4.21-EL-1.0.0-1.i686.rpm • oracleasmlib-1.0.0-1.i386.rpm Use the rpm –i command to install each of the RPMs above in that order.

2. Use the /etc/init.d/oracleasm configure command to configure your ASMLib.

Make sure that oracle is the default owner of the driver, and that dba is the default group owner of the driver. Also, see that the driver is started during the boot. When done, check that the ASMLib driver is correctly loaded.

3. Use the fdisk /dev/sda command to look at the shared devices that are available to you. Then, use the /etc/init.d/oracleasm createdisk command to create six ASM disks called VOL1, VOL2, …, VOL6, respectively. For that, use the last six shared devices available to you.

4. Telnet to the second node (rac2), and connect as root. When on rac2, install and configure the ASMLib driver like you did previously on rac1.

5. Still connected as root on rac2, use the /etc/init.d/oracleasm scandisks command to make sure that the previously defined volumes are also visible from rac2. When finished, you can exit from your rac2 session.

Oracle Database 10g: Real Application Clusters A-12

Practice 05-2: ASM Instances Creation
1. Connected as user oracle from a terminal window on your VNC session, you now create ASM

instances in your cluster. Execute the dbca command. Make sure that on the Welcome screen, the Oracle Real Application Clusters Database option is selected. Then, click Next.
2. On the Operations screen, select the Create a Database option. When finished, click Next. 3. On the Node Selection screen, select both nodes by using the Select All button. Then, click Next. 4. On the Database Templates screen, select the General Purpose option. Then, click Next. 5. On the Database Identification screen, specify the Global Database Name. Then, click Next. 6. On the Management Options screen, deselect the Configure the Database with Enterprise Manager check box. When finished, click Next. 7. On the Database Credentials screen, select the Use the Same Password for All Accounts option, and specify and confirm the passward for the SYS user. Use oracle as the password. When finished, click Next. 8. On the Storage Options screen, select the Automatic Storage Management (ASM) option. Click Next. 9. On the Create ASM Instance screen, select the Create initialization parameter file (IFILE) option, and click ASM Parameters. On the ASM Parameters screen, set the asm_diskstring value to ORCL:VOL*. Then make sure that you specify oracle as the password for SYS when back to the Create ASM Instance screen. When finished, click Next. 10. In the warning dialog box, click OK to start the ASM instance creation process. 11. When on the ASM Disk Groups screen, create a new ASM disk group called DBG1 by using the first four candidate disks. Make sure that the disk group that you create uses the Normal redundancy policy. When finished, click Next. 12. When on the Database File Locations screen, click Cancel to cancel the session and exit from dbca. 13. Using terminal sessions, and still connected as oracle, make sure that both ASM instances are started. If

an ASM instance is down, set its corresponding SID, and use SQL*Plus to start it up. Do that from the corresponding node.

Oracle Database 10g: Real Application Clusters A-13

Practice 05-3: Discovering ASM Instances with Database Control 1. Make sure that the asm1_target_RDBA.xml file is in your labs directory on rac1, and that asm2_target_RDBA.xml is in your labs directory on rac2. See that their content is correct. 2. On both nodes, check that you do not have any ASM target already registered in Database Control. 3. Go back to your terminal window, and use the emctl tool to add the new target and stop the agent. Then use emctl to restart the agent and the DB console. Perform this on both nodes. 4. Log in again by using Database Control. Make sure that both ASM instances are now recognized as Database Control targets on both nodes. 5. Use Database Control from rac1 to create a new ASM disk group called FRAG1. This disk group is used to migrate the flash recovery area to ASM. Make sure that this disk group uses the remaining two candidate ASM disks, and is using the Normal redundancy policy.

Oracle Database 10g: Real Application Clusters A-14

Practice 05-4: Understanding ASM Automatic Rebalancing
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus.
1. Connected as user oracle in your terminal emulator window on rac1, make sure that your

ASM instance is started up, and list the processes associated to it. Then determine the characteristics of: • The mounted disk groups • The associated ASM disks • The associated ASM files
2. Connected as SYSDBA under SQL*Plus in another terminal emulator window, determine the list

of disk groups that are visible from your database instance. Then list the processes associated to your database instance. When done, create a new tablespace called TBSASM that is stored inside the ASM disk group DGROUP1, and which has only one 500 MB data file. When finished, determine the list of processes associated to your database instance again, and list the data files associated to your database. What do you observe?
3. Back on your ASM instance, using SQL*Plus, determine the number of bytes written on the

DBG1 disk group. Then use Database Control to determine the amount of disk space used by TBSASM on each ASM disk. What is your conclusion?

4. Using Database Control, drop the ASM disk called VOL4. Using Database Control, look at the

performance charts relating to disk groups. What are your conclusions?

5. On your database instance, connected as SYSDBA, execute the lab_05_04_05.sql script.

This script creates and populates a new table called T, which is stored in TBSASM. When executed, set timing statistics in your SQL*Plus session, and execute the following query: SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) FROM t;
6. Using Database Control, navigate to the Add Disks page. Execute the lab_05_04_06.sql script from your terminal window on rac1. As soon as the execution of the script is started, add the ASM disk, which was dropped at step four, to DBG1. What do you observe, and what are your conclusions?

Oracle Database 10g: Real Application Clusters A-15

Practice 06-1: Changing Your Online Redo Log File Characteristics
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. 1. Use Database Control to create two new redo log groups in your database. The two groups must pertain to the thread number three, and each group must have only one 10240 KB member called redo05.log and redo06.log, respectively. Make sure that the members are created in the /ocfs/oradata/RACDB directory.

2. Use Database Control to set the RDBB2.THREAD initialization parameter to 3 in SPFILE only. 3. Use the SRVCTL control utility to stop the RDBB2 instance, and start it up again. What happens and why? If necessary, fix the situation. 4. Revert to the original situation where RDBB2 was using the redo thread two, and destroy redo thread number three. Make sure that in the end both instances are up and running and managed by CRS.

Oracle Database 10g: Real Application Clusters A-16

Practice 06-2: Migrating Tablespaces to ASM
1. Connected as SYSDBA under SQL*Plus in your database instance on rac1, create a new

tablespace called TBSASMMIG. This tablespace must contain only one 10 MB file stored in your file system (not using ASM). Create a table called T2 stored in TBSASMMIG. Insert one row inside T2.
2. From your database instance, migrate TBSASMMIG to ASM storage. When done, check that the

migration is successful.
3. Using Database Control, determine the name of the files created in DBG1 from the ASM pages.

4. From your database instance, clean up your environment by dropping the TBSASMMIG tablespace,

including its contents and data file. Perform the same with the TBSASM tablespace. Also, remove the file system file that was originally created to store TBSASMMIG. When finished, verify by using the ASM pages of Database Control that the ASM files are removed.

Oracle Database 10g: Real Application Clusters A-17

Practice 07-1: Using Services in RAC
The first step in creating a service configuration is to plan services for each application or application function that is using Oracle RAC 10g. In this practice, you create the following configuration: Service Name Usage SERV1 SERV2 Preferred Instances Client service RDBB1 Job scheduler RDBB2 Available Instances RDBB2 RDBB1 Priority HIGH STANDARD Response Time (sec)– Warning/Critical 0.4, 1.0 0.4, 2.0

1. Use DBCA to create the two services above. With DBCA, you can specify the service names as well as their preferred and available instances. 2. After you have created your services, make sure that they are taken into account by CRS. Use the crs_stat command from one of the nodes, and then use the SRVCTL command. 3. Make sure that DBCA has added your services to the tnsnames.ora files on both nodes, and that the listeners are aware of their existence. 4. Connect as SYSDBA under each instance, look at the current value of the SERVICE_NAMES initialization parameter, and check that it is set correctly. 5. Using Database Control, how can you check that the services are currently running as expected? 6. Using a telnet session connected as user oracle to the first node, kill the SMON process of the RDBB1 instance. After a while, use SRVCTL to determine the status of your database. Open a second terminal window by using user oracle on the second node, and look at the current value of the SERVICE_NAMES initialization parameter. What do you observe? 7. Using Database Control, check that both services are running on RDBB2.
8. Using Database Control, create a new user called JFV identified by the password JFV. The

default tablespace of this user should be USERS, and its temporary tablespace TEMP. Make sure that you grant this user with the CONNECT, RESOURCE, and DBA roles. Then, from rac1, using SQL*Plus, connect to SERV1 with user JFV. When connected, determine on which instance your session is currently running. Then, execute the following query: select count(*) from dba_objects,dba_objects,dba_objects
9. After a while, go to the EM Top Consumers page from the Cluster Database page from rac2,

and check that SERV1 is using more and more resources. Check statistics on your service with v$service_stats from a SQL*Plus session connected as SYSDBA.

Oracle Database 10g: Real Application Clusters A-18

10. Using Database Control, relocate SERV1 to its preferred instance RDBB1.

11. What happens to your already connected SERV1 session running on the second instance? If your

session is still executing the query, stop its execution by pressing [Ctrl] + [C].
12. Using Database Control, create the HIGH_PRIORITY and STANDARD_PRIORITY consumer

groups. Make sure that you map HIGH_PRIORITY to SERV1, and STANDARD_PRIORITY to SERV2.

13. Using a SQL*Plus session, check the mappings that you just defined, and make sure that you

grant PUBLIC the right to switch to HIGH_PRIORITY. You will no longer use STANDARD_PRIORITY in this practice.
14. Using Database Control, create a resource plan called SERVICE_PLAN. Make sure that this plan

is defined to dedicate 70% of the CPU to the HIGH_PRIORITY consumer group, and 30% to the OTHERS group. After you have created the plan, see that it is activated on both instances.
15. From a telnet session on the first node, connected as user oracle, change your directory to the

labs directory. From there, execute the lab_07_01_15_n1_RDBB.sh script. This script generates a workload on the first instance. It starts two sessions that use SERV1, and two sessions that use RDBB1. How would you verify that the resource plan is followed?
16. Deactivate SERVICE_PLAN by using DB Control.

17. Set alert thresholds for your service SERV1 by using DB Control. Specify the values defined at

the beginning of this practice.
18. Use SQL*Plus to connect to RDBB1 by using SERV1, and execute the lab_07_01_18.sql

script, connected as user JFV. Observe the graph. What is your conclusion?
19. Use Database Control to remove the thresholds that you specified during this practice.

Oracle Database 10g: Real Application Clusters A-19

Practice 08-1: Monitoring Connection High Availability
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. 1. Start both instances by using srvctl. 2. Using SRVCTL, change your service configuration to have RDBB1 and RDBB2 as the preferred instances for SERV1, and start SERV1. 3. Check that SERV1 is correctly started on each instance. 4. Check that dynamic service registration is enabled on both instances. 5. From a terminal window, and connected as user JFV in SQL*Plus, execute the createfan.sql script. This script creates a simple table used by the following scripts of this practice. This script is located in your $HOME/labs directory. 6. Connected as user oracle on the first terminal window on the first node, execute the sess_count_n1_RDBB.sh script located in your $HOME/labs directory. Then, connected as user oracle on the second terminal window, execute the startfanload.sh script. Looking at the first session, what is your conclusion? Note: When the workload is correctly balanced on both nodes, stop the workload by executing the stopfanlod.sh script from the third terminal window. Then you can also stop the sessions count by pressing [Ctrl] + [C] in the first session. 7. How would you make sure that the initial connection load is balanced on each instance? 8. Perform step six again. What is your conclusion? 9. Reconfigure SERV1 so that RACDB1 is its preferred instance and RACDB2 is its available instance, and start SERV1. 10. Perform step six again. What is your conclusion?

Oracle Database 10g: Real Application Clusters A-20

11. Perform step six again. After you start the execution of the startfanload script, wait for a while, and then kill the SMON process of the RDBB1 instance from another terminal window. You can use the kill_smon_n1_RDBB.sh script for that. This script is located in your $HOME/labs directory. What do you observe?
12. Using srvctl, make sure that you reassign SERV1 to its preferred instance.

Oracle Database 10g: Real Application Clusters A-21

Practice 08-2: Using Callouts
For this practice, you must start four terminal windows. You must connect as user oracle in each of them, and start two of them on rac1 and the remaining two on rac2. Open the four terminal windows now. For convenience, the first terminal window is called TW1R1. Start it on rac1. The second one is called TW2R1. Start it also on rac1. The last two are called TW3R2 and TW4R2, respectively. Start both on rac2. IMPORTANT NOTE: Before continuing, execute the callout_setup.sh script from TW1R1, and then from TW3R2. This script is located on your $HOME/labs directory.
1. Write a shell script that is able to trap FAN events generated by CRS.

The events that need to be trapped must be for the RDBB database, and the event types that need to be trapped are SERVICEMEMBER (up/down), SERVICE (up/down), and INSTANCE (up/down). The callout script should create a file that logs the events that are trapped. The log file must be located in the $ORA_CRS_HOME/racg/log directory. Make sure that the callout script is deployed on both nodes. Note: You can use TW1R1 for this step.
2. Run the following commands on both TW2R1, and TW4R2:

evmwatch -A -t "@timestamp @@" Do not wait; proceed with the next step.

3. From TW1R1, stop the SERV1 service by using srvctl. When finished, look at terminal windows TR2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? 4. From TW1R1, stop the SERV2 service by using srvctl. When finished, look at terminal windows TR2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? 5. From TW1R1, start both SERV1 and SERV2 services by using srvctl. When finished, look at terminal windows TW2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? Note: When finished, type [CTRL] [C] in both TW2R1 and TW4R2 to stop the evmwatch commands.

Oracle Database 10g: Real Application Clusters A-22

Practice 08-3: Using TAF
1. Using srvctl from rac1, add two new services to your database, and start them. The first one

is called AP and has RDBB1 and RDBB2 as its preferred instances. Its transparent application failover policy is BASIC. The second one is called ERP, and has its preferred instance as RDBB1 and its available instance as RDBB2. Its transparent application failover policy is defined to be PRECONNECT.
2. Using SQL*Plus from rac1, verify that both services are started correctly on their

corresponding instances. What do you observe?
3. In order to be able to connect to your database by using the AP and ERP services, what must you

do next?
4. Connect to the database by using the AP service from the second node (rac2). Use user JFV to

connect. Check which instance you are currently connected to. Also check which TAF policy you are using. How many sessions are started on each instance to support your connection?
5. Still connected as user JFV from your second node, insert a row into the FAN table, and commit

your modification. From another telnet session, kill all the server processes started on the instance to which you are connected. Go back to your session and try to insert a new row in the FAN table. What happens and why?
6. Try again. What happens and why?

7. Verify that you automatically failed over. 8. Exit from the SQL*Plus session that is connected as user JFV, and perform steps 4 through 7

again. This time use the ERP service instead of AP, and make sure that you also identify which service is used when you are asked to determine the failover method used by your session. What do you observe?
9. What is the current repartition of the ERP service on your cluster?

10. Exit from all your SQL*Plus sessions, and kill the SMON process of RDBB1. Wait for CRS to

restart RDBB1, and look at the service repartition across both instances for the ERP service. What is your conclusion?
11. Use srvctl to stop and remove both the AP and ERP services from your cluster.

Oracle Database 10g: Real Application Clusters A-23

Practice 09-1: Backing Up and Recovering Data
1. In this practice, you place the database in ARCHIVELOG mode and enable Flashback Recovery options. In addition, you set persistent RMAN configuration parameters that are RAC friendly. When the parameters have been saved, initiate a full cluster database backup. 2. From a terminal window, start an RMAN session, and adjust the following RMAN properties to take advantage of your RAC environment: • Disk parallelism • Disk channels • Snapshot controlfile name

3. Make a whole database backup including archived log files using Database Control. 4. Create a table called DEPARTMENTS2 as a select from HR.DEPARTMENTS. Add several records. Perform a point-in-time recovery of your cluster database to reflect the state of the database before the new records were added to the DEPARTMENTS2 table. 5. To conserve space, put your database in NOARCHIVELOG mode.

Oracle Database 10g: Real Application Clusters A-24

Practice 09-2: Working with the Cluster Registry In this exercise, you perform the Oracle Cluster Registry tasks.
1. List the most recent OCR backup. 2. Change the OCR autobackup location to the shared disk. 3. Create a manual backup of the cluster registry.

Oracle Database 10g: Real Application Clusters A-25

Practice 10-1: The Perfect Scalability Case
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. Before starting this lab, it is recommended that you stop your database as well Database Control. Once done, restart your database, as well Database Control. Make sure that this done on both nodes. This will ensure that you will see performance graphics that are very close to the ones shown in this solution. To do that, run the following commands from the first node: srvctl stop database –d RDBB Then issue the following commands from both nodes: emctl stop dbconsole Once done, start again your database from the first node: srvctl start database –d RDBB Finaly, restart Database Control on both nodes by executing the following commands: emctl start dbconsole 1. Execute the lab_10_01_01.sql script to create the necessary objects used throughout this practice. 2. Using Database Control, force the creation of an AWR snapshot. 3. Connected as user JFV to your database from both nodes, execute the lab_10_01_03_n1.sh script from the first node, and lab_10_01_03_n2.sh from the second node. Do not wait; proceed with the next step. 4. Using Database Control, determine the list of blocking locks in your database. 5. While the script are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Cluster wait class for the first node. What are your conclusions? 6. Using Database Control look at the Cluster Cache Coherency page. What are your conclusions? 7. While the scripts are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Application wait class for the first node. What are your conclusions? 8. Using Database Control, force the creation of an AWR snapshot. 9. Using Database Control, review the latest ADDM run. What are your conclusions?

Oracle Database 10g: Real Application Clusters A-26

10. Execute the lab_10_01_10.sql script to create the necessary objects used for the rest of this practice. 11. Using Database Control, force the creation of an AWR snapshot. 12. Connected as user JFV to your database from both nodes, execute the lab_10_01_12_n1.sh script from the first node and lab_10_01_12_n2.sh from the second node. Do not wait; proceed with the next step. 13. While the scripts are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Cluster wait class for the first node. What are your conclusions? 14. When both scripts are done, force the creation of an AWR snapshot by using Database Control. 15. Using Database Control, review the latest ADDM run. What are your conclusions? 16. Execute the lab_10_01_16.sql script to create the necessary objects used for the rest of this practice. 17. Connected as user JFV to your database from both nodes, execute the lab_10_01_17_n1.sh script from the first node and lab_10_01_17_n2.sh from the second node. Do not wait; proceed with the next step. 18. Until the scripts are executed, look at the Sessions: Waiting and Working graphic. What are your conclusions? 19. When both scripts are done, force the creation of an AWR snapshot by using Database Control. 20. Using Database Control, review the latest ADDM run. What are your conclusions?

Oracle Database 10g: Real Application Clusters A-27

Optional Practice: Migrating Your Database to ASM
1. From rac1, connect to RDBB1 as SYSDBA under SQL*Plus. Before migrating your database to

ASM, create PFILE as a backup of SPFILE of your database.
2. Change the configuration of your SPFILE to have DB_CREATE_FILE_DEST set to +DBG1, DB_RECOVERY_FILE_DEST set to +FRAG1, and CONTROL_FILES set to +DBG1/control01_RDBB.ctl. 3. See that you no longer have any connection to your database, and stop your database by using srvctl.

4. Make sure that you create a backup of one of the control files. Although this is not mandatory, it is the best practice. 5. Using Recovery Manager, restore an existing control file to ASM. When finished, back up your

database to ASM as a copy of the existing one, and switch the control file pointers to the backup. Finally, recover your database.
6. Using SQL*Plus, open your database, add a new temp file to ASM, and delete the old one. 7. Using SQL*Plus, add two new redo log groups for each thread. Make sure that those new groups are stored in ASM and all have the same size. Then, remove the old groups from each thread. When finished, see that both instances are shut down.

8. From rac1, use srvctl to start your database.

Oracle Database 10g: Real Application Clusters A-28

______________

Appendix B Solutions
______________

Conventions

For most of these practice exercises, you will be using various Oracle tools (OUI, DBCA, EMCA, and so on), and telnet sessions. Your instructor will give you details about your nodes and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. As indicated by your instructor, you will be paired with another student, and both of you will access the same cluster. We hope that you will coordinate the lab work with your coworker, and that you will form a great team. Each cluster is composed of two machines with names similar to the following: ed-toraclin1a, and ed-toraclin1b. In the above example, 1a represents the first node (a) of the first cluster (1), and 1b represents the second node (b) of the first cluster (1). To ease the connection to your cluster during certain labs, you will also be able to use a simplified version of the machine names: rac1a, rac1b, ... We highly recommend that you do not hesitate to look at the proposed solutions before you start a particular lab. When possible, we also provide a set of lab and solution scripts that can help you resolve a lab. If you get stuck on some of the questions, and you want to use a script, make sure you execute it on the right node of your cluster, and connected as the right user. The lab and solution scripts are located either in the $HOME/labs directory or $HOME/solutions directory of each node, and each solution script’s name as the following form: lab | sol_<lesson number>_<lab number>_<step number><sequence letter>_<n1 | n2>_<w1 | w2 | …>_<DB name>, where: • n1 represents the first node, and n2 represents the second node. • wn represents the telnet window n. For example, sol_08_02_04a_n1_w1_RDBA.sh is a shell script that should be executed first on the first node of the first cluster during the foorth step of the second lab of the eight’s lesson. In addition, whenever this guide references a database name like RDBA, you should replace the last letter with your corresponding letter. Note: If you want to execute the proposed solution scripts, make sure that you execute them from the $HOME/solutions directory of the oracle user, or from the $HOME/labs directory of the oracle user.

Oracle Database 10g: Real Application Clusters B-2

Practice 02-1 Solution: Prerequisites to the Installation Process For most of these practice exercises, you will be using various Oracle graphical tools (OUI, DBCA, EMCA, and so on), and telnet sessions. Your instructor will give you details about your node names and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Using a telnet session, connect as user root to your first RAC node. Make sure that you have enough memory, swap space, and temporary space to install your CRS and RAC software on both nodes.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-3

2. Using the redhat-config-network command, check your network configuration. Make sure that you have at least two correctly configured network adaptors. Repeat the check on both nodes.
a. Run the redhat-config-network command from a terminal session through VNC connected as root. Make sure that at least two Ethernet devices are active.

b. One at a time, select each Ethernet device that you will use, and make sure that they are activated when the computer starts.

Oracle Database 10g: Real Application Clusters B-4

c. Click the adapter to highlight it, then click Edit to review its configuration information.

d. Make sure to select the Activate device when computer starts check box. Make note of the IP address and Subnet Mask address.

Oracle Database 10g: Real Application Clusters B-5

e. Repeat the check of information for the other adapter (eth1). Again, make note of the IP address and Subnet Mask address. When you are finished, click on Cancel to return to the Network Configuration screen, and then click on File and then Quit to dismiss the tool.

f.

Make sure that the host names and associated IP addresses are correctly configured in the /etc/hosts file. See that the private interconnect as well as VIP names are also correctly configured. Repeat these steps on the second RAC node.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-6

3. Make sure that you can reach each node from the other one. Do this for both nodes.
From your first node, ping the second node using both its public and private names.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-7

4. Make sure that the required OS packages are loaded with a sufficient compatibility version. Check for this on both nodes.
Run the rpm command and grep for compat and gcc. Check that your package versions are at the same level or higher than those shown below.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-8

5. Make sure that the hangcheck timer is started and properly configured on each node.
Run the lsmod command as root and grep for hang. Make sure that the hangcheck module is loaded automatically at boot by checking the /etc/rc.local file.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-9

6. Make sure that the oracle and nobody users and the oinstall and dba groups exist. Check for this on both nodes.
Use the grep command to search the /etc/passwd and /etc/group files for the required users and groups.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-10

7. Edit the.bash_profile file of the oracle user, and set the environment variables. Do this on both nodes.
Use the su command to change to the oracle user and then use the cat or view commands to review the parameter values in the .bash_profile file. Exit back to the root shell when finished.

Note: After you have checked the first node, check the second node. Make sure the ORACLE_SID ends in “2”. In our example, the ORACLE_SID on the second node should be RDBB2. On your nodes however, please use the names assigned to you by your instructor.

Oracle Database 10g: Real Application Clusters B-11

8. As the root user, make sure that you have the following lines in your /etc/security/limits.conf file: • * soft nproc 2047 • * hard nproc 16384 • * soft nofile 1024 • * hard nofile 65536 In addition, check to see that you have the following entry in your /etc/pam.d/login file: session required /lib/security/pam_limits.so. Check the values above on the second node also.
Use the cat or view commands to review the parameter values in the /etc/security/limits.conf file. Check the values on the second node also.

Oracle Database 10g: Real Application Clusters B-12

Use the cat command to check your /etc/pam.d/login file for the following line: session required /lib/security/pam_limits.so. Check the values above on the second node also.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-13

9. Make sure that the /etc/hosts.equiv file contains the public node names for each node in your cluster. Repeat this step on the second node.
Use the cat command to check that the /etc/hosts.equiv file has the correct entries. If it does not, use vi to add the entries.

Note: After you have checked the first node, check the second node.

10. Connect as user oracle to the first node. Check that you can use the rsh command to connect to the other node of your cluster without being prompted for a password. Do this from the second node also.

Note: After you have checked the first node, check the second node.

11. Connect as user root on the first node. Change directory to /u01, and create the app directory. Make sure that it is owned by oracle and the group is oinstall. Change the effective user to oracle, and create the /u01/app/oracle directory. Do this on both nodes.
Oracle Database 10g: Real Application Clusters B-14

Use the mkdir command to create the app directory under /u01, and use the chown command to change the ownership of the newly created directory. Use the su command to change to the oracle user, and change directory to /u01/app. Use the mkdir command to create the oracle directory under the app directory. Change the permissions to 775 with the chmod command.

Note: Repeat this on the second node.

Oracle Database 10g: Real Application Clusters B-15

12. Connect as user root on the first node. Check that the Linux kernel parameters are set to at least the following values: • semmsl 250 • semmns 32000 • semopm 100 • semmni 128 • shmall 2097152 • shmmax Half the size of physical memory • shmmni 4096 • file-max 65536 • ip_local_port_name 1024-65000 When finished, do the same for the second node.
Run the sysctl command and grep for shm, sem, file-max, and ip_local, respectively. Make sure that the values are at least as large as those above.

Note: After you have checked the first node, check the second node.

Oracle Database 10g: Real Application Clusters B-16

Practice 02-2 Solution: OCFS Installation Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Connect as user root on the first node, and change your directory to the /stage directory. Make sure that you have a subdirectory called ocfs. Then change your directory to the ocfs subdirectory, and see that you have the following .rpm files. These files are downloaded from the OTN Web site, and you may need different ones depending on your machine. Check the OCFS Web pages on the OTN Web site for more information: • ocfs-EL • ocfs-support • ocfs-tools When finished, do the same for the second node.

Note: When you have checked the first node, check the second node.

2. Use the rpm command to install the following OCFS packages from the /stage/ocfs directory: • ocfs-support • ocfs-n-EL (single processor module) or ocfs-n-EL-smp (multiprocessor module) • ocfs-tools Install the ocfs-support rpm first, the OCFS kernel module rpm second, and the ocfstools rpm third. Note in the example, the single processor OCFS kernel module is used. When finished, repeat the steps on the second node.

Oracle Database 10g: Real Application Clusters B-17

a. Use the rpm command with the –i option to install the OCFS rpms. Note the order in which the rpms are installed.

b. Use the rpm command with the –qa options and grep for ocfs to ensure that the rpms are correctly installed before continuing. Do this on both nodes.

Oracle Database 10g: Real Application Clusters B-18

3. Launch the ocfstool utility as the root user. Use it to generate the OCFS configuration file, /etc/ocfs.conf. Then exit the utility, and check the ocfs.conf file that is generated. When done, perform the same steps on the second node.
a. Enter ocfstool at the root prompt, and then click Tasks. On the Tasks menu, click Generate Config.

Oracle Database 10g: Real Application Clusters B-19

b. On the OCFS Generate Config window, click the Interface drop-down button, and choose eth0.

Oracle Database 10g: Real Application Clusters B-20

c. Accept the default port and node name, and click the OK button.

d. Exit ocfstool and view the generated OCFS configuration file (/etc/ocfs.conf), using the cat command.

Note: Perform these steps on both nodes. Oracle Database 10g: Real Application Clusters B-21

4. Ensure that OCFS is automatically started by the system at the next reboot. Repeat this step on the second node.
View the /etc/rc5.d/S24ocfs file with the more command. This file is resposible for loading the ocfs module.

Note: Perform the steps above on the second node.

5. In order that the two nodes do not need to be rebooted, manually execute the load_ocfs command as the root user to load the OCFS module. Note that you may get warnings depending on the kernel version used by your Linux system. If the kernel version you are using is greater than the corresponding OCFS rpms, a forced load is done automatically. However, the reverse is not true. Repeat this step on the second node.
As the root user, load the OCFS kernel module by executing the load_ocfs command.

Oracle Database 10g: Real Application Clusters B-22

6. Using the fdisk command, ensure that a partition called sda1 exists on your shared disk, /dev/sda. The fdisk command must be run as the root user. Note: There is no need to do this on the second node because it is the shared disk!
As the root user, execute the fdisk command by using /dev/sda as an argument to the command. To view the partitions on the disk, press the P key. Your output should look similar to that shown below. Note that there are extra partitions in the example. This is normal; they will be used in a future practice. Press w to sync the disk. To quit fdisk, press q.

7. As the root user, create a new directory called /ocfs, and make sure that it is owned by the oracle user and the associated group is dba. Adjust the permissions appropriately. Repeat these actions on the second node.
Create the /ocfs directory by using the mkdir command as the root user. Use the chown command to change ownership as shown. Use the chmod command, and set the permissions to 775.

Note: Repeat the commands performed in step 7 on the second node. Oracle Database 10g: Real Application Clusters B-23

8. Next, the /dev/sda1 slice must be OCFS formatted by using the mkfs.ocfs command. This step must be performed from the first node only!
a. As the root user, execute the mkfs.ocfs command. The options and values used with the command are described below: -F Force format (Overwrites any existing OCFS volumes) -b Block size (128) -L Volume label (ocfs) -m Mount point (/ocfs) -u User id (500-oracle) -g Group id (502-dba) -p Permissions (0775) [/root] # mkfs.ocfs -F -b 128 -L oracle -m /ocfs -u 500 -g 502 -p 0775 /dev/sda1 Cleared volume header sectors Cleared node config sectors Cleared publish sectors Cleared vote sectors Cleared bitmap sectors Cleared data block Wrote volume header WARNING: PERFORM THIS STEP ON THE FIRST NODE ONLY!

9. The OCFS volume must be mounted on both nodes. Again, you can use ocfstool to do this. This step must be performed on both nodes.
a. Start ocfstool as root on the first node. Click the Mount button, and wait for the operation to complete.

Oracle Database 10g: Real Application Clusters B-24

b. When the mount operation is complete, /ocfs appears under the Mountpoint column heading, and the Mount button is disabled (greyed out). Quit ocfstool by clicking on File and then Exit.

Note: Repeat the operations in step 9 on the second node in your cluster.

10. On both nodes, make sure that the /ocfs directory is mounted automatically during the next reboot of the system. On both nodes, edit the /etc/fstab file to mount the OCFS volume whenever the system boots. When you have finished this step on the first node, go to your second node and repeat step 10.
Use the vi command to edit the /etc/fstab file, and add the following line: /dev/sda1 /ocfs _netdev

Note: Perform the step above on the second node.

11. Check that /dev/sda1 is mounted on /ocfs correctly. When finished, perform the same for the second node.
Oracle Database 10g: Real Application Clusters B-25

On the first node, execute the mount command, and check the output for /dev/sda.

Note: Repeat this step on the second node.

Oracle Database 10g: Real Application Clusters B-26

Practice 02-3 Solution: CRS Installation Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate. In this practice, you will install Oracle Cluster Ready Services on both nodes. The CRS software is located in the /stage directory under the crs10g_10.1.0.2.0 subdirectory. The CRS software is installed by using the Oracle Universal Installer (OUI). This is done on the first node only. 1. Before the OUI can be started, prepare the system to enable nonroot users to redirect the display of graphical applications.
1. As the root user, execute the xhost + command to enable the graphical application to display properly when being run by a nonroot user such as oracle.

2. Go to the CRS software staging location, and start OUI as oracle.
a. As the oracle user, change directory to the /stage/crs10g_10.1.0.2.0/Disk1 directory, and start the OUI by running the runInstaller command.

Oracle Database 10g: Real Application Clusters B-27

Oracle Database 10g: Real Application Clusters B-28

b. When the Oracle Universal Installer Welcome screen appears, click the Next button.

c. If ORACLE_BASE has been properly set in the oracle users .bash_profile file, the inventory directory points to /u01/app/oracle/oraInventory. If it does not, enter the correct value. The oinstall group should appear in the Operating System group name field. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-29

d. A dialog window appears, prompting you to run orainstRoot.sh as the root user to set inventory properties for the installation.

e. Open up a terminal window on the first node, and run the /u01/app/oracle/oraInventory/orainstRoot.sh script as the root user. This script is run from the first node only. When the script finishes running, click the Next button to close the dialog box and continue with the installation.

Oracle Database 10g: Real Application Clusters B-30

f.

The Specify File Locations window appears. The product Source Path should match that shown in the example below. The Installation Name should be OraCr10g_home1 and the Destination Path should be /u01/app/oracle/product/10.1.0/crs_1.

g. The Language Selection screen appears. By default, English is the only selected language.

Oracle Database 10g: Real Application Clusters B-31

h. Select all the available languages by clicking the >> (double greater than) button. Click the Next button to continue.

i.

The Cluster Configuration screen appears with the default Cluster Name of crs.

Oracle Database 10g: Real Application Clusters B-32

j.

Change your Cluster Name to dbclust0n, where n is equal to your cluster number. In this example, the cluster number is 1 (EDTDR1RAC1A, EDTDR1RAC1B). Enter the two public node names as they appear in your /etc/hosts file in the Public Node Name fields. Enter the two private node names in the Private Node Name fields. Again, these names should match those that appear in your /etc/hosts file. When you are finished, click the Next button to continue.

k. On the Private Interconnect Enforcement screen, you must choose the Ethernet interface and IP addresses to be used as interconnects on both nodes.

Oracle Database 10g: Real Application Clusters B-33

l.

Click the Interface Type drop-down menu for the Ethernet interface using the 192.168... network address. In this case, it is eth0. Warning: Please check this closely. Do not use the adapter and subnet that corresponds to your host address.

Oracle Database 10g: Real Application Clusters B-34

m. From the drop-down menu, click Private and allow the menu to collapse.

n. The Interface Type for the interconnect should now be Private. Click the Next button to continue.

Oracle Database 10g: Real Application Clusters B-35

o. The Oracle Cluster Registry screen appears. You must supply a directory name and a file name to be used as the cluster configuration repository for your RAC database. Open a terminal window, and create a directory called OCR under /OCFS. Give ownership to the oracle user and the group should be dba. Use the chmod command to set the permissions of the OCR directory to 775. Perform this step on the first node only!

p. After creating the OCR directory, return to the Oracle Cluster Registry screen, and enter /ocfs/OCR/ocr.dbf in the OCR Location field. Click the Next button to continue.

Oracle Database 10g: Real Application Clusters B-36

q. On the Voting Disk page, you must specify a directory location and the name of a file to be used as the cluster database voting file. From a terminal window, create a directory called CSS under the /ocfs directory. Give ownership to the oracle user and the group should be dba. Use the chmod command to set the permissions of the CSS directory to 775.

r.

After creating the CSS directory, return to the Voting Disk screen, and enter /ocfs/CSS/vdisk.dbf in the Voting Disk location field. Click the Next button to continue.

Oracle Database 10g: Real Application Clusters B-37

s.

A dialog box appears prompting you to run the orainstRoot.sh script as the root user. You must run it on both nodes.

t.

From a terminal window on the first node, run the /u01/app/oracle/oraInventory/orainstRoot.sh script as the root user.

Oracle Database 10g: Real Application Clusters B-38

u. From a terminal window on the second node, run the /u01/app/oracle/oraInventory/orainstRoot.sh script as the root user.

v. After the orainstRoot.sh script has been run on both nodes, click Continue to close the dialog box.

Oracle Database 10g: Real Application Clusters B-39

w. When the Summary screen appears, review the installation settings. Click the Install button when you are ready to proceed with the installation.

x. The Install screen enables you to monitor the progress of the installation and the actions being performed by the OUI.

Oracle Database 10g: Real Application Clusters B-40

y. When the progress reaches 100%, the files that are installed on the first node are copied remotely to the second node.

z. Before the installation is complete, a dialog box appears prompting you to run root.sh on both nodes.

Oracle Database 10g: Real Application Clusters B-41

aa. As the root user, run the /u01/app/oracle/product/10.1.0/crs_1/root.sh script from a terminal window on the first node. The script takes several minutes to execute. The formatting of the cluster configuration and voting disk files is a time-consuming task.

bb. When the root.sh script finishes running on the first node, open a terminal window on the second node as root, and run /u01/app/oracle/product/10.1.0/crs_1/root.sh there.

Oracle Database 10g: Real Application Clusters B-42

cc. When the root.sh script finishes running on the second node, click OK to close the dialog box.

dd. When the End of Installation screen appears, click Exit.

Oracle Database 10g: Real Application Clusters B-43

ee. Click Yes when the dialog box appears to confirm that you want to exit the OUI. Oracle Cluster Ready Services has been sucessfully installed on both nodes.

Oracle Database 10g: Real Application Clusters B-44

Practice 03-1 Solution: Oracle RAC Database Software Installation
In this practice, you use the Oracle Universal Installer (OUI) to install the Oracle Database 10g RAC software on both nodes in your cluster. 1. Go to the database software staging location, and start the OUI.
a. As the oracle user, go to the /stage/db10g_10.1.0.2.0/Disk1 directory. Execute the runInstaller command from that location.

2. Use the OUI to install the database software on both nodes in your cluster.
a. The first screen to appear is the Welcome screen. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-45

b.

Review the Installation Name and the Destination Path. Your values must match those in the example below.

c. The Cluster Installation Mode screen appears. By default, the first node is always selected.

Oracle Database 10g: Real Application Clusters B-46

d. Make sure that the Cluster Installation button is selected, and click the Select All button. Click Next to continue.

e. On the Select Installation Type screen, select the Enterprise Edition installation. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-47

f.

On the Product-specific Prerequisite Checks page, your cluster is checked to see that it meets the minimum requirements for a RAC installation. If your nodes have less than two gigabytes of shared memory (the classroom nodes actually have 500 megabytes of shared memory), the check may issue a warning, but it is OK to proceed. Click Next to continue.

g. On the Select Database configuration screen, click the Do not create a starter database button.

Oracle Database 10g: Real Application Clusters B-48

h. After selecting to skip database creation, click Next to continue.

i.

Take a few moments and review the installation settings on the Summary page. Click Install when you are ready to proceed with the installation.

Oracle Database 10g: Real Application Clusters B-49

j.

You may monitor the progress of the installation on the Install screen.

k. When the installation progress reaches 100%, the OUI copies the installed files remotely from the first node to the second node.

Oracle Database 10g: Real Application Clusters B-50

l.

A dialog box appears prompting that you run the root.sh script on both nodes.

m. Open a terminal window on the first node, and run the

/u01/app/oracle/product/10.1.0/db_1/root.sh script as the root user.

Oracle Database 10g: Real Application Clusters B-51

n. Accept the default value for ORACLE_OWNER , ORACLE_HOME, and the local bin directory questions by pressing <Enter>.

3. The VIP Configuration Assistant (VIPCA) is launched from the root.sh script on the first node.
a. The VIPCA Welcome screen appears first. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-52

b. On the Network Interfaces screen, choose the Ethernet adapter, which will be associated with the VIP address. All discovered adapters are highlighted.

c. Choose the adapter that is associated with your host address. In this example, the adapter is eth1. When you have done this, click Next to continue.

Oracle Database 10g: Real Application Clusters B-53

d. The Virtual IP screen appears next. Enter the IP Alias Name and IP address for each node. Refer to the /etc/hosts file for this information. Your eth1 subnet mask can be viewed by using the redhat-config-network tool or the ifconfig command-line utility. Enter this information carefully. Your VIP and Host information will be different from the example. Click Next when you have accurately filled in the required data.

e. Carefully review the information on the Summary page. Click Finish to complete the VIP configuration.

Oracle Database 10g: Real Application Clusters B-54

f.

You can monitor the VIP configuration progress on the Progress Dialog screen.

Oracle Database 10g: Real Application Clusters B-55

g. When the configuration completes successfully, click OK to close the Progress dialog screen.

h. Review configuration information on the Results page. Click Exit to continue.

Oracle Database 10g: Real Application Clusters B-56

4. After the VIP configuration finishes on the first node, run the root.sh script on the second node.
a. After the VIP configuration is completed, open a terminal window on the second node, and run /u01/app/oracle/product/10.1.0/db_1/root.sh as the root user.

Oracle Database 10g: Real Application Clusters B-57

b. When the root.sh execution is complete on both nodes, click OK to close the dialog box.

c. When the End of Installation screen appears, click the Exit button.

Oracle Database 10g: Real Application Clusters B-58

d. Click the Yes button in the Exit dialog box to exit the OUI.

Oracle Database 10g: Real Application Clusters B-59

5. Before continuing with database creation, make sure that all the nodeapps are running.
a. Run the crs_stat utility located in /u01/app/oracle/product/10.1.0/crs_1/bin to ensure that all the necessary nodeapps are running. All applications listed here should be online.

Oracle Database 10g: Real Application Clusters B-60

Practice 03-2 Solution: Database Creation
In this practice, you create a RAC database by using the Database Configuration Assistant (DBCA).
1. Start the DBCA to begin the creation of the RAC database.

a. As the oracle user, change directory to $ORACLE_HOME/bin, and execute DBCA by using the -datafileDestination argument. The directory passed with this argument is /ocfs/oradata. This simplifies the installation process by enabling the DBCA to put the data files on the shared disk.

b. When the DBCA Welcome screen appears, click Next to continue.

Oracle Database 10g: Real Application Clusters B-61

c. On the Operations page, click the Create a Database button, and then click Next to continue.

d. The Node Selection screen appears. By default, the local node is always selected.

Oracle Database 10g: Real Application Clusters B-62

e. Click the Select All button to choose both nodes. Click Next to continue.

f.

On the Database Templates page, choose the General Purpose template. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-63

g. On the Database Identification page, supply the Global Database Name and the SID Prefix. Your instructor will provide you with this information. Do not include the domain as part of the global database name. Click Next to continue.

h. On the Management Options page, make sure that Configure the Database with Enterprise Manager and Use Database Control for Database Management are both selected. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-64

i.

On the Database Credentials page, select Use the Same Password for All Accounts. Set the password for all accounts to oracle. Click Next to continue.

j.

On the Storage Options page, choose Cluster File System, and then click Next to continue.

Oracle Database 10g: Real Application Clusters B-65

k. On the Database File Locations page, select Use Common Location for All Database Files, and enter /ocfs/oradata in the Files Location field if it does not already appear there. Click Next to continue.

l.

On the Recovery Configuration page, select Specify Flash Recovery Area. The Flash Recovery Area should be /ocfs/flash_recovery_area and the Flash Recovery Area Size should be 2048 megabytes. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-66

m. When the Database Content page appears, select Sample Schemas, and then click Next to continue.

n. There is no need to configure additional services at this time. When the Database Services screen appears, click Next to continue.

Oracle Database 10g: Real Application Clusters B-67

o. Accept the default values on the Initialization Parameters page. Click Next to continue.

p. Review the information on the Database Storage page. Expand each storage component in the lefthand window. The corresponding values are displayed in the right-hand window. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-68

q. Select Create Database on the Creation Options page, and then click Finish.

r. Review the data on the Summary page. Displayed here are common options, initialization parameters, and database files.

Oracle Database 10g: Real Application Clusters B-69

s.

When you are finished reviewing the information, click OK to proceed with the installation.

Oracle Database 10g: Real Application Clusters B-70

t.

You may monitor the progress of database creation in the Progress dialog box.

Oracle Database 10g: Real Application Clusters B-71

u. If you want to perform password management for secondary database accounts, click Password Management when the DBCA dialog box appears. Otherwise, click Exit to continue.

v. A dialog box appears noting the start of the cluster database instances, 1 and 2 in this example. When the instances have been started, close the dialog box and exit the DBCA. The cluster database has been sucessfully created.

Oracle Database 10g: Real Application Clusters B-72

2. If DBCA reports an EM “Null argument” configuration error, you will have to correct this by running emca manually.
As the oracle user, change directory to $ORACLE_HOME/bin and run emca –c –x , then emca – c –r and enter the appropriate data. $ cd $ORACLE_HOME/bin $ ./emca -c -x RDBB $ ./emca -c -r STARTED EMCA at Thu Nov 04 06:10:12 EST 2004 Enter the following information about the cluster database to be configured Listener port number: 1521 Cluster name: dbclust01 Database name: RDBB Service name: RDBB Email address for notification: Email gateway for notification: Password for dbsnmp: **** Password for sysman: **** Password for sys: **** ----------------------------------------------------------------You have specified the following settings Database ORACLE_HOME ................ /u01/app/oracle/product/10.1.0/db_1 Enterprise Manager ORACLE_HOME ................ /u01/app/oracle/product/10.1.0/1/db_1 Database Instance host name ................ edtdr1rac1a Listener port number ................ 1521 Cluster name ................ dbclust01 Database name ................ RDBB Service name ................ RDBB Email address for notification ............... Email gateway for notification ............... ----------------------------------------------------------------Do you wish to continue? [yes/no]: yes Nov 4, 2004 6:11:12 AM oracle.sysman.emcp.EMConfig updateReposVars INFO: Updating file … Nov 4, 2004 6:11:31 AM oracle.sysman.emcp.EMConfig startOMS INFO: Starting the DBConsole in all cluster nodes. This will take a while ... Nov 4, 2004 6:14:46 AM oracle.sysman.emcp.EMConfig perform INFO: DBConsole is started successfully Nov 4, 2004 6:14:46 AM oracle.sysman.emcp.EMConfig perform INFO: >>>>>>>>>>> The Enterprise Manager URL is http://edtdr1rac1a:5500/em <<< Enterprise Manager configuration is completed successfully FINISHED EMCA at Thu Nov 04 06:14:46 EST 2004

Oracle Database 10g: Real Application Clusters B-73

3. Check that all necessary nodeapps and services are online.
a. As the oracle user, execute the crs_stat command located in the /u01/app/oracle/product/10.1.0/crs_1/bin directory. All nodeapps and services should be online. See the output below for a full list.

Oracle Database 10g: Real Application Clusters B-74

b. Finally, access the EM dbconsole for your cluster database. Do this on both nodes. If dbconsole is not started (it should be) execute the command: $ emctl start dbconsole

Oracle Database 10g: Real Application Clusters B-75

c. Accept the license agreement by clicking the I Agree button.

Oracle Database 10g: Real Application Clusters B-76

d. The cluster database home page should now be displayed. Remember to access dbconsole on the second node.

Oracle Database 10g: Real Application Clusters B-77

Practice 03-3 Solution: Identifying Components of Real Application Clusters For most of these practice exercises, you will be using Enterprise Manager Database Control. Your instructor will give you details about your nodes and accounts, as well as the SIDs of your instances. When working on these practices, you must use the provided values to prevent interference with your fellow students or other classes. Note: The solutions provided are not necessarily based on your assigned account, so you should substitute your designated user, database, instance, and related information as appropriate.
1. Connect as user oracle to the first node by using a telnet session. Determine the following: • List of processes associated with the instance on the first node. (For example, RDBA1, RDBB1, etc.) • List of processes running CRS • Notification Server processes When finished, perform the same for the second node.
$ ps –ef | grep RDBB1 | more oracle 23649 1 0 05:50 ? 00:00:00 ora_pmon_RDBB1 oracle 23651 1 0 05:50 ? 00:00:00 ora_diag_RDBB1 oracle 23654 1 0 05:50 ? 00:00:01 ora_lmon_RDBB1 oracle 23656 1 0 05:50 ? 00:00:00 ora_lmd0_RDBB1 oracle 23658 1 0 05:50 ? 00:00:00 ora_lms0_RDBB1 oracle 23660 1 0 05:50 ? 00:00:00 ora_lms1_RDBB1 oracle 23662 1 0 05:50 ? 00:00:00 ora_mman_RDBB1 oracle 23664 1 0 05:50 ? 00:00:00 ora_dbw0_RDBB1 oracle 23666 1 0 05:50 ? 00:00:00 ora_lgwr_RDBB1 oracle 23668 1 0 05:50 ? 00:00:00 ora_ckpt_RDBB1 oracle 23670 1 0 05:50 ? 00:00:00 ora_smon_RDBB1 oracle 23672 1 0 05:50 ? 00:00:00 ora_reco_RDBB1 oracle 23674 1 0 05:50 ? 00:00:00 ora_cjq0_RDBB1 oracle 23676 1 0 05:50 ? 00:00:00 ora_d000_RDBB1 oracle 23678 1 0 05:50 ? 00:00:00 ora_s000_RDBB1 oracle 23681 1 0 05:50 ? 00:00:01 ora_lck0_RDBB1 oracle 23920 1 0 05:50 ? 00:00:00 ora_qmnc_RDBB1 oracle 23923 1 0 05:50 ? 00:00:00 ora_mmon_RDBB1 oracle 23926 1 0 05:50 ? 00:00:00 ora_mmnl_RDBB1 oracle 23957 1 0 05:50 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 23958 23957 0 05:50 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 23960 23958 0 05:50 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 23975 1 0 05:50 ? 00:00:00 ora_q001_RDBB1 oracle 23978 23958 0 05:50 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 23979 23958 0 05:50 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 26281 23958 0 06:00 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/bin/racgimon daemon ora..1.inst oracle 28389 1 0 06:10 ? 00:00:00 ora_pz99_RDBB1 oracle 28694 1 0 06:11 ? 00:00:01 ora_j000_RDBB1 oracle 30943 1 0 06:11 ? 00:00:00 /u01/app/oracle/product/10.1.0/d b_1/perl/bin/perl /u01/app/oracle/product/10.1.0/db_1/bin/emwd.pl dbconsole /u01 /app/oracle/product/10.1.0/db_1/edtdr1rac1a_1/sysman/log/emdb.nohup oracle 31028 1 0 06:11 ? 00:00:00 oracle1 (LOCAL=NO) ... oracle 305 1 0 06:13 ? 00:00:02 oracle1 (LOCAL=NO)

Oracle Database 10g: Real Application Clusters B-78

oracle oracle oracle oracle oracle

310 328 336 1144 8645

1 1 1 1 8625

0 0 0 0 0

06:13 06:14 06:14 06:17 Nov04

? ? ? ? ?

00:00:00 00:00:00 00:00:00 00:00:01 00:00:01

oracle1 (LOCAL=NO) oracle1 (LOCAL=NO) ora_pz98_RDBB1 oracle1 (LOCAL=NO)

$ ps –ef|grep –i crs|more
oracle 8622 8505 0 Nov04 ? 00:00:01 /u01/app/oracle/product/10.1.0/crs_1/bin/evmd.bin ... oracle 8629 8621 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/ocssd.bin oracle 8630 8629 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/ocssd.bin ... root 8627 8508 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/crsd.bin root 23151 8627 0 09:36 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/crsd.bin ... oracle 8709 8708 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/evmlogger.bin -o /u01/app/oracle/product/1g oracle 8954 8709 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/evmlogger.bin -o /u01/app/oracle/product/1g ... root 8508 1 0 Nov04 ? 00:00:00 /u01/app/oracle/product/10.1.0/crs_1/bin/crsd.bin ps -ef|grep -i ons
oracle oracle oracle ... oracle 18381 1 0 Nov05 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/opmn/bin/ons -d 18382 18381 0 Nov05 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/opmn/bin/ons -d 18383 18382 0 Nov05 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/opmn/bin/ons -d 18391 18383 0 Nov05 ? 00:00:00 /u01/app/oracle/product/10.1.0/db_1/opmn/bin/ons -d

Repeat the commands executed above on your second node. Look for processes associated with the second instance and the CRS stack running there.
$ ps –ef | grep RDB2 | more $ ps –ef | grep crs | more

2. Using the same telnet session, execute the ocrdump command. When executed, look at the generated file to determine the list of components that are maintained in the OCR.
$ ocrdump $ more OCRDUMPFILE [SYSTEM] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [SYSTEM.css] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [SYSTEM.css.interfaces] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_CREATE_ SUB_KEY, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall} [SYSTEM.css.interfaces.global] UNDEF :

Oracle Database 10g: Real Application Clusters B-79

SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_ALL_ACC ESS, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall} [SYSTEM.css.interfaces.global.eth0] ... [DATABASE] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_CREATE_ SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : root, GROUP_NAME : root} [DATABASE.NODEAPPS] UNDEF : SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CRE ATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : oracle, GROUP_ NAME : oinstall} [DATABASE.NODEAPPS.edtdr1rac1a] ORATEXT : edtdr1rac1a SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_ALL_ACC ESS, OTHER_PERMISSION : PROCR_ALL_ACCESS, USER_NAME : root, GROUP_NAME : root} [DATABASE.NODEAPPS.edtdr1rac1a.ORACLE_HOME] ORATEXT : /u01/app/oracle/product/10.1.0/db_1 ... [CRS] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [CRS.CUR] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [CRS.CUR.ora!edtdr1rac1a!vip] UNDEF : SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [CRS.CUR.ora!edtdr1rac1a!vip.ACTION_SCRIPT] ORATEXT : /u01/app/oracle/product/10.1.0/crs_1/bin/racgwrap SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, O THER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root} [CRS.CUR.ora!edtdr1rac1a!vip.ACTIVE_PLACEMENT] ORATEXT : 1 ...

3. On the first node, execute the following command: evmwatch -A -t "@timestamp @@". Create a second telnet session connected as user oracle on the first node. In the second session, kill the SMON process on node 1 (1 in this example) when finished, what do you observe in the first session? After a few moments, determine the list of processes running 1. What is your conclusion? Note: Once done, type [CTRL] [C] in the first session to stop the evmwatch command. Answer: As soon as the SMON process is killed, CRS detects it, and starts up the 1 instance again. This is done automatically.
$ evmwatch -A -t "@timestamp @@" "04-Nov-2004 06:47:49 CRS is requested to perform action fail on resource ora.R"

Oracle Database 10g: Real Application Clusters B-80

"04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004 "04-Nov-2004

06:47:50 06:47:50 06:47:50 06:47:58 06:47:58 06:47:58 06:48:17 06:48:17 06:48:17 06:48:18

CRS ora.RDBB.RDBB1.inst is transitioning from state ONLIN" RAC: ora.RDBB.RDBB1.inst: down: " RAC: ora.RDBB.RDBB1.inst: imstop: " CRS ora.RDBB.RDBB1.inst stopped" CRS ora.RDBB.RDBB1.inst is transitioning from state OFFLI" CRS is requested to perform action check on resource ora." RAC: ora.RDBB.RDBB1.inst: up: " RAC: ora.RDBB.RDBB1.inst: imstop: " RAC: ora.RDBB.RDBB1.inst: imcheck: " CRS ora.RDBB.RDBB1.inst started on member edtdr1rac1a"

$ ps -ef |grep –i smon oracle 7738 1 0 05:50 ? oracle 7481 19878 0 01:28 pts/0 $ kill –9 7738 $ ps -ef | grep oracle 30943 oracle 8964 oracle 8966 oracle 8969 oracle 8971 oracle 8973 oracle 8975 oracle 8977 oracle 8979 oracle 8981 oracle 8983 oracle 8985 oracle 8987 oracle 8989 oracle 8991 oracle 8993 oracle 8996 oracle 9019 oracle 9022 oracle 9025 oracle 9053 oracle 9072 oracle 9073 oracle 9075 oracle 9086 oracle 9100 oracle 9101 oracle 9109 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9072 9073 1 9073 9073 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 06:11 06:47 06:47 06:47 06:47 06:47 06:47 06:47 06:47 06:47 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 06:48 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

00:00:02 ora_smon_RDBB1 00:00:00 grep ora_smon_RDBB1

00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00

/u01/app/oracle/product/10.1.0/p ora_pmon_RDBB1 ora_diag_RDBB1 ora_lmon_RDBB1 ora_lmd0_RDBB1 ora_lms0_RDBB1 ora_lms1_RDBB1 ora_mman_RDBB1 ora_dbw0_RDBB1 ora_lgwr_RDBB1 ora_ckpt_RDBB1 ora_smon_RDBB1 ora_reco_RDBB1 ora_cjq0_RDBB1 ora_d000_RDBB1 ora_s000_RDBB1 ora_lck0_RDBB1 ora_qmnc_RDBB1 ora_mmon_RDBB1 ora_mmnl_RDBB1 ora_pz99_RDBB1 /u01/app/oracle/product/10.1.0/t /u01/app/oracle/product/10.1.0/t /u01/app/oracle/product/10.1.0/t oracle1 (LOCAL=NO) /u01/app/oracle/product/10.1.0/t /u01/app/oracle/product/10.1.0/t ora_q000_RDBB1

Oracle Database 10g: Real Application Clusters B-81

4. Create two SQL*Plus sessions connected as SYSDBA to each instance. Determine the following for each instance: • The database name • The instance name • The instance number • The thread • The instance status • The list of redo log groups and their corresponding thread and members • The undo tablespaces What do you observe when looking at the status of each instance? Answer: Both instances have the same database open.
$ sqlplus /nolog SQL> connect sys/oracle@RDBB1 as sysdba SQL> select name from v$database; NAME --------RDBB SQL> col host_name format a15 SQL> select instance_name,instance_number,host_name,thread#,status 2 from v$instance; INSTANCE_NAME INSTANCE_NUMBER HOST_NAME THREAD# STATUS ---------------- --------------- --------------- ---------- -----------RDBB1 1 edtdr1rac1a 1 OPEN SQL> SQL> 2 3 col member format a30 select g.thread#,g.group#,l.member from v$log g, v$logfile l where g.group#=l.group#;

THREAD# GROUP# MEMBER ---------- ---------- -----------------------------1 2 /ocfs/oradata/RDBB/redo02.log 1 1 /ocfs/oradata/RDBB/redo01.log 2 3 /ocfs/oradata/RDBB/redo03.log 2 4 /ocfs/oradata/RDBB/redo04.log SQL> show parameter undo_tablespace NAME TYPE VALUE ------------------------------------ ----------- --------------------------undo_tablespace string UNDOTBS1

SQL> connect sys/oracle@RDBB2 as sysdba SQL> select name from v$database; NAME --------RDBB SQL> col host_name format a15 SQL> select instance_name,instance_number,host_name,thread#,status

Oracle Database 10g: Real Application Clusters B-82

2

from v$instance;

INSTANCE_NAME INSTANCE_NUMBER HOST_NAME THREAD# STATUS ---------------- --------------- --------------- ---------- -----------RDBB2 2 edtdr1rac1b 2 OPEN SQL> show parameter undo_tablespace NAME TYPE VALUE ------------------------------------ ----------- --------------------------undo_tablespace string UNDOTBS2

Oracle Database 10g: Real Application Clusters B-83

Practice 04-1 Solution: RAC Database Instances Administration
1. In this practice, you will use the srvctl utility to: • Stop the cluster database • Start the first instance only, then bring up the second instance • Start the first instance only in MOUNT mode • Start the second instance and open the database
a. Stop your database. $ srvctl stop database -d RDBB b. Start the first instance only, then start the second instance. $ srvctl start instance -d RDBB -i RDBB1 $ srvctl start instance -d RDBB -i RDBB2 c. After stopping the database, start the first instance in MOUNT mode, then start the second instance and open the database. $ srvctl stop database -d RDBB $ srvctl start instance -d RDBB -i RDBB1 -o mount $ srvctl start instance -d RDBB -i RDBB2 -o open

2. In this practice you will use EM Dbconsole to stop and start your database.
a. From the Cluster Database home page, click the Shutdown button.

b. On the Specify Credentials page, enter the operating system name and password, oracle/oracle then click Continue.

Oracle Database 10g: Real Application Clusters B-84

c. On the Select Operation page, click the Shutdown All button.

d. Next, click the Advanced Options button on the Confirmation page.

e. Click the Immediate button on the Advanced Options page and click OK. What is the default option?

Oracle Database 10g: Real Application Clusters B-85

f.

When you are returned to the Confirmation page, click Yes to begin the shutdown operation.

g. Wait several minutes, then click the Refresh button.

h. The next screen will allow you to restart the cluster database. Click the Startup button.

Oracle Database 10g: Real Application Clusters B-86

i.

On the Specify Credentials page, enter the operating system name and password, oracle/oracle then click Continue.

j.

On the Select Operation page, click the Startup All button.

Oracle Database 10g: Real Application Clusters B-87

k. Click the Advanced Options button on the Confirmation page.

l.

Make sure the Open the Database button is selected and click OK.

Oracle Database 10g: Real Application Clusters B-88

m. When you have been returned to the Confirmation page, click the Yes button.

n. The Startup screen appears next.

o. When the database has bee started, log back into Dbconsole.

Oracle Database 10g: Real Application Clusters B-89

p. You should be able to determine from the Cluster Database home page that the database is up and running.

Oracle Database 10g: Real Application Clusters B-90

Practice 05-1 Solution: ASMLib Installation
Unless specified otherwise, you must log in as root on a terminal session through VNC on the first node of your cluster (rac1). Before starting this lab, execute the setup_asm_partitions.sh script located in your labs directory.
1. Change your current directory to /stage/asmlib. Make sure that the following RPMs are in

that directory: • oracleasm-support-1.0.2-1.i386.rpm • oracleasm-2.4.21-EL-1.0.0-1.i686.rpm • oracleasmlib-1.0.0-1.i386.rpm Use the rpm –i command to install each of the RPMs above in that order.

Oracle Database 10g: Real Application Clusters B-91

2. Use the /etc/init.d/oracleasm configure command to configure your ASMLib.

Make sure that oracle is the default owner of the driver, and that dba is the default group owner of the driver. Also, see that the driver is started during the boot. When done, check that the ASMLib driver is correctly loaded.

Oracle Database 10g: Real Application Clusters B-92

Oracle Database 10g: Real Application Clusters B-93

Oracle Database 10g: Real Application Clusters B-94

3. Use the fdisk /dev/sda command to look at the shared devices that are available to you. Then, use the /etc/init.d/oracleasm createdisk command to create six ASM disks called VOL1, VOL2, …, VOL6, respectively. For that, use the last six shared devices available to you.

Oracle Database 10g: Real Application Clusters B-95

Oracle Database 10g: Real Application Clusters B-96

4. Telnet to the second node (rac2), and connect as root. When on rac2, install and configure the ASMLib driver like you did previously on rac1.

Oracle Database 10g: Real Application Clusters B-97

5. Still connected as root on rac2, use the /etc/init.d/oracleasm scandisks command to make sure that the previously defined volumes are also visible from rac2. When finished, you can exit from your rac2 session.

Oracle Database 10g: Real Application Clusters B-98

Practice 05-2 Solution: ASM Instances Creation
1. Connected as user oracle from a terminal window on your VNC session, you now create ASM

instances in your cluster. Execute the dbca command. Make sure that on the Welcome screen, the Oracle Real Application Clusters database option is selected. Then, click Next.

Oracle Database 10g: Real Application Clusters B-99

2. On the Operations screen, select the Create a Database option. When finished, click Next.

Oracle Database 10g: Real Application Clusters B-100

3. On the Node Selection screen, select both nodes by using the Select All button. Then, click Next.

Oracle Database 10g: Real Application Clusters B-101

4. On the Database Templates screen, select the General Purpose option. Then, click Next.

Oracle Database 10g: Real Application Clusters B-102

5. On the Database Identification screen, specify the Global Database Name. Then, click Next.

Oracle Database 10g: Real Application Clusters B-103

6. On the Management Options screen, deselect the Configure the Database with Enterprise Manager check box. When finished, click Next.

Oracle Database 10g: Real Application Clusters B-104

7. On the Database Credentials screen, select the Use the Same Password for All Accounts option, and specify and confirm the passward for the SYS user. Use oracle as the password. When finished, click Next.

Oracle Database 10g: Real Application Clusters B-105

8. On the Storage Options screen, select the Automatic Storage Management (ASM) option. Click Next.

Oracle Database 10g: Real Application Clusters B-106

9. On the Create ASM Instance screen, select the Create initialization parameter file (IFILE) option, and click ASM Parameters. On the ASM Parameters screen, set the asm_diskstring value to ORCL:VOL*. Then make sure that you specify oracle as the password for SYS when back to the Create ASM Instance screen. When finished, click Next.

Oracle Database 10g: Real Application Clusters B-107

Oracle Database 10g: Real Application Clusters B-108

Oracle Database 10g: Real Application Clusters B-109

10. In the warning dialog box, click OK to start the ASM instance creation process.

Oracle Database 10g: Real Application Clusters B-110

Oracle Database 10g: Real Application Clusters B-111

11. When on the ASM Disk Groups screen, create a new ASM disk group called DBG1 by using the first four candidate disks. Make sure that the disk group that you create uses the Normal redundancy policy. When finished, click Next.

Oracle Database 10g: Real Application Clusters B-112

Oracle Database 10g: Real Application Clusters B-113

Oracle Database 10g: Real Application Clusters B-114

Oracle Database 10g: Real Application Clusters B-115

12. When on the Database File Locations screen, click Cancel to cancel the session and exit from dbca.

Oracle Database 10g: Real Application Clusters B-116

Oracle Database 10g: Real Application Clusters B-117

13. Using terminal sessions, and still connected as oracle, make sure that both ASM instances are started. If

an ASM instance is down, set its corresponding SID, and use SQL*Plus to start it up. Do that from the corresponding node.

Oracle Database 10g: Real Application Clusters B-118

Oracle Database 10g: Real Application Clusters B-119

Practice 05-3 Solution: Discovering ASM Instances with Database Control 1. Make sure that the asm1_target_RDBA.xml file is in your labs directory on rac1, and that asm2_target_RDBA.xml is in your labs directory on rac2. See that their content is correct.
[oracle@EDTDR1RAC1A labs]$ ls asm1*.xml asm1_target_RDBB.xml [oracle@EDTDR1RAC1A labs]$ more asm1_target_RDBB.xml <Target TYPE="osm_instance" NAME="+ASM1" DISPLAY_NAME="+ASM1"> <Property NAME="SID" VALUE="+ASM1"/> <Property NAME="MachineName" VALUE="vip-TORAC1A"/> <Property NAME="OracleHome" VALUE="/u01/app/oracle/product/10.1.0/db_1"/> <Property NAME="UserName" VALUE="sys"/> <Property NAME="password" VALUE="oracle" ENCRYPTED="FALSE"/> <Property NAME="Role" VALUE="sysdba"/> <Property NAME="Port" VALUE="1521"/> </Target> [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1B labs]$ ls asm2*.xml asm2_target_RDBB.xml [oracle@EDTDR1RAC1B labs]$ more asm2_target_RDBB.xml <Target TYPE="osm_instance" NAME="+ASM2" DISPLAY_NAME="+ASM2"> <Property NAME="SID" VALUE="+ASM2"/> <Property NAME="MachineName" VALUE="vip-TORAC1B"/> <Property NAME="OracleHome" VALUE="/u01/app/oracle/product/10.1.0/db_1"/> <Property NAME="UserName" VALUE="sys"/> <Property NAME="password" VALUE="oracle" ENCRYPTED="FALSE"/> <Property NAME="Role" VALUE="sysdba"/> <Property NAME="Port" VALUE="1521"/> </Target> [oracle@EDTDR1RAC1B labs]$

2. On both nodes, check that you do not have any ASM target already registered in Database Control.
a. Connect as SYSDBA through Database Control on rac1, and then click the Cluster link in the General section of the Cluster Database page.

Oracle Database 10g: Real Application Clusters B-120

b. On the Cluster page, navigate to the Targets tab.

c. On the Targets page, check that all targets are currently up and running, and that you do not have any ASM target.

Oracle Database 10g: Real Application Clusters B-121

d. Repeat the same steps on rac2.

3. Go back to your terminal window, and use the emctl tool to add the new target and stop the agent. Then use emctl to restart the agent and the DB console. Perform this on both nodes.
[oracle@EDTDR1RAC1A labs]$ emctl config agent addtarget /home/oracle/labs/asm1_target_RDBB.xml TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. [oracle@EDTDR1RAC1A labs]$ emctl stop agent TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. This will stop the Oracle Enterprise Manager 10g Database Control ... ... Stopped. Agent is not running. [oracle@EDTDR1RAC1A labs]$ emctl start dbconsole TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. http://EDTDR1RAC1A:5500/em/console/aboutApplication Starting Oracle Enterprise Manager 10g Database Control ........................ started. --------------------------------------------------------------Logs are generated in directory /u01/app/oracle/product/10.1.0/db_1/EDTDR1RAC1A_RDBB1/sysman/log [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-122

[oracle@EDTDR1RAC1B labs]$ emctl config agent addtarget /home/oracle/labs/asm2_target_RDBB.xml TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. [oracle@EDTDR1RAC1B labs]$ emctl stop agent TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. This will stop the Oracle Enterprise Manager 10g Database Control ... ... Stopped. Agent is not running. [oracle@EDTDR1RAC1B labs]$ emctl start dbconsole TZ set to Canada/Eastern Oracle Enterprise Manager 10g Database Control Release 10.1.0.2.0 Copiright (c) 1996, 2004 Oracle Corporation. All rights reserved. http://EDTDR1RAC1B:5500/em/console/aboutApplication Starting Oracle Enterprise Manager 10g Database Control ........................ started. --------------------------------------------------------------Logs are generated in directory /u01/app/oracle/product/10.1.0/db_1/EDTDR1RAC1B_RDBB2/sysman/log [oracle@EDTDR1RAC1B labs]$

4. Log in again by using Database Control. Make sure that both ASM instances are now recognized as Database Control targets on both nodes.
a. Connect as SYSDBA through Database Control on rac1, and then click the Cluster link in the General section of the Cluster Database page.

Oracle Database 10g: Real Application Clusters B-123

b. On the Cluster page, navigate to the Targets tab.

c. On the Targets page, check that all targets are currently up and running, and that you have both ASM instances recognized.

Oracle Database 10g: Real Application Clusters B-124

d. Repeat the same steps on rac2.

5. Use Database Control from rac1 to create a new ASM disk group called FRAG1. This disk group is used to migrate the flash recovery area to ASM. Make sure that this disk group uses the remaining two candidate ASM disks, and is using the Normal redundancy policy.
a. On the Targets page on rac1, click the +ASM1 link.

Oracle Database 10g: Real Application Clusters B-125

b. When on the Automatic Storage Management page, click the Administration tab.

c. On the Automatic Storage Management Login page, specify SYS for the username and oracle for its password. Then, click Login.

d. When on the Administration page, click Create.

e. On the Create Disk Group page, specify FRAG1 in the Name field, and select the remaining two candidate disks. Deselect the Automatically Mount During Startup option if already set, and select the Mount this disk group on all ASM instances in the cluster option. Also, select the NORMAL redundancy policy. When finished, click OK.

Oracle Database 10g: Real Application Clusters B-126

Oracle Database 10g: Real Application Clusters B-127

Practice 05-4 Solution: Understanding ASM Automatic Rebalancing
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus.
1. Connected as user oracle in your terminal emulator window on rac1, make sure that your

ASM instance is started up, and list the processes associated to it. Then determine the characteristics of: • The mounted disk groups • The associated ASM disks • The associated ASM files

[oracle@EDTDR1RAC1A solutions]$ ./sol_05_04_01_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 3 09:47:07 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> startup ORA-01081: cannot start SQL> SQL> host ps -ef | grep oracle 19676 1 0 oracle 19678 1 0 oracle 19681 1 0 oracle 19683 1 0 oracle 19685 1 0 oracle 19687 1 0 oracle 19689 1 0 oracle 19691 1 0 oracle 19693 1 0 oracle 19695 1 0 oracle 19697 1 0 oracle 19700 1 0 oracle 1977 1 0 oracle 2017 1 0 oracle 18825 18817 0 _+ASM1 oracle 18827 18825 0

already-running ORACLE - shut it down first _+ASM1 Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? 06:10 ? 06:10 ? 09:47 pts/0 09:47 pts/0 00:00:00 00:00:00 00:00:01 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:05 00:00:00 00:00:03 00:00:02 00:00:00 asm_pmon_+ASM1 asm_diag_+ASM1 asm_lmon_+ASM1 asm_lmd0_+ASM1 asm_lms0_+ASM1 asm_mman_+ASM1 asm_dbw0_+ASM1 asm_lgwr_+ASM1 asm_ckpt_+ASM1 asm_smon_+ASM1 asm_rbal_+ASM1 asm_lck0_+ASM1 asm_pz99_+ASM1 asm_pz98_+ASM1 /bin/bash -c ps -ef | grep

00:00:00 grep _+ASM1

SQL> SQL> select * from v$asm_diskgroup; GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ------------ ------------------------------ ----------- ---------ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB -------------------- ----------- ------ ---------- ---------1 DBG1 512 4096 1048576 MOUNTED NORMAL 29336 29184 2 FRAG1 1048576 MOUNTED 512 NORMAL 9287 9187 4096

Oracle Database 10g: Real Application Clusters B-128

SQL> SQL> SQL> SQL> SQL> 2

col name format a20 col failgroup format a20 select name,free_mb,failgroup,bytes_read,bytes_written from v$asm_disk;

NAME FREE_MB FAILGROUP BYTES_READ BYTES_WRITTEN -------------------- ---------- -------------------- ---------- ------------VOL1 7709 VOL1 5820416 0 VOL2 6140 VOL2 4800512 4096 VOL3 7689 VOL3 4800512 4096 VOL4 7646 VOL4 4796416 4096 VOL5 7637 VOL5 5025792 57987072 VOL6 1550 VOL6 4096000 57987072 6 rows selected. SQL> SQL> select group_number,file_number,bytes,type,striped 2 from v$asm_file; no rows selected SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A solutions]$

2. Connected as SYSDBA under SQL*Plus in another terminal emulator window, determine the list

of disk groups that are visible from your database instance. Then list the processes associated to your database instance. When done, create a new tablespace called TBSASM that is stored inside the ASM disk group DGROUP1, and which has only one 500 MB data file. When finished, determine the list of processes associated to your database instance again, and list the data files associated to your database. What do you observe? Answer: As soon as the new tablespace is created, the ASM processes are started on the database instance. They are used to communicate with the ASM instance.
[oracle@EDTDR1RAC1A solutions]$ ./sol_05_04_02_n1_RDBA.sh SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 3 10:04:29 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> select * from v$asm_diskgroup;

Oracle Database 10g: Real Application Clusters B-129

GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ------------ ------------------------------ ----------- ---------ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB -------------------- ----------- ------ ---------- ---------1 DBG1 512 4096 1048576 MOUNTED NORMAL 2 FRAG1 1048576 MOUNTED 512 NORMAL 4096

SQL> SQL> host ps -ef | grep _RDBB1 oracle 9319 1 0 Sep02 ? 00:00:00 ora_pmon_RDBB1 oracle 9321 1 0 Sep02 ? 00:00:00 ora_diag_RDBB1 oracle 9323 1 0 Sep02 ? 00:00:04 ora_lmon_RDBB1 oracle 9325 1 0 Sep02 ? 00:00:05 ora_lmd0_RDBB1 oracle 9328 1 0 Sep02 ? 00:00:05 ora_lms0_RDBB1 oracle 9330 1 0 Sep02 ? 00:00:04 ora_lms1_RDBB1 oracle 9332 1 0 Sep02 ? 00:00:00 ora_mman_RDBB1 oracle 9334 1 0 Sep02 ? 00:00:01 ora_dbw0_RDBB1 oracle 9336 1 0 Sep02 ? 00:00:01 ora_lgwr_RDBB1 oracle 9338 1 0 Sep02 ? 00:00:00 ora_ckpt_RDBB1 oracle 9340 1 0 Sep02 ? 00:00:08 ora_smon_RDBB1 oracle 9342 1 0 Sep02 ? 00:00:00 ora_reco_RDBB1 oracle 9344 1 0 Sep02 ? 00:00:01 ora_cjq0_RDBB1 oracle 9346 1 0 Sep02 ? 00:00:00 ora_d000_RDBB1 oracle 9348 1 0 Sep02 ? 00:00:00 ora_s000_RDBB1 oracle 9351 1 0 Sep02 ? 00:00:05 ora_lck0_RDBB1 oracle 9370 1 0 Sep02 ? 00:00:00 ora_qmnc_RDBB1 oracle 9374 1 0 Sep02 ? 00:00:09 ora_mmon_RDBB1 oracle 9377 1 0 Sep02 ? 00:00:00 ora_mmnl_RDBB1 oracle 9471 1 0 Sep02 ? 00:00:00 ora_q001_RDBB1 oracle 9502 1 0 Sep02 ? 00:00:08 ora_pz99_RDBB1 oracle 3392 1 0 06:12 pts/0 00:00:00 /u01/app/oracle/product/10.1.0/db_1/perl/bin/perl /p oracle 6956 1 0 06:23 ? 00:00:00 ora_pz98_RDBB1 oracle 22572 1 3 10:04 ? 00:00:00 ora_j000_RDBB1 oracle 22581 22574 0 10:04 pts/0 00:00:00 /bin/bash -c ps -ef | grep _RDBB1 oracle 22583 22581 0 10:04 pts/0 00:00:00 grep _RDBB1 SQL> SQL> create tablespace tbsasm 2 datafile '+DBG1' size 500M; Tablespace created. SQL> SQL> host oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle oracle ps -ef | grep 9319 1 0 9321 1 0 9323 1 0 9325 1 0 9328 1 0 9330 1 0 9332 1 0 9334 1 0 9336 1 0 9338 1 0 9340 1 0 9342 1 0 9344 1 0 _RDBB1 Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ? Sep02 ?

00:00:00 00:00:00 00:00:04 00:00:05 00:00:05 00:00:04 00:00:00 00:00:01 00:00:01 00:00:00 00:00:08 00:00:00 00:00:01

ora_pmon_RDBB1 ora_diag_RDBB1 ora_lmon_RDBB1 ora_lmd0_RDBB1 ora_lms0_RDBB1 ora_lms1_RDBB1 ora_mman_RDBB1 ora_dbw0_RDBB1 ora_lgwr_RDBB1 ora_ckpt_RDBB1 ora_smon_RDBB1 ora_reco_RDBB1 ora_cjq0_RDBB1

Oracle Database 10g: Real Application Clusters B-130

oracle 9346 1 0 Sep02 ? 00:00:00 ora_d000_RDBB1 oracle 9348 1 0 Sep02 ? 00:00:00 ora_s000_RDBB1 oracle 9351 1 0 Sep02 ? 00:00:05 ora_lck0_RDBB1 oracle 9370 1 0 Sep02 ? 00:00:00 ora_qmnc_RDBB1 oracle 9374 1 0 Sep02 ? 00:00:09 ora_mmon_RDBB1 oracle 9377 1 0 Sep02 ? 00:00:00 ora_mmnl_RDBB1 oracle 9471 1 0 Sep02 ? 00:00:00 ora_q001_RDBB1 oracle 9502 1 0 Sep02 ? 00:00:08 ora_pz99_RDBB1 oracle 3392 1 0 06:12 pts/0 00:00:00 /u01/app/oracle/product/10.1.0/db_1/perl/bin/perl /p oracle 6956 1 0 06:23 ? 00:00:00 ora_pz98_RDBB1 oracle 22585 1 0 10:04 ? 00:00:00 ora_asmb_RDBB1 oracle 22590 1 0 10:04 ? 00:00:00 ora_rbal_RDBB1 oracle 22593 1 0 10:04 ? 00:00:00 ora_o000_RDBB1 oracle 22597 1 0 10:04 ? 00:00:00 ora_o001_RDBB1 oracle 22832 22574 0 10:05 pts/0 00:00:00 /bin/bash -c ps -ef | grep _RDBB1 oracle 22834 22832 0 10:05 pts/0 00:00:00 grep _RDBB1 SQL> SQL> col file_name format a46 SQL> SQL> select file_name,tablespace_name 2 from dba_data_files; FILE_NAME ---------------------------------------------/ocfs/oradata/RDBB/users01.dbf /ocfs/oradata/RDBB/sysaux01.dbf /ocfs/oradata/RDBB/undotbs01.dbf /ocfs/oradata/RDBB/system01.dbf /ocfs/oradata/RDBB/example01.dbf /ocfs/oradata/RDBB/undotbs02.dbf +DBG1/rdbb/datafile/tbsasm.256.1 7 rows selected. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A solutions]$ TABLESPACE_NAME -----------------------------USERS SYSAUX UNDOTBS1 SYSTEM EXAMPLE UNDOTBS2 TBSASM

3. Back on your ASM instance, using SQL*Plus, determine the number of bytes written on the

DBG1 disk group. Then use Database Control to determine the amount of disk space used by TBSASM on each ASM disk. What is your conclusion? Answer: You can see that the disk activity is almost equally distributed across all ASM disks during the creation of the tablespace. After the tablespace has been created, the free space is almost the same on each ASM disk. This is because ASM tries to stripe ASM extents across all ASM disks.
[oracle@EDTDR1RAC1A solutions]$ ./sol_05_04_03_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 3 10:17:54 2004

Oracle Database 10g: Real Application Clusters B-131

Copyright (c) 1982, 2004, Oracle.

All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> SQL> SQL> SQL> 2 col name format a20 col failgroup format a20 select name,free_mb,failgroup,bytes_read,bytes_written from v$asm_disk;

NAME FREE_MB FAILGROUP BYTES_READ BYTES_WRITTEN -------------------- ---------- -------------------- ---------- ------------VOL1 7443 VOL1 6582272 281145856 VOL2 5929 VOL2 5570560 224669696 VOL3 7425 VOL3 5595136 277180928 VOL4 7382 VOL4 5558272 279183360 VOL5 7637 VOL5 5779456 60428288 VOL6 1550 VOL6 4849664 60428288 6 rows selected. SQL> SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options

[oracle@EDTDR1RAC1A solutions]$

a. Connect as SYSDBA through Database Control on rac1. Click the Administration tab on the Automatic Storage Management page. On the Administration page, click the link corresponding to the DBG1 diskgroup.

b. On the Disk Group: DBG1 page, you can see that the used space is uniformly distributed over the four ASM disks.

Oracle Database 10g: Real Application Clusters B-132

4. Using Database Control, drop the ASM disk called VOL4. Using Database Control, look at the

performance charts relating to disk groups. What are your conclusions? Answer: You can see that the disk activity is almost equally distributed across all remaining ASM disks during the drop operation.

a. Connect as SYSDBA through Database Control on rac1. Click the Administration tab on the Automatic Storage Management page. On the Administration page, click the link corresponding to the DBG1 diskgroup. When on the Disk Group: DBG1 page, select the ASM disk called VOL4 in the Member Disks table. When finished, click Delete.

Oracle Database 10g: Real Application Clusters B-133

b. On the Confirmation page, click Yes.

c. Back on the Disk Group: DBG1 page, refresh your screen many times by using the refresh button of your browser. You should see the space used on VOL4 diminishing, while the space used on the other disks is rising. After a while, you should no longer see VOL4 on the page. This is because of the ongoing rebalance operation.

Oracle Database 10g: Real Application Clusters B-134

d. Click the Performance tab on the Disk Group: DBG1 page. Make sure that the Refresh field is set to Real Time: 15 Second Refresh. After a while, you should see the various graphics corresponding to the rebalance operation. The graphics clearly show that the response time is going down while there is an equal amount of read and write activity on the disk group. ASM reads the data from VOL4 and dispatches them on the remaining disks.

Oracle Database 10g: Real Application Clusters B-135

5. On your database instance, connected as SYSDBA, execute the lab_05_04_05.sql script.

This script creates and populates a new table called T, which is stored in TBSASM. When executed, set timing statistics in your SQL*Plus session, and execute the following query: SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) FROM t;
[oracle@EDTDR1RAC1A solutions]$ ./sol_05_04_05_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 3 14:57:08 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> drop table t purge; Table dropped. SQL> SQL> create table t(c number) tablespace tbsasm; Table created. SQL>

Oracle Database 10g: Real Application Clusters B-136

SQL> insert into t values(42); 1 row created. SQL> commit; Commit complete. SQL> SQL> insert into t select * from t; 1 row created. SQL> / 2 rows created. SQL> / 4 rows created. SQL> / 8 rows created. SQL> / 16 rows created. SQL> / 32 rows created. SQL> / 64 rows created. SQL> / 128 rows created. SQL> / 256 rows created. SQL> / 512 rows created. SQL> SQL> / 1024 rows created. SQL> / 2048 rows created. SQL> / 4096 rows created. SQL> /

Oracle Database 10g: Real Application Clusters B-137

8192 rows created. SQL> / 16384 rows created. SQL> SQL> / 32768 rows created. SQL> / 65536 rows created. SQL> / 131072 rows created. SQL> / 262144 rows created. SQL> SQL> commit; Commit complete. SQL> SQL> insert into t select * from t; 524288 rows created. SQL> SQL> commit; Commit complete. SQL> SQL> set timing on SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:21.81 SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A solutions]$

Oracle Database 10g: Real Application Clusters B-138

6.

Using Database Control, navigate to the Add Disks page. Execute the lab_05_04_06.sql script from your terminal window on rac1. As soon as the execution of the script is started, add the ASM disk, which was dropped at step four, to DBG1. What do you observe, and what are your conclusions?

Answer: While the ASM disk is added to DBG1, you can continue to execute queries against that disk group without experiencing any interruption. Because of this, the time to execute the queries may be longer during the add operation, but you are not blocked.

a. Click the General tab to go to the Disk Group: DBG1 page. Click Add Disks. This takes you to the Add Disks page. On that page, select ORCL:VOL4, but do not click OK. Go to your terminal window, and start the execution of the script.

b. When the script starts, go back quickly to the Add Disks page, and click OK.

[oracle@EDTDR1RAC1A solutions]$ ./sol_05_04_06_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 3 15:31:16 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> set timing on SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589

Oracle Database 10g: Real Application Clusters B-139

Elapsed: 00:00:19.64 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:29.30 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:25.39 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:22.48 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:18.55 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:17.33 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589 Elapsed: 00:00:18.39 SQL> SQL> SELECT count(distinct DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) 2 FROM t; COUNT(DISTINCTDBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID,'SMALLFILE')) --------------------------------------------------------------1589

Oracle Database 10g: Real Application Clusters B-140

Elapsed: 00:00:17.61 SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A solutions]$

c. While the script executes, refresh your Add Disks page multiple times by using the refresh button of your browser. Perform this until the rebalance operation finishes.

Oracle Database 10g: Real Application Clusters B-141

Practice 06-1 Solution: Changing Your Online Redo Log File Characteristics
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. 1. Use Database Control to create two new redo log groups in your database. The two groups must pertain to the thread number three, and each group must have only one 10240 KB member called redo05.log and redo06.log, respectively. Make sure that the members are created in the /ocfs/oradata/RACDB directory.

a. Connect as SYSDBA through Database Control, and navigate to the Administration tab on the Database Control home page. b. On the Administration page, click the Redo Log Groups link. c. On the Redo Log Groups page, click the Create button. d. On the Create Redo Log Group page, leave the current value of the Group# field as is it is (5). Make sure that the File size field is set to 10240 KB. Set the Thread# field to 3. Then set the File Name field in the Redo Log Members section to redo05.log. Make sure that the File Directory field in the Redo Log Members section is set to /ocfs/oradata/RDBB/.

e. Click OK. f. Go back to the Redo Log Groups page, and click the Create button. g. On the Create Redo Log Group page, leave the current value of the Group# field as is it is (6). Make sure that the File size field is set to 10240 KB. Set the Thread# field to 3. Then set the File Name field in the Redo Log Members section to redo06.log. See that the File Directory field in the Redo Log Members section is set to /ocfs/oradata/RDBB/.

Oracle Database 10g: Real Application Clusters B-142

h. Click OK.

2. Use Database Control to set the RDBB2.THREAD initialization parameter to 3 in SPFILE only.
a. Connect as SYSDBA through Database Control, and navigate to the Administration tab on the Database Control home page. b. On the Administration page, click the Initialization Parameters link in the Database section. c. On the Initialization Parameters page, click the SPFile tab. d. Click the Show All button. e. Make sure that the Apply changes in SPFile mode to the current running instance(s). For static parameters, you must restart the database check box is deselected. f. Scroll down the page to the end, and select 3 in the Value field for the RDBB2.thread field.

g. Click the Apply button.

3. Use the SRVCTL control utility to stop the RDBB2 instance, and start it up again. What happens and why? If necessary, fix the situation.
Answer: Although it is possible to stop the RDBB2 instance, it is not possible to start it up again because the redo thread number three is not enabled yet. To enable the redo thread number three, you can connect to the first instance and issue the ALTER DATABASE ENABLE THREAD 3 command. Then you can start up the RDBB2 instance again.
Oracle Database 10g: Real Application Clusters B-143

[oracle@EDTDR1RAC2A labs]$ srvctl stop instance -d RDBB -i RDBB2 [oracle@EDTDR1RAC2A labs]$ [oracle@EDTDR1RAC2A labs]$ srvctl start instance -d RDBB -i RDBB2 PRKP-1001 : Error starting instance RDBB2 on node edtdr1rac1b CRS-0215: Could not start resource ora.RDBB.RDBB2.inst. [oracle@EDTDR1RAC2A labs]$

[oracle@EDTDR1RAC2A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Sun Sep 5 03:36:30 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> ALTER DATABASE ENABLE THREAD 3; Database altered. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A labs]$ srvctl start instance -d RDBB -i RDBB2 [oracle@EDTDR1RAC2A labs]$

4. Revert to the original situation where RDBB2 was using the redo thread two, and destroy redo thread number three. Make sure that in the end both instances are up and running and managed by CRS.
[oracle@EDTDR1RAC2A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Sun Sep 5 03:41:01 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> ALTER SYSTEM SET thread = 2 SCOPE=SPFILE SID='RDBB2'; System altered. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A labs]$ srvctl stop instance -d RDBB -i RDBB2

Oracle Database 10g: Real Application Clusters B-144

[oracle@EDTDR1RAC2A labs]$ srvctl start instance -d RDBB -i RDBB2 [oracle@EDTDR1RAC2A labs]$ [oracle@EDTDR1RAC2A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Sun Sep 5 03:43:56 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> alter database disable thread 3; Database altered. SQL> alter database drop logfile group 5; Database altered. SQL> alter database drop logfile group 6; Database altered. SQL> host rm /ocfs/oradata/RDBB/redo05.log SQL> host rm /ocfs/oradata/RDBB/redo06.log SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A labs]$ [oracle@EDTDR1RAC2A labs]$ crs_stat NAME=ora.RDBB.RDBB1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.RDBB.RDBB2.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b NAME=ora.RDBB.db TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac2a.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac1a.LISTENER_EDTDR1RAC2A.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac2a.gsd TYPE=application TARGET=ONLINE

Oracle Database 10g: Real Application Clusters B-145

STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac2a.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac2a.vip TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2a NAME=ora.edtdr1rac2b.ASM2.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b NAME=ora.edtdr1rac1b.LISTENER_EDTDR1RAC2B.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b NAME=ora.edtdr1rac2b.gsd TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b NAME=ora.edtdr1rac2b.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b NAME=ora.edtdr1rac2b.vip TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac2b [oracle@EDTDR1RAC2A labs]$

Oracle Database 10g: Real Application Clusters B-146

Practice 06-2 Solution: Migrating Tablespaces to ASM
1. Connected as SYSDBA under SQL*Plus in your database instance on rac1, create a new

tablespace called TBSASMMIG. This tablespace must contain only one 10 MB file stored in your file system (not using ASM). Create a table called T2 stored in TBSASMMIG. Insert one row inside T2.
[oracle@EDTDR1RAC2A solutions]$ ./sol_06_02_01_n1_RDBB.sh SQL*Plus: Release 10.1.0.2.0 - Production on Wed Sep 8 09:41:49 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> create tablespace tbsasmmig 2 datafile '/ocfs/oradata/RDBB/asmmig1.dbf' size 10M; Tablespace created. SQL> SQL> SQL> SQL> SQL> 2 col file_name format a52 col tablespace_name format a10 select file_name,tablespace_name from dba_data_files; TABLESPACE ---------USERS SYSAUX UNDOTBS1 SYSTEM EXAMPLE UNDOTBS2 TBSASM TBSASMMIG

FILE_NAME ---------------------------------------------------/ocfs/oradata/RDBB/users01.dbf /ocfs/oradata/RDBB/sysaux01.dbf /ocfs/oradata/RDBB/undotbs01.dbf /ocfs/oradata/RDBB/system01.dbf /ocfs/oradata/RDBB/example01.dbf /ocfs/oradata/RDBB/undotbs02.dbf +DBG1/rdbb/datafile/tbsasm.256.1 /ocfs/oradata/RDBB/asmmig1.dbf 8 rows selected. SQL> SQL> create table t2(c number) tablespace tbsasmmig; Table created. SQL> SQL> insert into t2 values(1); 1 row created. SQL> SQL> commit; Commit complete. SQL>

Oracle Database 10g: Real Application Clusters B-147

SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A solutions]$

2. From your database instance, migrate TBSASMMIG to ASM storage. When done, check that the

migration is successful.
[oracle@EDTDR1RAC2A solutions]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Wed Sep 8 09:53:50 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> set echo on SQL> host rman target / nocatalog Recovery Manager: Release 10.1.0.2.0 - Production Copyright (c) 1995, 2004, Oracle. All rights reserved.

connected to target database: RDBB (DBID=772184773) using target database controlfile instead of recovery catalog RMAN> SQL "alter tablespace tbsasmmig offline"; sql statement: alter tablespace tbsasmmig offline RMAN> backup as copy tablespace tbsasmmig format '+DBG1'; Starting backup at 08-SEP-04 allocated channel: ORA_DISK_1 channel ORA_DISK_1: sid=226 devtype=DISK channel ORA_DISK_1: starting datafile copy input datafile fno=00008 name=/ocfs/oradata/RDBB/asmmig1.dbf output filename=+DBG1/rdbb/datafile/tbsasmmig.257.1 tag=TAG20040908T095431 reci4 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03 Finished backup at 08-SEP-04 RMAN> switch tablespace tbsasmmig to copy; datafile 8 switched to datafile copy "+DBG1/rdbb/datafile/tbsasmmig.257.1" RMAN> SQL "alter tablespace tbsasmmig online"; sql statement: alter tablespace tbsasmmig online RMAN> exit Recovery Manager complete. SQL> col file_name format a52 SQL> col tablespace_name format a10

Oracle Database 10g: Real Application Clusters B-148

SQL> select file_name,tablespace_name 2 from dba_data_files; FILE_NAME ---------------------------------------------------/ocfs/oradata/RDBB/users01.dbf /ocfs/oradata/RDBB/sysaux01.dbf /ocfs/oradata/RDBB/undotbs01.dbf /ocfs/oradata/RDBB/system01.dbf /ocfs/oradata/RDBB/example01.dbf /ocfs/oradata/RDBB/undotbs02.dbf +DBG1/rdbb/datafile/tbsasm.256.1 +DBG1/rdbb/datafile/tbsasmmig.257.1 8 rows selected. SQL> select * from t2; C ---------1 SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A solutions]$ TABLESPACE ---------USERS SYSAUX UNDOTBS1 SYSTEM EXAMPLE UNDOTBS2 TBSASM TBSASMMIG

3. Using Database Control, determine the name of the files created in DBG1 from the ASM pages.

a. b. c. d. e.

On the Cluster Database Home page, click the Cluster link in the General section. On the Cluster page, click the Targets tab. On the Targets page, click the +ASM1 link. On the Automatic Storage Management:+ASM1 Home page, click the Administration tab. On the Automatic Storage Management Login page, enter SYS in the Username field, and oracle in the Password field. Click Login. f. On the Automatic Storage Management:+ASM1 Administration page, click the DBG1 link. g. On the Disk Group:DBG1 page, click the Files tab. h. On the Files page, click the Expand All link, and here you can see all the related files.

Oracle Database 10g: Real Application Clusters B-149

4. From your database instance, clean up your environment by dropping the TBSASMMIG tablespace,

including its contents and data file. Perform the same with the TBSASM tablespace. Also, remove the file system file that was originally created to store TBSASMMIG. When finished, verify by using the ASM pages of Database Control that the ASM files are removed.

[oracle@EDTDR1RAC2A solutions]$ ./sol_06_02_04_n1_RDBB.sh SQL*Plus: Release 10.1.0.2.0 - Production on Wed Sep 8 10:20:17 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> drop tablespace tbsasmmig including contents and datafiles; Tablespace dropped. SQL> SQL> drop tablespace tbsasm including contents and datafiles; Tablespace dropped. SQL> SQL> host rm /ocfs/oradata/RDBB/asmmig1.dbf SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC2A solutions]$

Oracle Database 10g: Real Application Clusters B-150

a. On the Disk Group:DBG1 Files page, refresh your browser window.

Oracle Database 10g: Real Application Clusters B-151

Practice 07-1 Solution: Using Services in RAC
The first step in creating a service configuration is to plan services for each application or application function that is using Oracle RAC 10g. In this practice, you create the following configuration: Service Name Usage SERV1 SERV2 Preferred Instances Client service RDBB1 Job scheduler RDBB2 Available Instances RDBB2 RDBB1 Priority HIGH STANDARD Response Time (sec)– Warning/Critical 0.4, 1.0 0.4, 2.0

1. Use DBCA to create the two services above. With DBCA, you can specify the service names as well as their preferred and available instances.
a. From your xWindow terminal emulator, run the dbca command.
[oracle@EDTDR1RAC1A labs]$ dbca

b. On the Welcome screen, select the Oracle Real Application Clusters database option.

Oracle Database 10g: Real Application Clusters B-152

c. Click Next. d. On the Operations screen, select the Services Management option.

e. Click Next. f. On the List of cluster databases screen, select your database.

Oracle Database 10g: Real Application Clusters B-153

g. Click Next. h. On the Database Services screen, click Add.

Oracle Database 10g: Real Application Clusters B-154

i.

In the Add a Service window, enter SERV1 in the Enter Service Name field.

Oracle Database 10g: Real Application Clusters B-155

j. Click OK. k. Go back to the Database Services screen, and select SERV1. Also select RDBB1 as the preferred instance and RDBB2 as the available instance. Make sure that the TAF Policy is set to None.

Oracle Database 10g: Real Application Clusters B-156

l. On the Database Services screen, click Add. m. In the Add a Service window, enter SERV2 in the Enter Service Name field.

Oracle Database 10g: Real Application Clusters B-157

n. Click OK. o. Go back to the Database Services screen, and select SERV2. Also select RDBB1 as the available instance and RDBB2 as the preferred instance. Make sure that the TAF Policy is set to None.

Oracle Database 10g: Real Application Clusters B-158

p. When done, click Finish. q. In the confirmation window, click OK.

Oracle Database 10g: Real Application Clusters B-159

r.

After the services are configured, click No in the warning dialog box.

Oracle Database 10g: Real Application Clusters B-160

2. After you have created your services, make sure that they are taken into account by CRS. Use the crs_stat command from one of the nodes, and then use the SRVCTL command.
[oracle@EDTDR1RAC1A labs]$ crs_stat NAME=ora.RDBB.RDBB1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.RDBB2.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV1.RDBB1.sa TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV1.RDBB1.srv TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a

Oracle Database 10g: Real Application Clusters B-161

NAME=ora.RDBB.SERV1.cs TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.SERV2.RDBB2.sa TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.SERV2.RDBB2.srv TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV2.cs TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.db TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.LISTENER_EDTDR1RAC1A.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.gsd TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.vip TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1b.ASM2.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.LISTENER_EDTDR1RAC1B.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b

Oracle Database 10g: Real Application Clusters B-162

NAME=ora.edtdr1rac1b.gsd TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.vip TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b [oracle@EDTDR1RAC1A labs]$ [oracle@EDTDR1RAC1A labs]$ srvctl status service -d RDBB -s SERV1 Service SERV1 is running on instance(s) RDBB1 [oracle@EDTDR1RAC1A labs]$ srvctl status service -d RDBB -s SERV2 Service SERV2 is running on instance(s) RDBB2 [oracle@EDTDR1RAC1A labs]$

3. Make sure that DBCA has added your services to the tnsnames.ora files on both nodes, and that the listeners are aware of their existence.
[oracle@EDTDR1RAC1A labs]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /u01/app/oracle/product/10.1.0/db_1/a # Generated by Oracle configuration tools. RDBB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB1) ) ) RDBB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) ) ) SERV2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes)

Oracle Database 10g: Real Application Clusters B-163

(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV2) ) ) SERV1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV1) ) ) LISTENERS_RDBB = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) ) RDBB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB2) ) ) [oracle@EDTDR1RAC1A labs]$ lsnrctl services LISTENER_EDTDR1RAC1A LSNRCTL for Linux: Version 10.1.0.2.0 - Production on 06-SEP-2004 04:54:35 Copyright (c) 1991, 2004, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vipRAC1A)(PORT=1521))) Services Summary... Service "+ASM1" has 1 instance(s). Instance "+ASM1", status BLOCKED, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:7 refused:0 state:ready LOCAL SERVER Service "RDBB" has 2 instance(s). Instance "RDBB1", status READY, has 2 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=EDTDR1RAC1A)(PORT=1521)) "DEDICATED" established:11 refused:0 state:ready LOCAL SERVER Instance "RDBB2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:1 refused:0 state:ready REMOTE SERVER

Oracle Database 10g: Real Application Clusters B-164

(ADDRESS=(PROTOCOL=TCP)(HOST=EDTDR1RAC1B)(PORT=1521)) Service "RDBBXDB" has 2 instance(s). Instance "RDBB1", status READY, has 1 handler(s) for this service... Handler(s): "D000" established:0 refused:0 current:0 max:972 state:ready DISPATCHER <machine: EDTDR1RAC1A, pid: 9346> (ADDRESS=(PROTOCOL=tcp)(HOST=EDTDR1RAC1A)(PORT=32783)) Instance "RDBB2", status READY, has 1 handler(s) for this service... Handler(s): "D000" established:0 refused:0 current:0 max:972 state:ready DISPATCHER <machine: EDTDR1RAC1B, pid: 6828> (ADDRESS=(PROTOCOL=tcp)(HOST=EDTDR1RAC1B)(PORT=44721)) Service "SERV1" has 1 instance(s). Instance "RDBB1", status READY, has 2 handler(s) for this service... Handler(s): "DEDICATED" established:0 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=EDTDR1RAC1A)(PORT=1521)) "DEDICATED" established:11 refused:0 state:ready LOCAL SERVER Service "SERV2" has 1 instance(s). Instance "RDBB2", status READY, has 1 handler(s) for this service... Handler(s): "DEDICATED" established:1 refused:0 state:ready REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=EDTDR1RAC1B)(PORT=1521)) The command completed successfully [oracle@EDTDR1RAC1A labs]$ rsh rac1b cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora.edtdr1rac1b Network Configuration File: /u01/app/oracle/product/b # Generated by Oracle configuration tools. RDBB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB1) ) ) RDBB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) ) ) SERV2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA =

Oracle Database 10g: Real Application Clusters B-165

(SERVER = DEDICATED) (SERVICE_NAME = SERV2) ) ) SERV1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV1) ) ) LISTENERS_RDBB = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) ) RDBB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB2) ) ) [oracle@EDTDR1RAC1A labs]$

4. Connect as SYSDBA under each instance, look at the current value of the SERVICE_NAMES initialization parameter, and check that it is set correctly.
SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 04:44:13 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> set echo on; SQL> SQL> show parameter service

NAME TYPE VALUE ------------------------------------ ----------- ---------------------------

Oracle Database 10g: Real Application Clusters B-166

service_names string SQL> SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> show parameter service

RDBB, SERV1

NAME TYPE VALUE ------------------------------------ ----------- --------------------------service_names string RDBB, SERV2 SQL>

5. Using Database Control, how can you check that the services are currently running as expected?
a. On the Cluster Database Home page, click the Administration tab. b. In the High Availability section of the Administration page, click the Cluster Managed Database Services link. c. On the Cluster Managed Database Services: Cluster Login page, specify the oracle OS username and password. Check the Save as Preferred Credential box, and click the Login button. d. On the Cluster Managed Database Services page, see that SERV1 and SERV2 are correctly running on their preferred instances.

e. On the Cluster Managed Database Services page, select one of the services that you want to check, and click the Manage Service button. Choose SERV1. f. On the Cluster Managed Database Service: SERV1 page, you can now see the details broken down per instance.

Oracle Database 10g: Real Application Clusters B-167

6. Using a telnet session connected as user oracle to the first node, kill the SMON process of the RDBB1 instance. After a while, use SRVCTL to determine the status of your database. Open a second terminal window by using user oracle on the second node, and look at the current value of the SERVICE_NAMES initialization parameter. What do you observe? Answer: Immediately after the SMON process is killed, CRS automatically moves the SERV1 service to its available instance RDBB2. Although CRS also restarts RDBB1, SERV1 does not fall back to RDBB1.

[oracle@EDTDR1RAC1A labs]$ srvctl status database -d RDBB Instance RDBB1 is running on node edtdr1rac1a Instance RDBB2 is running on node edtdr1rac1b [oracle@EDTDR1RAC1A labs]$ ps -ef | grep smon_RDBB1 oracle 9340 1 0 Sep02 ? 00:00:29 ora_smon_RDBB1 oracle 28685 29452 0 05:12 pts/0 00:00:00 grep smon_RDBB1 [oracle@EDTDR1RAC1A labs]$ kill -9 9340 [oracle@EDTDR1RAC1A labs]$ srvctl status database -d RDBB Instance RDBB1 is running on node edtdr1rac1a Instance RDBB2 is running on node edtdr1rac1b [oracle@EDTDR1RAC1A labs]$ srvctl status database -d RDBB Instance RDBB1 is not running on node edtdr1rac1a Instance RDBB2 is running on node edtdr1rac1b [oracle@EDTDR1RAC1A labs]$ ps -ef | grep smon_RDBB1 oracle 29161 29452 0 05:13 pts/0 00:00:00 grep smon_RDBB1 [oracle@EDTDR1RAC1A labs]$ [oracle@EDTDR1RAC1A labs]$ ps -ef | grep smon_RDBB1 oracle 29055 1 0 05:13 ? 00:00:00 ora_smon_RDBB1 oracle 29161 29452 0 05:13 pts/0 00:00:00 grep smon_RDBB1

[oracle@EDTDR1RAC1A labs]$ [oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba
SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 05:15:53 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Oracle Database 10g: Real Application Clusters B-168

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- ----------------------------service_names string RDBB SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- ----------------------------service_names string RDBB, SERV2, SERV1 SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A labs]$

7. Using Database Control, check that both services are running on RDBB2.
a. On the Cluster Managed Database Services page, click SERV1 in the Service Name column.

Oracle Database 10g: Real Application Clusters B-169

b. On the Cluster Managed Database Service: SERV1 page, you can now see the details broken down per instance. SERV1 is now running on RDBB2.

8. Using Database Control, create a new user called JFV identified by the password JFV. The

default tablespace of this user should be USERS, and its temporary tablespace TEMP. Make sure that you grant this user with the CONNECT, RESOURCE, and DBA roles. Then, from rac1, using SQL*Plus, connect to SERV1 with user JFV. When connected, determine on which instance your session is currently running. Then, execute the following query: select count(*) from dba_objects,dba_objects,dba_objects

a. From the Cluster Database Home page: Administration > Users b. On the Users page, click the Create button. c. On the Create User page, specify the Name, Enter Password, Confirm Password, Default Tablespace, and Temporary Tablespace fields. d. Click the Roles tab, and then the Modify button. e. On the Modify Roles page, double-click in succession DBA, and RESOURCE roles in the Available Roles table. f. Click the OK button. g. Back on the Create User page, click the OK button.

[oracle@EDTDR1RAC1A labs]$ sqlplus jfv/jfv@SERV1 SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 05:33:33 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining

Oracle Database 10g: Real Application Clusters B-170

options SQL> set echo on; SQL> SQL> select instance_name, host_name from v$instance; INSTANCE_NAME ---------------HOST_NAME ---------------------------------------------------------------RDBB2 EDTDR1RAC1B SQL> select count(*) from dba_objects,dba_objects,dba_objects;

9. After a while, go to the EM Top Consumers page from the Cluster Database page from rac2,

and check that SERV1 is using more and more resources. Check statistics on your service with v$service_stats from a SQL*Plus session connected as SYSDBA.

From the Cluster Home page on rac2, click the Performance tab. On the middle of the Performance page, click the Top Consumers link. On the Top Consumers Overview page, refresh the page several times.

Oracle Database 10g: Real Application Clusters B-171

Oracle Database 10g: Real Application Clusters B-172

You can also see the detailed statistics by clicking the Top Services tab.

… and then by clicking on the SERV1 link.

Oracle Database 10g: Real Application Clusters B-173

… and then by clicking on the Statistics tab.

Oracle Database 10g: Real Application Clusters B-174

[oracle@ed-toraclin1b solutions]$ ./sol_07_01_09_n2.sh SQL*Plus: Release 10.1.0.2.0 - Production on Tue Nov 2 08:20:32 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> select stat_name, value from v$service_stats where service_name = 'SERV1'; STAT_NAME VALUE ---------------------------------------------------------------- ---------logons cumulative 1 user calls 23 DB time 282813818 DB CPU 259309478 parse count (total) 102 parse time elapsed 131637 execute count 604 sql execute elapsed time 282737128 opened cursors cumulative 93 session logical reads 10657 physical reads 704 STAT_NAME VALUE ---------------------------------------------------------------- ---------physical writes 0 redo size 520 user commits 1 workarea executions - optimal 29 workarea executions - onepass 0 workarea executions - multipass 0 session cursor cache hits 0 user rollbacks 0 db block changes 4 gc cr blocks received 0 gc cr block receive time 0 STAT_NAME VALUE ---------------------------------------------------------------- ---------gc current blocks received 39 gc current block receive time 6 cluster wait time 16 concurrency wait time 1 application wait time 0 user I/O wait time 47 28 rows selected. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Pn

Oracle Database 10g: Real Application Clusters B-175

With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@ed-toraclin1b solutions]$

10. Using Database Control, relocate SERV1 to its preferred instance RDBB1.

a. On the Cluster Managed Database Service page, select SERV1, and click Manage Service. b. On the Cluster Managed Database Service: SERV1 page, select the instance that is currently running SERV1 (RDBB2), and click Relocate.

c. On the Relocate Service from Instance: RDBB2 page, click OK to relocate SERV1 to RDBB1.

Oracle Database 10g: Real Application Clusters B-176

d. On the Cluster Managed Database Service: SERV1 page, you can now see the relocation confirmation.

Note: Relocating a service stops the service on the preferred or available instance it is running on and starts it on the available or preferred instance that you specify. This feature can be used to perform manual load balancing of services. You may only relocate a started service from a preferred instance to an available instance where the service is stopped, or from an available instance to a preferred instance where the service is stopped.

11. What happens to your already connected SERV1 session running on the second instance? If your

session is still executing the query, stop its execution by pressing [Ctrl] + [C]. Answer: You can see that although the service has been switched to RDBB1, your session is still executing under SERV1 on RDBB2. So, if you want to manually relocate a service, you must make sure that no session is currently connected.
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 05:33:33 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> set echo on; SQL> SQL> connect sys/oracle@RDBA2 as sysdba Connected. SQL>

Oracle Database 10g: Real Application Clusters B-177

SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- ----------------------------service_names string RDBB, SERV2 SQL> col service_name format a30 SQL> select username,service_name from v$session; USERNAME -----------------------------SYSMAN SYSMAN JFV DBSNMP DBSNMP SYSMAN SERVICE_NAME -----------------------------RDBB RDBB SERV1 SYS$USERS SYS$USERS SYS$USERS SYS$BACKGROUND RDBB SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND

USERNAME SERVICE_NAME ------------------------------ -----------------------------SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND USERNAME SERVICE_NAME ------------------------------ -----------------------------SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND SYS$BACKGROUND 28 rows selected. SQL> select instance_name, host_name from v$instance; INSTANCE_NAME ---------------HOST_NAME ---------------------------------------------------------------RDBB2 EDTDR1RAC1B SQL>

Oracle Database 10g: Real Application Clusters B-178

12. Using Database Control, create the HIGH_PRIORITY and STANDARD_PRIORITY consumer

groups. Make sure that you map HIGH_PRIORITY to SERV1, and STANDARD_PRIORITY to SERV2.
a. On the Cluster Database Home page, click the Administration tab. b. On the Administration page, click the Resource Consumer Groups link in the Resource Manager section. c. On the Resource Consumer Groups page, click the Create button. d. On the Create Resource Consumer Group page, enter HIGH_PRIORITY in the Consumer Group field, and High priority in the Description field.

e. Click the OK button. f. Go back to the Resource Consumer Groups page, and click the Create button. g. On the Create Resource Consumer Group page, enter STANDARD_PRIORITY in the Consumer Group field, and Standard priority in the Description field.

h. Click the OK button. i. Click the Cluster Database: RDBB link to return to the Administration page. j. On the Administration page, click the Resource Consumer Group Mappings link in the Resource Manager section. Oracle Database 10g: Real Application Clusters B-179

k. On the Resource Consumer Group Mapping page, click the Add Another Row button in the Service Name Map section near the end of the page. l. Select the HIGH_PRIORITY value from the Consumer Group drop-down menu of the Service Name Map section. Select the SERV1 value from the Service Name drop-down menu of the Service Name Map section.

m. On the Resource Consumer Group Mapping page, click the Add Another Row button in the Service Name Map section. n. Select the STANDARD_PRIORITY value from the Consumer Group drop-down menu of the Service Name Map section. Select the SERV2 value from the Service Name drop-down menu of the Service Name Map section.

o. On the Resource Consumer Group Mapping page, click the Apply button.

Oracle Database 10g: Real Application Clusters B-180

13. Using a SQL*Plus session, check the mappings that you just defined, and make sure that you

grant PUBLIC the right to switch to HIGH_PRIORITY. You will no longer use STANDARD_PRIORITY in this practice.
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 06:27:37 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> SQL> SQL> col value format a30 trunc col attribute format a20 trunc col consumer_group format a20 trunc SELECT ATTRIBUTE, VALUE, CONSUMER_GROUP from DBA_RSRC_GROUP_MAPPINGS; VALUE -----------------------------SERV1 SERV2 SYS SYSTEM CONSUMER_GROUP -------------------HIGH_PRIORITY STANDARD_PRIORITY SYS_GROUP SYS_GROUP

ATTRIBUTE -------------------SERVICE_NAME SERVICE_NAME ORACLE_USER ORACLE_USER

SQL> execute DBMS_RESOURCE_MANAGER_PRIVS.GRANT_SWITCH_CONSUMER_GROUP> (GRANTEE_NAME =>'PUBLIC',CONSUMER_GROUP=>'HIGH_PRIORITY',GRANT_OPTION=>FALSE); PL/SQL procedure successfully completed.

SQL>

14. Using Database Control, create a resource plan called SERVICE_PLAN. Make sure that this plan

is defined to dedicate 70% of the CPU to the HIGH_PRIORITY consumer group, and 30% to the OTHERS group. After you have created the plan, see that it is activated on both instances.
a. b. c. d.
On the Cluster Database Home page, click the Administration tab. On the Administration page, click the Resource Plans link in the Resource Manager section. On the Resource Plans page, click the Create button. On the Create Resource Plan page, enter SERVICE_PLAN in the Plan field.

Oracle Database 10g: Real Application Clusters B-181

e. In the Selected Groups/Subplans section, click the Modify button. f. On the Select Groups/Subplans page, double-click the HIGH_PRIORITY consumer group in the Available Groups/Subplans list.

g. Click OK. h. Back on the Create Resource Plan page, enter 70 in the Level 1 field of the HIGH_PRIORITY consumer group in the Selected Groups/Subplans table. Then enter 30 in the Level 1 field of the OTHER_GROUPS consumer group in the Selected Groups/Subplans table.

Oracle Database 10g: Real Application Clusters B-182

i. j.

When finished, click the OK button on the Create Resource Plan page. Back on the Resource Plans page, select SERVICE_PLAN, and select the Activate/Deactivate value from the Actions drop-down list. Then, click Go.

Oracle Database 10g: Real Application Clusters B-183

k. On the Active/Deactivate Plan: SERVICE_PLAN page, move both instances from the Available Instances list to the Selected Instances list.

l.

Click OK. SERVICE_PLAN is now activated on both instances.

15. From a telnet session on the first node, connected as user oracle, change your directory to the

labs directory. From there, execute the lab_07_01_15_n1_RDBB.sh script. This script generates a workload on the first instance. It starts two sessions that use SERV1, and two sessions that use RDBB1. How would you verify that the resource plan is followed?
Oracle Database 10g: Real Application Clusters B-184

a. b. c. d.

On the Cluster Database: RDBB Home page, click the RDBB_RDBB1 link in the Instances section. On the Cluster Database Instance: RDBB_RACDBB1 Home page, click the Administration tab. On the Administration page, click the Resource Monitors link in the Resource Manager section. On the Resource Monitors page, you should see that SERVICE_PLAN is now active, and that all resources are assigned to the current active sessions of OTHER_GROUPS.

e. Execute the lab_07_01_15_n1_RDBB.sh script from a telnet session, connected as user oracle on rac1.
[oracle@EDTDR1RAC1A labs]$ ./lab_07_01_15_n1_RDBB.sh

f.

On the Resource Monitors page, click Refresh several times until you reach almost 70% for SERV1.

Oracle Database 10g: Real Application Clusters B-185

Oracle Database 10g: Real Application Clusters B-186

g. When the number of active sessions in the HIGH_PRIORITY group is zero, many more resources are again assigned to sessions still active in OTHER_GROUPS.

Oracle Database 10g: Real Application Clusters B-187

16. Deactivate SERVICE_PLAN by using DB Control.

a. On the Cluster Database Home page, click the Administration tab. b. On the Administration page, click the Resource Plans link in the Resource Manager section. c. On the Resource Plans page, select SERVICE_PLAN, and select the Activate/Deactivate value from the Actions drop-down list. Then, click Go. d. On the Active/Deactivate Plan: SERVICE_PLAN page, move both instances from the Selected Instances list to the Available Instances list.

e. Click OK.

Oracle Database 10g: Real Application Clusters B-188

17. Set alert thresholds for your service SERV1 by using DB Control. Specify the values defined at

the beginning of this practice.
a. On the Cluster Database: RDBB Home page, click the RDBB_RDBB1 link in the Instances section of the page. b. On the Cluster Database Instance: RDBB_RBB1 page, click the Manage Metrics link in the Related Links section. c. On the Manage Metrics page, click the Edit Thresholds button. d. On the Edit Thresholds page, select the metric called Service Response Time (per user call) (microseconds).

e. Click the Specify Multiple Thresholds button.

f.

On the Specify Multiple Thresholds: Service Response Time (per user call) (microseconds) page, enter SERV1 in the Service Name field, 40000000 in the Warning Threshold field, and 100000000 in the Critical Threshold field.

g. Click OK.

Oracle Database 10g: Real Application Clusters B-189

h. Back on the Edit Thresholds page, click OK.

i. j.

On the Manage Metrics page, click the Database: RDBB_RDBB1 link. Back on the Cluster Database Instance: RDBB_RDBB1 page, click the All Metrics link in the Related Links section. k. On the All Metrics page, expand the Database Services link.

Oracle Database 10g: Real Application Clusters B-190

l. On the All Metrics page, click the Service Response Time (per user call) (microseconds) link. m. On the Service Response Time (per user call) (microseconds) page, click the SERV1 link in the Service Name column.

Oracle Database 10g: Real Application Clusters B-191

n. On the Service Response Time (per user call) (microseconds): Service Name SERV1 page, select Real Time: 30 Second Refresh from the View Data drop-down list.

18. Use SQL*Plus to connect to RDBB1 by using SERV1, and execute the lab_07_01_18.sql

script, connected as user JFV. Observe the graph. What is your conclusion? Answer: You must see that the metric passes the warning threshold. Soon after, you must see an alert raised on the DB Console.
a. Execute the script. It takes a while to complete. Do not wait its end. Look at the corresponding threshold history by using Database Control.
[oracle@EDTDR1RAC1A labs]$ sqlplus /nolog SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 09:05:31 2004 Copyright (c) 1982, 2004, Oracle. SQL> @lab_07_01_18.sql SQL> SQL> connect jfv/jfv@SERV1 Connected. SQL> SQL> SQL> DECLARE 2 t number; 3 BEGIN 4 for i in 1..10000 loop All rights reserved.

Oracle Database 10g: Real Application Clusters B-192

5 6 7 8

select count(*) into t from dba_objects; end loop; END; /

PL/SQL procedure successfully completed. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options

[oracle@EDTDR1RAC1A labs]$ b. On the Cluster Database Instance: RDBB_RDBB1 Home page, click the All Metrics link in the Related Links section. c. On the All Metrics page, expand the Database Services folder. Then, click the Service Response Time (per user call) (microseconds) link. d. On the Service Response Time (per user call) (microseconds) page, click the SERV1 link. e. On the Service Response Time (per user call) (microseconds): Service Name SERV1 page, select Real Time: 30 Seconds Refresh from the View Data drop-down list. Then monitor the Metric Value graph.

Oracle Database 10g: Real Application Clusters B-193

Oracle Database 10g: Real Application Clusters B-194

19. Use Database Control to remove the thresholds that you specified during this practice.

a. On the Cluster Database: RDBB Home page, click the RDBB_RDBB1 link in the Instances section of the page. b. On the Cluster Database Instance: RDBB_RBB1 page, click the Manage Metrics link in the Related Links section. c. On the Manage Metrics page, click the Edit Thresholds button. d. On the Edit Thresholds page, remove the thresholds that you previously set for the metric called Service Response Time (per user call) (microseconds).

Oracle Database 10g: Real Application Clusters B-195

e. Click OK.

Oracle Database 10g: Real Application Clusters B-196

Practice 08-1 Solution: Monitoring Connection High Availability
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. 1. Start both instances by using srvctl.
[oracle@EDTDR1RAC1A labs]$ srvctl start instance -d rdbb -i rdbb1,rdbb2

[oracle@EDTDR1RAC1A labs]$

2. Using SRVCTL, change your service configuration to have RDBB1 and RDBB2 as the preferred instances for SERV1, and start SERV1.
[oracle@EDTDR1RAC1A labs]$ srvctl stop service -d RDBB -s SERV1 -f [oracle@EDTDR1RAC1A labs]$ srvctl modify service -d RDBB -s SERV1 -n -i RDBB1,RDBB2 [oracle@EDTDR1RAC1A labs]$ srvctl start service -d RDBB -s SERV1

[oracle@EDTDR1RAC1A labs]$

3. Check that SERV1 is correctly started on each instance.
[oracle@EDTDR1RAC1A solutions]$ crs_stat NAME=ora.RDBB.RDBB1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.RDBB2.inst TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV1.RDBB1.srv TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.SERV1.RDBB2.srv TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV1.cs TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.RDBB.SERV2.RDBB2.sa TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a

Oracle Database 10g: Real Application Clusters B-197

NAME=ora.RDBB.SERV2.RDBB2.srv TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.SERV2.cs TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.RDBB.db TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.LISTENER_EDTDR1RAC1A.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.gsd TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1a.vip TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1a NAME=ora.edtdr1rac1b.ASM2.asm TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.LISTENER_EDTDR1RAC1B.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.gsd TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.ons TYPE=application TARGET=ONLINE STATE=ONLINE on edtdr1rac1b NAME=ora.edtdr1rac1b.vip TYPE=application

Oracle Database 10g: Real Application Clusters B-198

TARGET=ONLINE STATE=ONLINE on edtdr1rac1b

[oracle@EDTDR1RAC1A solutions]$
[oracle@EDTDR1RAC1A solutions]$ sqlplus / as sysdba @sol_08_01_03_n1_RDBB.sql SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 10:11:20 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV1 SQL> SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV2, SERV1 SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options

[oracle@EDTDR1RAC1A solutions]$

4. Check that dynamic service registration is enabled on both instances.
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 10:47:02 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> set echo on; SQL> show parameter remote_listener NAME TYPE VALUE ------------------------------------ ----------- -----------------------------remote_listener string LISTENERS_RDBB SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> show parameter remote_listener

Oracle Database 10g: Real Application Clusters B-199

NAME TYPE VALUE ------------------------------------ ----------- -----------------------------remote_listener string LISTENERS_RDBB SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A labs]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /u01/app/oracle/product/10.1.0/db_1/network/ada # Generated by Oracle configuration tools. RDBB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB1) ) ) RDBB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) ) ) SERV2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV2) ) ) SERV1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV1) ) ) LISTENERS_RDBB = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) )

Oracle Database 10g: Real Application Clusters B-200

RDBB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB2) ) )

[oracle@EDTDR1RAC1A labs]$

5. From a terminal window, and connected as user JFV in SQL*Plus, execute the createfan.sql script. This script creates a simple table used by the following scripts of this practice. This script is located in your $HOME/labs directory.

[oracle@EDTDR1RAC1A solutions]$ ./sol_08_01_05_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Mon Sep 6 11:13:33 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> connect jfv/jfv Connected. SQL> SQL> drop table fan purge; drop table fan purge * ERROR at line 1: ORA-00942: table or view does not exist SQL> SQL> create table fan(c number); Table created. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options

[oracle@EDTDR1RAC1A solutions]$

Oracle Database 10g: Real Application Clusters B-201

6. Connected as user oracle on the first terminal window on the first node, execute the sess_count_n1_RDBB.sh script located in your $HOME/labs directory. Then, connected as user oracle on the second terminal window, execute the startfanload.sh script. Looking at the first session, what is your conclusion? Note: When the workload is correctly balanced on both nodes, stop the workload by executing the stopfanlod.sh script from the third terminal window. Then you can also stop the sessions count by pressing [Ctrl] + [C] in the first session. Answer: At the beginning, one of the nodes gets almost all connections. After a while, connections are spread across both nodes.

[oracle@EDTDR1RAC1A labs]$ ./sess_count_n1_RDBB.sh RDBB1 ----15 15 15 15 15 After startfanload! 18 19 29 30 34 35 35 35 34 35 35 33 20 18 17 16 15 16 15 15 16 17 16 16 15 15 16 16 16 15 15 17 16 16 13 13 13 13 13 13 13 13 13 13 13 13 26 30 31 32 32 32 30 33 30 30 32 32 33 33 32 32 32 33 33 31 32 32 RDBB2 ----13 13 13 13 13

Oracle Database 10g: Real Application Clusters B-202

16 16 16 15 15 15 15 15 15 15 15 15 15 15 15 16 20 19 20 21 22 22 21 23 29 29 29 29 29 29 29 28 28 31 31 35 35 35 35 35 After stopfanload! 33 24 21 15 15 15 15 15 15 15 15 14 14 14 15 14 15 14 15 14

32 32 32 29 33 33 33 33 33 29 30 33 33 33 33 32 29 29 28 28 27 26 26 25 19 19 19 19 19 19 19 19 19 17 17 13 13 13 13 13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13

Oracle Database 10g: Real Application Clusters B-203

14 14 15 14 15 14 15 [oracle@ed-toraclin1a

13 13 13 13 13 13 13 labs]$

[oracle@EDTDR1RAC1A labs]$ ./startfanload.sh … PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ ./stopfanload.sh ./stopfanload.sh: line 21: kill: (11600) - No such process [oracle@EDTDR1RAC1A labs]$

7. How would you make sure that the initial connection load is balanced on each instance? Answer: Make sure that both listeners use the PREFER_LEAST_LOADED_NODE_<listener_name> parameter set to OFF. Then, restart both listeners. Note that the listener.ora file is located in $ORACLE_HOME/network/admin.
[oracle@EDTDR1RAC1A admin]$ cp listener.ora listener.ora.old [oracle@EDTDR1RAC1A admin]$ vi listener.ora # listener.ora.edtdr1rac1a Network Configuration File: /u01/app/oracle/product/10.1.0/db_1 /network/admin/listener.ora.edtdr1rac1a # Generated by Oracle configuration tools.

Oracle Database 10g: Real Application Clusters B-204

LISTENER_EDTDR1RAC1A = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 144.23.230.180)(PORT = 1521)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) PREFER_LEAST_LOADED_NODE_LISTENER_EDTDR1RAC1A=OFF ~ ~ "listener.ora" 20L, 583C written

[oracle@EDTDR1RAC1A admin]$
[oracle@EDTDR1RAC1A admin]$ lsnrctl reload LISTENER_EDTDR1RAC1A LSNRCTL for Linux: Version 10.1.0.2.0 - Production on 06-SEP-2004 13:19:24 Copyright (c) 1991, 2004, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1A)(PORT=1521))) The command completed successfully

[oracle@EDTDR1RAC1A admin]$

[oracle@EDTDR1RAC1B admin]$ cp listener.ora listener.ora.old [oracle@EDTDR1RAC1B admin]$ ls listener.ora listener.ora.old samples shrept.lst tnsnames.ora [oracle@EDTDR1RAC1B admin]$ vi listener.ora # listener.ora.edtdr1rac1b Network Configuration File: /u01/app/oracle/product/1 0.1.0/db_1/network/admin/listener.ora.edtdr1rac1b # Generated by Oracle configuration tools. LISTENER_EDTDR1RAC1B = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 144.23.230.181)(PORT = 1521)) ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC)) ) ) ) PREFER_LEAST_LOADED_NODE_LISTENER_EDTDR1RAC1B=OFF ~

Oracle Database 10g: Real Application Clusters B-205

~ "listener.ora" 20L, 583C written

[oracle@EDTDR1RAC1B admin]$
[oracle@EDTDR1RAC1B admin]$ lsnrctl reload LISTENER_EDTDR1RAC1B LSNRCTL for Linux: Version 10.1.0.2.0 - Production on 06-SEP-2004 13:19:44 Copyright (c) 1991, 2004, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1B)(PORT=1521))) The command completed successfully

[oracle@EDTDR1RAC1B admin]$

8. Perform step six again. What is your conclusion? Answer: In this case, the number of sessions is spread almost evenly across the two instances from the very beginning.

[oracle@EDTDR1RAC1A labs]$ ./sess_count_n1_RDBB.sh RDBB1 ----15 15 15 15 After startfanload! 15 16 17 17 18 18 17 19 19 19 19 19 19 21 23 24 29 31 35 31 31 29 26 25 24 21 32 32 31 31 30 30 30 29 29 29 29 29 29 27 25 24 19 17 13 17 17 19 21 23 24 26 RDBB2 ----13 13 13 13

Oracle Database 10g: Real Application Clusters B-206

18 18 16 16 17 17 19 19 22 22 21 23 23 24 25 26 26 26 26 24 21 21 16 16 15 15 15 17 17 17 19 19 19 20 21 21 21 22 22 20 After stopfanload! 18 17 16 15 15 15 15 15 14 15 [oracle@ed-toraclin1a

28 30 30 27 31 31 29 29 26 26 26 25 25 24 23 22 22 22 22 22 24 22 31 31 33 33 33 31 31 31 29 29 29 28 27 27 27 25 25 23

23 19 18 16 13 13 13 13 13 13 labs]$

[oracle@EDTDR1RAC1A labs]$ ./startfanload.sh …

Oracle Database 10g: Real Application Clusters B-207

PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ ./stopfanload.sh ./stopfanload.sh: line 21: kill: (11600) - No such process [oracle@EDTDR1RAC1A labs]$

9. Reconfigure SERV1 so that RACDB1 is its preferred instance and RACDB2 is its available instance, and start SERV1.
[oracle@EDTDR1RAC1A [oracle@EDTDR1RAC1A [oracle@EDTDR1RAC1A RDBB2 [oracle@EDTDR1RAC1A [oracle@EDTDR1RAC1A labs]$ labs]$ srvctl stop service -d RDBB -s SERV1 -f labs]$ srvctl modify service -d RDBB -s SERV1 -n -i RDBB1 -a labs]$ srvctl start service -d RDBB -s SERV1 labs]$

10. Perform step six again. What is your conclusion? Answer: All connections are directed only to instance one, which is the preferred instance.
[oracle@EDTDR1RAC1A labs]$ ./sess_count_n1_RDBB.sh RDBB1 ----15 15 15 15 After startfanload! 20 23 35 35 13 13 13 13 RDBB2 ----13 13 13 13

Oracle Database 10g: Real Application Clusters B-208

35 34 32 33 35 34 30 31 35 35 32 35 30 35 35 34 35 35 30 35 After stopfanload! 28 26 28 28 27 24 22 21 15 14 15 15 14 15 14 15 14 15 14 [oracle@ed-toraclin1a

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13

13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 labs]$

[oracle@EDTDR1RAC1A labs]$ ./startfanload.sh … PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed.

Oracle Database 10g: Real Application Clusters B-209

[oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ ./stopfanload.sh ./stopfanload.sh: line 21: kill: (11600) - No such process [oracle@EDTDR1RAC1A labs]$

11. Perform step six again. After you start the execution of the startfanload script, wait for a while, and then kill the SMON process of the RDBB1 instance from another terminal window. You can use the kill_smon_n1_RDBB.sh script for that. This script is located in your $HOME/labs directory. What do you observe? Answer: All of a sudden, the session count drops on the first node, and at the same time, sessions are no longer able to connect to the service for a while. This is because SERV1 is failing over to RDBB2. When that is done by CRS, the listeners know the new instance on which SERV1 is available. At that point, the number of connections on RDBB2 starts to rise. Meanwhile, the number of connections also starts to rise on the first instance. This means that CRS has also restarted RDBB1.
[oracle@EDTDR1RAC1A labs]$ ./sess_count_n1_RDBB.sh RDBB1 ----13 13 13 13 13 After startfanload! 18 33 33 32 33 21 27 30 33 20 After kill smon! 2 1 1 1 1 1 1 1 1 15 15 15 15 16 17 17 17 17 14 14 14 14 14 14 14 14 14 14 RDBB2 ----14 14 14 14 14

Oracle Database 10g: Real Application Clusters B-210

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1

17 17 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18

After SERV2 is failed over to RDBB2! 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 After stopfanload! 2 1 31 31 35 36 35 38 38 38 36 38 34 38 38 29 38 38 38 38 38 38 29 38 38 38 38 38 36

Oracle Database 10g: Real Application Clusters B-211

2 29 2 29 1 28 2 28 1 28 2 28 1 28 1 28 2 28 1 28 2 26 1 25 2 24 2 24 2 22 1 22 2 20 2 20 2 16 2 15 2 15 2 14 3 14 3 14 10 14 12 14 11 14 11 14 12 14 12 14 12 14 11 14 [oracle@ed-toraclin1a labs]$

[oracle@EDTDR1RAC1A labs]$ ./startfanload.sh … PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. After kill_smon1! ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Oracle Database 10g: Real Application Clusters B-212

ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. PL/SQL procedure successfully completed. [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ ./kill_smon_n1_RDBB.sh

[oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ ./stopfanload.sh ./stopfanload.sh: line 21: kill: (11600) - No such process [oracle@EDTDR1RAC1A labs]$

12. Using srvctl, make sure that you reassign SERV1 to its preferred instance.
[oracle@EDTDR1RAC1A labs]$ srvctl stop service -d RDBB -s SERV1 -f [oracle@EDTDR1RAC1A labs]$ srvctl start service -d RDBB -s SERV1 [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-213

Practice 08-2 Solution: Using Callouts
For this practice, you must start four terminal windows. You must connect as user oracle in each of them, and start two of them on rac1 and the remaining two on rac2. Open the four terminal windows now. For convenience, the first terminal window is called TW1R1. Start it on rac1. The second one is called TW2R1. Start it also on rac1. The last two are called TW3R2 and TW4R2, respectively. Start both on rac2. IMPORTANT NOTE: Before continuing, execute the callout_setup.sh script from TW1R1, and then from TW3R2. This script is located on your $HOME/labs directory.
1. Write a shell script that is able to trap FAN events generated by CRS.

The events that need to be trapped must be for the RDBB database, and the event types that need to be trapped are SERVICEMEMBER (up/down), SERVICE (up/down), and INSTANCE (up/down). The callout script should create a file that logs the events that are trapped. The log file must be located in the $ORA_CRS_HOME/racg/log directory. Make sure that the callout script is deployed on both nodes. Note: You can use TW1R1 for this step.
[oracle@EDTDR1RAC1A labs]$ cat callout1_RDBB.sh #!/bin/sh NOTIFY_EVENTTYPE=$1 AWK=/usr/bin/awk for ARGS in $*; do PROPERTY=`echo $ARGS | $AWK -F"=" '{print $1}'` VALUE=`echo $ARGS | $AWK -F"=" '{print $2}'` case $PROPERTY in VERSION|version) NOTIFY_VERSION=$VALUE ;; SERVICE|service) NOTIFY_SERVICE=$VALUE ;; DATABASE|database) NOTIFY_DATABASE=$VALUE ;; INSTANCE|instance) NOTIFY_INSTANCE=$VALUE ;; HOST|host) NOTIFY_HOST=$VALUE ;; STATUS|status) NOTIFY_STATUS=$VALUE ;; REASON|reason) NOTIFY_REASON=$VALUE ;; CARD|card) NOTIFY_CARDINALITY=$VALUE ;; TIMESTAMP|timestamp) NOTIFY_LOGDATE=$VALUE ;; ??:??:??) NOTIFY_LOGTIME=$PROPERTY ;; esac done if ((( [ $NOTIFY_EVENTTYPE = "SERVICEMEMBER" ] || [ $NOTIFY_EVENTTYPE = "SERVICE" ] || [ $NOTIFY_EVENTTYPE = "INSTANCE" ] \ ) && \ ( [ $NOTIFY_STATUS = "up" ] || \ [ $NOTIFY_STATUS = "down" ] \ )) && \ ( [ $NOTIFY_DATABASE = "RDBB" ] \ )) then echo $* >> $ORA_CRS_HOME/racg/log/crsevtco.log fi [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-214

[oracle@EDTDR1RAC1A labs]$ cp $HOME/labs/callout1_RDBB.sh $ORA_CRS_HOME/racg/usrco [oracle@EDTDR1RAC1A labs]$ chmod 777 $ORA_CRS_HOME/racg/usrco/callout1_RDBB.sh [oracle@EDTDR1RAC1A labs]$ rsh rac1b cp $HOME/labs/callout1_RDBB.sh $ORA_CRS_HOME/racg/usrco [oracle@EDTDR1RAC1A labs]$ rsh rac1b chmod 777 $ORA_CRS_HOME/racg/usrco/callout1_RDBB.sh [oracle@EDTDR1RAC1A labs]$

2. Run the following commands on both TW2R1, and TW4R2:

evmwatch -A -t "@timestamp @@" Do not wait; proceed with the next step.
TW2R1! [oracle@EDTDR1RAC1A oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: "

TW4R2!
[oracle@EDTDR1RAC1B oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: "

3. From TW1R1, stop the SERV1 service by using srvctl. When finished, look at terminal windows TR2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? Answer: You can see that on both nodes, CRS is aware of the shutdown of SERV1. However, only one log file is generated. The log file is generated on rac1 only because SERV1 is running on its preferred instance RDBB1.
TW1R1! [oracle@EDTDR1RAC1A labs]$ srvctl stop service -d RDBB -s SERV1 [oracle@EDTDR1RAC1A labs]$

TW2R1! [oracle@EDTDR1RAC1A oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state "

Oracle Database 10g: Real Application Clusters B-215

"10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004

04:34:03 04:34:03 04:34:04 04:34:07 04:34:04 04:34:08 04:34:08 04:34:13 04:34:14

RAC: ora.RDBB.SERV1.RDBB1.srv: down: " CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" CRS ora.RDBB.SERV1.RDBB1.srv stopped" RAC: ora.RDBB.SERV1.RDBB1.sa: down: " CRS ora.RDBB.SERV1.RDBB1.sa stopped" RAC: ora.RDBA.RDBA2.inst: imcheck: " RAC: ora.RDBA.RDBA2.inst: imup: " RAC: ora.RDBB.SERV1.cs: down: " CRS ora.RDBB.SERV1.cs stopped"

TW4R2!
[oracle@EDTDR1RAC1B oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " "10-Sep-2004 04:34:03 RAC: ora.RDBB.SERV1.RDBB1.srv: down: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.srv stopped" "10-Sep-2004 04:34:07 RAC: ora.RDBB.SERV1.RDBB1.sa: down: " "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.sa stopped" "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imcheck: " "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imup: " "10-Sep-2004 04:34:13 RAC: ora.RDBB.SERV1.cs: down: " "10-Sep-2004 04:34:14 CRS ora.RDBB.SERV1.cs stopped"

TW1R1!
[oracle@EDTDR1RAC1A labs]$ cd $ORA_CRS_HOME/racg/log [oracle@EDTDR1RAC1A log]$ ls crsevtco.log [oracle@EDTDR1RAC1A log]$ more crsevtco.log SERVICEMEMBER VERSION=1.0 service=SERV1 database=RDBB instance=RDBB1 host=edtdr1 rac1a status=down reason=user timestamp=10-Sep-2004 04:34:03 SERVICE VERSION=1.0 service=SERV1 database=RDBB instance= host=edtdr1rac1a statu s=down reason=user timestamp=10-Sep-2004 04:34:13 [oracle@EDTDR1RAC1A log]$

TW3R2!
[oracle@EDTDR1RAC1B labs]$ cd $ORA_CRS_HOME/racg/log [oracle@EDTDR1RAC1B log]$ ls [oracle@EDTDR1RAC1B log]$

Oracle Database 10g: Real Application Clusters B-216

4. From TW1R1, stop the SERV2 service by using srvctl. When finished, look at terminal windows TR2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? Answer: You can see that on both nodes, CRS is aware of the shutdown of SERV2. However, only the log file on rac2 shows the shutdown of SERV2. This is because SERV2 is running on its preferred instance RDBB2.

TW1R1!
[oracle@EDTDR1RAC1A log]$ srvctl stop service -d RDBB -s SERV2 [oracle@EDTDR1RAC1A log]$

TW2R1!
[oracle@EDTDR1RAC1A oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " "10-Sep-2004 04:34:03 RAC: ora.RDBB.SERV1.RDBB1.srv: down: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.srv stopped" "10-Sep-2004 04:34:07 RAC: ora.RDBB.SERV1.RDBB1.sa: down: " "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.sa stopped" "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imcheck: " "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imup: " "10-Sep-2004 04:34:13 RAC: ora.RDBB.SERV1.cs: down: " "10-Sep-2004 04:34:14 CRS ora.RDBB.SERV1.cs stopped" "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 04:37:04 04:37:04 04:39:34 04:39:34 04:40:16 04:40:16 04:40:21 04:40:21 04:40:37 04:40:40 04:40:40 04:40:41 04:40:41 04:40:38 04:40:41 04:40:50 04:40:47 04:40:56 04:40:56 RAC: ora.edtdr1rac1b.ASM2.asm: imcheck: " RAC: ora.edtdr1rac1b.ASM2.asm: imup: " RAC: ora.edtdr1rac1a.ASM1.asm: imcheck: " RAC: ora.edtdr1rac1a.ASM1.asm: imup: " RAC: ora.RDBA.RDBA1.inst: imcheck: " RAC: ora.RDBA.RDBA1.inst: imup: " RAC: ora.RDBB.RDBB2.inst: imcheck: " RAC: ora.RDBB.RDBB2.inst: imup: " CRS ora.RDBB.SERV2.cs is transitioning from state ONLINE " CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: down: " CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O" CRS ora.RDBB.SERV2.RDBB2.srv stopped" RAC: ora.RDBB.SERV2.RDBB2.sa: down: " CRS ora.RDBB.SERV2.RDBB2.sa stopped" RAC: ora.RDBB.SERV2.cs: down: " CRS ora.RDBB.SERV2.cs stopped" RAC: ora.RDBB.RDBB1.inst: imcheck: " RAC: ora.RDBB.RDBB1.inst: imup: "

Oracle Database 10g: Real Application Clusters B-217

TW4R2!
[oracle@EDTDR1RAC1B oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " "10-Sep-2004 04:34:03 RAC: ora.RDBB.SERV1.RDBB1.srv: down: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.srv stopped" "10-Sep-2004 04:34:07 RAC: ora.RDBB.SERV1.RDBB1.sa: down: " "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.sa stopped" "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imcheck: " "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imup: " "10-Sep-2004 04:34:13 RAC: ora.RDBB.SERV1.cs: down: " "10-Sep-2004 04:34:14 CRS ora.RDBB.SERV1.cs stopped" "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 04:37:04 04:37:04 04:39:34 04:39:34 04:40:16 04:40:16 04:40:21 04:40:21 04:40:37 04:40:40 04:40:40 04:40:41 04:40:41 04:40:38 04:40:41 04:40:50 04:40:47 04:40:56 04:40:56 RAC: ora.edtdr1rac1b.ASM2.asm: imcheck: " RAC: ora.edtdr1rac1b.ASM2.asm: imup: " RAC: ora.edtdr1rac1a.ASM1.asm: imcheck: " RAC: ora.edtdr1rac1a.ASM1.asm: imup: " RAC: ora.RDBA.RDBA1.inst: imcheck: " RAC: ora.RDBA.RDBA1.inst: imup: " RAC: ora.RDBB.RDBB2.inst: imcheck: " RAC: ora.RDBB.RDBB2.inst: imup: " CRS ora.RDBB.SERV2.cs is transitioning from state ONLINE " CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: down: " CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O" CRS ora.RDBB.SERV2.RDBB2.srv stopped" RAC: ora.RDBB.SERV2.RDBB2.sa: down: " CRS ora.RDBB.SERV2.RDBB2.sa stopped" RAC: ora.RDBB.SERV2.cs: down: " CRS ora.RDBB.SERV2.cs stopped" RAC: ora.RDBB.RDBB1.inst: imcheck: " RAC: ora.RDBB.RDBB1.inst: imup: "

TW1R1!
[oracle@EDTDR1RAC1A log]$ more crsevtco.log SERVICEMEMBER VERSION=1.0 service=SERV1 database=RDBB instance=RDBB1 host=edtdr1 rac1a status=down reason=user timestamp=10-Sep-2004 04:34:03 SERVICE VERSION=1.0 service=SERV1 database=RDBB instance= host=edtdr1rac1a statu s=down reason=user timestamp=10-Sep-2004 04:34:13 [oracle@EDTDR1RAC1A log]$

TW3R2!
[oracle@EDTDR1RAC1B log]$ ls crsevtco.log [oracle@EDTDR1RAC1B log]$ more crsevtco.log SERVICEMEMBER VERSION=1.0 service=SERV2 database=RDBB instance=RDBB2 host=edtdr1 rac1b status=down reason=user timestamp=10-Sep-2004 04:40:40

Oracle Database 10g: Real Application Clusters B-218

SERVICE VERSION=1.0 service=SERV2 database=RDBB instance= host=edtdr1rac1b statu s=down reason=user timestamp=10-Sep-2004 04:40:50 [oracle@EDTDR1RAC1B log]$

5. From TW1R1, start both SERV1 and SERV2 services by using srvctl. When finished, look at terminal windows TW2R1 and TW4R2, and then look at the generated log files in the $ORA_CRS_HOME/racg/log directory by using TW1R1 and TW3R2. What do you observe? Note: When finished, type [CTRL] [C] in both TW2R1 and TW4R2 to stop the evmwatch commands. Answer: You can see that on both nodes, CRS is aware of the startup of both services. Each startup appears in a different log file based on the preferred instance.

TW1R1!
[oracle@EDTDR1RAC1A log]$ srvctl start service -d RDBB -s SERV1 [oracle@EDTDR1RAC1A log]$ srvctl start service -d RDBB -s SERV2 [oracle@EDTDR1RAC1A log]$

TW2R1!
[oracle@EDTDR1RAC1A oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " "10-Sep-2004 04:34:03 RAC: ora.RDBB.SERV1.RDBB1.srv: down: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.srv stopped" "10-Sep-2004 04:34:07 RAC: ora.RDBB.SERV1.RDBB1.sa: down: " "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.sa stopped" "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imcheck: " "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imup: " "10-Sep-2004 04:34:13 RAC: ora.RDBB.SERV1.cs: down: " "10-Sep-2004 04:34:14 CRS ora.RDBB.SERV1.cs stopped" "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 04:37:04 04:37:04 04:39:34 04:39:34 04:40:16 04:40:16 04:40:21 04:40:21 04:40:37 04:40:40 04:40:40 04:40:41 04:40:41 RAC: ora.edtdr1rac1b.ASM2.asm: imcheck: " RAC: ora.edtdr1rac1b.ASM2.asm: imup: " RAC: ora.edtdr1rac1a.ASM1.asm: imcheck: " RAC: ora.edtdr1rac1a.ASM1.asm: imup: " RAC: ora.RDBA.RDBA1.inst: imcheck: " RAC: ora.RDBA.RDBA1.inst: imup: " RAC: ora.RDBB.RDBB2.inst: imcheck: " RAC: ora.RDBB.RDBB2.inst: imup: " CRS ora.RDBB.SERV2.cs is transitioning from state ONLINE " CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: down: " CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O" CRS ora.RDBB.SERV2.RDBB2.srv stopped"

Oracle Database 10g: Real Application Clusters B-219

"10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004

04:40:38 04:40:41 04:40:50 04:40:47 04:40:56 04:40:56 04:44:09 04:44:09 04:45:39 04:45:40 04:45:40 04:45:40 04:45:40 04:45:40 04:45:41 04:45:44 04:45:41 04:45:55 04:45:59 04:45:59 04:45:59 04:45:59 04:45:57 04:46:00 04:45:57 04:46:00

RAC: ora.RDBB.SERV2.RDBB2.sa: down: " CRS ora.RDBB.SERV2.RDBB2.sa stopped" RAC: ora.RDBB.SERV2.cs: down: " CRS ora.RDBB.SERV2.cs stopped" RAC: ora.RDBB.RDBB1.inst: imcheck: " RAC: ora.RDBB.RDBB1.inst: imup: " RAC: ora.RDBA.RDBA2.inst: imcheck: " RAC: ora.RDBA.RDBA2.inst: imup: " CRS ora.RDBB.SERV1.cs is transitioning from state OFFLINE" CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " RAC: ora.RDBB.SERV1.RDBB1.srv: up: " RAC: ora.RDBB.SERV1.cs: up: " CRS ora.RDBB.SERV1.RDBB1.srv started on member edtdr1rac1" CRS ora.RDBB.SERV1.cs started on member edtdr1rac1a" CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" RAC: ora.RDBB.SERV1.RDBB1.sa: up: " CRS ora.RDBB.SERV1.RDBB1.sa started on member edtdr1rac1b" CRS ora.RDBB.SERV2.cs is transitioning from state OFFLINE" CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: up: " CRS ora.RDBB.SERV2.RDBB2.srv started on member edtdr1rac1" RAC: ora.RDBB.SERV2.cs: up: " CRS ora.RDBB.SERV2.cs started on member edtdr1rac1b" CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O" RAC: ora.RDBB.SERV2.RDBB2.sa: up: " CRS ora.RDBB.SERV2.RDBB2.sa started on member edtdr1rac1a"

TW4R2!
[oracle@EDTDR1RAC1B oracle]$ evmwatch -A -t "@timestamp @@" "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imcheck: " "10-Sep-2004 04:30:55 RAC: ora.RDBB.RDBB1.inst: imup: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.cs is transitioning from state ONLINE " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " "10-Sep-2004 04:34:03 RAC: ora.RDBB.SERV1.RDBB1.srv: down: " "10-Sep-2004 04:34:03 CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.srv stopped" "10-Sep-2004 04:34:07 RAC: ora.RDBB.SERV1.RDBB1.sa: down: " "10-Sep-2004 04:34:04 CRS ora.RDBB.SERV1.RDBB1.sa stopped" "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imcheck: " "10-Sep-2004 04:34:08 RAC: ora.RDBA.RDBA2.inst: imup: " "10-Sep-2004 04:34:13 RAC: ora.RDBB.SERV1.cs: down: " "10-Sep-2004 04:34:14 CRS ora.RDBB.SERV1.cs stopped" "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 04:37:04 04:37:04 04:39:34 04:39:34 04:40:16 04:40:16 04:40:21 04:40:21 04:40:37 04:40:40 04:40:40 04:40:41 RAC: ora.edtdr1rac1b.ASM2.asm: imcheck: " RAC: ora.edtdr1rac1b.ASM2.asm: imup: " RAC: ora.edtdr1rac1a.ASM1.asm: imcheck: " RAC: ora.edtdr1rac1a.ASM1.asm: imup: " RAC: ora.RDBA.RDBA1.inst: imcheck: " RAC: ora.RDBA.RDBA1.inst: imup: " RAC: ora.RDBB.RDBB2.inst: imcheck: " RAC: ora.RDBB.RDBB2.inst: imup: " CRS ora.RDBB.SERV2.cs is transitioning from state ONLINE " CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: down: " CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O"

Oracle Database 10g: Real Application Clusters B-220

"10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004 "10-Sep-2004

04:40:41 04:40:38 04:40:41 04:40:50 04:40:47 04:40:56 04:40:56 04:44:09 04:44:09 04:45:39 04:45:40 04:45:40 04:45:40 04:45:40 04:45:40 04:45:41 04:45:44 04:45:41 04:45:55 04:45:59 04:45:59 04:45:59 04:45:59 04:45:57 04:46:00 04:45:57 04:46:00

CRS ora.RDBB.SERV2.RDBB2.srv stopped" RAC: ora.RDBB.SERV2.RDBB2.sa: down: " CRS ora.RDBB.SERV2.RDBB2.sa stopped" RAC: ora.RDBB.SERV2.cs: down: " CRS ora.RDBB.SERV2.cs stopped" RAC: ora.RDBB.RDBB1.inst: imcheck: " RAC: ora.RDBB.RDBB1.inst: imup: " RAC: ora.RDBA.RDBA2.inst: imcheck: " RAC: ora.RDBA.RDBA2.inst: imup: " CRS ora.RDBB.SERV1.cs is transitioning from state OFFLINE" CRS ora.RDBB.SERV1.RDBB1.srv is transitioning from state " RAC: ora.RDBB.SERV1.RDBB1.srv: up: " RAC: ora.RDBB.SERV1.cs: up: " CRS ora.RDBB.SERV1.RDBB1.srv started on member edtdr1rac1" CRS ora.RDBB.SERV1.cs started on member edtdr1rac1a" CRS ora.RDBB.SERV1.RDBB1.sa is transitioning from state O" RAC: ora.RDBB.SERV1.RDBB1.sa: up: " CRS ora.RDBB.SERV1.RDBB1.sa started on member edtdr1rac1b" CRS ora.RDBB.SERV2.cs is transitioning from state OFFLINE" CRS ora.RDBB.SERV2.RDBB2.srv is transitioning from state " RAC: ora.RDBB.SERV2.RDBB2.srv: up: " CRS ora.RDBB.SERV2.RDBB2.srv started on member edtdr1rac1" RAC: ora.RDBB.SERV2.cs: up: " CRS ora.RDBB.SERV2.cs started on member edtdr1rac1b" CRS ora.RDBB.SERV2.RDBB2.sa is transitioning from state O" RAC: ora.RDBB.SERV2.RDBB2.sa: up: " CRS ora.RDBB.SERV2.RDBB2.sa started on member edtdr1rac1a"

TW1R1!
[oracle@EDTDR1RAC1A log]$ more crsevtco.log SERVICEMEMBER VERSION=1.0 service=SERV1 database=RDBB instance=RDBB1 host=edtdr1 rac1a status=down reason=user timestamp=10-Sep-2004 04:34:03 SERVICE VERSION=1.0 service=SERV1 database=RDBB instance= host=edtdr1rac1a statu s=down reason=user timestamp=10-Sep-2004 04:34:13 SERVICEMEMBER VERSION=1.0 service=SERV1 database=RDBB instance=RDBB1 host=edtdr1 rac1a status=up reason=user timestamp=10-Sep-2004 04:45:40 SERVICE VERSION=1.0 service=SERV1 database=RDBB instance= host=edtdr1rac1a statu s=up reason=user timestamp=10-Sep-2004 04:45:40 [oracle@EDTDR1RAC1A log]$

TW3R2!
[oracle@EDTDR1RAC1B log]$ more crsevtco.log SERVICEMEMBER VERSION=1.0 service=SERV2 database=RDBB instance=RDBB2 host=edtdr1 rac1b status=down reason=user timestamp=10-Sep-2004 04:40:40 SERVICE VERSION=1.0 service=SERV2 database=RDBB instance= host=edtdr1rac1b statu s=down reason=user timestamp=10-Sep-2004 04:40:50 SERVICE VERSION=1.0 service=SERV2 database=RDBB instance= host=edtdr1rac1b statu s=up reason=user timestamp=10-Sep-2004 04:45:59 SERVICEMEMBER VERSION=1.0 service=SERV2 database=RDBB instance=RDBB2 host=edtdr1 rac1b status=up reason=user timestamp=10-Sep-2004 04:45:59 [oracle@EDTDR1RAC1B log]$

Oracle Database 10g: Real Application Clusters B-221

Practice 08-3 Solution: Using TAF
1.

Using srvctl from rac1, add two new services to your database, and start them. The first one is called AP and has RDBB1 and RDBB2 as its preferred instances. Its transparent application failover policy is BASIC. The second one is called ERP, and has its preferred instance as RDBB1 and its available instance as RDBB2. Its transparent application failover policy is defined to be PRECONNECT.
labs]$ srvctl add service -d RDBB -s AP -r RDBB1,RDBB2 -P labs]$ srvctl start service -d RDBB -s AP labs]$ srvctl add service -d RDBB -s ERP -r RDBB1 –a RDBB2 labs]$ srvctl start service -d RDBB -s ERP labs]$

[oracle@EDTDR1RAC1A BASIC [oracle@EDTDR1RAC1A [oracle@EDTDR1RAC1A P PRECONNECT [oracle@EDTDR1RAC1A [oracle@EDTDR1RAC1A

2. Using SQL*Plus from rac1, verify that both services are started correctly on their

corresponding instances. What do you observe? Answer: AP is started on both instances. ERP is started on its preferred instance RDBB1, and ERP_PRECONNECT is started on RDBB2.
[oracle@EDTDR1RAC1A labs]$ ./sol_08_03_02_n1_RDBB.sh SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 03:22:54 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV1, AP, ERP SQL> SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV2, AP, ERP_PRECONNEC T SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-222

3. In order to be able to connect to your database by using the AP and ERP services, what must you

do next? Answer: Add corresponding TNS name entries for the two new services on both nodes. This is not done automatically if you are not using DBCA.
[oracle@EDTDR1RAC1A labs]$ cp $ORACLE_HOME/network/admin/tnsnames.ora $ORACLE_HOME/network/admin/tnsnames.ora.old [oracle@EDTDR1RAC1A labs]$ [oracle@EDTDR1RAC1A labs]$ cat tnsnames_new.ora >> $ORACLE_HOME/network/admin/tnsnames.ora [oracle@EDTDR1RAC1A labs]$ [oracle@EDTDR1RAC1A labs]$ cat $ORACLE_HOME/network/admin/tnsnames.ora # tnsnames.ora Network Configuration File: /u01/app/oracle/product/10.1.0/db_1/a # Generated by Oracle configuration tools. RDBB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB1) ) ) RDBB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) ) ) SERV2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV2) ) ) SERV1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SERV1) ) ) LISTENERS_RDBB =

Oracle Database 10g: Real Application Clusters B-223

(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) ) LISTENERS_RDBA = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1A)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) ) RDBB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = vip-RAC1B)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = RDBB) (INSTANCE_NAME = RDBB2) ) )

AP = (DESCRIPTION=(FAILOVER=ON)(LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1A)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1B)(PORT=1521)) (CONNECT_DATA = (SERVICE_NAME = AP) (FAILOVER_MODE = (TYPE=SESSION) (METHOD=BASIC) ))) ERP = (DESCRIPTION=(FAILOVER=ON)(LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1A)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1B)(PORT=1521)) (CONNECT_DATA = (SERVICE_NAME = ERP) (FAILOVER_MODE = (BACKUP=ERP_PRECONNECT) (TYPE=SESSION) (METHOD=PRECONNECT) ))) ERP_PRECONNECT = (DESCRIPTION=(FAILOVER=ON)(LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1A)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=vip-RAC1B)(PORT=1521)) (CONNECT_DATA = (SERVICE_NAME = ERP_PRECONNECT))) [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1B labs]$ cp $ORACLE_HOME/network/admin/tnsnames.ora $ORACLE_HOME/network/admin/tnsnames.ora.old [oracle@EDTDR1RAC1B labs]$ [oracle@EDTDR1RAC1B labs]$ cat tnsnames_new.ora >> $ORACLE_HOME/network/admin/tnsnames.ora [oracle@EDTDR1RAC1B labs]$

Oracle Database 10g: Real Application Clusters B-224

4. Connect to the database by using the AP service from the second node (rac2). Use user JFV to

connect. Check which instance you are currently connected to. Also check which TAF policy you are using. How many sessions are started on each instance to support your connection? Answer: Only one session is created to support your connection. Note that this session can be started on either RDBB1 or RDBB2. In this example, it is started on RDBB2, and no corresponding session on RACDB1 exists.
On the second node! [oracle@EDTDR1RAC1B labs]$ sqlplus jfv/jfv@AP SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 04:12:16 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> select instance_name from v$instance; INSTANCE_NAME ---------------RDBB2 SQL> col service_name format a10 SQL> select failover_type,failover_method,failed_over 2 from v$session where username='JFV'; FAILOVER_TYPE FAILOVER_M FAI ------------- ---------- --SESSION BASIC NO SQL>

On the first node!
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 04:23:36 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> col service_name format a10 SQL> select failover_type,failover_method,failed_over 2 from v$session where username='JFV'; no rows selected

SQL> Oracle Database 10g: Real Application Clusters B-225

5. Still connected as user JFV from your second node, insert a row into the FAN table, and commit

your modification. From another telnet session, kill all the server processes started on the instance to which you are connected. Go back to your session and try to insert a new row in the FAN table. What happens and why? Answer: Because your server process has been killed, you are no longer able to execute correctly a SQL statement from your session.
SQL> insert into fan values(1); 1 row created. SQL> commit; Commit complete.

SQL>

[oracle@EDTDR1RAC1B labs]$ ps -ef | grep "RDBB2 (LOCAL=NO)" | awk '{print "kill -9 " $2 }' > y.sh [oracle@EDTDR1RAC1B LABS]$ . ./y.sh -bash: kill: (10405) - No such process [oracle@EDTDR1RAC1B labs]$

SQL> insert into fan values(1); insert into fan values(1) * ERROR at line 1: ORA-03113: end-of-file on communication channel SQL>

6. Try again. What happens and why?

Answer: This time it is possible to execute again a SQL command because you have been automatically failed over to an instance that offers the AP service.
SQL> insert into fan values(1); 1 row created. SQL> commit; Commit complete.

SQL>

Oracle Database 10g: Real Application Clusters B-226

7. Verify that you automatically failed over.
SQL> select instance_name from v$instance; INSTANCE_NAME ---------------RDBB2 SQL> select failover_type,failover_method,failed_over 2 from v$session where username='JFV'; FAILOVER_TYPE FAILOVER_M FAI ------------- ---------- --SESSION BASIC YES SQL>

8. Exit from the SQL*Plus session that is connected as user JFV, and perform steps 4 through 7

again. This time use the ERP service instead of AP, and make sure that you also identify which service is used when you are asked to determine the failover method used by your session. What do you observe? Answer: Because your server process has been killed in the previous step, you are no longer able to execute correctly a SQL statement from your session. The next time you insert, it is possible to execute again a SQL command because you have been automatically failed over to the RACDB2 instance that offers the ERP_PRECONNECT service.
On the second node! [oracle@EDTDR1RAC1B labs]$ sqlplus jfv/jfv@ERP SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 05:03:16 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> select instance_name from v$instance; INSTANCE_NAME ---------------RDBB1 SQL> col service_name format a15 SQL> select failover_type,failover_method,failed_over,service_name 2 from v$session where username='JFV'; FAILOVER_TYPE FAILOVER_M FAI SERVICE_NAME ------------- ---------- --- -----------SESSION PRECONNECT NO ERP SQL>

Oracle Database 10g: Real Application Clusters B-227

On the first node!
SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> col service_name format a15 SQL> select failover_type,failover_method,failed_over,service_name 2 from v$session where username='JFV'; FAILOVER_TYPE FAILOVER_M FAI SERVICE_NAME ------------- ---------- --- --------------NONE NONE NO ERP_PRECONNECT

SQL>

SQL> insert into fan values(1); 1 row created. SQL> commit; Commit complete.

SQL>

Use the kill_servers1.sh script
[oracle@EDTDR1RAC1A labs]$ ps -ef | grep "RDBB1 (LOCAL=NO)" | awk '{print "kill -9 " $2 }' > y.sh [oracle@EDTDR1RAC1B labs]$ . ./y.sh -bash: kill: (10405) - No such process [oracle@EDTDR1RAC1B labs]$

SQL> insert into fan values(1); insert into fan values(1) * ERROR at line 1: ORA-03113: end-of-file on communication channel SQL>

SQL> insert into fan values(1); 1 row created. SQL> commit; Commit complete.

SQL> Oracle Database 10g: Real Application Clusters B-228

SQL> select instance_name from v$instance; INSTANCE_NAME ---------------RDBB2 SQL> SQL> 2 3 col service_name format a15 select failover_type,failover_method,failed_over,service_name from v$session where username='JFV';

FAILOVER_TYPE FAILOVER_M FAI SERVICE_NAME ------------- ---------- --- --------------SESSION PRECONNECT YES ERP_PRECONNECT SQL>

9. What is the current repartition of the ERP service on your cluster?

Answer: ERP_PRECONNECT is started on RDBB2, and ERP is started on RDBB1.
[oracle@EDTDR1RAC1A labs]$ sqlplus /nolog SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 05:42:52 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

SQL> connect sys/oracle@RDBB1 as sysdba Connected. SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV1, AP, ERP SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV2, AP, ERP_PRECONNEC T SQL>

10. Exit from all your SQL*Plus sessions, and kill the SMON process of RDBB1. Wait for CRS to

restart RDBB1, and look at the service repartition across both instances for the ERP service. What is your conclusion? Answer: ERP failed over to RDBB2, and thus ERP_PRECONNECT is now available on RDBB1.

Oracle Database 10g: Real Application Clusters B-229

Use the kill_smon_n1.sh script. [oracle@EDTDR1RAC1A labs]$ ps -ef | grep "ora_smon_RDBB1" | awk '{print "kill -9 " $2 }' > x.sh [oracle@EDTDR1RAC1A labs]$ . ./x.sh -bash: kill: (14123) - No such process [oracle@EDTDR1RAC1A labs]$

[oracle@EDTDR1RAC1A labs]$ sqlplus /nolog SQL*Plus: Release 10.1.0.2.0 - Production on Thu Sep 9 06:01:03 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

SQL> connect sys/oracle@RDBB1 as sysdba Connected. SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, AP, ERP_PRECONNECT SQL> connect sys/oracle@RDBB2 as sysdba Connected. SQL> show parameter service NAME TYPE VALUE ------------------------------------ ----------- -----------------------------service_names string RDBB, SERV2, AP, SERV1, ERP SQL>

11. Use srvctl to stop and remove both the AP and ERP services from your cluster.

[oracle@EDTDR1RAC1A labs]$ srvctl stop service -d RDBB -s AP -f [oracle@EDTDR1RAC1A labs]$ srvctl stop service -d RDBB -s ERP -f [oracle@EDTDR1RAC1A labs]$ srvctl disable service -d RDBB -s AP,ERP [oracle@EDTDR1RAC1A labs]$ srvctl remove service -d RDBB -s AP Remove service AP for the database RDBB? (y/[n]) y [oracle@EDTDR1RAC1A labs]$ srvctl remove service -d RDBB -s ERP Remove service ERP for the database RDBB? (y/[n]) y [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-230

Practice 09-1 Solution: Backing Up and Recovering Data
1. In this practice, you place the database in ARCHIVELOG mode and enable Flashback Recovery options. In addition, you set persistent RMAN configuration parameters that are RAC friendly. When the parameters have been saved, initiate a full cluster database backup.
a. From a terminal session, create two directories, /ocfs/ARCHIVE1 and /ocfs/ARCHIVE2 as the oracle user.
$ cd /ocfs $ mkdir ARCHIVE1 ARCHIVE2 $ ls -la total 65671 drwxrwxr-x 1 oracle dba drwxr-xr-x 24 root root drwxr-xr-x 1 oracle oinstall drwxr-xr-x 1 oracle oinstall drwxrwxr-x 1 oracle dba drwxr-xr-x 1 oracle oinstall drwxrwxr-x 1 oracle dba drwxr-xr-x 1 oracle oinstall

131072 4096 131072 131072 131072 131072 131072 131072

Nov Nov Nov Nov Nov Nov Nov Nov

4 4 4 4 4 4 4 4

10:00 09:34 13:28 13:28 11:20 12:07 11:18 12:08

. .. ARCHIVE1 ARCHIVE2 CSS flash_recovery_area OCR oradata

b. Log in to Database Control, and click the Maintenance folder tab. Click Configure Recovery Settings under Backup/Recovery. Click the ARCHIVELOG mode button. Add your database name to the beginning of the archive log file name format string, %t_%s_%r.dbf in the example below. Enter /ocfs/ARCHIVE1 for the value of LOG_ARCHIVE_DEST_1 and enter /ocfs/ARCHIVE2 for the value of LOG_ARCHIVE_DEST_2. Delete entry 10.

Oracle Database 10g: Real Application Clusters B-231

c. At the bottom of the page, click the Enable flashback logging… button, and set the Flash Recovery Area size to 4 GB. Click the Apply button.

d. When the Confirmation screen appears, click the Yes button to initiate shutdown and restart of your cluster database. This is needed for the changes to take effect.

e. On the Specify Credentials page, supply the username and password of the operating system user (oracle/oracle) and Oracle database user (sys/oracle) to be used to shutdown and restart the database. Click Continue.

Oracle Database 10g: Real Application Clusters B-232

f.

The Request In Process screen appears. After waiting for a few minutes for the shutdown and restart to take place, click Refresh to continue.

g. You can now log back in to Database Control.

h. Log in to Database Control, and click the Maintenance folder tab. Click Configure Recovery Settings under Backup/Recovery. Verify that the database is in ARCHIVELOG mode and that the ARCHIVE_LOG_DEST_ parameters are properly set.

Oracle Database 10g: Real Application Clusters B-233

i.

Make sure that log archiving is taking place as configured by starting a SQL*Plus session and performing several log switches. Look in the ARCHIVE1 and ARCHIVE2 directories and make sure there are thread specific arcived logs in each directory.

$ sqlplus /nolog SQL> connect / as sysdba SQL> alter system switch logfile; System altered. SQL> alter system switch logfile; System altered. SQL> exit $ cd /ocfs $ ls -la ARCHIVE1 total 77898 drwxr-xr-x 1 oracle oinstall 131072 drwxrwxr-x 1 oracle dba 131072 -rw-r----- 1 oracle oinstall 901632 -rw-r----- 1 oracle oinstall 3072 -rw-r----- 1 oracle oinstall 2958336 -rw-r----- 1 oracle oinstall 8662016 $ ls -la ARCHIVE2 total 77898 drwxr-xr-x 1 oracle oinstall 131072 drwxrwxr-x 1 oracle dba 131072 -rw-r----- 1 oracle oinstall 901632 -rw-r----- 1 oracle oinstall 3072 -rw-r----- 1 oracle oinstall 2958336 -rw-r----- 1 oracle oinstall 8662016

Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov Nov

4 4 4 4 4 4 4 4 4 4 4 4

13:28 10:00 13:59 13:59 14:00 13:59 13:28 10:00 13:59 13:59 14:00 13:59

. .. RDBB1_10_541339807.dbf RDBB1_11_541339807.dbf RDBB1_12_541339807.dbf RDBB2_2_541339807.dbf . .. RDBB1_10_541339807.dbf RDBB1_11_541339807.dbf RDBB1_12_541339807.dbf RDBB2_2_541339807.dbf

2. From a terminal window, start an RMAN session, and adjust the following RMAN properties to take advantage of your RAC environment: • Disk parallelism • Disk channels • Snapshot controlfile name
a. Execute the rman command, and connect as sys to your RAC database. View the current values of the RMAN configuration with the show all command. $ rman
RMAN> connect target sys/oracle@RDBB connected to target database: (DBID=775469082) RMAN> show all; using target database controlfile instead of recovery catalog RMAN configuration parameters are: CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default CONFIGURE BACKUP OPTIMIZATION OFF; # default CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE MAXSETSIZE TO UNLIMITED; # default CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/10.1.0/db_1/dbs/snapcf_1.f'; # default CONFIGURE

Oracle Database 10g: Real Application Clusters B-234

ARCHIVELOG DELETION POLICY TO NONE; # default CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/10.1.0/db_1/dbs/snapcf_1.f'; # default

b. Run the RMAN configure commands as follows:
RMAN> CONFIGURE DEVICE TYPE DISK PARALLELISM 2; RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON; RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/ocfs/flash_recovery_area/RDBB/backupset/snap_rac.cf'; RMAN> CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT 'sys/oracle@RDBB1' FORMAT '/ocfs/flash_recovery_area/RDBB/backupset/%d_%T_%t_%s'; RMAN> CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT 'sys/oracle@RDBB2' FORMAT '/ocfs/flash_recovery_area/RDBB/backupset/%d_%T_%t_%s'; RMAN> show all; RMAN configuration parameters are: CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default CONFIGURE BACKUP OPTIMIZATION OFF; # default CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO BACKUPSET; CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT 'sys/oracle@RDBB1' FORMAT '/ocfs/flash_recovery_area/RDBB/backupset/%d_%T_%t_%s'; CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT 'sys/oracle@RDBB2' FORMAT '/ocfs/flash_recovery_area/RDBB/backupset/%d_%T_%t_%s'; CONFIGURE MAXSETSIZE TO UNLIMITED; # default CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/ocfs/flash_recovery_area/RDBB/backupset/snap_rac.cf';

c. Exit RMAN and create the /ocfs/flash_recovery_area/global_db_name/backupset directory as the oracle user:
$ mkdir /ocfs/flash_recovery_area/RDBB/backupset

Oracle Database 10g: Real Application Clusters B-235

3. Make a whole database backup including archived log files using Database Control.
a. On the Database Control home page, click the Maintenance folder tab, and click Schedule Backup under Backup/Recovery. Select Customized for Backup Strategy, and click the Whole Database button beside Object Type. Provide the operating system username and password (oracle/oracle) to be used, and click Next.

b. On the Options page, click the Use as the base of an incremental backup strategy button. Select Online Backup for Backup mode, and click Back up all archived logs on disk and Delete all archived logs from disk after they are successfully backup up under Advanced. Click Next to continue.

Oracle Database 10g: Real Application Clusters B-236

c. Select Disk on the Settings page. Click Next to continue.

d. On the Job Schedule page, select Immediately for the start time. Click Next to continue.

e. Make sure that the information entered on the Review page is correct. /backupset Click Submit Job.

Oracle Database 10g: Real Application Clusters B-237

f.

After the job has been submitted, you may observe the progress of the job by clicking the View job button.

g. You can drill down on the Job Name and watch the RMAN code for each job if you would like.

h. Under /ocfs, list the contents of the flash_recovery_area/RDBB/backupset directory and the ARCHIVE1 and ARCHIVE2 directories after the backup is finished. You should be able to verify the RMAN backup files and that the archived logs were deleted after they were backed up.
$ cd /ocfs $ ls –al flash_recovery_area/RDBB/backupset total 566843 drwxr-xr-x 1 oracle oinstall 131072 Nov 4 15:00 . drwxr-x--- 1 oracle oinstall 131072 Nov 4 13:59 .. -rw-r----- 1 oracle oinstall 223240192 Nov 4 15:03 RDBB_20041104_541350132_2 -rw-r----- 1 oracle oinstall 347521024 Nov 4 15:02 RDBB_20041104_541350158_1 -rw-r----- 1 oracle oinstall 14482432 Nov 4 15:05 RDBB_20041104_541350342_5 -rw-r----- 1 oracle oinstall 19585536 Nov 4 15:06 RDBB_20041104_541350368_4 -rw-r----- 1 oracle oinstall 873984 Nov 4 15:06 RDBB_20041104_541350383_6 -rw-r----- 1 oracle oinstall 9420800 Nov 4 15:04 snap_rac.cf $ ls -la ARCHIVE1 total 65665 drwxr-xr-x 1 oracle oinstall 131072 Nov 4 13:28 . drwxrwxr-x 1 oracle dba 131072 Nov 4 10:00 .. $ ls -la ARCHIVE2 total 65665 drwxr-xr-x 1 oracle oinstall 131072 Nov 4 13:28 .

drwxrwxr-x

1 oracle

dba

131072 Nov

4 10:00 ..

Oracle Database 10g: Real Application Clusters B-238

4. Create a table called DEPARTMENTS2 as a select from HR.DEPARTMENTS. Add several records. Perform a point-in-time recovery of your cluster database to reflect the state of the database before the new records were added to the DEPARTMENTS2 table.
a. On your first node, start a SQL*Plus session and create a table called DEPARTMENTS 2 from the HR.DEPARTMENTS table. Get a row count from the newly created table. Perform three log switches, then determine the current sequence.

$ sqlplus /nolog SQL> connect / as sysdba SQL> create table departments2 as select * from hr.departments; SQL> select count(*) from departments2; COUNT(*) ---------27 SQL> alter system switch logfile; System altered. SQL> alter system switch logfile; System altered. SQL> alter system switch logfile; SQL> select sequence#, thread#, status from v$log; SEQUENCE# THREAD# STATUS ---------- ---------- ---------------43 1 CURRENT 42 1 ACTIVE (Record this value for future reference) 16 2 CURRENT 15 2 ACTIVE

b. Insert 3 records into the DEPARTMENTS2 table. Commit the work and get another row count of the table. Perform a log switch when finished and then take note of the current log file sequence.
SQL> insert into departments2 values (280, 'DUMMY1','',''); SQL> insert into departments2 values (290, 'DUMMY2','',''); SQL> insert into departments2 values (300, 'DUMMY3','',''); SQL> commit; SQL> select count(*) from departments2; COUNT(*) ---------30 SQL> alter system switch logfile;

Oracle Database 10g: Real Application Clusters B-239

System altered. SQL> select sequence#, thread#, status from v$log; SEQUENCE# THREAD# STATUS ---------- ---------- ---------------43 1 ACTIVE 44 1 CURRENT 16 2 CURRENT 15 2 ACTIVE

c. Exit SQL*Plus and delete all archived log files under ARCHIVE_LOG_DEST_1 (/ocfs/ARCHIVE1) and then stop the database. Restart the database in MOUNT mode and perform recovery.
SQL> exit $ rm /ocfs/ARCHIVE1/* $ srvctl stop database -d RDBB $ srvctl start database -d RDBB -o mount $ rman target sys/oracle@RDBB connected to target database: RDBB (DBID=622888697) RMAN> run { 2> set until logseq 41 thread 1; *** use the active seq# - 1 on thread 1 (42 –1) from solutions step a. 3> set autolocate on; *** for the logseq value 4> allocate channel c1 type disk; 5> restore database; 6> recover database; 7> } executing command: SET until clause executing command: SET autolocate allocated channel: c1 channel c1: sid=261 devtype=DISK Starting restore at 07-NOV-04 channel c1: starting datafile backupset restore channel c1: specifying datafile(s) to restore from backup set restoring datafile 00003 to /ocfs/oradata/RDBB/sysaux01.dbf restoring datafile 00005 to /ocfs/oradata/RDBB/example01.dbf restoring datafile 00006 to /ocfs/oradata/RDBB/undotbs02.dbf channel c1: restored backup piece 1 piece handle=/ocfs/flash_recovery_area/RDBB/backupset/RDBB_20041107_541580142_41 tag=%TAG channel c1: restore complete channel c1: starting datafile backupset restore channel c1: specifying datafile(s) to restore from backup set restoring datafile 00001 to /ocfs/oradata/RDBB/system01.dbf restoring datafile 00002 to /ocfs/oradata/RDBB/undotbs01.dbf restoring datafile 00004 to /ocfs/oradata/RDBB/users01.dbf channel c1: restored backup piece 1 piece handle=/ocfs/flash_recovery_area/RDBB/backupset/RDBB_20041107_541583684_40 tag=%TAG channel c1: restore complete Finished restore at 07-NOV-04

Oracle Database 10g: Real Application Clusters B-240

Starting recover at 07-NOV-04 starting media recovery archive log thread 1 sequence 40 is already on disk as file /ocfs/ARCHIVE2/RDBB1_40_541499030.dbf archive log thread 2 sequence 15 is already on disk as file /ocfs/ARCHIVE2/RDBB2_15_541499030.dbf channel c1: starting archive log restore to default destination channel c1: restoring archive log archive log thread=1 sequence=39 channel c1: restoring archive log archive log thread=2 sequence=14 channel c1: restored backup piece 1 piece handle=/ocfs/flash_recovery_area/RDBB/backupset/RDBB_20041107_541581009_44 tag=%TAG channel c1: restore complete archive log filename=/ocfs/ARCHIVE2/RDBB1_39_541499030.dbf thread=1 sequence=39 archive log filename=/ocfs/ARCHIVE2/RDBB2_14_541499030.dbf thread=2 sequence=14 archive log filename=/ocfs/ARCHIVE2/RDBB2_15_541499030.dbf thread=2 sequence=15 archive log filename=/ocfs/ARCHIVE2/RDBB1_40_541499030.dbf thread=1 sequence=40 media recovery complete Finished recover at 07-NOV-04 released channel: c1 RMAN> exit

d. Stop the second instance and open the database with the resetlogs option. Get a row count from the DEPARTMENTS2 table to verify the recovery and then restart the instance on the second node.
$ srvctl stop instance -d RDBB -i RDBB2 $ sqlplus /nolog SQL> connect / as sysdba Connected. SQL> alter database open resetlogs; Database altered. SQL> select count(*) from departments2; COUNT(*) ---------27 SQL> exit $ srvctl start instance -d RDBB -i RDBB2

5. To conserve space, put your database in NOARCHIVELOG mode.
a. Stop your database. Restart the first instance in MOUNT mode and place the database in NOARCHIVELOG mode. $ srvctl stop database -d RDBB $ srvctl start instance -d RDBB -i RDBB1 -o mount Oracle Database 10g: Real Application Clusters B-241

$ sqlplus /nolog SQL> connect / as sysdba SQL> alter database noarchivelog; SQL> exit b. Shut down the instance and restart the database. Ensure the database is in NORCHIVELOG mode using SQL*Plus. $ srvctl stop instance -d RDBB -i RDBB1 $ srvctl start database -d RDBB $ sqlplus /nolog SQL> connect / as sysdba SQL> SELECT log_mode FROM v$database; LOG_MODE -----------NOARCHIVELOG

Oracle Database 10g: Real Application Clusters B-242

Practice 09-2 Solution: Working with the Cluster Registry In this exercise, you perform the Oracle Cluster Registry tasks.
1. List the most recent OCR backup. Using the ocrconfig command, locate the most recent automatic backups of the cluster registry.
$ ocrconfig -showbackup racic02 2004/10/09 15:03:44 /u01/app/oracle/product/10.1.0/crs_1/cdata/dbclust01 racic02 2004/10/09 11:03:44 /u01/app/oracle/product/10.1.0/crs_1/cdata/dbclust01 racic02 2004/10/09 07:03:43 /u01/app/oracle/product/10.1.0/crs_1/cdata/dbclust01 racic02 2004/10/08 15:03:41 /u01/app/oracle/product/10.1.0/crs_1/cdata/dbclust01 racic02 2004/10/08 15:03:41 /u01/app/oracle/product/10.1.0/crs_1/cdata/dbclust01

2. Change the OCR autobackup location to the shared disk. As the oracle user, create a directory in /ocfs/OCR called backup. After this, make the new directory the cluster registry auto backup location. $ mkdir /ocfs/OCR/backup $ ocrconfig –backuploc /ocfs/OCR/backup

3. Create a manual backup of the cluster registry. Create a manual OCR backup by using the import function of ocrconfig. $ ocrconfig -export /ocfs/OCR/backup/ocr.dbf.bk $ ls -la /ocfs/OCR/backup/ocr.dbf.bk
-rw-r--r-- 1 oracle oinstall 94227 Oct 9 17:31 /ocfs/OCR/backup/ocr.dbf.bk

Oracle Database 10g: Real Application Clusters B-243

Practice 10-1 Solution: The Perfect Scalability Case
Unless specified otherwise, you must log in as SYSDBA either through Database Control or through SQL*Plus. Before starting this lab, it is recommended that you stop your database as well Database Control. Once done, restart your database, as well Database Control. Make sure that this done on both nodes. This will ensure that you will see performance graphics that are very close to the ones shown in this solution. To do that, run the following commands from the first node: srvctl stop database –d RDBB Then issue the following commands from both nodes: emctl stop dbconsole Once done, start again your database from the first node: srvctl start database –d RDBB Finaly, restart Database Control on both nodes by executing the following commands: emctl start dbconsole 1. Execute the lab_10_01_01.sql script to create the necessary objects used throughout this practice.
[oracle@ed-toraclin4a solutions]$ ./sol_10_01_01_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Wed Nov 10 06:59:54 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> drop sequence s; drop sequence s * ERROR at line 1: ORA-02289: sequence does not exist SQL> drop table s purge; drop table s purge * ERROR at line 1: ORA-00942: table or view does not exist SQL> drop table t purge; drop table t purge * ERROR at line 1: ORA-00942: table or view does not exist SQL> SQL> create table s(sn number); Table created.

Oracle Database 10g: Real Application Clusters B-244

SQL> create table t(c number,d varchar2(20)); Table created. SQL> create index it on t(c); Index created. SQL> SQL> SQL> insert into s values(1); 1 row created. SQL> commit; Commit complete. SQL> SQL> 2 3 4 5 6 begin for i in 1..90000 loop insert into t values (i,'initial'); end loop; end; /

PL/SQL procedure successfully completed. SQL> SQL> connect / as sysdba Connected. SQL> SQL> exec dbms_advisor.set_default_task_parameter('ADDM','DB_ACTIVITY_MIN',30); PL/SQL procedure successfully completed. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@ed-toraclin4a solutions]$

2. Using Database Control, force the creation of an AWR snapshot.
a. On the Cluster Database Home page, click the first instance in the Instances section at the bottom of the page. b. On the Cluster Database Instance Home page, click the Administration tab. c. In the Workload section, click the Automatic Workload Repository link. d. On the Automatic Workload Repository page, click the Snapshots link representing the number of snapshots retained thus far. e. On the Snapshots page, click the Create button. f. On the Confirmation page, click the Yes button.

Oracle Database 10g: Real Application Clusters B-245

3. Connected as user JFV to your database from both nodes, execute the lab_10_01_03_n1.sh script from the first node, and lab_10_01_03_n2.sh from the second node. Do not wait; proceed with the next step.
-- On first instance: declare v number; begin for i in 1..40000 loop lock table s in exclusive mode; select sn into v from s; insert into t values(v,'first'); update s set sn=sn+1; commit; end loop; end; /

-- On second instance: declare v number; begin for i in 1..40000 loop lock table s in exclusive mode; select sn into v from s; insert into t values(v,'second'); update s set sn=sn+1; commit; end loop; end; /

4. Using Database Control, determine the list of blocking locks in your database.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, click the Database Locks link. c. On the Database Locks page, if you do not see any lock, refresh the page by clicking Refresh. Perform this until you see a lock. When you see a session lock, you should also see that the other session is waiting for that same lock. By clicking Refresh several times, you must see that both sessions are alternatively waiting for the other to release the exclusive lock on table S.

Oracle Database 10g: Real Application Clusters B-246

5. While the script are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Cluster wait class for the first node. What are your conclusions? Answer: By using the drilldown method of Database Control, you can quickly identify the top waiting SQL statements and the top waiting sessions on both instances. Here it appears that an UPDATE statement on table S is causing most of the waits for the Cluster wait class.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, make sure that the View Data field is set to Real Time:15 Seconds Refresh. After a few seconds, you should see something similar to the following screenshot. The graphic must clearly show that the Cluster and Application wait classes are causing most waits. You should also notice that the transaction rate is about 100 per second.

Oracle Database 10g: Real Application Clusters B-247

c. In the Sessions: Waiting and Working graphic, click the Cluster link on the right. This takes you to the Active Sessions By Instance: Cluster page.

d. On the Active Sessions By Instance: Cluster page, you must see that the number of active sessions is almost the same on both nodes. Click the RDBD1 link. This takes you to the Active Sessions Waiting: Cluster page of the first node.

Oracle Database 10g: Real Application Clusters B-248

e. On the Active Sessions Waiting: Cluster page, you can see the most important wait events causing most of the waits in the Cluster wait class on RDBD1. In the Top Waiting SQL section, click the SQL identifier that uses most of the resources (40% in this example). This takes you to the SQL Details page for the corresponding statement. You must see that the script running on RDBD1 is executing an UPDATE statement on table S that causes most of the Cluster waits.

f.

Back on the Active Sessions Waiting: Cluster page, click the link corresponding to the top waiting session in the Top Waiting Sessions section. This takes you to the Session Details page, which shows you the corresponding details.

Oracle Database 10g: Real Application Clusters B-249

6. Using Database Control look at the Cluster Cache Coherency page. What are your conclusions? Answer: The Cluster Cache Coherency page cleary shows that there are a lot of blocks transferred per second on the system. This represents more than 17% of the total logical reads. This is reflected in both the Block Access Statistics section and the Cache Coherency vs. Session Logical Reads graphic.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, click the Cluster Cache Coherency link in the Additional Monitoring Links section.

Oracle Database 10g: Real Application Clusters B-250

7. While the scripts are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Application wait class for the first node. What are your conclusions? Answer: By using the drilldown method of Database Control, you can quickly identify the top waiting SQL statements and the top waiting sessions on both instances. Here it appears that a LOCK statement on table S is causing most of the waits for the Application wait class.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, make sure that the View Data field is set to Real Time:15 Seconds Refresh. After a few seconds, you must see something similar to the following screenshot. The graphic should clearly show that the Cluster and Application wait classes are causing most waits. You must also notice that the transaction rate is about 100 per second.

c. In the Sessions: Waiting and Working graphic, click the Application link on the right. This takes you to the Active Sessions By Instance: Application page.

Oracle Database 10g: Real Application Clusters B-251

d. On the Active Sessions By Instance: Application page, you must see that the number of active sessions is almost the same on both nodes. Click the RDBD1 link. This takes you to the Active Sessions Waiting: Application page of the first node.

e. On the Active Sessions Waiting: Application page, you can see the most important wait events causing most of the waits in the Application wait class on RDBD1. In the Top Waiting SQL section, click the SQL identifier that uses most of the resources (100% in this example). This takes you to the SQL Details page for the corresponding statement. You must see that the script running on RDBD1 is executing a LOCK statement on table S that causes most of the Application waits.

Oracle Database 10g: Real Application Clusters B-252

f.

After a while, you can see that both scripts are executed by looking at the Sessions: Waiting and Working graphic again.

8. Using Database Control, force the creation of an AWR snapshot.
a. On the Cluster Database Home page, click the first instance in the Instances section at the bottom of the page. b. On the Cluster Database Instance Home page, click the Administration tab. c. In the Workload section, click the Automatic Workload Repository link. d. On the Automatic Workload Repository page, click the Snapshots link representing the number of snapshots retained thus far. e. On the Snapshots page, click the Create button. f. On the Confirmation page, click the Yes button.

9. Using Database Control, review the latest ADDM run. What are your conclusions? Answer: Looking at the last ADDM run on the first instance, it is clear that table S is locked during a significant part of the database time. By looking at the scripts, you realize that this table is used as a sequence generator, and must be locked each time a session needs a new value from the sequence.

Oracle Database 10g: Real Application Clusters B-253

a. On the Cluster Database Home page, click the Advisor Central link. b. On the Advisor Central page, make sure that the Advisory Type field is set to All Types, and that the Advisor Runs field is set to Last Run. c. In the Results table, select the latest ADDM run corresponding to Instance Number 1. Then click View Result. This takes you to the Automatic Database Diagnostic Monitor (ADDM) page.

d. On the Automatic Database Diagnostic Monitor (ADDM) page, select the Contention on table lock waits was cossuming significant database time link in the Finding column. This takes you to the corresponding Performance Finding Details page.

Oracle Database 10g: Real Application Clusters B-254

e. On the Performance Finding Details page, click the View Rationale link in the Recommendations section. This takes you to the View Rationale page, which clearly shows you the object causing the problem.

10. Execute the lab_10_01_10.sql script to create the necessary objects used for the rest of this practice.
[oracle@ed-toraclin4a solutions]$ ./sol_10_01_10_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Wed Nov 10 07:12:33 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> connect jfv/jfv Connected.

Oracle Database 10g: Real Application Clusters B-255

SQL> SQL> drop table s purge; Table dropped. SQL> drop sequence s; drop sequence s * ERROR at line 1: ORA-02289: sequence does not exist SQL> drop table t purge; Table dropped. SQL> SQL> SQL> create table t(c number, d varchar2(20)); Table created. SQL> create index it on t(c); Index created. SQL> SQL> 2 3 4 5 create sequence s start with 1 increment by 1 nomaxvalue nocache;

Sequence created. SQL> SQL> 2 3 4 5 6 begin for i in 1..90000 loop insert into t values (i,'initial'); end loop; end; /

PL/SQL procedure successfully completed. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@ed-toraclin4a solutions]$

11. Using Database Control, force the creation of an AWR snapshot.
a. On the Cluster Database Home page, click the first instance in the Instances section at the bottom of the page. b. On the Cluster Database Instance Home page, click the Administration tab. c. In the Workload section, click the Automatic Workload Repository link. Oracle Database 10g: Real Application Clusters B-256

d. On the Automatic Workload Repository page, click the Snapshots link representing the number of snapshots retained thus far. e. On the Snapshots page, click the Create button. f. On the Confirmation page, click the Yes button.

12. Connected as user JFV to your database from both nodes, execute the lab_10_01_12_n1.sh script from the first node and lab_10_01_12_n2.sh from the second node. Do not wait; proceed with the next step.

-- On first instance: begin for i in 1..40000 loop insert into t values(s.nextval,'first'); commit; end loop; end; /

-- On second instance: begin for i in 1..40000 loop insert into t values(s.nextval,'second'); commit; end loop; end; /

13. While the scripts are still executing, look at the Sessions: Waiting and Working graphic. Then, drill down to the Cluster wait class for the first node. What are your conclusions? Answer: This time, looking at the Sessions: Waiting and Working graphic, it is clear that there are no significant Application waits. Cluster is now the class on which most of the waits are occurring. Using the drilldown method of Database Control, you can quickly identify the top waiting SQL statements on both instances. Here it appears that an INSERT statement on table T is causing most of the waits for the Cluster wait class.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, make sure that the View Data field is set to Real Time:15 Seconds Refresh. After a few seconds, you must see something similar to the following screenshot. The graphic must clearly show that the Cluster wait class is causing most waits. You must also notice that the transaction rate is now around 400 per second.

Oracle Database 10g: Real Application Clusters B-257

c. In the Sessions: Waiting and Working graphic, click the Cluster link on the right. This takes you to the Active Sessions By Instance: Cluster page.

d. On the Active Sessions By Instance: Cluster page, you must see that the number of active sessions is almost the same on both nodes and is higher than the previous run. Click the RDBD1 link. This takes you to the Active Sessions Waiting: Cluster page of the first node.

Oracle Database 10g: Real Application Clusters B-258

e. On the Active Sessions Waiting: Cluster page, you can see the most important wait events causing most of the waits in the Cluster wait class on RDBC1. In the Top Waiting SQL section, click the SQL identifier that uses most of the resources (99% in this example). This takes you to the SQL Details page for the corresponding statement. You must see that the script running on RDBD1 is executing an INSERT statement on table T that causes most of the cluster waits.

f.

After a while, you can see that both scripts are executed by looking at the Sessions: Waiting and Working graphic again.

Oracle Database 10g: Real Application Clusters B-259

14. When both scripts are done, force the creation of an AWR snapshot by using Database Control.
a. On the Cluster Database Home page, click the first instance in the Instances section at the bottom of the page. b. On the Cluster Database Instance Home page, click the Administration tab. c. In the Workload section, click the Automatic Workload Repository link. d. On the Automatic Workload Repository page, click the Snapshots link representing the number of snapshots retained thus far. e. On the Snapshots page, click the Create button. f. On the Confirmation page, click the Yes button.

15. Using Database Control, review the latest ADDM run. What are your conclusions? Answer: Looking at the last ADDM run on the first instance, it is clear that the INSERT statement is causing most waits. As shown by the ADDM report, this is probably because of the cache misses for S.
a. On the Cluster Database Home page, click the Advisor Central link. b. On the Advisor Central page, make sure that the Advisory Type field is set to All Types and that the Advisor Runs field is set to Last Run. c. In the Results table, select the latest ADDM run corresponding to Instance Number 1. Then, click View Result. This takes you to the Automatic Database Diagnostic Monitor (ADDM) page. Oracle Database 10g: Real Application Clusters B-260

d. You see a Read and write contention on database blocks was consuming significant database time in the cluster link in the Finding column. This should correspond to the index always being updated on its right part.

Oracle Database 10g: Real Application Clusters B-261

e. Go back to the Automatic Database Diagnostic Monitor (ADDM) page. Click the Sequence cache misses were consuming significant database time link in the Finding column. This takes you to the corresponding Performance Finding Details page.

f.

Go back to the Automatic Database Diagnostic Monitor (ADDM) page. Then, click the SQL statements consuming significant database time were found link in the Finding column. This takes you to the corresponding Performance Finding Details page.

Oracle Database 10g: Real Application Clusters B-262

16. Execute the lab_10_01_16.sql script to create the necessary objects used for the rest of this practice.
[oracle@ed-toraclin4a solutions]$ ./sol_10_01_16_n1.sh SQL*Plus: Release 10.1.0.2.0 - Production on Wed Nov 10 07:24:34 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> SQL> drop table s purge; drop table s purge * ERROR at line 1: ORA-00942: table or view does not exist SQL> drop sequence s; Sequence dropped. SQL> drop table t purge; Table dropped. SQL> SQL> create table t(c number,d varchar2(20)); Table created. SQL> create index it on t(c); Index created. SQL> SQL> 2 3 4 5 create sequence s start with 1 increment by 1 nomaxvalue cache 90000;

Sequence created. SQL> SQL> 2 3 4 5 6 begin for i in 1..90000 loop insert into t values (i,'initial'); end loop; end; /

PL/SQL procedure successfully completed. SQL> SQL> exit; Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options

[oracle@ed-toraclin4a solutions]$ Oracle Database 10g: Real Application Clusters B-263

17. Connected as user JFV to your database from both nodes, execute the lab_10_01_17_n1.sh script from the first node and lab_10_01_17_n2.sh from the second node. Do not wait; proceed with the next step.
-- On first instance: begin for i in 1..40000 loop insert into t values(s.nextval,'first'); commit; end loop; end; /

-- On second instance: begin for i in 1..40000 loop insert into t values(s.nextval,'second'); commit; end loop; end; /

18. Until the scripts are executed, look at the Sessions: Waiting and Working graphic. What are your conclusions? Answer: This time, looking at the Sessions: Waiting and Working graphic, it is clear that there are no significant waits. The sequence has a big enough cache value to avoid the most significant waits.
a. On the Cluster Database Home page, click the Performance tab. b. On the Performance page, make sure that the View Data field is set to Real Time:15 Seconds Refresh. After both scripts have finished their execution, you must see something similar to the following screenshot. The graphic must clearly show that there are no significant waits on your cluster. You must also notice that the transaction rate is now around 1500 per second.

Oracle Database 10g: Real Application Clusters B-264

19. When both scripts are done, force the creation of an AWR snapshot by using Database Control.
a. On the Cluster Database Home page, click the first instance in the Instances section at the bottom of the page. b. On the Cluster Database Instance Home page, click the Administration tab. c. In the Workload section, click the Automatic Workload Repository link. d. On the Automatic Workload Repository page, click the Snapshots link representing the number of snapshots retained thus far. e. On the Snapshots page, click the Create button. f. On the Confirmation page, click the Yes button.

Oracle Database 10g: Real Application Clusters B-265

20. Using Database Control, review the latest ADDM run. What are your conclusions? Answer: Looking at the last ADDM run on the first instance, it is clear that there is no longer any significant issues relating to sequences.
a. On the Cluster Database Home page, click the Advisor Central link. b. On the Advisor Central page, make sure that the Advisory Type field is set to All Types and that the Advisor Runs field is set to Last Run. c. In the Results table, select the latest ADDM run corresponding to Instance Number 1. Then, click View Result. This takes you to the Automatic Database Diagnostic Monitor (ADDM) page.

Oracle Database 10g: Real Application Clusters B-266

Optional Practice Solution: Migrating Your Database to ASM
1. From rac1, connect to RDBB1 as SYSDBA under SQL*Plus. Before migrating your database to

ASM, create PFILE as a backup of SPFILE of your database.
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 10 08:40:14 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> create pfile='initRDBB.ora' from spfile; File created. SQL> SQL> host cat $ORACLE_HOME/dbs/initRDBB.ora *.background_dump_dest='/u01/app/oracle/admin/RDBB/bdump' *.cluster_database_instances=2 *.cluster_database=true *.compatible='10.1.0.2.0' *.control_files='/ocfs/oradata/RDBB/control01.ctl','/ocfs/oradata/RDBB/control0' *.core_dump_dest='/u01/app/oracle/admin/RDBB/cdump' *.db_block_size=8192 *.db_cache_size=25165824 *.db_domain='' *.db_file_multiblock_read_count=16 *.db_name='RDBB' *.db_recovery_file_dest='/ocfs/flash_recovery_area' *.db_recovery_file_dest_size=2147483648 *.dispatchers='(PROTOCOL=TCP) (SERVICE=RDBBXDB)' RDBB1.instance_number=1 RDBB2.instance_number=2 *.java_pool_size=50331648 *.job_queue_processes=10 *.large_pool_size=8388608 *.open_cursors=300 *.pga_aggregate_target=25165824 *.processes=250 *.remote_listener='LISTENERS_RDBB' *.remote_login_passwordfile='exclusive' RDBB1.resource_manager_plan='' RDBB2.resource_manager_plan='' *.shared_pool_size=99614720 *.sort_area_size=65536 RDBB1.thread=1 RDBB2.thread=2 *.undo_management='AUTO' RDBB1.undo_tablespace='UNDOTBS1' RDBB2.undo_tablespace='UNDOTBS2' *.user_dump_dest='/u01/app/oracle/admin/RDBB/udump' SQL>

Oracle Database 10g: Real Application Clusters B-267

2. Change the configuration of your SPFILE to have DB_CREATE_FILE_DEST set to +DBG1, DB_RECOVERY_FILE_DEST set to +FRAG1, and CONTROL_FILES set to +DBG1/control01_RDBB.ctl.
SQL> ALTER SYSTEM SET db_create_file_dest='+DBG1' SCOPE=SPFILE sid='*'; System altered. SQL> ALTER SYSTEM SET db_recovery_file_dest='+FRAG1' SCOPE=SPFILE sid='*'; System altered. SQL> ALTER SYSTEM SET control_files='+DBG1/control01_RDBB.ctl' SCOPE=SPFILE sid='*'; System altered. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1A labs]$

3. See that you no longer have any connection to your database, and stop your database by using srvctl.
[oracle@EDTDR1RAC1A labs]$ srvctl stop database -d RDBB [oracle@EDTDR1RAC1A labs]$

4. Make sure that you create a backup of one of the control files. Although this is not mandatory, it is the best practice.
[oracle@EDTDR1RAC1A labs]$ cp /ocfs/oradata/RDBB/control01.ctl . [oracle@EDTDR1RAC1A labs]$

5. Using Recovery Manager, restore an existing control file to ASM. When finished, back up your

database to ASM as a copy of the existing one, and switch the control file pointers to the backup. Finally, recover your database.
[oracle@EDTDR1RAC1A labs]$ rman target / nocatalog Recovery Manager: Release 10.1.0.2.0 - Production Copyright (c) 1995, 2004, Oracle. All rights reserved.

connected to target database (not started) RMAN> STARTUP NOMOUNT;

Oracle Database 10g: Real Application Clusters B-268

Oracle instance started Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers 188743680 bytes 778036 162537676 25165824 262144 bytes bytes bytes bytes

RMAN> restore controlfile to '+DBG1/control01_RDBB.ctl' from '/ocfs/oradata/RDBB/control01.ctl'; Starting restore at 10-SEP-04 allocated channel: ORA_DISK_1 channel ORA_DISK_1: sid=262 devtype=DISK channel ORA_DISK_1: copied controlfile copy Finished restore at 10-SEP-04 RMAN> RMAN> ALTER DATABASE MOUNT; database mounted released channel: ORA_DISK_1 RMAN> RMAN> BACKUP AS COPY DATABASE FORMAT '+DBG1'; Starting backup at 10-SEP-04 allocated channel: ORA_DISK_1 channel ORA_DISK_1: sid=262 devtype=DISK channel ORA_DISK_1: starting datafile copy input datafile fno=00003 name=/ocfs/oradata/RDBB/sysaux01.dbf output filename=+DBG1/rdbb/datafile/sysaux.257.3 tag=TAG20040910T095054 recid=44 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:16 channel ORA_DISK_1: starting datafile copy input datafile fno=00001 name=/ocfs/oradata/RDBB/system01.dbf output filename=+DBG1/rdbb/datafile/system.258.1 tag=TAG20040910T095054 recid=51 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:05 channel ORA_DISK_1: starting datafile copy input datafile fno=00005 name=/ocfs/oradata/RDBB/example01.dbf output filename=+DBG1/rdbb/datafile/example.259.1 tag=TAG20040910T095054 recid=3 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:46 channel ORA_DISK_1: starting datafile copy input datafile fno=00002 name=/ocfs/oradata/RDBB/undotbs01.dbf output filename=+DBG1/rdbb/datafile/undotbs1.260.1 tag=TAG20040910T095054 recid9 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35 channel ORA_DISK_1: starting datafile copy input datafile fno=00006 name=/ocfs/oradata/RDBB/undotbs02.dbf output filename=+DBG1/rdbb/datafile/undotbs2.261.1 tag=TAG20040910T095054 recid2 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07 channel ORA_DISK_1: starting datafile copy input datafile fno=00004 name=/ocfs/oradata/RDBB/users01.dbf output filename=+DBG1/rdbb/datafile/users.262.1 tag=TAG20040910T095054 recid=9 9 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:08 channel ORA_DISK_1: starting datafile copy copying current controlfile output filename=+DBG1/rdbb/controlfile/backup.263.1 tag=TAG20040910T095054 reci8 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15 channel ORA_DISK_1: starting full datafile backupset channel ORA_DISK_1: specifying datafile(s) in backupset including current SPFILE in backupset channel ORA_DISK_1: starting piece 1 at 10-SEP-04

Oracle Database 10g: Real Application Clusters B-269

channel ORA_DISK_1: finished piece 1 at 10-SEP-04 piece handle=+DBG1/rdbb/backupset/2004_09_10/nnsnf0_tag20040910t095054_0.264.1 E channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02 Finished backup at 10-SEP-04 RMAN> RMAN> SWITCH DATABASE TO COPY; datafile datafile datafile datafile datafile datafile 1 2 3 4 5 6 switched switched switched switched switched switched to to to to to to datafile datafile datafile datafile datafile datafile copy copy copy copy copy copy "+DBG1/rdbb/datafile/system.258.1" "+DBG1/rdbb/datafile/undotbs1.260.1" "+DBG1/rdbb/datafile/sysaux.257.3" "+DBG1/rdbb/datafile/users.262.1" "+DBG1/rdbb/datafile/example.259.1" "+DBG1/rdbb/datafile/undotbs2.261.1"

RMAN> RMAN> RECOVER DATABASE; Starting recover at 10-SEP-04 using channel ORA_DISK_1 starting media recovery media recovery complete Finished recover at 10-SEP-04 RMAN> RMAN> exit Recovery Manager complete. [oracle@EDTDR1RAC1A labs]$

6. Using SQL*Plus, open your database, add a new temp file to ASM, and delete the old one.
[oracle@EDTDR1RAC1A labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 10 10:01:31 2004 Copyright (c) 1982, 2004, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> ALTER DATABASE OPEN; Database altered. SQL> select tablespace_name from dba_tablespaces; TABLESPACE_NAME -----------------------------SYSTEM UNDOTBS1 SYSAUX TEMP

Oracle Database 10g: Real Application Clusters B-270

USERS UNDOTBS2 EXAMPLE 7 rows selected. SQL> SQL> ALTER TABLESPACE temp ADD TEMPFILE; Tablespace altered. SQL> select name from v$tempfile; NAME -------------------------------------------------------------------------------/ocfs/oradata/RDBB/temp01.dbf +DBG1/rdbb/tempfile/temp.265.1 SQL> ALTER DATABASE TEMPFILE '/ocfs/oradata/RDBB/temp01.dbf' DROP; Database altered. SQL>

7. Using SQL*Plus, add two new redo log groups for each thread. Make sure that those new groups are stored in ASM and all have the same size. Then, remove the old groups from each thread. When finished, see that both instances are shut down.
From rac1! SQL> select l.thread#, l.group#, l.bytes 2 from v$log l; THREAD# GROUP# BYTES ---------- ---------- ---------1 1 10485760 1 2 10485760 2 3 10485760 2 4 10485760 SQL> SQL> alter database add logfile thread 1 group 5 size 10485760; Database altered. SQL> alter database add logfile thread 1 group 6 size 10485760; Database altered. SQL> alter database add logfile thread 2 group 7 size 10485760; Database altered. SQL> alter database add logfile thread 2 group 8 size 10485760; Database altered.

Oracle Database 10g: Real Application Clusters B-271

SQL> 2 3 4 5 6 7 8 9 10 11 12 13 14

declare stmt varchar2(1024):='alter database drop logfile group 1'; asalc varchar2(1024):='alter system switch logfile'; begin for i in 1 .. 5 loop begin execute immediate stmt; exit; exception when others then execute immediate asalc; end; end loop; end; /

PL/SQL procedure successfully completed. SQL> 2 3 4 5 6 7 8 9 10 11 12 13 14 declare stmt varchar2(1024):='alter database drop logfile group 2'; asalc varchar2(1024):='alter system switch logfile'; begin for i in 1 .. 5 loop begin execute immediate stmt; exit; exception when others then execute immediate asalc; end; end loop; end; /

PL/SQL procedure successfully completed. SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> SQL> exit [oracle@EDTDR1RAC1A labs]$

From rac2! [oracle@EDTDR1RAC1B labs]$ sqlplus / as sysdba SQL*Plus: Release 10.1.0.2.0 - Production on Fri Sep 10 11:06:46 2004 Copyright (c) 1982, 2004, Oracle. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers 188743680 778036 162537676 25165824 bytes bytes bytes bytes All rights reserved.

Oracle Database 10g: Real Application Clusters B-272

Redo Buffers 262144 bytes Database mounted. Database opened. SQL> SQL> declare 2 stmt varchar2(1024):='alter database drop logfile group 3'; 3 asalc varchar2(1024):='alter system switch logfile'; 4 begin 5 for i in 1 .. 5 loop 6 begin 7 execute immediate stmt; 8 exit; 9 exception 10 when others then execute immediate asalc; 11 end; 12 end loop; 13 end; 14 / PL/SQL procedure successfully completed. SQL> SQL> 2 3 4 5 6 7 8 9 10 11 12 13 14 declare stmt varchar2(1024):='alter database drop logfile group 4'; asalc varchar2(1024):='alter system switch logfile'; begin for i in 1 .. 5 loop begin execute immediate stmt; exit; exception when others then execute immediate asalc; end; end loop; end; /

PL/SQL procedure successfully completed. SQL> SQL> select thread#,group#,status from v$log order by 1; THREAD# GROUP# STATUS ---------- ---------- ---------------1 5 CURRENT 1 6 INACTIVE 2 7 CURRENT 2 8 INACTIVE SQL> SQL> col member format a50 SQL> select group#,member from v$logfile order by 1; GROUP# MEMBER ---------- -------------------------------------------------5 +DBG1/rdbb/onlinelog/group_5.267.3 5 +FRAG1/rdbb/onlinelog/group_5.257.3 6 +DBG1/rdbb/onlinelog/group_6.268.3 6 +FRAG1/rdbb/onlinelog/group_6.258.3 7 +DBG1/rdbb/onlinelog/group_7.269.1 7 +FRAG1/rdbb/onlinelog/group_7.259.1 8 +DBG1/rdbb/onlinelog/group_8.266.1 8 +FRAG1/rdbb/onlinelog/group_8.256.1

Oracle Database 10g: Real Application Clusters B-273

8 rows selected. SQL> SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 - Pn With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EDTDR1RAC1B labs]$

8. From rac1, use srvctl to start your database.
[oracle@EDTDR1RAC1A labs]$ srvctl start database -d RDBB [oracle@EDTDR1RAC1A labs]$

Oracle Database 10g: Real Application Clusters B-274

________________________

Appendix C RAC on Windows Installation
________________________

Installing Oracle Database 10g on Windows Using Real Application Clusters (RAC) and Automated Storage Management (ASM)

Purpose This appendix describes how to install Oracle Database 10g on a Windows two-node cluster by using Real Application Clusters (RAC) and Automated Storage Management (ASM). The example in this lesson uses Oracle Database 10g Standard Edition; however, the installation process is similar to that of Oracle Database 10g Enterprise Edition. Topics This appendix discusses the following topics: • Overview • Prerequisites • Configure Disk System • Install Cluster Ready Services • Stamp the Logical Drives for ASM • Install Oracle Database 10g with Real Application Clusters • Verify Valid Path Name Exists on Both Nodes • Review Your Installation in Oracle Enterprise Manager Database Control I. Overview The installation has two phases. • In the first phase, you install and configure the Cluster Ready Services (CRS) software. • In the second phase, you install the Oracle database with RAC software. II. Prerequisites To make sure that the installation is successful, you must check the following requirements: • Software • Web browser • Hardware • Disk space • Network

Oracle Database 10g: Real Application Clusters C-2

II.1 Software The software requirements for Oracle Database 10g Real Application Clusters are: Requirement System architecture Operating system Network protocol Value 32-bit Windows 2000 with service pack 1 or later, or Windows Server 2003 TCP/IP

To check that the requirements above have been met, perform the following steps: 1. Click Start > Settings > Control Panel. Double-click System. 2. You notice that the operating system requirements have been met.

Note: To perform the installation, you need the CD-ROMs for Oracle Cluster Services and Oracle Database 10g.

Oracle Database 10g: Real Application Clusters C-3

II.2 Web Browser The following Web browsers are supported for iSQL*Plus and Oracle Enterprise Manager Database Control: • Netscape Navigator 4.78, 4.79, 7.01, or 7.1.0 • Microsoft Internet Explorer 5.5 with service pack 1 • Microsoft Internet Explorer 6.0 with service pack 2 To check that the requirements above have been met, perform the following steps: 1. Open your browser, and select Help > About <your browser>. 2. You notice that a supported Web browser version has been installed.

Oracle Database 10g: Real Application Clusters C-4

II.3 Hardware To ensure that both nodes in the cluster meet the minimum requirements to install Oracle Database 10g Real Application Clusters, perform the following steps: 1. Check that the physical RAM size is at least 512 MB. Click Start > Settings > Control Panel. Double-click System. 2. You notice that the memory requirements have been met.

Oracle Database 10g: Real Application Clusters C-5

3. Confirm that the size of the configured swap space is at least twice the physical RAM size, which in this case is 2 MB. Click Start > Settings > Control Panel. Double-click System, and then click the Advanced tab. 4. Click Performance Options.

5. Notice that Virtual Memory is set to 2 GB. Click Change.

Oracle Database 10g: Real Application Clusters C-6

6. Notice that the maximum size is set to 4 GB or twice the physical RAM size. Click OK three times.

Oracle Database 10g: Real Application Clusters C-7

II.4 Disk Space To support your RAC database, your database files must be stored on disks that are shared by both nodes in your cluster. Any shared disks supported by your hardware vendor, other than Network Attached Storage (NAS), can be used. The shared disks must be attached to both nodes in your cluster, and both nodes must be able to read and write to them. For performance and availability reasons, you must use at least two shared disks for your database files. The requirements for Oracle Real Application Clusters 10g installation and database creation are: Requirement Ancillary Oracle files Temporary space Cluster Ready Services software Database software Database files Free Space 100 MB on both nodes 50 MB on both nodes 500 MB on both nodes 1 GB on both nodes 7 GB Needed Location Any local system drive Any local system drive Any local system drive; but must be the same named drive on both nodes Any local system drive; but must be the same named drive on both nodes Shared disks

Oracle Database 10g: Real Application Clusters C-8

To confirm that you have the required amount of free disk space available, perform the following steps: 1. Open My Computer, right-click the drive that you are verifying, and click Properties.

Oracle Database 10g: Real Application Clusters C-9

2. You notice that the disk space requirements have been met.

Oracle Database 10g: Real Application Clusters C-10

II.5 Network The two nodes in the cluster must be able to communicate with each other and with external clients by using the TCP/IP protocol. Communication between clients and the nodes in the cluster is across the public network. Both nodes need a network adapter configured for the public network. To enable availability and failover, a virtual IP (VIP) address is also required for each of your nodes. A VIP address can be moved between nodes in case of a failure. CRS manages the VIP addresses for you. To support a VIP address, both nodes require an unused IP address that is compatible with the public network's subnet and netmask. The VIP address and host name must also be registered in the domain name system (DNS). For communications between the instances running on the two nodes, a private network is required. This private network connects only the nodes in the cluster and cannot be accessed from outside the cluster. Both nodes need a separate network adapter configured for this private network. Public Versus Private Network Both nodes must meet the following public and private network requirements: 1. Support for two network adapters Public network interface used for client connections Private network interfaces used for communication between database instances 2. Support of interface name restrictions Names are case sensitive. Names must not contain any multibyte language characters. Public and private network interface names must be different from each other . Names for each interface must be same on both nodes. Public and private IP addresses must be on different subnets. 3. An IP address and host name registered in the DNS for the public network interface 4. A private IP address and, optionally, a private host name for each private network interface Note: Oracle recommends that you use private network IP addresses for these interfaces; for example, 10.*.*.* or 192.168.*.*. You can use the %SystemRoot%\system32\drivers\etc\hosts file on both nodes to associate private host names with private IP addresses.

Oracle Database 10g: Real Application Clusters C-11

Host Name iwinrca01.us.oracle.co m iwinrca02.us.oracle.co m viprca01.us.oracle.com viprca02.us.oracle.com iwinrca01-node iwinrca02-node

Type Public Public Virtual Virtual Private Private

IP Address 139.185.150.201 139.185.150.202 139.185.150.54 139.185.150.55 10.10.10.2 10.10.10.3

Registered In DNS DNS DNS DNS Hosts file Hosts file

To configure or determine the IP addresses associated with your cluster nodes, perform the following steps: 1. Determine the IP addresses and names for your two public and two virtual IP addresses. On the first node, click Start > Run. Enter cmd and press [Enter]. 2. Enter ipconfig and press [Enter].

In this window, you can see the public and private network interface names, the public and virtual IP addresses, and the private IP address for node one. 3. Switch to your other node and perform steps 1 and 2 again. You see the public and private network interface names, the public and virtual IP addresses, and the private IP address for node two. Make sure that the interface names match those on node one.

Oracle Database 10g: Real Application Clusters C-12

4. Confirm that the private IP addresses are listed in the %SystemRoot%\system32\drivers\etc\hosts file.

Oracle Database 10g: Real Application Clusters C-13

5. Confirm that the two nodes can actually communicate. On both nodes, execute ping by using the IP address or alias name for the other node's private IP address and public IP address. Repeat this process on the other node.

Oracle Database 10g: Real Application Clusters C-14

III. Configure Disk System To configure the disk system, you must perform the following tasks: • Disable write caching • Prepare disks for Cluster Ready Services • Prepare disks for data storage III.1 Disable Write Caching To disable write caching, perform the following steps: 1. Click Start > Settings > Control Panel > Administrative Tools > Computer Management > Device Manager > Disk Drives. Right-click the first disk drive, and select Properties.

Oracle Database 10g: Real Application Clusters C-15

2. Click the Disk Properties tab, and deselect Write cache enabled. Then click OK.

3. Repeat the previous step for all disk drives.

Oracle Database 10g: Real Application Clusters C-16

III.2 Prepare Disks for Cluster Ready Services Cluster Ready Services (CRS) provides overall management of the cluster activities. It requires the following key files to be located in logical drives on shared disks: • Oracle Cluster Registry • Voting disk From one of the existing nodes of the cluster, perform the following steps: 1. Click Start > Settings > Control Panel > Administrative Tools > Computer Management. Expand Storage and select Disk Management.

Oracle Database 10g: Real Application Clusters C-17

2. Right-click in the free space area on Disk1, and select Create Logical Drive.

Oracle Database 10g: Real Application Clusters C-18

3. The Create Partition Wizard appears. Click Next.

4. The partition type that you want to create is Logical Drive. Click Next.

Oracle Database 10g: Real Application Clusters C-19

5. Enter 100 (MB) for the size for the first logical partition. This is the size that you want for the Oracle Cluster Registry logical drive. Click Next.

6. Select the Do not assign a drive letter or drive path option, and click Next.

Oracle Database 10g: Real Application Clusters C-20

7. Select the Do not format this partition option, and click Next.

8. Review your selections, and click Finish.

Oracle Database 10g: Real Application Clusters C-21

9. The logical drive is created. Repeat the previous steps to create a second logical drive of 20 MB for the voting disk.

10. Check the two nodes in the cluster to make sure that the partitions are visible on both nodes, and that none of the Oracle partitions have drive letters assigned.

Oracle Database 10g: Real Application Clusters C-22

III.3 Prepare Disks for Database Storage You must configure disk storage for use with Automatic Storage Management (ASM). ASM storage consists of one or more disk groups, each of which spans multiple disks. To prepare the shared disks that you identified in the prerequisites listed at the beginning of the section titled “Disk Space,” you must create two or more logical drives, each on a different shared disk, for a total of at least 7 GB. To prepare each logical drive, perform the following steps: 1. In this example, because only one disk is available, two additional logical drives are created on Disk1. All the partitions that you create for your ASM disk groups must be of at least 7 GB, must be of equal size, and must include as much free space on the disk as possible. Repeat the steps that you performed in the previous section to create the two logical drives.

Note: The amount of disk space to be used is in megabytes. So you must enter the amount accordingly.

Oracle Database 10g: Real Application Clusters C-23

IV. Install Cluster Ready Services To install Cluster Ready Services, perform the following steps: 1. Run setup.exe from the Oracle Cluster Ready Services CD-ROM. 2. When the Welcome page appears, click Next.

Oracle Database 10g: Real Application Clusters C-24

3. On the Specify File Locations page, enter the name of your Oracle Home and directory where you want to place the files. Make sure that you enter a location on the drive that contains the free space that you identified previously. Then click Next.

Note: If this is the first installation of any Oracle product, the Specify Inventory directory page appears, which provides the location where Oracle inventory of installed products is stored. Accept the default and click Next.

Oracle Database 10g: Real Application Clusters C-25

4. Accept the default selection of English on the Language Selection page, and click Next.

Oracle Database 10g: Real Application Clusters C-26

5. On the Cluster Configuration page, enter a Cluster Name and your public and private node names for both nodes. Click Next. Note: You may need to click Next more than once to go to the next page.

Oracle Database 10g: Real Application Clusters C-27

6. On the Specify Network Interface Usage page, the Oracle Universal Installer displays a list of clusterwide interfaces. Select Public for the first Local Area Connection interface and Private for the Local Area Connection 2 interface, and click Next.

Oracle Database 10g: Real Application Clusters C-28

7. On the Select Disk Formatting Options page, select the Do not format any logical drives option, and click Next.

Oracle Database 10g: Real Application Clusters C-29

8. On the Disk Configuration - Oracle Cluster Registry (OCR) page, select the partition’s disk number to hold the OCR (100 MB), and click Next.

Oracle Database 10g: Real Application Clusters C-30

9. On the Disk Configuration - Voting Disk page, select the partition’s disk number to hold the voting disk (20 MB), and click Next.

Oracle Database 10g: Real Application Clusters C-31

10. Review the list of products that will be installed, and click Install.

Oracle Database 10g: Real Application Clusters C-32

11. The progress window appears.

Oracle Database 10g: Real Application Clusters C-33

12. The Configuration Assistants page appears.

Oracle Database 10g: Real Application Clusters C-34

13. After the configuration tools complete their processing, the End of Installation page appears. Click Exit, and then Yes to close the Oracle Universal Installer.

Oracle Database 10g: Real Application Clusters C-35

V. Stamp the Logical Drives for ASM To enable disk discovery during database installation, the logical drives used to store your database files must be stamped with an ASM header using asmtoolg. All disk names created by the tool begin with the prefix ORCLDISK for identification purposes. To stamp the logical drives for ASM, perform the following steps: 1. Run <CRS_home>\BIN\asmtoolg.exe.

Oracle Database 10g: Real Application Clusters C-36

2. Accept the default to Add or change label, and click Next.

3. On the Stamp Disks page, select a disk to stamp and optionally modify the prefix. Then click Next.

Oracle Database 10g: Real Application Clusters C-37

4. Click Next.

5. Click Finish.

Oracle Database 10g: Real Application Clusters C-38

6. Repeat the steps above for the second logical disk. Note: Alternatively, you can select both disks in step 3.

Oracle Database 10g: Real Application Clusters C-39

VI. Install Oracle Database 10g with Real Application Clusters To install Oracle Database 10g with RAC, perform the following steps: 1. Run setup.exe from the Oracle Database 10g CD-ROM.

Oracle Database 10g: Real Application Clusters C-40

2. When the Welcome page appears, click Next.

Oracle Database 10g: Real Application Clusters C-41

3. On the Specify File Locations page, enter the name of your Oracle Home and directory where you want to place the files. The Name and Path in the Destination section should be different than what you specified when you installed Cluster Ready Services. Make sure that you enter a location on the drive that contains the free space that you identified previously. Then click Next.

Oracle Database 10g: Real Application Clusters C-42

4. On the Specify Hardware Cluster Installation Mode page, the Cluster Installation mode is selected by default. In addition, the local node, from which you are running the Oracle Universal Installer, is always selected. Select the additional node that is to be part of this installation session, and click Next.

Note: If the Oracle Universal Installer detects a network problem on any node that you have included in this installation, then a warning is displayed. To resolve problems, examine the installation log file: \Program
Files\Oracle\Inventory\logs\installActions<date_time>.log

Oracle Database 10g: Real Application Clusters C-43

5. On the Select Installation Type page, select the Standard Edition option, and click Next.

Oracle Database 10g: Real Application Clusters C-44

6. On the Select Database Configuration page, accept the default options to create a General Purpose starter database, and click Next.

Oracle Database 10g: Real Application Clusters C-45

7. On the Specify Database Configuration Options page, you can change the default names for your database and character set. Select the Create database with sample schemas option, and click Next.

Oracle Database 10g: Real Application Clusters C-46

8. On the Select Database Management Option page, accept the default option, and click Next.

Oracle Database 10g: Real Application Clusters C-47

9. On the Specify Database File Storage Option page, select Automatic Storage Management (ASM), and click Next.

Note: The other options are supported only by the Enterprise Edition.

Oracle Database 10g: Real Application Clusters C-48

10. On the Specify Backup and Recovery Options page, select Enable Automated Backups. Automatic Storage Management is selected by default. In the Backup Job Credentials section, enter the name and password of your current windows session user. Then click Next.

Oracle Database 10g: Real Application Clusters C-49

11. On the Configure Automatic Storage Management page, select the column for each disk that you prepared for your ASM disk groups. The Oracle Universal Installer lists only the partitions for logical drives located on disks that have been stamped with asmtoolg. Then click Next.

Oracle Database 10g: Real Application Clusters C-50

12. On the Specify Database Schema Passwords page, enter and confirm passwords for all the privileges database accounts, and click Next.

Oracle Database 10g: Real Application Clusters C-51

13. On the Summary page, review the list of software components that will be installed. Then click Install.

Oracle Database 10g: Real Application Clusters C-52

14. The progress window appears.

Oracle Database 10g: Real Application Clusters C-53

15. When the installation completes, the Welcome page for the Virtual IP Configuration Assistant appears. Click Next.

Oracle Database 10g: Real Application Clusters C-54

16. On the Network Interfaces page, select the network interface card (NIC) to which you want to assign your VIP address. Then click Next. Note: Make sure that your public NIC is selected for this purpose. If you select your private NIC, your cluster will not function correctly.

Oracle Database 10g: Real Application Clusters C-55

17. On the Virtual IPs for Cluster Nodes page, enter the virtual IP (VIP) addresses that you identified in the “Prerequisites” section, and click Next.

Oracle Database 10g: Real Application Clusters C-56

18. On the Summary page, review your entries, and click Finish.

19. A Configuration Assistant Progress Dialog page appears.

Oracle Database 10g: Real Application Clusters C-57

20. When the configuration completes, click OK.

Oracle Database 10g: Real Application Clusters C-58

21. The Configuration Results window appears. Review the results, scroll down to the bottom of the page, and click Exit.

Oracle Database 10g: Real Application Clusters C-59

22. After you exit from the VIPCA, the Net Configuration Assistant and Database Configuration Assistants start running.

Oracle Database 10g: Real Application Clusters C-60

23. The Database Configuration Assistant progress window appears.

24. When the database creation is complete, click OK.

Oracle Database 10g: Real Application Clusters C-61

25. The cluster database instances on both nodes are being started.

26. The installation is successful. Click Exit and then Yes to terminate the Oracle Universal Installer.

Oracle Database 10g: Real Application Clusters C-62

VII. Verify Valid Path Name Exists on Both Nodes To verify the path names on both nodes, perform the following steps: 1. Click Start > Settings > Control Panel > System > Advanced > Environment Variables. Select the Path variable. Make sure that the value is set to <Oracle Home>\bin. If the variable does not contain the value, add the value to the start of the path variable definition.

2. Repeat the previous step for your other node.

Oracle Database 10g: Real Application Clusters C-63

VIII. Review Your Installation in Oracle Enterprise Manager Database Control To review the results of your installation, perform the following steps: 1. Open a browser and enter the following URL: http://<hostname>.<domain>:5500/em Enter sys/oracle as SYSDBA, and click Login.

Oracle Database 10g: Real Application Clusters C-64

2. You see that your cluster database is up with two instances. In the General section, click the link next to Cluster.

Oracle Database 10g: Real Application Clusters C-65

3. The cluster database is up. Scroll down to see the list of Hosts.

Oracle Database 10g: Real Application Clusters C-66

4. You have just completed the creation of a two-node RAC database in which both nodes have an Oracle Instance, an Oracle Net Services listener, and the Oracle Enterprise Manager components for Database Control.

Oracle Database 10g: Real Application Clusters C-67

Sign up to vote on this title
UsefulNot useful