Grid Control Playbook GDMS

Version: 1.00 Date: 9/11/07

Adding a Node To 10gR2 RAC cluster
Presented by: Mengmeng Tang Sreekanth Chintala

Summary:
=============================================================

This presentation will provide detail steps for Oracle DBA and Linux Engineer to add new node to existing 10gR2 (10.2.0.3) database RAC.
The most critical steps that need to be followed are:


• • • •

Pre-install checking
Adding an Oracle clusterware home to new nodes using OUI in interactive node Adding an Oracle home to new nodes using OUI in interactive mode Reconfigure listener on new node using NETCA Adding ASM instance to new nodes manually

1 Dell Confidential Printed 2/13/2012 Printed: 02/13/12

Grid Control Playbook GDMS

Version: 1.00 Date: 9/11/07

Part I: Pre-install checking
1. Verify cluster healthy on existing nodes (ocrcheck, cluvfy) 2. Check OS version, Kernel parameters, /etc/hosts file and ensure they are identical on all nodes 3. Cross check ssh login on all nodes 4. Cross ping hostname and hostname-pn on all nodes 5. Copy .profile from existing node to new node 6. Ensure ocr, css files and shared ASM disk are visible from new node and have right permission 7. Backup crs, ocr and oraInventory on existing nodes 8. Change permission on $ORA_CRS_HOME/bin/vipca.orig and $ORA_CRS_HOME/bin/srvctl.orig on existing node change permission from $ ls -lrat *.orig -rwxr-x--x 1 root users 5380 Feb 27 2007 vipca.orig -rwxr-x--x 1 root users 5800 Feb 27 2007 srvctl.orig to $ls -lrat *.orig -rwxr-x--x 1 oracle oinstall 5380 2007-02-27 18:47 vipca.orig -rwxr-x--x 1 oracle oinstall 5800 2007-02-27 18:47 srvctl.orig 9. Comment out two lines below in $ORA_CRS_HOME/bin/srvctl on existing nodes #LD_ASSUME_KERNEL=2.4.19 #export LD_ASSUME_KERNEL 10. Add "unset LD_ASSUME_KERNEL" in $ORA_CRS_HOME/bin/vipca line 126
Step9 and 10 are 10.2.0.3 specific. The issue is fixed on 10.2.0.4. Please reference Oracle metalink Doc ID: Note:414163.1 -- 10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures)

11. Ensure $ORACLE_BASE directory exist on new node with correct permission

2 Dell Confidential Printed 2/13/2012 Printed: 02/13/12

sh script from existing node (node1) ./addNode. your $ORA_CRS_HOME environment variable must identify your successfully installed Oracle Clusterware home. . 1. Set the DISPLAY environment variable and run the addNode.00 Date: 9/11/07 Part II: Adding an Oracle Clusterware Home to a New Node Using OUI in Interactive Mode =============================================================Ensure that you have successfully installed Oracle Clusterware on at least one node in your cluster environment.Grid Control Playbook GDMS Version: 1.profile DISPLAY=ipaddress:0./ .sh 3 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .0. export DISPLAY cd $ORA_CRS_HOME/oui/bin . To use these procedures as shown.

Grid Control Playbook GDMS Version: 1.00 Date: 9/11/07 4 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .

00 Date: 9/11/07 3. 5 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . Enter the node that you want to add and verify the entries that OUI displays on the Summary Page click Next.The Oracle Universal Installer (OUI) displays the Node Selection Page Version: 1.Grid Control Playbook GDMS 2.

00 Date: 9/11/07 4. Monitor the progress of the copy crs home to new node and verify the total size of the CRS directory 6 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .Grid Control Playbook GDMS Version: 1.

00 Date: 9/11/07 5. Execute configuration scripts by root users 7 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .Grid Control Playbook GDMS Version: 1.

Run the orainstRoot.Grid Control Playbook GDMS Version: 1.2. 8 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . The execution of the script is complete b. ausdfsgriddb01:/u01/app/oracle/product/10.00 Date: 9/11/07 a.sh script on the new node if OUI prompts you to do so. ausdfsgriddb10:/u01/app/oracle/oraInventory # . Changing groupname of /u01/app/oracle/oraInventory to oinstall.0/crs_1/install # .sh script from the $ORA_CRS_HOME/install/ directory on the node from which you are running OUI.sh Changing permissions of /u01/app/oracle/oraInventory to 770./orainstRoot. Run the rootaddNode./rootaddnode.sh clscfg: EXISTING configuration version 3 detected.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.2. privgrp 'root'. Successfully accumulated necessary OCR keys./root.0/eth0 -o /u01/app/oracle/product/10.com/255.240. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: ausdfsgriddb01 ausdfsgriddb01-pn ausdfsgriddb01 node 2: ausdfsgriddb02 ausdfsgriddb02-pn ausdfsgriddb02 node 3: ausdfsgriddb03 ausdfsgriddb03-pn ausdfsgriddb03 9 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . Operation successful.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root Checking to see if Oracle CRS stack is already configured /etc/oracle does not exist. Attempting to add 1 new nodes to the configuration Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.2.us.dell.0/crs_1 # .00 Date: 9/11/07 node <nodenumber>: <nodename> <private interconnect name> <hostname> node 10: ausdfsgriddb10 ausdfsgriddb10-pn ausdfsgriddb10 Creating OCR keys for user 'root'.0/crs_1/cdata/dfsddeamercrs' does not exist./u02/oradata/ocr2 OCR backup directory '/u01/app/oracle/product/10.. Version: 1.2.2.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root clscfg: EXISTING configuration version 3 detected. Creating it now.2. Run the root.Grid Control Playbook GDMS clscfg: version 3 is 10G Release 2. Creating now Setting the permissions on OCR backup directory Setting up NS directories Oracle Cluster Registry configuration upgraded successfully WARNING: directory '/u01/app/oracle/product/10.255.sh WARNING: directory '/u01/app/oracle/product/10. clscfg: version 3 is 10G Release 2.0/crs_1/bin/srvctl add nodeapps -n ausdfsgriddb10 -A ausdfsgriddb10-vip. ausdfsgriddb10:/u01/app/oracle/product/10. OCR LOCATIONS = /u02/oradata/ocr1.2.0/crs_1 c. /u01/app/oracle/product/10.sh script on the new node from $ORA_CRS_HOME to start Oracle Clusterware on the new node.

.. NO KEYS WERE WRITTEN. ausdfsgriddb01 ausdfsgriddb02 ausdfsgriddb03 ausdfsgriddb04 ausdfsgriddb05 ausdfsgriddb06 ausdfsgriddb07 ausdfsgriddb08 ausdfsgriddb09 ausdfsgriddb10 CSS is active on all nodes...... CSS is active on these nodes.Grid Control Playbook GDMS node 4: ausdfsgriddb04 ausdfsgriddb04-pn ausdfsgriddb04 node 5: ausdfsgriddb05 ausdfsgriddb05-pn ausdfsgriddb05 node 6: ausdfsgriddb06 ausdfsgriddb06-pn ausdfsgriddb06 node 7: ausdfsgriddb07 ausdfsgriddb07-pn ausdfsgriddb07 node 8: ausdfsgriddb08 ausdfsgriddb08-pn ausdfsgriddb08 clscfg: Arguments check out successfully... Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds.. Creating GSD application resource on (0) nodes.. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds.. Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating VIP application resource on (0) nodes. 10 Dell Confidential Version: 1. Starting GSD application resource on (8) nodes.. Creating ONS application resource on (0) nodes.. Starting VIP application resource on (8) nodes.. Supply -force parameter to override... -force is destructive and will destroy any previous cluster configuration.00 Date: 9/11/07 Printed 2/13/2012 Printed: 02/13/12 ....... Starting ONS application resource on (8) nodes..

Grid Control Playbook GDMS Version: 1. and then exit OUI.00 Date: 9/11/07 Done. 11 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . Verify crs is started on new node and nodeapps are started except for listener. 6.

00 Date: 9/11/07 7. Finally.config localport=6113 remoteport=6201 loglevel=3 useocr=on 9. (SE) 12 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . run the Oracle Notification Service (RACGONS) utility as in the following example where remote_port is the port number from the previous step and node2 is the name of the node that you are adding: .crs off and then back on to ensure CRS will start properly during a reboot.From the $ORA_CRS_HOME/bin directory on an existing node. Move the Oracle created S96init. by running the following command on the existing node from the $ORA_CRS_HOME/opmn/conf directory: $ cat $ORA_CRS_HOME/opmn/conf/ons.crs script to S11 and chkconfig init./racgons add_config New_Node:Remote_Port Ex: $ . Obtain the remote port identifier. reboot the node and confirm./racgons add_config ausdfsgriddb10:6201 10.Grid Control Playbook GDMS Version: 1. which you need to know for the next step.

crs (Now within vi change to on and “:wq”) Version: 1./ .crs (Now within vi change to on and “:wq”) chkconfig -e init.d/rc3.0. To use these procedures as shown.crs chkconfig -e init. When OUI displays the Node Selection Page.d/S96init. select the node to be added and click Next.crs (Now within vi change to off and “:wq”) chkconfig -e init. export DISPLAY cd $ORACLE_HOME/oui/bin . 1.sh 2.profile DISPLAY=ipaddress:0. .sh script on node1. your $ORACLE_HOME environment variable must identify your successfully installed Oracle home. .d/rc3.d/S11init.crs /etc/init.Grid Control Playbook GDMS mv /etc/init. 13 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .00 Date: 9/11/07 Part III: Adding an Oracle Home to a New Node Using OUI in Interactive Mode ================================================ Ensure that you have successfully installed Oracle with the Oracle RAC software on at least one node in your cluster environment./addNode. Go to $ORACLE_HOME/oui/bin and run the addNode.

00 Date: 9/11/07 3. 14 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . Verify the entries that OUI displays on the Summary Page and click Next.Grid Control Playbook GDMS Version: 1.

Run the root.00 Date: 9/11/07 4.Grid Control Playbook GDMS Version: 1.sh script on the new node from Oracle_home when OUI prompts you to do so. 15 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .

.... Copying coraenv to /usr/local/bin ..sh Running Oracle10 root.0/db_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ..00 Date: 9/11/07 ausdfsgriddb10:/u01/app/oracle/product/10..Grid Control Playbook GDMS Version: 1.sh script. Creating /etc/oratab file.../root.0/db_1 # .sh script.2.2. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root. 16 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . Copying oraenv to /usr/local/bin .. The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/10.

export DISPLAY=ipaddress:0./netca & 17 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . run the Oracle Net Configuration Assistant (NETCA) to add a Listener.0 .Grid Control Playbook GDMS Now product-specific root actions will be performed. Version: 1.00 Date: 9/11/07 Part IV: reconfigure listener on new node On the new node.

00 Date: 9/11/07 18 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .Grid Control Playbook GDMS Version: 1.

00 Date: 9/11/07 19 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 .Grid Control Playbook GDMS Version: 1.

ora init+ASM10.00 Date: 9/11/07 Waiting for listener configuration to complete.ora in /u01/app/oracle/product/10.2. Verify listener is up using srvctl Part V: add ASM instance on new node manually ====================================================== a.0/db_1/dbs mv init+ASM1.0/db_1/dbs cd /u01/app/oracle/product/10.ora 20 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . add ASM entry in /etc/oratab on new node +ASM10:/u01/app/oracle/product/10.2.0/db_1:N b.Grid Control Playbook GDMS Version: 1. modify init+ASM10.2.

2.Grid Control Playbook GDMS mv orapw+ASM1 orapw+ASM10 c. Version: 1.ora to new node cd /u01/app/oracle/admin/+ASM/pfile scp ausdfsgriddb01:/u01/app/oracle/admin/+ASM/pfile/init. add ASM to cluster srvctl add asm -n ausdfsgriddb10 -i +ASM10 –o /u01/app/oracle/product/10. copy init.instance_number=10 f. It may not apply to you.00 Date: 9/11/07 e.1 Oracle Metalink: CLUSTER VERIFICATION UTILITY:Doc ID:316817.0/db_1 g.ora on all RAC nodes +ASM10.ora .1 21 Dell Confidential Printed 2/13/2012 Printed: 02/13/12 . create admin directories for ASM instance mkdir /u01/app/oracle/admin/+ASM/udump cdump hdump bdump pfile d. start asm instance srvctl start asm -n ausdfsgriddb10 Some steps in the document are specific for Dell Linux standard environment setup.1 Oracle metalink : Doc ID: 10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10 (VIPCA Failures) Note:414163. add new ASM instance in /u01/app/oracle/admin/+ASM/pfile/init. Reference ================= Oracle Metalink: Adding a Node to a 10g RAC Cluster: Note:270512.