You are on page 1of 7

Upgrading Clusterware to patchset 3 (10.2.0.

4)

Posted on 10:44 PM by Rishi Mahajan

Recently upgraded Oracle 10g RAC (10.2.0.1) to 10.2.0.4 patchset. Noticed something
interesting while applying the patchset. I have my 10g RAC running on 10.2.0.1 (base
version).I planned to apply patch as a ROLLING Upgrade. The patchset document says
the patch can be applied as rolling upgrade but when you invoke the runInstaller for
cluster upgrade, you won't get the option to select one of the two nodes (in 2 node
cluster). Both the nodes are automatically selected and grayed out (we can't make
any change). Initially i thought that my cluster is not being detected by
runInstaller.Cancelled it and set the ORACLE_HOME,PATH etc...properly but got the
same thing again...But later realised that it works like that only. The runInstaller
completes the installation and prompts to shutdown the CRS on node 1 and then run
root102.sh.This scripts bring up the cluster after execution and the process is
repeated on node 2...So here i go...

I have my 10.2.0.1 clusterware running


[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

Node-1
[root@rac1 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac1] is [10.2.0.1.0]
[root@rac1 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]

Node-2
[root@rac2 bin]# ./crsctl query crs softwareversion
CRS software version on node [rac2] is [10.2.0.1.0]
[root@rac2 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
Applying the patchset.
1) Stop service, db instance, asm and nodeapps on node 1 (rac1)
[oracle@rac1 ~]$ srvctl stop instance -i orcl1 -d orcl
[oracle@rac1 ~]$ srvctl stop asm -n rac1
[oracle@rac1 ~]$ srvctl stop nodeapps -n rac1
[oracle@rac1 ~]$ crs_stat -t

Name Type Target State Host


------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application OFFLINE OFFLINE
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip application OFFLINE OFFLINE
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2

Don't stop the CRS daemons. They would be brought down later.
[root@rac1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

2) Unzip the patch (p6810189_10204_Linux-x86.zip) and invoke runInstaller

3) In the screenshot as shown below both the nodes are grayed out and there is not
option to select one node (this was my assumpiton intially that rolling should apply
one by one with node selection option)..Click Next to move ahead...
4) Once the installer completes, it will prompt to run the following two scripts.(Each
node one at a time)
/u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs
/u01/app/oracle/product/10.2.0/crs/install/root102.sh

The first script stops the clusterware and after that second script completesthe
patching...
[root@rac1 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rac1 bin]#

[root@rac1 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl check crs


Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
[root@rac1 bin]#

[oracle@rac2 ~]$ crs_stat -t


Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac2
ora....l1.inst application OFFLINE OFFLINE
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application OFFLINE OFFLINE
ora.rac1.vip application OFFLINE OFFLINE
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac2 ~]$

[root@rac1 install]# /u01/app/oracle/product/10.2.0/crs/install/root102.sh


Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /u01/app/oracle/product/10.2.0/crs
Relinking some shared libraries.
Relinking of patched files is complete.
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
Startup will be queued to init within 30 seconds.
/etc/profile: line 59: ulimit: open files: cannot modify limit: Operation not permitted
Starting up the CRS daemons.
Waiting for the patched CRS daemons to start.
This may take a while on some systems.
.
10204 patch successfully applied.
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
[root@rac1 install]#

The above scripts will automatically startup the clusterware on node 1 (and
nodeapps,asm and instance if they are enabled for startup)

root@rac1 bin]# ./crs_stat -t


Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac2
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[root@rac1 bin]#

This is what i have in alertrac1.log under $CRS_HOME/log/rac1


[cssd(30267)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 .
2009-09-10 14:32:18.880
[evmd(29572)]CRS-1401:EVMD started on node rac1.
2009-09-10 14:32:18.927
[crsd(29700)]CRS-1012:The OCR service started on node rac1.
2009-09-10 14:32:20.661
[crsd(29700)]CRS-1201:CRSD started on node rac1.
2009-09-10 14:32:31.424
[client(30665)]CRS-1804:Node rac1 upgraded to version 10.2.0.4.0.
2009-09-10 14:32:33.507

So the software version of clusterware change to 10.2.0.4 but it would be active only
after upgrade of second node.

[root@rac1 bin]# ./crsctl query crs softwareversion


CRS software version on node [rac1] is [10.2.0.4.0]
[root@rac1 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]
[root@rac1 bin]#

5) Now is the time to repeat the scripts on node 2 (rac2). Before that shutdown the
resources i.e. Stop service, db instance, asm and nodeapps on node 2 (rac2)

[root@rac2 bin]# /u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs


Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[oracle@rac1 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE OFFLINE
ora....l1.inst application ONLINE OFFLINE
ora....l2.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE OFFLINE
ora....C2.lsnr application ONLINE OFFLINE
ora.rac2.gsd application ONLINE OFFLINE
ora.rac2.ons application ONLINE OFFLINE
ora.rac2.vip application ONLINE OFFLINE

[root@rac2 install]# /u01/app/oracle/product/10.2.0/crs/install/root102.sh


Creating pre-patch directory for saving pre-patch clusterware files
Completed patching clusterware files to /u01/app/oracle/product/10.2.0/crs
Relinking some shared libraries.
Relinking of patched files is complete.
WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/app/oracle/product' is not owned by root
WARNING: directory '/u01/app/oracle' is not owned by root
WARNING: directory '/u01/app' is not owned by root
WARNING: directory '/u01' is not owned by root
Preparing to recopy patched init and RC scripts.
Recopying init and RC scripts.
Startup will be queued to init within 30 seconds.
/etc/profile: line 59: ulimit: open files: cannot modify limit: Operation not pe
rmitted
Starting up the CRS daemons.
Waiting for the patched CRS daemons to start.
This may take a while on some systems.
.
10204 patch successfully applied.
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
clscfg -upgrade completed successfully
[root@rac2 install]#

[root@rac2 bin]# ./crsctl query crs softwareversion


CRS software version on node [rac2] is [10.2.0.4.0]
[root@rac2 bin]# ./crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.4.0]
[root@rac2 bin]#

At the end of this script, everyhting would be brought up automatically.

[oracle@rac1 ~]$ crs_stat -t


Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
[oracle@rac1 ~]$

This patched the CRS to 10.2.0.4...

You might also like