Professional Documents
Culture Documents
I had a 3 node RAC. One of the nodes had to be completely rebuilt as a result of a
problem. As there are no backups, What is the proper procedure to remove the 3rd node
from the cluster so it can be added back in?
What combinations of Oracle Clusterware, RAC and ASM versions can I use?
Is relink required for CRS_HOME after OS upgrade?
Does Oracle Clusterware or Real Application Clusters support heterogeneous platforms?
Is Infiniband supported for the RAC interconnect?
Can I run more than one clustered database on a single RAC cluster?
Can I run 9i RAC and RAC 10g in the same cluster?
I could not get the user equivalence check to work on my Solaris 10 server when trying to
install 10.2.0.1 Oracle Clusterware. The install ran fine without issue. << Message:
Result: User equivalence check failed for user "oracle". >>
Does changing uid or gid of the Oracle User affect Oracle Clusterware?
How many NICs do I need to implement RAC?
Can we output the backupset onto regular file system directly (not onto flash recovery
area) using RMAN command, when we use SE RAC?
Should the SCSI-3 reservation bit be set for our Oracle Clusterware only installation?
A client is a new RAC user and are using it in conjunction with BEA weblogic. Can they
use Connection Load Balancing and Services? What about FCF, FAN, RCLB?
Why is validateUserEquiv failing during install (or cluvfy run)?
What are the restrictions on the SID with a RAC database? Is it limited to 5 characters?
What storage is supported with Standard Edition RAC?
What would you recomend to customer, Oracle clusterware or Vendor Clusterware (I.E.
MC Service Guard, HACMP, Sun Cluster, Veritas etc.) with Oracle Database 10g Real
Application Clusters?
Can I use RAC in a distributed transaction processing environment?
Is it a good idea to add anti-virus software to my RAC cluster?
When configuring the NIC cards and switch for a GigE Interconnect should it be set to
FULL or Half duplex in RAC?
RAC Assistance
High Availability
Why am I seeing the following warnings in my listener.log for my RAC 10g environment?
WARNING: Subscription for node down event still pending
Will FAN work with SQLPlus?
Do I need to install the ONS on all my mid-tier serves in order to enable JDBC Fast
Connection Failover (FCF)?
Will FAN/FCF work with the default database service?
Can I use the 10.2 JDBC driver with 10.1 database for FCF?
What clients provide integration with FAN through FCF?
Can I use TAF and FAN/FCF?
How does the datasource properties initialLimit, minLimit, and maxLimit affect Fast
Connection Failover processing with JDBC?
Will FAN/OCI work with Instant Client?
What type of callbacks are supported with OCI when using FAN/FCF?
Does FCF for OCI react to FAN HA UP events?
Can I use FAN/OCI with Pro*C?
Do I have to link my OCI application with a thread library? Why?
Scalability
I am seeing the wait events 'ges remote message', 'gcs remote message', and/or 'gcs for
action'. What should I do about these?
What are the changes in memory requirements from moving from single instance to
RAC?
Will adding a new instance to my Oracle RAC database (new node to the cluster) allow
me to scale the workload?
What do I do if I see GC CR BLOCK LOST in my top 5 Timed Events in my AWR
Report?
Manageability
I found in 10.2 that the EM "Convert to Cluster Database" wizard would always fall over
on the last step where it runs emca and needs to log into the new cluster database as
dbsnmp to create the cluster database targets etc. I changed the password for the
dbsnmp account to be dbsnmp (same as username) and it worked OK. Is this a known
issue?
What storage option should I use for RAC 10g on Linux? ASM / OCFS / Raw Devices /
Block Devices / Ext3 ?
How do I stop the GSD?
What is the purpose of the gsd service in Oracle 9i RAC?
How should I deal with space management? Do I need to set free lists and free list
groups?
I was installing RAC and my Oracle files did not get copied to the remote node(s). What
went wrong?
If I am using Vendor Clusterware such as Veritas, IBM, Sun or HP, do I still need Oracle
Clusterware to run Oracle RAC 10g?
Srvctl cannot start instance, I get the following error PRKP-1001 CRS-0215, however
sqlplus can start it on both nodes? What is the problem?
When I look at ALL_SERVICES view in my database I see services I did not create, what
are they for?
Does RAC work with NTP (Network Time Protocol)?
I have 2 clusters named "crs" (the default), how do I get Grid Control to recognize them
as targets?
If using plsql native code, the plsql_native_library_dir needs to be defined. In RAC
environement, must the directory be in the shared storage?
How do I determine whether or not an OneOff patch is "rolling upgradeable"?
What is the Cluster Verification Utiltiy (cluvfy)?
What versions of the database can I use the cluster verification utility (cluvfy) with?
What are the implications of using srvctl disable for an instance in my RAC cluster? I
want to have it available to start if I need it but at this time to not want to run this extra
instance for this database.
Platform Specific
Does the Oracle Cluster File System (OCFS) support network access through NFS or
Windows Network Shares?
The OracleCRService does not start with my windows RAC implementation, what do I
do?
When running Oracle RAC on Windows 2003, what is the recommended OS level?
How do I verify that Host Bus Adapter Node Local Caching has been disabled for the
disks I will be using in my RAC cluster?
My customer has a failsafe cluster installed, what are the benefits of moving their system
to RAC?
When I try to login to the +ASM2 on node2 with asmcmd (after setting ORACLE_HOME
and ORACLE_SID correctly) I get: ORA-01031: insufficient privileges (DBD ERROR: OCI
SessionBegin). When I try to login to +ASM2 using sqlplus (connect / as sysdba) I get the
same ORA-01031: insufficient privileges. When I try to login to +ASM2 using sqlplus
(connect sys/passwd as sysdba) I get connected successfully.
Can I run my 9i RAC and RAC 10g on the same Windows cluster?
My customer wants to understand what type of disk caching they can use with their
Windows RAC Cluster, the install guide tells them to disable disk caching?
Is HACMP needed for RAC on AIX 5.2 using GPFS file system?
Do I need HACMP/GPFS to store my OCR/Voting file on a shared device.
Is VIO supported with RAC on IBM AIX?
Can I run Oracle RAC 10g on my IBM Mainframe Sysplex environment (z/OS)?
Can I use Oracle Clusterware for failover of the SAP Enqueue and VIP services when
running SAP in a RAC environment?
Are Oracle Applications certified with RAC?
Diagnosibility
How do I gather all relevant Oracle and OS log/trace files in a RAC cluster to provide to
Support?
What are the cdmp directories in the background_dump_dest used for?
Oracle Clusterware
Can the Network Interface Card (NIC) device names be different on the nodes in a
cluster, for both public and private?
Can I use Oracle Clusterware to provide cold failover of my single instance Oracle
Databases?
What are the licensing rules for Oracle Clusterware? Can I run it without RAC?
How do I protect the OCR and Voting in case of media failure?
What happens if I lose my voting disk(s)?
During Oracle Clusterware installation, I am asked to define a private node name, and
then on the next screen asked to define which interfaces should be used as private and
public interfaces. What information is required to answer these questions?
Can I change the name of my cluster after I have created it when I am using Oracle
Clusterware?
Which processes access the OCR ?
Why does Oracle Clusterware use an additional 'heartbeat' via the voting disk, when
other cluster software products do not?
Why does Oracle still use the voting disks when other cluster sofware is present?
How do I identify the voting file location ?
How much I/O activity should the voting disk have?
What is the voting disk used for?
Does Oracle Clusterware have to be the same or higher release than all instances
running on the cluster?
Can I use Oracle Clusterware to monitor my EM Agent?
When ct run the command 'onsctl start' receives the message "Unable to open
libhasgen10.so". Any idea why the message "unable to open libhasgen10.so" ?
What are the IP requirements for the private interconnect?
How to Restore a Lost Voting Disk used by Oracle Clusterware 10g
How can I register the listener with Oracle Clusterware in RAC 10g Release 2?
How is the voting disk used by Oracle Clusterware?
Does Oracle Clusterware support application vips?
Why is the home for Oracle Clusterware not recommended to be subdirectory of the
Oracle base directory?
How do I put my application under the control of Oracle Clusterware to achieve higher
availability?
With Oracle Clusterware 10g, how do you backup the OCR?
Is it a requirement to have the public interface linked to ETH0 or does it only need to be
on a ETH lower than the private interface?: - public on ETH1 - private on ETH2
How do I restore OCR from a backup? On Windows, can I use ocopy?
What should the permissions be set to for the voting disk and ocr when doing a RAC
Install?
How do I check the Oracle Clusterware stack and other sub-components of it?
Is there a way to verify that the Oracle Clusterware is working properly before proceeding
with RAC install?
At what point cluvfy is usable? Can I use cluvfy before installing Oracle Clusterware?
What is CVU? What are its objectives and features?
What is a stage?
What is a component?
What is nodelist?
Do I have to be root to use CVU?
What about discovery? Does CVU discover installed components?
How do I report a(or tons of) bug?
What are the requirements for CVU?
How do I install 'cvuqdisk' package?
How do I know about cluvfy commands? The usage text of cluvfy does not show
individual commands.
Do I have to type the nodelist every time for the CVU commands? Is there any shortcut?
How do I get detail output of a check?
How do I check network or node connectivity related issues?
How do I check whether OCFS is properly configured?
How do I check user accounts and administrative permissions related issues?
How do I check minimal system requirements on the nodes?
Is there a way to compare nodes?
Why the peer comparison with -refnode says passed when the group or user does not
exist?
How do I turn on tracing?
Where can I find the CVU trace files?
Why cluvfy reports "unknown" on a particular node?
What are the known issues with this release?
When I run 10.2 CLUVFY on a system where RAC 10g Release 1 is running I get
following output:
Package existence check failed for "SUNWscucm:3.1".
Package existence check failed for "SUNWudlmr:3.1".
Package existence check failed for "SUNWudlm:3.1".
Package existence check failed for
"ORCLudlm:Dev_Release_06/11/04,_64bit_3.3.4.8_reentrant".
Package existence check failed for "SUNWscr:3.1".
Package existence check failed for "SUNWscu:3.1".
Checking this Solaris system I don't see those packages installed. Can I continue my
install?
What is 'cvuqdisk' rpm? Why should I install this rpm?
Answers
I have changed my spfile with alter system set <parameter_name>
=.... scope=spfile. The spfile is on ASM storage and the database
will not start.
How to recover:
In $ORACLE_HOME/dbs
. oraenv <instance_name>
sqlplus "/ as sysdba"
startup nomount
create pfile='recoversp' from spfile
/
shutdown immediate
quit
Now edit the newly created pfile to change the parameter to something sensible.
Then:
sqlplus "/ as sysdba"
startup pfile='recoversp' (or whatever you called it in step one).
create spfile='+DATA/GASM/spfileGASM.ora' from pfile='recoversp' / N.B.The name of the
spfile is in your original init<instance_name>.ora so adjust to suit shutdown immediate
startup
quit
Note that if hardware is not identical you'll have to provide each node with it's own correct value, if
it's identical hardware you can use the -global switch.
Can we designate the place of archive logs on both ASM disk and
regular file system, when we use SE RAC?
Yes, - customers may want to create a standby database for their SE RAC database so placing
the archive logs additionally outside ASM is OK.
Why does netca always creates the listener which listens to public
ip and not VIP only?
This is for backward compatibility with existing clients: consider pre-10g to 10g server upgrade. If
we made upgraded listener to only listen on VIP, then clients that didn't upgrade will not be able
to reach this listener anymore.
How can a NAS storage vendor certify their storage solution for
RAC ?
As of January 2007 the OSCP has been discontinued!!
Please refer to this link on OTN for details on RAC Technologies Matrix (storage being part of it).
Old Answer text:
They should obtain an OCE test kit and complete the required RAC tests. They can submit the
request for an OCE kit to ocesup_ie@oracle.com.
The list of certified NAS vendors/solutions is posted on OTN under the OSCP program
Should the SCSI-3 reservation bit be set for our Oracle Clusterware
only installation?
If you are using only Oracle Clusterware(no Veritas CM), then you don't need to have SCSI-3
PGR enabled, since Oracle Clusterware does not require it for IO fencing. If the reservation is set,
then you'll get the inconsistent results. So ask your storage vendor to disable the reservation.
Veritas RAC requires that the storage array support SCSI-3 PGR, since this is how Veritas
handles IO fencing. This SCSI-3 PGR is set at the array level; for example EMC hypervolume
level.
proper procedure to remove the 3rd node from the cluster so it can
be added back in?
Follow the documentation for removing a node but you can skip all the steps in the node-removal
doc that need to be run on the node being removed, like steps 4, 6 and 7 (See Chapter 10 of
RAC Admin and Deployment Guide). Make sure that you remove any database instances that
were configured on the failed node with srvctl, and listener resources also, otherwise
rootdeltenode.sh will have trouble removing the nodeapps.
Just running rootdeletenode.sh isn't really enough, because you need to update the installer
inventory as well, otherwise you won't be able to add back the node using addNode.sh. And if
you don't remove the instances and listeners you'll also have problems adding the node and
instance back again.
Probably a better alternative (than the generic documentation, bug 5929611 filed) for a remove
node is Note 269320.1
I/O so a minimum of 3. Anything else will cause performance and stability problems under load.
From an HA perspective, you want these to be redundant, thus needing a total of 6.
Can we output the backupset onto regular file system directly (not
onto flash recovery area) using RMAN command, when we use SE
RAC?
Yes, - customers might want to backup their database to offline storage so this is also supported.
For details on the support of SFRAC and Veritas Agents with RAC 10g, please see <> Oracle's
Policy for Supporting RAC 10g with Symantec SFRAC on Unix and <> Using Oracle Clusterware
with Vendor Clusterware FAQ
Bonding
More information: Metalink Note 298891.1
Related RAC FAQ entries: How do I use multiple network interfaces to provide High Availability
and/or Load Balancing for my interconnect with Oracle Clusterware?
Windows - available solutions:
Teaming
On Windows teaming solutions to ensure NIC availability are usually part of the network card
driver.
Thus, they depend on the network card used. Please, contact the respective hardware vendor for
more information.
Are there any issues for the interconnect when sharing the same
switch as the public network by using VLAN to separate the
network?
RAC and Clusterware deployment best practices recommend that the interconnect be deployed
on a stand-alone, physically seperate, dedicated switch. Many customers have consolidated
these stand-alone switches into larger managed switches. A consequence of this consolidation is
a merging of IP networks on a single shared switch, segmented by VLANs. There are caveats
associated with such deployments. RAC cache fusion exercises the IP network more rigorously
than non-RAC Oracle databases. The latency and bandwidth requirements as well as availability
requirements of the RAC/Clusterware interconnect IP network are more in-line with high
performance computing. Deploying the RAC/Clusterware interconnect on a shared switch,
segmented VLAN may expose the interconnect links to congestion and instability in the larger IP
network topology. If deploying the interconnect on a VLAN, there should be a 1:1 mapping of
VLAN to non-routable subnet and the VLAN should not span multiple VLANs (tagged) or multiple
switches. Deployment concerns in this environment include Spanning Tree loops when the larger
IP network topology changes, Assymetric routing that may cause packet flooding, and lack of fine
grained monitoring of the VLAN/port.
Are block devices supported for OCR, Voting Disks, ASM devices?
Block Devices are only supported on Linux. For Unix platforms, the directio symantics not
applicable (or rather not implemented) for the block devices on these platforms.
Note: On Linux, raw devices are being deprecated so you should move to using block devices.
Note the Oracle Database 10g OUI does not support block devices however Oracle Clusterware
and ASM do.
Why does the NOAC attribute need to be set on NFS mounted RAC
Binaries?
The noac attribute is required because the installer determines sharedness by creating a file and
checking for that file s existance on remote node. If the noac attribute is not enabled then this
test will incorrectly fail. This will confuse installer and opatch. Some other minor issues issues
with spfile in the default $ORACLE_HOME/dbs will definitely be affected.
example, while OCR is running when the device is down, OCR assign 2 vote to the surviving
device and that is why this surviving device now with two votes can start after the cluster is
down). See warning below
c) This method is not recommend to be performed by customers. It is possible to manually modify
ocr.loc to delete the failed device and restart the cluster. OCR won't do the vote check if the
mirror is not configured. See warning below
EXTREME CAUTION should be excersized if chosing option b or c above since data loss can
occur if the wrong file is manipulated, please contact Oracle Support for assistance before
proceeding.
Bug 5055145 was the basis for this FAQ, also thanks to Ken Lee for his valuable feedback.
In the case of existing SQL conenctions, errors will typically be in the form of ORA-3113 errors,
while a new connection using an address list will select the next entry in the list. Without using
VIPs, clients connected to a node that died will often wait for a TCP/IP timeout period before
getting an error. This can be as long as 10 minutes or more. As a result, you don't really have a
good HA solution without using VIPs.
What are my options for load balancing with RAC? Why do I get an
uneven number of connections on my instances?
All the types of load balancing available currently (9i-10g) occur at connect time.
This means that it is very important how one balances connections and what these connections
do on a long term basis.
Since establishing connections can be very expensive for your application, it is good
programming practice to connect once and stay connected. This means one needs to be careful
as to what option one uses. Oracle Net Services provides load balancing or you can use external
methods such as hardware based or clusterware solutions.
The following options exist prior to Oracle RAC 10g Releae 2 (for 10g Release 2 see Load
Balancing Advisory):
Random
Either client side load balancing or hardware based methods will randomize the connections to
the instances.
On the negative side this method is unaware of load on the connections or even if they are up
meaning they might cause waits on TCP/IP timeouts.
Load Based
Server side load balancing (by the listener) redirects connections by default depending on the
RunQ length of each of the instances. This is great for short lived connections. Terrible for
persistent connections or login storms. Do not use this method for connections from connection
pools or applicaton servers
Session Based
Server side load balancing can also be used to balance the number of connections to each
instance. Session count balancing is method used when you set a listener parameter,
prefer_least_loaded_node_listener-name=off. Note listener name is the actual name of the
listener which is different on each node in your cluster and by default is listener_nodename.
Session based load balancing takes into account the number of sessions connected to each node
and then distributes the connections to balance the number of sessions across the different
nodes.
Can our 10g VIP fail over from NIC to NIC as well as from node to
node ?
Yes the 10g VIP implementation is capable from failing over within a node from NIC to NIC and
back if the failed NIC is back online again, and also we fail over between nodes. The NIC to NIC
failover is fully redundant if redundant switches are installed.
With three primary load balancing options (client-side connecttime LB, server-side connect-time LB, and the runtime connection
load balancing) Is it fair to say Runtime Connection Load
Balancing is the only option to leverage FAN up/down events?
No. The listener is a subscriber to all FAN events (both from the load balancing advisory and the
HA events). Therefore server side connection load balancing leverages FAN HA events as well
as laod balancing advisory events.
With the Oracle JDBC driver 10g Release 2, if you enable Fast Connection Failover, you also
enable Runtime Connection Load Balancing (one knob for both).
The client gets this error message in Production in the ons.log file
every minute or so: 06/11/10 10:11:14 [2] Connection
0,129.86.186.58,6200 SSL handshake failed 06/11/10 10:11:14 [2]
Handshake for 0,129.86.186.58,6200: nz error = 29049 interval = 0
(180 max)
These annoying messages in ons.log are telling you that you have a configuration mismatch for
ONS somewhere in the farm. RAC has its own ONS server for which SSL is disabled by default.
You must either enable SSL for RAC ONS, or disable it for OID ONS(OPMN). You need to create
a wallet for each RAC ONS server, or copy one of the wallets from OPMN on the OID instances.
In ons.conf you need to specify the wallet file and password:
walletfile=
walletpassword=
ONS only uses SSL between servers, and so ONS clients will not be affected. You specify the
wallet password when you create the wallet. If you copy a wallet from an OPMN instance, then
use the same password configured in opmn.xml. If there is no wallet password configured in
opmn.xml, then you don't need to specify a wallet password in ons.conf either.
How do I configure FCF with BPEL so I can use RAC 10g in the
backend?
** Note:372456.1 describes the procedure to set up BPEL with a Oracle RAC 10g Release 1
database.
If you are using SSL, ensure the SSL enable attribute of ONS in opmn.xml file has same value,
either true or false, for all OPMN servers in the Farm. To troubleshoot OPMN at the application
server level, look at appendix A in Oracle Process Manager and Notification Server
Administrator's Guide.
An UP event is processed for both (a) new instance joins, as well as (b) down followed by an
instance UP. This has no relevance to initialLimit, or even minLimit. When a UP event comes into
our jdbc Implicit Connection Cache, we will create some new connections. Assuming you have
your listener load balancing set up properly, then those connections should go to the instance
that was just started. When your application does a get connection to the pool, it will be given an
idle connection, if you are running 10.2 and have the load balancing advisory turned on for the
service, we will allocate the session based on the defined goal to provide the best service level
MaxLimit, when set, defines the upper boundary limit for the connection cache. By default,
maxLimit is unbounded - your database sets the limit.
Can I use the 10.2 JDBC driver with 10.1 database for FCF?
Yes with the patch for Bug 5657975 for 10.2.0.3,the 10.2 JDBC driver will work with a 10.1
database. The fix will be part of the 10.2.0.4 patchset. If you do not have the patch then using
FCF, use the 10.2 JDBC driver with 10.2 database. If database is 10.1, use 10.1 JDBC driver.
THROUGHPUT Work requests are directed based on throughput. This should be used when
the work in a service completes at homogenous rates. An example is a trading system where
work requests are similar lengths.
SERVICE_TIME Work requests are directed based on response time. This should be used
when the work in a service completes at various rates. An example is as internet shopping
system where work requests are various lengths
Note: If using GOAL, you should set CLB_GOAL=SHORT
a:
needs to log into the new cluster database as dbsnmp to create the
cluster database targets etc. I changed the password for the
dbsnmp account to be dbsnmp (same as username) and it worked
OK. Is this a known issue?
The conversion to cluster happens successfully but the EM monitoring credentials for the
converted database are not properly set due to this bug. This is resolved in next patchset. In the
interim, user can set the monitoring password from the "monitoring configuration" screen for the
RAC DB from GC console and proceed.
This issue has been fixed in 10.2.0.3 database and to get the complete functionality you will need
10.2.0.2 Grid Control patch also, as the fix is spread between the two pieces of software. For now
you can proceed with setting password for dbsnmp user same as that of sys user.
What storage option should I use for RAC 10g on Linux? ASM /
OCFS / Raw Devices / Block Devices / Ext3 ?
The recommended way to manage large amounts of storage in a RAC environment is ASM
(Automatic Storage Management) really need/want a clustered filesystem, then Oracle offers
OCFS (Oracle Clustered File System); for 2.4 kernel (RHEL3/SLES8) use OCFS Version 1 and
for 2.6 kernel (RHEL4/SLES9) use OCFS2. All these options are free to use and completely
supported, ASM is bundled in the RDBMS software, and OCFS as well as ASMLib are freely
downloadable from Oracle's OSS (Open Source Software) website.
EXT3 is out of the question, since it's data structures are not cluster aware, that is, if you mount
an ext3 filesystem from multiple nodes, it will quickly get corrupted.
Another option of course is NFS and iSCSI both are outside the scope of this FAQ but included
for completeness.
If for any reason the above options (ASM/OCFS) are not good enough and you insist on using
'raw devices' or 'block devices' here are the details on the two (This information is still very useful
to know in the context of ASM and OCFS).
On Unix/Linux there are two types of devices:
block devices (/dev/sde9) are **BUFFERED** devices!! unless you explicitly open them in
O_DIRECT you will get buffered (linux buffer cache) IO.
character devices (/dev/raw/raw9) are *UN-BUFFERRED** devices!! no matter how you open
them, you always get unbufferred IO, hence no need to specify O_DIRECT on the file open call.
Above is not a typo, block devices on Unix do buffered IO by default (cached in linux buffer
cache), this means that RAC can not operate on it (unless opened with O_DIRECT), since the
IO's will not be immediately visible to other nodes.
You may check if a device is block or character device by the first letter printed with the "ls -l"
command:
crw-rw---- 1 root disk 162, 1 Jan 23 19:53 /dev/raw/raw1
I was installing RAC and my Oracle files did not get copied to the
remote node(s). What went wrong?
First make sure the cluster is running and is available on all nodes. You should be able to see all
nodes when running an 'lsnodes -v' command.
If lsnodes shows that all members of the cluster are available, then you may have an rcp/rsh
problem on Unix or shares have not been configured on Windows.
You can test rcp/rsh on Unix by issuing the following from each node:
[node1]/tmp> touch test.tst
[node1]/tmp> rcp test.tst node2:/tmp
[node2]/tmp> touch test.tst
[node2]/tmp> rcp test.tst node1:/tmp
On Windows, ensure that each node has administrative access to all these directories within the
Windows environment by running the following at the command prompt:
NET USE \\host_name\C$
Clustercheck.exe also checks for this.
More information can be found in the Step-by-Step RAC notes available on Metalink. To find
these search Metalink for 'Step-by-Step Installation of RAC'.
failure of a verification task, does not enter into areas of performance tuning or monitoring, does
not perform any cluster or RAC operation, and does not attempt to verify the internals of cluster
database or cluster elements.
Minor changes in time (in the seconds range) are harmless for RAC and the Oracle
clusterware. If you intend on making large time changes it is best to shutdown the instances and
the entire Clusterware stack on that node to avoid a false eviction, especially if you are using the
10g low-brownout patches, which allow really low misscount settings.
Backup/recovery aspect of large time changes are documented in note 77370.1, basically you
can't use RECOVER DATABASE UNTIL TIME to reach the second recovery point, It is possible
to overcome with RECOVER DATABASE UNTIL CANCEL or UNTIL CHANGE. If you are doing
complete recovery (most of the times) then this is not an issue since the Oracle recovery code
uses SCN (System Change Numbers) to advance in the redo/archive logs. The SCN numbers
never go back in time (unless a reset-logs operation is performed), there is always an association
of an SCN to a human readable timestamp (which may change forward or backwards), hence the
issue with recovery until point in time vs. until SCN/Cancel.
If DBMS_SCHEDULER is in usage it will be affected by time changes, as it's using actual clock
rather than SCN.
On platforms with OPROCD get fix for bug 5015469 "OPROCD REBOOTS NODE WHEN
TIME IS SET BACK BY XNTPD"
Apart from these issues, the Oracle server is immuned to time changes, i.e. will not affect
transaction/read consistency operations.
On Linux the "-x" flag can be added to the ntpd daemon to prevent the clock from going
backwards.
point to an instance local directory. Alternately, if the RAC configuration supports a shared
(cluster) file system, you can use a common directory (on the shared file system) for all instances.
You can also check out the PL/SQL Native Compilation FAQ on OTN:
www.oracle.com/technology/tech/pl_sql/htdocs/ncomp_faq.html
DBCA can be used to create databases on raw devices in 9i RAC Release 1 and 9i Release 2.
Standard database creation scripts using SQL commands will work with file system and raw.
DBCA cannot be used to create databases on file systems on Oracle 9i Release 1. The user can
choose to set up a database on raw devices, and have DBCA output a script. The script can then
be modified to use cluster file systems instead.
With Oracle 9i RAC Release 2 (Oracle 9.2), DBCA can be used to create databases on a cluster
filesystem. If the ORACLE_HOME is stored on the cluster filesystem, the tool will work directly. If
ORACLE_HOME is on local drives on each system, and the customer wishes to place database
files onto a cluster file system, they must invoke DBCA as follows: dbca -datafileDestination
/oradata where /oradata is on the CFS filesystem. See 9iR2 README and bug 2300874 for more
info.
No, there should be only one Oracle Cluster Manager (ORACM) running on each node. All RAC
databases should run out of the $ORACLE_HOME that ORACM is installed in.
Please carefully read the following new information about configuring Oracle Cluster
Management on Linux, provided as part of the patch README:
Three parameters affect the startup time:
soft_margin (defined at watchdog module load)
-m (watchdogd startup option)
WatchdogMarginWait (defined in nmcfg.ora).
WatchdogMarginWait is calculated using the formula:
WatchdogMarginWait = soft_margin(msec) + -m + 5000(msec).
[5000(msec) is hardcoded]
Note that the soft_margin is measured in seconds, -m and WatchMarginWait are measured in
milliseconds.
Based on benchmarking, it is recommended to set soft_margin between 10 and 20 seconds. Use
the same value for -m (converted to milliseconds) as used for soft_margin. Here is an example:
soft_margin=10 -m=10000 WatchdogMarginWait = 10000+10000+5000=25000
If CPU utilizatn in your system is high and you experience unexpected node reboots, check the
wdd.log file. If there are any 'ping came too late' messages, increase the value of the above
parameters.
Yes, OCFS (Oracle Cluster Filesystem) is now available for Linux. The following Metalink note
has information for obtaining the latest version of OCFS:
<> - How to find the current OCFS version for Linux
Can RAC 10g and 9i RAC be installed and run on the same
physical Linux cluster?
Yes - However Oracle Clusterware (CRS) will not support a 9i RAC database so you will have to
leave the current configuration in place. You can install Oracle Clusterware and RAC 10g into the
same cluster. On Windows and Linux, you must run the 9i Cluster Manager for the 9i Database
and the Oracle Clusterware for the 10g Database. When you install Oracle Clusterware, your 9i
srvconfig file will be converted to the OCR. Both 9i RAC and 10g will use the OCR. Do not restart
the 9i gsd after you have installed Oracle Clusterware. Remember to check certify for details of
what vendor clusterware can be run with Oracle Clusterware.
Is the hangcheck timer still needed with Oracle RAC 10g and 11g?
YES! The hangcheck-timer module monitors the Linux kernel for extended operating system
hangs that could affect the reliability of the RAC node ( I/O fencing) and cause database
corruption. To verify the hangcheck-timer module is running on every node:
as root user:
/sbin/lsmod | grep hangcheck
If the hangcheck-timer module is not listed enter the following command
as the root user:
9i: /sbin/insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
hangcheck_reboot=1
10g & 11g: /sbin/insmod hangcheck-timer hangcheck_tick=1
hangcheck_margin=10 hangcheck_reboot=1
To ensure the module is loaded every time the system reboots, verify that the local system
startup file (/etc/rc.d/rc.local) contains the command above.
For additional information please review the Oracle RAC Install and Configuration Guide (5-41)
and note:726833.1.
The way to fix this is on RHEL4, OEL4 and SLES9 is to create /etc/udev/permission.d/40udev.permissions (you must choose a number that's lower than 50). You can do this by copying
/etc/udev/permission.d/50-udev.permissions, and removing the lines that are not needed (50udev.permissions gets replaced with upgrades so you do not want to edit it directly, also a typo in
the 50-udev.permissions can render the system non-usable). Example permissions file:
# raw devices
raw/raw[1-2]:root:oinstall:0640
raw/raw[3-5]:oracle:oinstall:0660
Note that this applied to all raw device files, here just the voting and OCR devices were specified.
On RHEL5, OEL5 and SLES10 a different file is used /etc/udev/rules.d/99-raw.rules, notice that
now the number must be (any number) higher than 50. Also the syntax of the rules is different
than the permissions file, here's an example:
KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="640"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="660"
A customer installed 10g R2 on Linux RH4 Update 2, 2.6.922.ELsmp #1 SMP x86_64 GNU/Linux, and got the error Error in
invoking target 'all_no_orcl'. Customer ignored the error and the
install succeeded without any other errors and oracle pparently
worked fine. What should they do?
Because of compatibility with their storage array (EMC DMX with
Powerpath 4.5) they must use update 2. Oracle install guide states
that RH4 64 bits update 1 "or higher" should be used for 10g R2.
The binutils patch binutils-2.15.92.0.2-13.0.0.0.2.x86_64.rpm is needed to relink without error.
Red hat is aware of the bug and has a test version and they will have it fixed for update3 however
it is taking them time. so using the binutils rpm you mentioned you should be all good to go. if the
customer wants they can update that package again after install. at least we were able to provide
a fix directly to get folks going.
cluster
Although this is rarely needed, since most hardware will detect the cards in the correct order on
all nodes, if you still need to change/control the ordering, see external website, here is more help
on writing UDEV rules.
The answer is the same if you are using Vendor Clusterware such as Veritas SF RAC or Sun
Cluster.
Can I run my 9i RAC and RAC 10g on the same Windows cluster?
Yes but the 9i RAC database must have the 9i Cluster Manager and you must run Oracle
Clusterware for the Oracle Database 10g. 9i Cluster Manager can coexsist with Oracle
Clusterware 10g.
Be sure to use the same 'cluster name' in the appropriate OUI field for both 9i and 10g when you
install both together in the same cluster.
The OracleCMService9i service will remain intact during the Oracle Clusterware 10g install, as a
9i RAC database would require that the 9i OracleCMService9i, it should be left running. The
information for the 9i database will get migrated to the OCR during the Oracle Clusterware
installation. Then, for future database management, you would use the 9i srvctl to manage the 9i
database, and the 10g srvctl to manage any new 10g databases. Both srvctl commands will use
the OCR.
This sounds like the ORA_DBA group on Node2 is empty, or else does not have the correct
username in it. Double-check what user account you are using to logon to Node2 as ( a 'set'
command will show you the USERNAME and USERDOMAIN values) and then make sure that
this account is part of ORA_DBA.
The other issue to check is that SQLNET.AUTHENTICATION_SERVICES=(NTS) is set in the
SQLNET.ORA
How do I verify that Host Bus Adapter Node Local Caching has
been disabled for the disks I will be using in my RAC cluster?
Disabling write caching is a standard practice while using the volume managers/file systems are
shared. Go to My computer -> Manage->Storage->Disk Management->Disk-Properties->Policies> and uncheck the "Enable Write Caching on Disk". This will disable the write caching.
3rd party HBA's may have their own management tools to modify these settings. Just remember
that centralized, shared cache is generally OK. It's the node local cache that you need to turn off.
How exactly you do this will vary from HBA vendor to HBA vendor.
(for ESS, EMC, HDS, CLARiiON, and MPIO-capable devices you have to do an
chdev -l hdiskn -a reserve_policy=no_reserve)
Is HACMP needed for RAC on AIX 5.2 using GPFS file system?
The newest version of GPFS can be used without HACMP, if it is available for AIX 5.2 then you
do not need HACMP.
Can I use Oracle Clusterware for failover of the Sue and VIP
services when running SAP in a RAC environment?
Oracle has created sapctl to do this and it is available for certain platforms. SAPCTL will be
available for download on SAP Services Marketplace on AIX and Linux. For Solaris, it will not be
available in 2007, use Veritas or Sun Cluster.
Oracle Clusterware files OCR and Voting Disk can be put on OCFS2 however Best Practice is to
put them on raw or block devices.
What are the Best Practices for using a clustered file system with
Oracle RAC?
Can I use it for OCR, Voting Disk, Binaries as well as database
files?
Oracle Best Practice for usingCluster File Systems (CFS) with Oracle RAC 10g
* Oracle Clusterware binaries should not be placed on a CFS as this reduces cluster functionality
while CFS is recovering, and also limits the ability to perform rolling upgrades of Oracle
Clusterware.
* Oracle Clusterware voting disks and the Oracle Cluster Registry (OCR) should not be placed on
a CFS as the I/O freeze during CFS reconfiguration can lead to node eviction, or cluster
management activities to fail (I.E start, stop, or check of a resource).
* Oracle Database 10g binaries are supported on CFS for Oracle RAC 10g and for Oracle
Database. The system should be configured to support multiple ORACLE_HOME s in order to
maintain the ability to perform a rolling patch application.
* Oracle Database 10g database files (e.g. datafiles, trace files, and archive log files) are
supported on CFS.
Check Certify for certified cluster file systems.
Rolling Upgrades with Cluster File Systems in General
It is not recommended to use a cluster file system (CFS) for the Oracle Clusterware binaries.
Oracle Clusterware supports in-place rolling upgrades. Using a shared Oracle Clusterware home
results in a global outage during patch application and upgrades. A workaround is available to
clone the Oracle Clusterware home for each upgrade. This is not common practice.
If a patch is marked for rolling upgrade, then it can be applied to a RAC database in a rolling
fashion. Oracle supports rolling upgrades for the Oracle Database Automated Storage
Management (ASM) after you have upgraded to Oracle Database 11g. When using a CFS for the
database and ASM Oracle homes, the CFS should be configured to use of context dependent
links (CDSLs) or equivalent and these should configured to work in conjunction with rolling
upgrades and downgrades. This includes updating the database and ASM homes in the OCR to
point to the current home.
This is included in Metalink Note 444134.1
(value)'.
Can anyone please explain the meaning of these two timeouts in
addition to disktimeout?
Having a short and long disktimeout, and no longer just one disktimeout, is due to patch for bug
4748797 (included in 10.2.0.2). The long disktimeout is 200 sec by default unless set differently
via 'crsctl set css disktimeout', and applies to time outside a reconfiguration. The short
disktimeout is in effect during a reconfiguration and is misscount-3s. The point is that we can
tolerate a long disktimeout when all nodes are just running fine, but have to revert back to a short
disktimeout if there's a reconfiguration.
** Note:399482.1 describes how to recreate your OCR/Voting Disk which you have accidently
deleted and cannot recover from backups
What should the permissions be set to for the voting disk and ocr
when doing a RAC Install?
The Oracle Real Application Clusters install guide is correct. It describes the PRE INSTALL
ownership/permission requirements for ocr and voting disk. This step is needed to make sure that
the CRS install succeeds. Please don't use those values to determine what the
ownership/permmission should be POST INSTALL. The root script will change the
ownership/permission of ocr and voting disk as part of install. The POST INSTALL permissions
will end up being : OCR - root:oinstall - 640 Voting Disk - oracle:oinstall - 644
As long as you can confirm via the CSS daemon logfile that it thinks the voting disk is bad, you
can restore the voting disk from backup while the cluster is online. This is the backup that you
took with dd (by the manual's request) after the most recent addnode, deletenode, or install
operation. If by accident you restore a voting disk that the CSS daemon thinks is NOT bad, then
the entire cluster will probably go down.
crsctl add css votedisk - adds a new voting disk
crsctl delete css votedisk - removes a voting disk
Note: the cluster has to be down. You can also restore the backup via dd when the cluster is
down.
Starting with Oracle Database 10g Release 2 Oracle Clusterware will multiplex the OCR and
Voting Disk (two for the OCR and three for the Voting).
Please read Note:279793.1 and Note:268937.1 regarding backup and restore a lost Voting/OCR
and FAQ 10734 regarding OCR backup.
As far as voting disks are concerned, a node must be able to access strictly more than half of the
voting disks at any time. So if you want to be able to tolerate a failure of n voting disks, you must
have at least 2n+1 configured. (n=1 means 3 voting disks). You can configure up to 32 voting
disks, providing protection against 15 simultaneous disk failures, however it's unlikely that any
customer would have enough disk systems with statistically independent failure characteristics
that such a configuration is meaningful. At any rate, configuring multiple voting disks increases
the system's tolerance of disk failures (i.e. increases reliability).
Configuring a smaller number of voting disks on some kind of RAID system can allow a customer
to use some other means of reliability than the CSS's multiple voting disk mechanisms. However
there seem to be quite a few RAID systems that decide that 30-60 second (or 45 minutes in the
case of veritas) IO latencies are acceptable. However we have to wait for at least the longest IO
latency before we can declare a node dead and allow the database to reassign database blocks.
So while using an independent RAID system for the voting disk may appear appealing,
sometimes there are failover latency consequenecs.
How can I register the listener with Oracle Clusterware in RAC 10g
Release 2?
NetCA is the only tool that configures listener and you should be always using it. It will register
the listener with Oracle Clusterware. There are no other supported alternatives.
Do not mix NICs with different interface types (infiniband, ethernet, hyperfabric, etc.) for the same
subnet/network.
Why does Oracle still use the voting disks when other cluster
sofware is present?
Voting disks are still used when 3rd party vendor clusterware is present, because vendor
clusterware is not able to monitor/detect all failures that matter to Oracle Clusterware and the
database. For example one known case is when the vendor clusterware is set to have its
heartbeat go over a different network than RAC traffic. Continuing to use the voting disks allows
CSS to resolve situations which would otherwise end up in cluster hangs.
I made a mistake when I created the VIP during the install of Oracle
Clusterware, can I change the VIP?
Yes The details of how to do this are described in <>
What are the licensing rules for Oracle Clusterware? Can I run it
without RAC?
Check the Oracle Database Licensing Information 11g Release 1 (11.1) Part Number B2828701 Look in the Special Use section under Oracle Database Editions.
http://www.oracle.com/technology/products/database/clustering/pdf/thirdvoteonnfs.pdf Otherwise
just as with database files, we only support voting files on certified NAS devices, with the
appropriate mount options. Pls refer to Metalink Note 359515.1 for a full description of the
required mount options. For a complete list of supported NAS vendors refer to OTN at:
http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html
cables from a switch. If a DWDM or CWDM is used then then these are directly connected via a
dedicated switch on either side.
Note of caution: Do not do RAC Interconnect over a WAN. This is a the same as doing it over the
public network which is not supported and other uses of the network (i.e. large FTPs) can cause
performance degradations or even node evictions.
For SAN networks make sure you are using SAN buffer credits if the distance is over 10km.
At the moment in Oracle 10g, if Oracle Clusterware is being used, we also require that a single
subnet be setup for the public connections so we can fail over VIPs from one side to another.
What is a stage?
CVU supports the notion of Stage verification. It identifies all the important stages in RAC
deployment and provides each stage with its own entry and exit criteria. The entry criteria for a
stage define a specific set of verification tasks to be performed before initiating that stage. This
pre-check saves the user from entering into a stage unless its pre-requisite conditions are met.
The exit criteria for a stage define another specific set of verification tasks to be performed after
completion of the stage. The post-check ensures that the activities for that stage have been
completed successfully. It identifies any stage specific problem before it propagates to
subsequent stages; thus making it difficult to find its root cause. An example of a stage is "precheck of database installation", which checks whether the system meets the criteria for RAC
install.
What is a component?
CVU supports the notion of Component verification. The verifications in this category are not
associated with any specific stage. The user can verify the correctness of a specific cluster
component. A component can range from a basic one, like free disk space to a complex one like
CRS Stack. The integrity check for CRS stack will transparently span over verification of multiple
sub-components associated with CRS stack. This encapsulation of a set of tasks within specific
component verification should be of a great ease to the user.
What is nodelist?
Nodelist is a comma separated list of hostnames without domain. Cluvfy will ignore any domain
while processing the nodelist. If duplicate entities after removing the domain exist, cluvfy will
eliminate the duplicate names while processing. Wherever supported, you can use '-n all' to
check on all the cluster nodes. Check this for more information on nodelist and shortcuts.
location exists on all nodes and it has write permission for CVU user. This dir is set through the
CV_DESTLOC environment variable. If this variable does not exist, CVU will use "/tmp" as the
work dir. 4._ On RedHat Linux 3.0, an optional package 'cvuqdisk' is required on all the nodes.
This assists CVU in finding scsi disks and helps CVU to perform storage checks on disks. Please
refer to What is 'cvuqdisk' rpm? for detail. Note that, this package should be installed only on
RedHat Linux 3.0 distribution.
What are the default values for the command line arguments?
Here are the default values and behavior for different stage and component commands:
Do I have to type the nodelist every time for the CVU commands?
Is there any shortcut?
You do not have to type the nodelist every time for the CVU commands. Typing the nodelist for a
large cluster is painful and error prone. Here are few short cuts. To provide all the nodes of the
cluster, type '-n all'. Cluvfy will attempt to get the nodelist in the following order: 1. If a vendor
clusterware is available, it will pick all the configured nodes from the vendor clusterware using
lsnodes utility. 2. If CRS is installed, it will pick all the configured nodes from Oracle clusterware
using olsnodes utility. 3. In none of the above, it will look for the CV_NODE_ALL environmental
variable. If this variable is not defined, it will complain. To provide a partial list(some of the nodes
of the cluster) of nodes, you can set an environmental variable and use it in the CVU command.
For example: setenv MYNODES node1,node3,node5 cluvfy comp nodecon -n $MYNODES
How do I check the Oracle Clusterware stack and other subcomponents of it?
Cluvfy provides commands to check a particular sub-component of the CRS stack as well as the
whole CRS stack. You can use the 'comp ocr' command to check the integrity of OCR. Similarly,
you can use 'comp crs' and 'comp clumgr' commands to check integrity of crs and
clustermanager sub-components. To check the entire CRS stack, run the stage command 'clucvy
stage -post crsinst'.
Why the peer comparison with -refnode says passed when the
group or user does not exist?
Peer comparison with the -refnode feature acts like a baseline feature. It compares the system
properties of other nodes against the reference node. If the value does not match( not equal to
reference node value ), then it flags that as a deviation from the reference node. If a group or user
does not exist on reference node as well as on the other node, it will report this as 'matched'
since there is no deviation from the reference node. Similarly, it will report as 'mismatched' for a
node with higher total memory than the reference node for the above reason.