racnode1 APA group Public VIP SCAN IP Cluster Interconnect S.

No 1 Network Interface lan900 lan900:801 lan900:803 lan901 Test Scenario

Failover within public lan Auto Port Aggregation(APA) Bonding

Failover within public lan Auto Port Aggregation(APA) Bonding

3 Failover within cluster Interconnect lan Auto Port Aggregation(APA) Bonding

4 Failover within cluster Interconnect lan Auto Port Aggregation(APA) Bonding

Public Network Failure

Public Network Failure

Interconnect Network Failure (11.2.0.2 and higher)

Interconnect Network Failure (11.2.0.2 and higher)

racnode1 IP Address 172.20.1.1 172.20.1.3 172.20.1.5 192.1.1.36 Action Removed cable 1 of PUBLIC lan900 Connected back cable 1 of PUBLIC lan 900 Removed cable 2 of PUBLIC lan900 Connected back cable 2 of PUBLIC lan 900 Removed cable 1 of PUBLIC lan900 Connected back cable 1 of PUBLIC lan 900 Removed cable 2 of PUBLIC lan900 Connected back cable 2 of PUBLIC lan 900 Removed cable 1 of CLUSTER INTERCONNECT lan901 Connected back cable 1 of CLUSTER INTERCONNECT lan 900 Removed cable 2 of CLUSTER INTERCONNECT lan901 Connected back cable 2 of CLUSTER INTERCONNECT lan 900 Removed cable 1 of CLUSTER INTERCONNECT lan901 Connected back cable 1 of CLUSTER INTERCONNECT lan 900 Removed cable 2 of CLUSTER INTERCONNECT lan901 Connected back cable 2 of CLUSTER INTERCONNECT lan 900 172.20.1.2 APA group Public VIP SCAN IP Cluster Interconnect Host 172.20.1.1

racnode2

172.20.1.2

172.20.1.1

Unplug all network cables for public network of node 1 (172.20.1.1)

172.20.1.1

Unplug all network cables for public network of node 2 (172.20.1.2)

172.20.1.2

Unplug all network cables for the interconnect network of node 1

172.20.1.1

Unplug all network cables for the interconnect network of node 2

172.20.1.2

racnode2 Network Interface lan900 lan900:801 NA lan901 Expected Results

IP Address 172.20.1.2 172.20.1.4 NA 192.1.1.40 Observed Results Status SUCCESS

Failover of IP within in APA. No Failover of IP within in APA. No impact to Oracle impact to Oracle SUCCESS

SUCCESS Failover of IP within in APA. No Failover of IP within in APA. No impact to Oracle impact to Oracle SUCCESS

Check with crsctl stat res t o The ora.*.network and listener resources will go offline for the node. o SCAN VIPs and SCAN LISTENERs running on the node will fail over to a surviving node. o The VIP for the node will fail over to a surviving node. The database instance will remain up but will be unregistered with the remote listeners. Database services will fail over to one of the other available nodes. If TAF is configured, clients should fail over to an available

* ora.*.network and listener resources went offline as expected * SCAN VIP and SCAN LISTENER failed over to node 2 as expected. * VIP was in INTERMEDIATE status although it failed over to node 2( This is expected behavior starting from 11g) * Database instance was running and unregistered from remote listener. * TAF connections with uncommitted transactions failed over to node 2 as expected.

SUCCESS

Check with crsctl stat res t o The ora.*.network and listener resources will go offline for the node. o SCAN VIPs and SCAN LISTENERs running on the node will fail over to a surviving node. o The VIP for the node will fail over to a surviving node. The database instance will remain up but will be unregistered with the remote listeners. Database services will fail over to one of the other available nodes. If TAF is configured, clients should fail over to an available

* ora.*.network and listener resources went offline as expected * SCAN VIP and SCAN LISTENER failed over to node 1 as expected. * VIP was in INTERMEDIATE status although it failed over to node 2( This is expected behavior starting from 11g) * Database instance was running and unregistered from remote listener. * TAF connections with uncommitted transactions failed over to node 1 as expected.

SUCCESS

For 11.2.0.2 and above: CSSD will detect split-brain situation and perform one of the following: o In a two-node cluster the node with the lowest node number will survive. o In a multiple node cluster the largest sub-cluster will survive. On the node(s) that is being evicted, a graceful shutdown of Oracle Clusterware will be attempted. o All I/O capable client processes will be terminated and all resources will be cleaned up. If process termination and/or resource cleanup does not complete successfully the node will be rebooted. o Assuming that the above has completed successfully, OHASD will attempt to restart the stack. In this case the stack will be restarted once the network connectivity of the private interconnect network has been restored.

* Since this is 2 node cluster, the 2nd node got evicted from the cluster as expected. Node was not rebooted, as starting from 11g, eviction type is reboot less. If reboot less eviction fails for some reason, then node will be rebooted. * Cluster services on node 2 went down, database instance and ASM were abnormally shutdown. * VIP was in INTERMEDIATE status although it failed over to node 2( This is expected behavior starting from 11g) * After reconnecting the private interconnect cables, crs stack and resources started.

SUCCESS

For 11.2.0.2 and above: CSSD will detect split-brain situation and perform one of the following: o In a two-node cluster the node with the lowest node number will survive. o In a multiple node cluster the largest sub-cluster will survive. On the node(s) that is being evicted, a graceful shutdown of Oracle Clusterware will be attempted. o All I/O capable client processes will be terminated and all resources will be cleaned up. If process termination and/or resource cleanup does not complete successfully the node will be rebooted. o Assuming that the above has completed successfully, OHASD will attempt to restart the stack. In this case the stack will be restarted once the network connectivity of the private interconnect network has been restored.

* Since this is 2 node cluster, the 2nd node got evicted from the cluster as expected. * Cluster services on node 2 went down, database instance and ASM were abnormally shutdown * After reconnecting the private interconnect cables, some of the cluster daemons started but ASM and Database did not start automatically( Not as expected) * VIP Failed over but was in INTERMEDIATE state(not as expected) * DB sessions connected using TAF over VIP and not having uncommitted transactions failed over to node 1.

SUCCESS