TEST RULE CASE – 1 : FIREWALL FAILOVER SCENARIO

FINDINGS:

During real-time monitoring, we found events related to firewall failover with

event IDs 103005 and 104003.

 103005: It happens when the primary firewall sends events saying that
the secondary firewall is reporting a failure due the SSM card failure.
 104003: It happens when the Primary firewall s switching to FAILED state.

THEORETICAL CONCEPT

Cisco ASA: Cisco ASA is a security device that provides the combined capabilities of a firewall, an
antivirus, and an intrusion prevention system. It also facilitates virtual private network (VPN)
connections. It helps to detect threats and stop attacks before they spread through the network.

CISCO ASA ACTIVE/PASSIVE FAILOVER CONFIGURATION

High Availability:
 HA is essential for ensuring seamless network operations by providing various mechanisms
for redundancy and reliability in network configurations.
 In case of switches and routers, some of the HA options available are:
- Switch-Stack
- vPC
- VSS
- HSRP
 In case of firewalls, available options related to HA are limited because of the stateful nature
of the firewall appliances.
 Most organizations opt for an Active/Passive setup, where the Standby firewall remains idle,
ready to take over in case of failure.
 In IVFRT, we are using Cisco ASA and Cisco FTD firewalls in the Active/Passive design in all
major ICPs.

FIREWALL MODELS USED IN IVFRT NETWORK

Cisco ASA models deployed Cisco FTD models deployed
 ASA 5516X  Firepower 2110 FPR2110-NGFW-K9
 ASA 5525  Firepower 1140 NGFW
 ASA 5555-X  Firepower 4110 NGFW
 ASA 5525-FPWR-K9  Firepower 4115
 ASA5512-K9 - FMC-2600

ASA Failover Modes

Cisco ASA supports 2 failover modes:

1. Active/Standby failover mode:

- One device acts as the Active unit, handling traffic, while the Standby
unit remains inactive.
- Upon failover, the Standby unit takes over as the Active unit and begins
processing traffic.

2. Active/Active failover mode:

- Both ASAs can handle traffic, but Active/Active failover is only supported
in multiple context modes.
- In this setup, security contexts are divided into two failover groups, with
one group active on the primary ASA and the other on the secondary
ASA.
- Failover occurs at the group level when needed.

HARDWARE & SOFTWARE REQUIREMENTS

- Both units in a failover configuration must be identical models with the

same number and types of interfaces, installed modules, and RAM.
- They should also run the same major and minor software versions and
have matching AnyConnect images.

NOTE: The two firewalls in a failover setup do not need to have identical
licenses; their licenses combine to create a single failover cluster
license.

FAILOVER & STATEFUL FAILOVER LINKS

- Like any firewall cluster, the firewalls must be connected to exchange information,
including heartbeat and state data.
For ASAs, two links are required: a failover link and an optional stateful failover
link. Cisco advises using the same interface on both firewalls; for instance, if you
use Gi0/1 on one device, you should also use Gi0/1 on the other.

1. Failover Link

The two units in a failover pair continuously communicate over the failover
link to monitor each other's operational status. The data exchanged over
this link includes:
- The unit state (Active or Standby)
- Hello messages/keep-alives
- Link status
- MAC address exchange
- Configuration replication & synchronization

2. Stateful Failover Link

 To implement Stateful Failover, a Stateful Failover link must be configured
to exchange connection state information. This link can utilize a dedicated
data interface, either physical or Port-Channel.

NOTE: Cisco recommends that the latency for the stateful failover link
should be under 10 milliseconds and no more than 250 milliseconds.

FAILOVER LINK DESIGN

Cisco advises using separate paths for failover links and data interfaces to reduce
the risk of simultaneous failures. If the failover link goes down, the ASA can utilize
data interfaces to assess the need for a failover, which will be paused until the
failover link is restored.

 NOT Recommended: If a single switch or a set of switches connects both

failover and data interfaces between two ASAs, a failure in the switch or
inter-switch link can lead to both ASAs becoming active simultaneously,
resulting in a split-brain scenario.

 Recommended: Cisco recommends that failover links should not share the
same switch as data interfaces. Instead, you can either use a different
switch or connect the failover link directly between the two units.
MAC ADDRESSES & IP ADDRESSES

Standby IP Address

When configuring ASA interfaces, you can optionally assign a

standby IP address on the same subnet as the active IP address.
During a failover, the new active unit takes over both the active IP
and MAC addresses.

IP/MAC ADDRESS BEHAVIOR DURING FAILOVER

The active unit always uses the primary unit's IP and MAC addresses.
Upon failover, the standby unit takes over the IP and MAC addresses
of the failed active unit and starts handling traffic.

When the failed unit comes back online, it remains in standby mode
and assumes the standby IP and MAC addresses.

FAILOVER SYSLOG MESSAGES

ASA generates several syslog messages related to failover, with message IDs in
the following ranges:
  101xxx
 102xxx
 103xxx
 104xxx
 105xxx
 210xxx
 311xxx
 709xxx
 727xxx

Cisco ASA & FTD: System Health & Network Diagnostic Messages Listed by
Severity Level
Severity Syslog Message Type
1 Alert
2 Critical
3 Error
4 Warning
5 Notification
6 Informational
7 Debugging

SYSLOG MESSAGE IDs Related to Firewall Failover

ASA FTD
103005 103005
104003 104003

CORRELATION RULE CREATION:

Based on these Syslog message IDs, we have created a correlation rule in

ArcSight Console to detect the ASA/FTD failover events.
RULE ACTION:

When the rule is triggered, it will send notifications to the SOC Operators user
group, where the analysts can further drill down the event details and do
further analysis.

NOTIFICATIONS IN CONSOLE:
NOTIFICATIONS IN COMMAND CENTER:

Here we can see that we are notifications when the rule is triggered.
We can further take actions on this by drilling down on the base events as
below:

Event ID: 103005

 EVENT ID: 104003

RECOMMENDATIONS:

 It is recommended to tag this rule with ArcSight SOAR and further open a
 case in SMAX.
 Your kind suggestion is required in this regard to take further actions.
 If this event is tagged with SMAX, we can work on that ticket by
coordinating with the concerned firewall team and do further investigation
to find out if it caused due to any suspicious activities or any kind of
hardware failure.
 We can close the ticket in SMAX which will automatically change the status
of the case in SOAR.
 In Console, as there is also an inbuilt feature of acknowledging the incident,
we can acknowledge the pending events, once the case is closed/resolved
in SMAX.

CISCO ASA FAILOVER MESSAGES FOR WHICH WE CAN CREATE MORE

RULES AND INTEGRATE THE RULES WITH SOAR AND SMAX
SYSLOG MESSAGE ID
ASA FTD SYSLOG EVENT
101002 101002 (Primary) Bad failover cable.
101003 101003 (Primary) Failover cable not connected (this unit).
101004 101004 (Primary) Failover cable not connected (other unit).
101005 101005 (Primary) Error reading failover cable status.
103001 103001 (Primary) No response from other firewall (reasonw code =
code).
103003 103003 (Primary) Other firewall network interface interface_number
failed.
103004 103004 (Primary) Other firewall reports this firewall failed. Reason:
reason-string.
103005 103005 (Primary) Other firewall reporting failure. Reason: SSM card
failure
104001 %threat (Primary) Switching to ACTIVE (cause: string).
defense-1-
104001
105001 105001 (Primary) Disabling failover.
105002 105002 (Primary) Enabling failover.
105005 105005 (Primary) Lost Failover communications with mate on
interface interface_name.
105011 105011 (Primary) Failover cable communication failure
105021 105021 (failover_unit) Standby unit failed to sync due to a locked
context_name config. Lock held by lock_owner_name
105022 105022 (host) Config replication failed with reason = (reason)
105032 105032 LAN Failover interface is down.
105034 105034 Receive a LAN_FAILOVER_UP message from peer.
105035 105035 Receive a LAN failover interface down msg from peer.
105036 105036 dropped a LAN Failover command message.
105037 105037 The primary and standby units are switching back and forth
as the active unit.
105038 105038 (Primary) Interface count mismatch
105039 105039 (Primary) Unable to verify the Interface count with mate.
Failover may be disabled in mate.
105043 105043 (Primary) Failover interface failed
105044 105044 (Primary) Mate operational mode mode is not compatible
with my mode mode.
105045 105045 (Primary) Mate license (number contexts) is not compatible
with my license (number contexts).
106101 106101 The number of ACL log deny-flows has reached limit
(number).