PIW Cisco ACI Upgrade Best Practices

ACI Firmware Upgrade
Architecture and Best Practices (Series)
Takuya Kishida, Joseph Ezerski, Alejandro De Alda, Jakub Horbacewicz

Cloud Networking Business Unit
May 2022
• Understand the fundamentals of
the ACI Upgrade Process
• Detail the architecture and
options for an upgrade
In this session... • Learn how to configure ACI
Upgrade Options and Tools
• Understand feature
enhancements for upgrade in
recent code releases
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Software Release Guidelines
Long Lived Releases
1 Two Long Lived Releases At Any Given Point of Time
2 Active Maintenance Will Be Primarily Focused On Long Lived Release
3 Target Duration Of Long -Lived Release Support: Up to 18 Months From FCS
4 Direct Upgrade From One Long Lived To Next Long -Lived Release Will Be Supported
5 Long Lived Releases Are Recommended For Networks That Will Not be Upgraded Frequently
Short Lived Releases
1 No Active Maintenance Beyond Six Months From FCS
Where to Start?
Recommended Guides !
Cisco ACI Upgrade Checklist – Important Starting Point
ACI Upgrade Checklist:
https://www.cisco.com/c/en/us/td/docs/swi
tches/datacenter/aci/apic/sw/kb/Cisco-ACI-
Upgrade-Checklist.html
Detailed Upgrade Guide (the basis for this

presentation)
https://www.cisco.com/c/en/us/td/docs/dc
n/aci/apic/all/apic-installation-aci-upgrade-
downgrade/Cisco-APIC-Installation-ACI-
Upgrade-Downgrade-Guide.html
Agenda
1 ACI Firmware Upgrade Best Practice 101
2 ACI Firmware Upgrade Architecture
3 ACI Firmware Upgrade Configuration
4 ACI Firmware Upgrade Enhancements
Agenda
ACI Firmware Upgrade Best Practice 101
Consider the fabric as
one huge switch
Spine
(Fabric Card)
Leaf (Line Card)
APIC (Supervisor)
ACI is a solution to manage multiple switches as if it’s one huge switch
➢ APIC (i.e. SUP of the fabric) can be upgraded non-disruptively

➢ Each switch (i.e. modules of the fabric) can intelligently choose appropriate switch nodes for non-disruptive traffic
flow
Always keep hardware redundancy to achieve zero-to-minimum traffic disruption

1. Upgrade Green switch groups
2. Upgrade Red switch groups
APIC and CIMC Versions
One: Check your version of APIC to match UCS Platform
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-installation-aci-upgrade-downgrade/Cisco-APIC-Installation-ACI-Upgrade-Downgrade-Guide/m-installing-or-
recovering-cisco-apic-images.html#id_22801
APIC and CIMC Versions
Two: Check APIC Release Notes (for your version) for HUU ISO
Best to use latest version of HUU

called out in your version’s release
notes
Boot APIC from HUU ISO and

upgrade EVERYTHING (not just CIMC
code)
Takes about 30-40 min per APIC (not

disruptive to APIC cluster)
This is a rare procedure and there is

wide version support
Example: APIC v5.2.4
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/release-notes/cisco-apic-release-notes-524.html#HardwareCompatibilityInformation
Agenda
ACI Switch Upgrade Flow
• The switch downloads the image from APIC No Traffic Impact

Image Download
• The download is via infra TEP
Download the image

Scheduler
Preparation
Reboot
Boot Up
Image Download No Traffic Impact
Upgrade Token
• The switch receives approval from APIC (Approval)
Scheduler
• Controls switches that are upgraded in parallel
Preparation • One leaf at a time in each vPC pair
Since APIC 4.1(1) • Not all spines in each pod if graceful

option is used
Reboot • One pod at a time

Not Applicable since
APIC 4.2(5) • 20 switches at a time (configurable)
Boot Up
Image Download No Traffic Impact
Scheduler
• The switch extracts the image.

Preparation Preparation
• The switch sets the boot var and so on.
Reboot
Boot Up
ACI Switch Upgrade Flow •
Depends on other conditions such as:
Link failure detection time on the external device
• Routing protocol
< 0.1 sec Traffic Impact

Image Download
in the best case
Scheduler ISIS detects

the tunnel down
Preparation
Reboot
• Wipe the config and reboot (i.e. clean reboot)

Reboot
• Traffic failover relies on link failure
Boot Up
Fail over with
the link down
Image Download
Scheduler
Preparation Boot Up
Reboot
• Various traffic flow optimizations

Boot Up
• (Continue to next slides)
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations
• Bring up fabric links

No Traffic Flow Change
01 • Bring up APIC connected down links
• Admin down other down links
02
03
Bring up
fabric ports
04
05
06
07

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned
03
TEP IP is
restored
04
05
06
07


• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
✓ ISIS does not advertise BD mcast groups to join Infra reachability is restored
04
05
06
07



✓ ISIS does not advertise BD mcast groups to join Config from APIC
(Takes several min)
04 • Starts downloading configurations from an APIC
05
06
07



Flood traffic starts coming
✓ ISIS does not advertise BD mcast groups to join
but no impact due to admin-
down down links
• ISIS multicast overload mode complete (i.e. flood)

05 • vPC peer is established at the same time
06
ISIS multicast overload timer
• Leaf nodes – Fixed 1min
• Spine nodes – When FTAG tree is created
07 (Fixed 1 min prior to Switch 14.2(1))
Boot Up • Various traffic flow optimizations Ready to receive traffic
• VLANs are deployed
• Bring up fabric links • For VMM, depends on Resolution Immediacy
01 • Bring up APIC connected down links • Contracts are deployed
• Admin down other down links • Depends on Deployment Immediacy
• Spine-Proxy is ready
• An APIC discovers the switch via DHCP/LLDP • Flood handling (FTAG) is ready


• Bring up down access links

06 ✓ once configurations are installed
✓ and vPC restore delay timer expires (for vPC ports)
• vPC restore delay timer is fixed to 120s since Switch 12.0(2)
• vPC restore delay timer starts when vPC peer is established.
07


02 • The same TEP IP is assigned Traffic flow is back to the previous status


• Bring up down access links

06 ✓ once configurations are installed
✓ and vPC restore delay timer expires (for vPC ports)
• ISIS unicast overload mode complete

07 ✓ The TEP IP is advertised with a normal metric
ISIS unicast overload timer - 10 min fixed for all nodes
ACI Switch Upgrade
Considerations with Multi-x
Helpful Tips for Multi-Pod / Multi-Site
Verify Spines are Sending Routes to the IPN after upgrade
IPN
Node Upgrade Group 1 Node Upgrade Group 1
Spine 1,4 Spine 1,4

Remaining spines
(Rebooting) (Rebooting)
sending pod TEP routes to IPN
Node Upgrade Group 2

Spine 2,3
• When Node Upgrade Group 1 finishes, Spines may show as “completed” in upgrade UI but routes
towards IPN/ISN may still be in hold down period (up to 10 min)
• Before starting Spine Node Upgrade Group 2, verify that TEP routes of pods / sites are being sent /
received from newly upgraded spines in Group 1
Helpful Tips for Multi-Pod / Multi-Site
Set default IS-IS fabric policy to a lower value before upgrade
IPN
ISIS Overload in process ISIS Overload in process

(in hold down) (in hold down)
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd75131
• For Multi-Pod be aware of CSCvd75131

• Easily avoided with IS-IS best practice
• Non-disruptive change
• Short Explanation:
• Default fabric wide IS-IS metric is set at 63 (max value)
• During upgrade, spines want to overload this value by design Set this value to < 63
• If fabric-wide value is already at max, there is no room to overload before any upgrade
• Can create unexpected traffic interruption in a strict HA upgrade window
ACI Switch Upgrade
with Graceful Option
ACI Switch Upgrade with graceful option
Image Download
Scheduler
Graceful Option is to gracefully isolate
the switch before the switch goes
down for the upgrade
Preparation
• Wipe the config and reboot (i.e. clean reboot)

Reboot
• Traffic failover relies on link failure
Boot Up The rest is the same as

without graceful option.
Additional reboot sequence with graceful option
• Graceful option disabled
1. Wipe the config and reboot (i.e. clean reboot) As shown

Reboot earlier
2. Traffic failover relies on user configured link failure mechanism
• Graceful option enabled

1. Put the switch into MMode (Maintenance Mode)
1. ISIS Overload Mode enabled Graceful
2. Graceful Shutdown on Routing Protocols Process
differences
✓ Leaf - BGP, EIGRP, OSPF for L3Out
✓ Spine – BGP, OSPF for IPN, GOLF
3. vPC informs its peer that this switch is going down
Reboot
4. LACP sends PDUs with aggregation bit zero (starting from 3.1(2))
➢ External devices can exclude the link from the port-channel before the link physically goes down.
5. Shutdown front panel ports
✓ Leaf – all down links including APIC connected links
✓ Spine – all IPN links
2. Wipe the config and reboot (i.e. clean reboot)
Warnings for ACI switch graceful upgrade
A graceful upgrade isolates the switch and assumes there are redundant paths via other nodes.
Never use the graceful option to upgrade switches within a redundancy pair simultaneously.
Examples of when NOT to use the graceful option:
Upgrading both leaf nodes connected to an

APIC with graceful option Upgrading all spines in a pod with graceful option
IPN links and BGP/OSPF are shut down before the upgrade.
All down links are shut down before the upgrade.
➢ Spines can no longer communicate to APICs to finish the
➢ The entire fabric will lose the access to the APIC and
upgrade.
the APIC cluster will be broken that could also
impact switch upgrades.
IPN
GIR and Graceful Upgrade in ACI
Both GIR (Graceful Insertion and Removal) and an upgrade using the graceful option put the
switch in MMode (Maintenance Mode) to isolate the switch from the fabric.
However, the use case for these two features are completely different.
GIR (Graceful Insertion and Removal) An upgrade with the graceful option
Use Case: Use Case:

• To isolate a switch for further debugging • To upgrade a switch after isolating the switch
• To quickly restore service by isolating a malfunctioning
switch
Difference:
Difference: • The switch will communicate to APIC and perform an
• It is not supported to upgrade a switch in MMode via GIR upgrade immediately after the switch was put into MMode.
Agenda
ACI Switch Upgrade Configuration Improvements
Prior to ACI 4.0/14.0 Since ACI 4.0/14.0

• Firmware group to specify a target version • A single group to manage:
• Maintenance group to control when to upgrade ✓ which switches
➢ Unnecessary flexibility ✓ which version
because all ACI switches should use the same version ✓ when
Firmware Group Maintenance Group Upgrade Group
• Name • Name • Name

• Node ID List • Node ID List • Node ID List
• Target Firmware Version • Scheduler • Target Firmware Version
Merge
• Ignore Compatibility Check • Run Mode • Scheduler
• Graceful option • Ignore Compatibility Check
• Graceful option
• Run Mode
Recommended Upgrade Procedure crossing 4.0
Although APIC takes care of the merging of firmware and maintenance groups for pre-4.0 to post-4.0 APIC upgrade
seamlessly, there were defects in the past.
➢ To avoid any unexpected behavior, the following procedure is recommended:
01 Remove all existing firmware and maintenance groups
02 Upgrade APICs from pre-4.0 to post-4.0 (ex. 3.2 -> 4.2)
03 Create new upgrade groups for switches
Upgrade switches from pre-14.0 to post-14.0 (ex. 13.2 -> 14.2) with the
04
new upgrade groups
Agenda
ACI Upgrade Enhancement Quick Summary
Supported APIC Version 3.2 4.1(1) 4.2(*) 4.2(4) 4.2(5) 5.0(1) Switch version requirements
APIC Detailed Install Stage N/A
Switch Image Pre-download

Switches also need to be on 14.1(1) or later
(with a scheduler)
Switch Image
Switches also need to be on 14.5(1) or later
Download Progress
Multi-Pod
No requirements*
Parallel Switch Upgrade
Unlimited
Parallel Switch Upgrade No requirements*
By Default
Pre-Upgrade Validation
No requirements*
(Specific Faults)
No requirements*
(Config)
Backported from 5.0(1) * APIC is what enforces/validate those rules
Upgrade Time Reduction
Switch Image Download Upgrade multiple

from APIC to switches pods/switches in parallel
4.1(1)
Switch Image Pre-Download with a scheduler
New label in ACI 14.2(5).

The functionality of pre-download has been the same since ACI 4.1. Long time ahead
Prior to 14.2(5), it was labeled as “Schedule for Later” with the same functionality..
1. Schedule for a long time ahead just to trigger pre-download of a switch image.
2. During the actual maintenance window, come back to this same window (maintenance group) and select “Now” to trigger the upgrade on
demand. Switches don’t need to re-download images and can proceed with the upgrade immediately.
4.2(5)
Switch Image Download Progress
New in ACI 4.2(5), download progress

(switches need to be 14.2(5) for this functionality)
• All switches (regardless of pods or vPC) in the update group download the switch image from APICs in parallel. During this period, the
Upgrade Progress remains 0 %.
• With the new Download Progress bar, users can see if switches finished the download and ready to upgrade.
• If it was triggered with a scheduler, all switches wait after they completed their download.
• If it was triggered with “Upgrade Now”, each switch proceed with the upgrade as soon as it has completed its download.
OLD
Upgrade multiple pods/switches in parallel
One pod at a time
When the actual upgrade starts, APICs allow each switch to upgrade based on the following rules;
• One Pod at a time (14.2(5) has an update)

• When triggered with “Upgrade Now”, 20 switches at a time (14.2(5) has an update)
• When a vPC pair leaf nodes are in the same group, only one of the pair at a time
4.2(5)
Unlimited Parallel Upgrade
All pods at once
• From APIC 14.2(5) or later, any switches in any pods can be upgraded in parallel
• “Upgrade Now” is no longer limited to 20 switches at a time
Useability
Detailed APIC install

progress
4.2(5)
APIC “Install Stage”
• This is available when your APIC is already on APIC 4.2(5) or higher.

New in ACI 4.2(5) and 5.0
1. Ready for next upgrade • When upgrading from an older version such as 4.2(4) to 4.2(5), you will not
2. Checking compatibility see this. The value of this feature is seen after APICs are upgraded to 4.2(5)
3. Checking controller health or later.
4. Performing upgrade (i.e. pre-loading the image)
5. Waiting for other controllers to upgrade • If you are downgrading from 4.2(5) to an older version then you will see this,
6. Migrating configuration however after the downgrade is complete you will be back to the previous
(i.e. data conversion from old object model to the new one) version’s UI functionality.
• If you downgrade from 5.0(1) to 4.2(5) this UI enhancement is still available.

3.2 - continuing
Pre-upgrade validation
APIC 3.2, 4.0, 4.1
• Prior to 4.2, the APIC upgrade simply warned about the
number of all critical and major faults
• On 4.2(1) – 4.2(3), the APIC upgrade warned about

APIC 4.2(1) – 4.2(3) ✓ config related critical faults
✓ some specific faults that are known to cause issues
during upgrades.
• On 4.2(4), the APIC upgrade warns about

✓ config related critical faults
✓ some specific faults that are known to cause issues
during upgrades
✓ A few nonoptimal configurations that may disrupt
traffic during the upgrade.
APIC 4.2(4)
• Additional validation items are being added on each release.
For older APIC versions to run some of the validations added in later release:
https://dcappcenter.cisco.com/pre-upgrade-validator.html
Pre-upgrade validation (AppCenter App)
The goal of the app
https://dcappcenter.cisco.com/pre-upgrade-validator.html
To be able to apply the latest validations on any APIC
versions via AppCenter app
(Currently only for APIC upgrade validations)
Note - This screen capture is from the latest development code

Pre-upgrade validation – Script (Preferred)
https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script
The goal of the script
To be able to apply the latest validations on any APIC

versions via a script
(Currently only for APIC upgrade validations)
Both app and script are fully supported by TAC
Why the script may be a better choice:
• Github script is updated more frequently

• Has added checks the app does not yet have
• With Github account, you can submit issues or
features requests directly
Live Demo With Script
Nexus Dashboard Insights (Optional)
Benefit of Nexus Insights
Does both a pre-check and a post-check to alert on effects
and changes in the upgrade window
• Pre-Update Verifications and Alerting

• Detailed list of bugs addressed in the upgrade
• Post-upgrade Delta analysis of Anomalies, Edits and
Operations changes in the upgrade process
Final Tips
Back-up with Disable APIC Avoid overlapping VLANs

Encryption Apps
Back Up Configuration with AES File Encryption !
In the Checklist But A Commonly Missed Step
Setting Global AES Encryption allows all the
• Turn this on before taking backup secure properties of the configuration (like
credentials) to be successfully imported when
• The AES passphrase that generates the encryption restoring the fabric
keys cannot be recovered or read by an ACI
administrator or any other user. The AES passphrase
is not stored. Copy your passphrase somewhere
safe!
• On import, be sure the “Fail Import if secure fields

cannot be decrypted” check box is enabled (on by
default) or you may lock yourself out
Pre ACI v4.0.1 Setting Location: ACI v4.0.1 and later Location:
Admin > AAA > AES Encryption Passphrase and Keys for Config System > System Settings > Global AES Passphrase Encryption
Export (and Import) Settings
Technote For Import/Export:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Using_Import_Export_to_Recover_Config_States.html
Temporarily Disable APIC Applications
Also in the Checklist But Easy To Forget
• Installed Apps found in Top Level APIC Apps Tab
• Click For Each Running App

• Running apps may slow down or impact the upgrade
process
• NI Base app (now called “Nexus Insights Cloud
Connector” is Exempt (leave it enabled)
• Re-enable apps after successful upgrade

• Be aware of Advisory Notice for CSCvv12524 (ACI
version < 4.1.1)
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw
/advisory/sw-advisory-CSCvv12524.html
The following applications send the uribv4Nexthop.type query that

can induce the crashes during mixed operating system (OS) mode:
• The ConnectivityCompliance APIC App by AlgoSec
• The AlgoSec Firewall Analyzer
Remember Overlapping VLANs?
Why avoiding overlapping VLAN pools
• Operational result when overlapping VLAN occurs is
unpredictable
• Incorrect configuration may lead to outages after a reboot

• Traffic being dropped
• Layer 2 loop
• Different situations can trigger the issues:

• Switch hard reboot
• Switch upgrade (we wipe and reload by design)
We covered the Overlapping VLAN use case and solution in a previous event on Access Policy Best Practices
Recording: https://players.brightcove.net/1384193102001/NJgI8K0ie_default/index.html?videoId=6274036977001
Remote Leaf Upgrade Order
CSCvs167670 – Only when going from 4.1(2) to 4.2(2)
Specific corner case if your customer has the following:
• ACI v 4.1(2) as current version
• Remote Leafs already deployed
Workaround: Upgrade Remote Leaf Nodes first
Why: ACI 4.1.(2) was the first release where we enabled direct forwarding between Remote Leafs (versus
hairpin via Spines) resulting in some internal enhancements
• ACI Upgrades can be done with minimal
to no impact to traffic forwarding with
Graceful Option
Key points to • ACI Pre-Upgrade Checklist can help with

planning and risk avoidance
remember • Recent enhancements in 4.2 and 5.x
releases can improve upgrade time and
experience
• Tools available to help
• Cisco APIC Installation and ACI Upgrade and Downgrade
Guide https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-installation-
aci-upgrade-downgrade/Cisco-APIC-Installation-ACI-Upgrade-Downgrade-Guide.html
• Cisco ACI Upgrade Checklist

Reference https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/Cisco-
ACI-Upgrade-Checklist.html
• Cisco APIC Release Notes

https://www.cisco.com/c/en/us/support/cloud-systems-management/application-
policy-infrastructure-controller-apic/tsd-products-support-series-home.html
• Release Notes for Cisco Nexus 9000 Series Switches in

ACI Mode https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-
switches/products-release-notes-list.html
• Access Policies (Delivered: Sept 2021)
Forwarding and Data-Plane
Coming Topics •
Cisco ACI Best Practices Series • External Connectivity

• Segmentation and Contracts

PIW Cisco ACI Upgrade Best Practices

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PIW Cisco ACI Upgrade Best Practices

Uploaded by

Copyright:

Available Formats

ACI Firmware Upgrade

Architecture and Best Practices (Series)

Takuya Kishida, Joseph Ezerski, Alejandro De Alda, Jakub Horbacewicz

1 Two Long Lived Releases At Any Given Point of Time

2 Active Maintenance Will Be Primarily Focused On Long Lived Release

3 Target Duration Of Long -Lived Release Support: Up to 18 Months From FCS

Short Lived Releases

1 No Active Maintenance Beyond Six Months From FCS

ACI Upgrade Checklist:

Detailed Upgrade Guide (the basis for this

1 ACI Firmware Upgrade Best Practice 101

2 ACI Firmware Upgrade Architecture

3 ACI Firmware Upgrade Configuration

4 ACI Firmware Upgrade Enhancements

1 ACI Firmware Upgrade Best Practice 101

2 ACI Firmware Upgrade Architecture

3 ACI Firmware Upgrade Configuration

4 ACI Firmware Upgrade Enhancements

Leaf (Line Card)

➢ APIC (i.e. SUP of the fabric) can be upgraded non-disruptively

Always keep hardware redundancy to achieve zero-to-minimum traffic disruption

Best to use latest version of HUU

Boot APIC from HUU ISO and

Takes about 30-40 min per APIC (not

This is a rare procedure and there is

1 ACI Firmware Upgrade Best Practice 101

2 ACI Firmware Upgrade Architecture

3 ACI Firmware Upgrade Configuration

4 ACI Firmware Upgrade Enhancements

• The switch downloads the image from APIC No Traffic Impact

Download the image

Image Download No Traffic Impact

Preparation • One leaf at a time in each vPC pair

Since APIC 4.1(1) • Not all spines in each pod if graceful

Reboot • One pod at a time

Image Download No Traffic Impact

• The switch extracts the image.

< 0.1 sec Traffic Impact

Scheduler ISIS detects

• Wipe the config and reboot (i.e. clean reboot)

• Various traffic flow optimizations

• Bring up fabric links

• Bring up fabric links

• An APIC discovers the switch via DHCP/LLDP

• Bring up fabric links

• An APIC discovers the switch via DHCP/LLDP

• ISIS overload mode is activated

• Bring up fabric links

• An APIC discovers the switch via DHCP/LLDP

• ISIS overload mode is activated

04 • Starts downloading configurations from an APIC

• Bring up fabric links

• An APIC discovers the switch via DHCP/LLDP

• ISIS overload mode is activated

• ISIS multicast overload mode complete (i.e. flood)

• ISIS overload mode is activated

04 • Starts downloading configurations from an APIC

• ISIS multicast overload mode complete (i.e. flood)

• Bring up down access links

• Bring up fabric links

• An APIC discovers the switch via DHCP/LLDP

• ISIS overload mode is activated

04 • Starts downloading configurations from an APIC