ACI Fabric Operations

Building an ACI Fabric from Scratch

Joseph Ezerski, Konrad “Smith” Rządziński
IBNG
v. 2021
Today’s Goal
We want to prove how easy it is to go from zero to fully functional ACI fabric in less than one
day… on a real ACI Fabric
• Initial Fabric Setup / Discovery
• Housekeeping
• Connecting things to the fabric
• Virtual Machine Integration
• Bare Metal servers
• Tenants, VRFs, BDs, Contracts/Filters
• Connecting things to the outside world
• L3Out to the Internet example
• Opportunities for Automation
Good News / Bad News
Bad News: Good News:
• We admit this is a A LOT of info for • There are very few slides
one day
• We will show everything on a live
• We also believe it is possible to learn
and do a basic fabric setup in a very ACI Fabric
short period of time • We will record everything and
• There are no labs for you share it in a series of video files
• There is no time plus I believe labs are (and any slides)
not something that you remember after
the day is done. It only proves you
can follow instructions.
But I was really hoping for labs….

dCloud has a lot of ACI specific labs you can do at your own pace…
Day-0 Setup
Prerequisites
• Before starting, you should have:
• At least 6 routable IP addresses for APIC OOB mgmt and APIC CIMC
• Functional NTP server
• Serial number of all leafs and spines
• Optionally but recommended:
• 1 IP per leaf and spine for OOB
• SCP / FTP / HTTP server
• Console / serial server
• Infrastructure VLAN / VTEP pool
• vCenter IP address and credentials

6
ACI Topology – Proper Production View
SPINE 201 SPINE 202

LEAF 101 LEAF 102 LEAF 103 LEAF 104 LEAF 105 LEAF 106

APIC-1 APIC-2 APIC-3

SFP+ 10G Ports

APIC M3

1G Mgmt Ports CIMC Port

 Domain VLAN Range

Today’s ACI Lab Topology ESX

HYPER-V
900-940

800-810

OSPF L3 OUT 333

SPINE 201 10.50.129.241 Bare Metal 99

APIC

HyperV-Server
HYPER1

1/45-46
LEAF 103
LEAF 101 LEAF 102
“border leaf”
1/45-46 1/17 1/18 1/17 1/18

1/19
L3 OUT
(OSPF)
g0/1
UCS FI-A UCS FI-B ASA 5512X Internet

mgmt: 10.50.129.120

Servers
VMs
VMs
A production fabric has a min. of 3 APICs and two
spines. My lab uses one which is not HA
APIC Console Access First (Configure CIMC)

• CIMC interface default uses DHCP

• Recommend to set a fixed IP for server level management access
• Can be changed from start up screen interrupt (shown above), or if DHCP, from the CIMC UI
• This IP is also how we launch KVM or SoL access to APIC (needed for Day-0 Setup)
APIC Day 0 Setup Questions Explained
Cluster configuration ... Cluster configuration ...
Enter the fabric name [ACI Fabric1]: ams-fab Fabric Name: Choose whatever you want
Enter the fabric ID (1-128) [1]: 1 Fabric ID: Best to begin with “1”
Enter the number of active controllers in the fabric (1-9) [3]: 3 Controllers: Production uses 3 nodes
Enter the POD ID (1-254) [1]: 1 POD ID: Just choose 1 (related to something called Multipod, not shown)
Is this a standby controller? [NO]: Standby: We offer an optional 4th controller in standby mode
Enter the controller ID (1-3) [1]: Controller ID: First controller is node 1 (others are 2 and 3 in order)
Enter the controller name [apic1]: apic-ams Controller name: Whatever you want
Enter address pool for TEP addresses [10.0.0.0/16]: Enter address pool for TEP addresses [10.0.0.0/16]:
Note: The infra VLAN ID should not be used elsewhere in your environment and TAC recommends /16 with minimum of /19
should not overlap with any other reserved VLANs on other platforms.
Enter the VLAN ID for infra network (1-4094): 3456 VLAN ID for infra network (1-4094): Choose a non-overlapping vlan
Enter address pool for BD multicast addresses (GIPO) [225.0.0.0/15]: BD multicast addresses (GIPO) [225.0.0.0/15]:
Must be a /15 in the multicast range
Out-of-band management configuration ... Out-of-band management configuration ...
Enable IPv6 for Out of Band Mgmt Interface? [N]: N Enable IPv6: Your Choice
Enter the IPv4 address [192.168.10.1/24]: 10.50.129.241/24 IPv4 address: Your own OOB IP for this APIC
Enter the IPv4 address of the default gateway [None]: 10.50.129.254 IPv4 address of the default gateway: Your OOB Gateway
Enter the interface speed/duplex mode [auto]: Interface speed/duplex mode: Auto

admin user configuration ... admin user configuration ...

Enable strong passwords? [Y]: N Enable strong passwords? [Y]: Your choice
Enter the password for admin: Enter the password for admin: Your choice

Reenter the password for admin: Reenter the password for admin: Your choice
APIC Day 0 Setup – Remaining Nodes
APIC 2 APIC 3
Cluster configuration ... Cluster configuration ...
Enter the fabric name [ACI Fabric1]: ams-fab Enter the fabric name [ACI Fabric1]: ams-fab
Enter the fabric ID (1-128) [1]: 1 Enter the fabric ID (1-128) [1]: 1
Enter the number of active controllers in the fabric (1-9) [3]: 3 Enter the number of active controllers in the fabric (1-9) [3]: 3
Enter the POD ID (1-254) [1]: 1 Enter the POD ID (1-254) [1]: 1
Is this a standby controller? [NO]: Is this a standby controller? [NO]:
Enter the controller ID (1-3) [2]: Enter the controller ID (1-3) [3]:
Enter the controller name [apic1]: apic2-ams Enter the controller name [apic1]: apic3-ams
Enter address pool for TEP addresses [10.0.0.0/16]: Enter address pool for TEP addresses [10.0.0.0/16]:
Note: The infra VLAN ID should not be used elsewhere in your Note: The infra VLAN ID should not be used elsewhere in your
environment environment
and should not overlap with any other reserved VLANs on other and should not overlap with any other reserved VLANs on other
platforms. platforms.
Enter the VLAN ID for infra network (1-4094): 3456 Enter the VLAN ID for infra network (1-4094): 3456
Enter address pool for BD multicast addresses (GIPO) [225.0.0.0/15]: Enter address pool for BD multicast addresses (GIPO) [225.0.0.0/15]:

Out-of-band management configuration ... Out-of-band management configuration ...

Enable IPv6 for Out of Band Mgmt Interface? [N]: N Enable IPv6 for Out of Band Mgmt Interface? [N]: N
Enter the IPv4 address [192.168.10.1/24]: 10.50.129.211/24 Enter the IPv4 address [192.168.10.1/24]: 10.50.129.212/24
Enter the IPv4 address of the default gateway [None]: 10.50.129.254 Enter the IPv4 address of the default gateway [None]: 10.50.129.254
Enter the interface speed/duplex mode [auto]: Enter the interface speed/duplex mode [auto]:

Add APIC nodes 2 & 3 after fabric discovery

Fabric Discovery and
“Housekeeping”
 Node Info For Reference (Best Practice)
Device Name Node Number Serial # OOB Mgmt IP Address

Leaf-1 101 FDO22220TF1 10.50.129.242/24

Leaf-2 102 FDO22232PL6 10.50.129.243/24

Leaf-3 103 FDO21162N03 10.50.129.244/24

Spine-1 201 FDO24090Y29 10.50.129.247/24

APIC 1 10.50.129.241/24

NTP 173.38.201.67 DNS 173.38.200.100

A production fabric has a min. of 3 APICs. My lab uses one which is not HA
 ACI Lab Topology – OOB Network (Best Practice)
10.50.129.245 SPINE 201

APIC

10.50.129.241
OOB
10.50.129.x/24

LEAF 103
LEAF 101 LEAF 102
“border leaf”
1/17 1/18 1/17 1/18 1/1
10.50.129.242 10.50.129.243 10.50.129.244 10.50.129.204

vCenter

1/3 DC
UCS FI-A UCS FI-B ASA5512X traditional VMs
network VMs
UCSM: 10.50.129.15 10.50.129.120

Servers

VMs
VMs
Connecting things to the
fabric…
 Domain VLAN Range

ACI Lab Topology – Connecting Stuff ESX

HYPER-V
900-940

800-810

OSPF L3 OUT 333

Bare Metal 99

Goal #1: Connect a blade server chassis to ACI Leafs with Virtual Port Channel
LEAF 101 LEAF 102
1/17 1/18 1/17 1/18
This example uses a UCS-B Series Chassis but any vendor is supported

VPC-A VPC-B • VPC_A: Eth1/17 (same on both leafs in the VPC pair)
• VPC_B: Eth1/18 (same on both leafs in the VPC pair)

UCS FI-A UCS FI-B Note: Blade Chassis parameters already configured by server team

Goal #2: Integrate APIC with Hypervisor running on blade in chassis (vCenter)
Blade Servers • Also called “VMM Integration”

ESXi Host
on blade

VC_DVS
VMs
VMs
 ACI Objects and Relationships
VM “Tells ACI what is at the other
M end of the cable”
VLAN
Pools Domain

Phys

This effectively glues together the

AAEP “The Glue” configuration of the physical ports
below to what is actually connected to
those ports above

Interface policy group CDP, LLDP, Bcast, etc “Select how interface should
behave”

Switch
“Select where interface profile
Interface “Select who the interface
profile
lives” is”

-switch selectors- -Interface selectors-

leaf x, VPC pair, etc e1/1, 1/x …
What about Bare Metal?
 Domain VLAN Range

ACI Lab Topology ESX

HYPER-V
900-940

800-810

OSPF L3 OUT 333

SPINE 201 10.50.129.241 Bare Metal 99

APIC

HyperV-Server
HYPER1

1/45-46
LEAF 103
LEAF 101 LEAF 102
“border leaf”
1/45-46 1/17 1/18 1/17 1/18

1/19
L3 OUT
(OSPF VLAN-333)
g0/1
UCS FI-A UCS FI-B ASA 5512X Internet

mgmt: 10.50.129.120

Servers
VMs
VMs
Now Get those VMs to talk…
ACI Tenant Hierarchy (Basic Building Blocks)
TENANT: Simply a container to keep things organized

TENANT VRF: IPv4 or IPv6 Routing Space. A tenant can

have multiple VRFs (if you want)
VRF
BRIDGE DOMAIN: Container for subnets plus settings for
BRIDGE DOMAIN L2, flooding and/or unicast routing. Default gateways
configured here. Usually one BD per subnet

Subnet(s) & Default Gateway(s) END POINT GROUP: This is where you put actual end
points (physical or virtual) and apply forwarding and
security policy. EPGs attach to BDs. An ACI fabric will
have many EPGs

END POINT GROUP

Domain
Virtual or Physical
MAPPING

DOMAIN: An object to group EPs by type and how

they physically connect to the fabric, like Bare Metal
Servers or Virtual Machines. We map this to an EPG to
establish EP membership
Contracts and ACI
• The entire ACI Fabric is a stateless firewall out of the box (zero trust model)
• If EPGs are groupings of endpoints with a similar policy, we use contracts to define
how EPGs can talk to each other and what they can talk about
• We can use contracts for many types of policy
• Security
• Quality of Service
• Service Insertion
END POINT GROUP 1 Contract END POINT GROUP 2
• Policy Based Redirect
• Microsegmentation Filters:

• SD-WAN Policy Permit 80,443

Permit ICMP
Permit TCP x-y
Permit UDP x-y
A Word about Contracts
• Contracts are similar to ACLs
• We don’t use IP address or subnet
• We use EPG Membership Protocol and Port
• Contracts ARE needed when you want to talk between EPGs
• We call this a “whitelist model”
• Contracts are NOT needed if talking within the same EPG
• Contracts need “consumers” and “providers”
• Why? We need to match up who is talking to whom based on contract configuration
• Contract must match for consumer to talk to provider and vice versa
Contracts Example

END POINT GROUP 1 Contract END POINT GROUP 2

I am a webserver group and I

I am a client and I want to
provide web services on Ports
talk to a webserver in EPG2
80 and 443

One contract but deployed in

both EPGs.
CONSUMING SERVICES The contract name must PROVIDING SERVICES
match! (i.e we need to know
which policy to apply
between EPs that are talking)
What about external routing?
L3Out – Leaf to External ASA (Internet Example)
Goal: Connect endpoints to the Internet

LEAF 103
“border leaf”
1/19
5.5.1.2/24 (router ID 51.1.1.1)
L3 Out OSPF P2P
• MTU Ignore Gig0/1 (inside) 5.5.1.1/24
• No BFD support on ASA
ASA 5512-WEST
Mgmt0: 10.50.129.120
Gig0/0 (outside) 10.50.128.x (NAT)

Interwebs
0.0.0.0/0
General Best Practices
• Highly advised to set up OOB Management
• Choose a useful naming system ahead of time
• If you use the wizards, know what gets created in the background too!
• Prefer symmetry whenever possible (but optional, of course)
• When something is not working, check your faults first!
• Use ACI Inspector for Rest API code generation
• Automate the easy stuff!
General Order of Operation
• Run APIC Initial Setup Script (via CIMC on APIC or SoL)
• Discover and Register leaves and spines
• Stage One: Housekeeping Tasks
• Set up OOB addresses to nodes
• Set up DNS (optional)
• Set up Date/Time and NTP
• Set up BGP RRs
General Order of Operation
• Stage Two: Physical Connectivity
• Understand the object model about port configurations
• Set up VPC to UCS Fabric Interconnects
• VMM integration with vCenter
• Use script to push any remaining port configurations
• To save time and not repeat the same steps over again
General Order of Operation
• Stage Three: Build a tenant and consume what we connected
• Create Tenant, ANP and three EPGs
• Silver, Gold, and Migration EPGs
• Map to VMM Domain
• Show the use of a physdom and static port binding
• Contract Basics
• Filters, Subjects, Contracts, Preferred Groups and vzAny
• Set up L3 Out with OSPF to ASA Firewall
• Understand the concept of the External EPG and the need for contracts here