BRKCRS 2825

Cisco SD-Access
Scaling the fabric to multiple sites
Scott Hodgdon
Senior Technical Marketing Engineer
BRKCRS-2825
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKCRS-2825 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Who is Scott ?
Personal
• Based in Raleigh, NC (US)
• 19-year-old daughter in university (she’s smarter than I)
Career
• 19 years as a Technical Marketing Engineer
• 13 Years focused on just Catalyst 6K Family
• 15 years as a Cisco Live Speaker
• 9 years as Cisco Live Session Group Manager for US and EMEA
• 2 Years as a Cisco Partner SE
• 2 Years Lead Network Engineer for 15-site Health Care network
in North Carolina
• No formal technology schooling … I have a Business Degree with
a Finance Concentration
Current Focus
• Cisco SDA Network Design and Partner Enablement

• Catalyst 9K Network Design and Partner Enablement
Session Goals
• This session assumes that there is a basic understanding

of Cisco SD-Access and is recommended that you
attend BRKCRS-2810, BRKCRS-2821, BRKCRS-2815
before this.
• To enable the audience to imagine a fully functional

Large enterprise network with hundreds of campuses
and remote locations running on Cisco SD-Access
fabric network with wired and wireless clients.
Agenda
• Introduction to Cisco SD-Access
Fabric Roles and Constructs
• Scaling the fabric in a single site
Vertical scaling of fabric end points , Cisco DNA Center (for
automation and assurance) and Cisco ISE (for policy)
• Deploying Cisco SD-Access in small branches/sites
Collapsed fabric roles ( Border + edge + control plane + WLC )
• Scaling the fabric across multiple sites
Expanding the fabric across multiple metro/WAN regions
Horizontal scaling of fabric end points , Cisco DNA Center (for
automation and assurance) and Cisco ISE (for policy)
• Cisco SD-Access fabric across Geographical Locations ( Design Overivew)
• Conclusion
Fabric Roles and
Constructs
Cisco SD-Access
Fabric Roles & Terminology
Cisco DNA  Cisco DNA Automation – provides simple
GUI management and intent based
Identity NCP Automation
automation (e.g. NCP) and context sharing
Services
ISE NDP  Cisco DNA Assurance – Data Collectors
Cisco DNA (e.g. NDP) analyze Endpoint to App flows
Cisco DNA
Center Assurance and monitor fabric status
 Identity Services – NAC & ID Systems
(e.g. ISE) for dynamic Endpoint to Group
Fabric Border Fabric Wireless mapping and Policy definition
Nodes Controller
B B  Control-Plane Nodes – Map System that
manages Endpoint to Device relationships
Intermediate Control-Plane
C Nodes  Fabric Border Nodes – A Fabric device
Nodes (Underlay) (e.g. Core) that connects External L3
network(s) to the SDA Fabric
Campus  Fabric Edge Nodes – A Fabric device

(e.g. Access or Distribution) that connects
Fabric Edge
Nodes Fabric Wired Endpoints to the SDA Fabric
 Fabric Wireless Controller – A Fabric device
(WLC) that connects APs and Wireless
Endpoints to the SDA Fabric
Cisco SD-Access Fabric
Control-Plane Nodes – A Closer Look
Control-Plane Node runs a Host Tracking Database to map location information
• A simple Host Database that maps Endpoint IDs to C

Unknown
a current Location, along with other attributes
Known
Networks Networks
B B
• Host Database supports multiple types of Endpoint
ID lookup types (IPv4, IPv6 or MAC)
• Receives Endpoint ID map registrations from Edge

and/or Border Nodes for “known” IP prefixes
• Resolves lookup requests from Edge and/or Border

Nodes, to locate destination Endpoint IDs
Control-Plane Nodes – Scale Considerations
Control-Plane Node runs a Host Tracking Database to map location information
• Each fabric site can support up to six control plane C

Unknown
nodes .
Known
Networks Networks
• For a wired only network we can support a B B

maximum of six control plane nodes
• For a wireless only or wired + wireless network
we can support a maximum of two control
plane nodes.
• All the control planes nodes in a given state work in

an active-active mode without any synchronization
between them.
Edge Nodes – A Closer Look
Edge Node provides first-hop services for Users / Devices connected to a Fabric
• Responsible for Identifying and Authenticating C

Endpoints (e.g. Static, 802.1X, Active Directory)
Known Unknown
Networks Networks
B B
• Register specific Endpoint ID info (e.g. /32 or /128)
with the Control-Plane Node(s)
• Provide an Anycast L3 Gateway for the connected

Endpoints (same IP address on all Edge nodes)
• Performs encapsulation / de-encapsulation of data

traffic to and from all connected Endpoints
Border Nodes
Border Node is an Entry & Exit point for data traffic going Into & Out of a Fabric
There are 3 Types of Border Node! C

Known Unknown
Networks Networks
B B
• Rest of Company/Internal Border Used for
“Known” Routes inside your company
• Outside World/External Border Used for

“Unknown” Routes outside your company
• Anywhere/External + Internal Border Used

for “Known” and “UnKnown” Routes for your company
Border Nodes – Rest of Company/Internal
Rest of Company/Internal Border advertises Endpoints to outside, and known

Subnets to inside
• Connects to any “known” IP subnets available from C

the outside network (e.g. DC, WLC, FW, etc.)
Known Unknown
Networks Networks
B B
• Exports all internal IP Pools to outside (as
aggregate), using a traditional IP routing protocol(s).
• Imports and registers (known) IP subnets from

outside, into the Control-Plane Map System except
the default route.
• Hand-off requires mapping the context (VRF & SGT)

from one domain to another.
Border Nodes – Outside World/External
Outside World/External Border is a “Gateway of Last Resort” for any unknown

destinations
• Connects to any “unknown” IP subnets, outside of C

the network (e.g. Internet, Public Cloud)
Known Unknown
Networks Networks
B B
• Exports all internal IP Pools outside (as aggregate)
into traditional IP routing protocol(s).
• Does NOT import any routes! It is a “default” exit, if

no entry is available in Control-Plane.
• Hand-off requires mapping the context (VRF & SGT)

from one domain to another.
Border Nodes – Outside World/External
Anywhere/ Internal + External Border is a “One all exit point” for any known
and unknown destinations
• Connects to any “unknown” IP subnets, outside of

Unknown
the network (e.g. Internet, Public Cloud) and
Known
Networks Networks
C
“known” IP subnets available from the outside B
network (e.g. DC, WLC, FW, etc.)
• Imports and registers (known) IP subnets from

outside, into the Control-Plane Map System except
the default route.
• Exports all internal IP Pools outside (as aggregate)

into traditional IP routing protocol(s).
Border Nodes – Scale Considerations
Border Node is an Entry and Exit point for data traffic going Into and Out of a
Fabric
• Each fabric site supports a maximum of four Outside
World / External Border nodes. C
Known Unknown
Networks Networks
• Each fabric site supports a maximum of four outside B B
Anywhere / Internal+ External Border nodes.
• The above two borders are cumulative in a given

fabric site.
• EX: if we have two Outside World borders in a fabric site
then we can only have two more Anywhere Borders.
• Each fabric site can support hundreds (up to fabric site

device scale) of Rest of Company / Internal Border nodes.
Scaling the fabric
in a single site
General Scaling Strategy for Network Designs
How do I achieve higher scale?
Vertical Scaling Horizontal Scaling
• Using larger capacity devices / platforms to • Using more devices to achieve higher scale.
achieve higher scale.
• Multiple devices can aggregately provide
• Larger capacity devices provide more control greater control and data plane scale.
plane as well as more data plane scale.
• Distributed model of scaling.
• Centralized model of scaling.
• Using multiple C9400’s as core platform or
• Using an C9600 as a core platform or an using C9300’s as BGP route reflector etc.
C9500 as a BGP route reflector etc.
Scaling Strategy for Cisco SD-Access Networks
• Using a larger capacity control plane node • Using more control plane nodes and wireless
and wireless controller to achieve scale for controllers to achieve scale for the fabric
the fabric infrastructure. infrastructure.
• In a given single site by increasing the control planes
nodes we cannot provide higher scale. To achieve
• Using a larger Cisco DNA Center appliance to higher scale in this model we have to split a single fabric
achieve higher automation and assurance site into multiple sites.
scale for fabric.
• Using multiple clusters of Cisco DNA Center
• Using a larger ISE node to achieve higher appliances to achieve higher automation and
scale for authentications and policy. assurance scale for fabric.
• Using multiple ISE nodes and with load

balancing can achieve higher scale for
authentications and policy.
Scaling Strategy for within a Fabric Site
• Using a larger capacity control plane node • Using more control plane nodes to achieve
and wireless control to achieve scale for the scale for the fabric infrastructure.
fabric infrastructure. • In a given single site by increasing the control planes
higher scale in this model we have to split a single fabric
• Using a larger Cisco DNA Center appliance to site into multiple sites.
achieve higher automation and assurance
scale for fabric. • Using multiple clusters of Cisco DNA Center
appliance to achieve higher automation and
• Using a larger ISE Node to achieve higher assurance scale for fabric.
scale for authentications and policy.
• Using multiple ISE Node’s and with load
Scaling Strategy for Cisco SD-Access Networks
CP redundancy for equivalency and not scale
• No Synchronization between CP nodes

Branch • Edges must Register with ALL CP nodes
• Map Requests are round-robin to CP nodes
Border
LISP Map Server: 5.2.2.2

Map Server: 5.1.1.1
Mapping DB
2.1.1.1 2.1.2.1 3.1.1.1 3.1.2.1
Edge Edge Edge Edge

10.2.0.0 /16
Campus
Scaling Strategy for Fabric within a site
Design for a small size fabric site
Key Decision Points
Cisco DNAC
DC 1 NCP + NDP
Cluster
ISP
ISE
1 PAN + PXG
Internet
• Tends to be Building or Office
+ PSN
DDI
IP
with < 1000 endpoints and
1 DHCP + DNS
+ IPAM
< 100 IP Pools/Groups
• 1-2 Collocated CP +
CP EB CP EB
External Border (Single Exit)
Site
• Tends to be local WLC
connected to Border
+ SD-Access Wireless
• Looking at <1000 dynamic

authentications and <250
group based policies
Design for a medium size fabric site
Key Decision Points
Cisco DNAC
DC 3 NCP + NDP
Cluster
ISP
ISE
2 PAN + PXG
Internet
• Tends to be Multiple Buildings
2 PSN
DDI
1 DHCP + DNS
1 IPAM IP with < 10,000 endpoints and
• Can choose a Co-located or a

CP EB CP EB
Distributed/Dedicated CP +
External Border(Single Exit)
Site CP CP
design
• Tends to be WLC + SD-

Access Wireless via Services
Block or a local Data Center
• Looking at < 10,000 dynamic

authentications and < 1000
Design for a Large size fabric site
DC
Cisco DNAC
5-7 NCP + NDP
Cluster
WAN ISP Key Decision Points
ISE
• Tends to be Many Buildings

2 PAN 2 PXG
5-10 PSN Internet
DDI
1 DHCP 1 DNS
1 IPAM Internal External with < 25,000 endpoints and
• Can choose a Co-located or a

CP IB IB CP EB EB
Distributed/Dedicated CP + 2-
4 Borders (Multiple Exits)
Site
CP CP • Tends to be WLC + SD-
Access Wireless via local DC
• Looking at < 25,000 dynamic

authentications and < 2000
Scaling Strategy for Fabric within a Site
and wireless controller to achieve scale for the scale for the fabric infrastructure.
appliances to achieve higher automation and
• Using a larger ISE node to achieve higher assurance scale for fabric.
SD-Access Platforms
For more details: cs.co/sda-compatibility-matrix
The Channelco®
Fabric Control Plane CRN®

Products of the Year
2017, 2018
NEW
Catalyst 9300 Catalyst 9400 Catalyst 9500 Catalyst 9600
• Catalyst 9300 • Catalyst 9400 • Catalyst 9500 • Catalyst 9600

• 1/mG RJ45 • Sup1XL • 40/100G QSFP • Sup1
• 10/25/40/mG NM • 9400 Cards • 1/10/25G SFP • 9600 Cards
SD-Access Platforms
Fabric Control Plane
Catalyst 3K Catalyst 6K ISR 4K & ENCS ASR1K
NEW
• Catalyst 3650/3850 • Catalyst 6500/6800 • ISR 4430/4450 • ASR 1000-X

• 1/mG RJ45 • Sup2T/Sup6T • ISR 4330/4450 • ASR 1000-HX
• 1/10G SFP • C6800 Cards • ENCS 5400 • 1/10G RJ45
• C6880/6840-X • ISRv / CSRv • 1/10G SFP
• 1/10/40G NM Cards
SD-Access – CP Scale
ASR1K
Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Nexus or CSR1K
Scale
3850(XS) 9300 9400 9500 9500H 9600 6800 N7700 ISR4K V
Control- 200K
Plane (16GB)
3K 16K 80K (XL) 80K 150K 150K 50K N/A 200K
(LISP) 100K
Entries (8GB)
SD-Access – Border Scale
ASR1K
Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Nexus
Scale or CSR1KV
3850(XS) 9300 9300L 9400 9500 9500H 9600 6800 N7700
ISR4K
Virtual
64 256 256 256 256 256 1K 500 500 4K 4K
Networks
IPv4 Fabric 4M
1M (XL)
Routes (16GB)
8K 8K 8K 80K (XL) 48K 48K 200K 256K 500K 200K
(LPM 1M
(LE)
IP/mask) (8GB)
4M
IPv4 Host 1M (XL)
(16GB)
Entries 16K 16K 16K 80K (XL) 80K 150K 150K 512K 32K 100K
1M
(Host /32) (LE)
(8GB)
IPv4:SGT
12K 10K 10K 40K 40K 40K 200K 256K 200K 750K 750K
Bindings
SGT/DGT
4K 8K 8K 8K 8K 16K 32K 30K 16K 64K N/A
Policies
SGACEs
30K (XL)
(Contract 1500 5K 5K 18K 18K 13K IPv4 27K 128K 64K N/A
12K (LE)
Actions)
SD-Access – Edge Scale These are 1D numbers
Catalyst
Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst
Scale 4K
3650 3850 9200L 9200 9300 9300L 9400 9500
(Sup8/9E)
Virtual
64 64 1* 4* 256 256 64 256 256
Networks
Local End
2K 4K 2K 4K 4K 4K 4K 4K 4K
Points/Hosts
IPv4:SGT
12K 12K 8K 10K 10K 10K 128K 40K 40K
Bindings
SGT/DGT
4K 4K 2K 2K 8K 8K 2K 8K 8K
Policies
SGACEs
(Contract 1350 1350 1K 1K 5K 5K 64K 18K 18K
Actions)
* 9200 = 1 DEFAULT_VN + 1 INFRA_VN + 3 User-Configured VNs

* 9200L = 1 DEFAULT_VN + 1 INFRA_VN - No extra User VN possible
SD-Access Platforms
Fabric Enabled Wireless

* No IPv6, AVC, FNF
AireOS WLC Catalyst 9800 Wifi 6, 11ac Wave 2 Wave 1*AP

NEW
NEW
• AIR-CT3504 • Catalyst 9800-40/80 • Catalyst 9100 • AIR-CAP1700, 2700

• AIR-CT5520 • Catalyst 9800-CL • AIR-CAP1800, 2800, and 3700
• AIR-CT8540 • C9K Embedded WLC 3800 and 4800 • AIR-CAP1540, 1560
• 802.11ax, 11ac Wave2 • 802.11ac Wave1*
Wireless Controller Design
Active Standby
Scale remains same
Client updates
SSO pair
Control Plane redundancy is supported
in Active / Active configuration
C C
B
WLC is configured with two CP nodes
with information sync across both
Stateful redundancy with WLC SSO

pair. Active WLC updates Control nodes
SD-Access – WLC Scale
Number of end
Platform Number of AP’s SDA Design
points
3504 150 3000 Small
5520 1500 20,000 Small or Medium
8540 6000 40,000 Medium or Large
Catalyst 9800L 250 5000 Small
Catalyst 9800-40 2000 32000 Medium
Catalyst 9800-80 6000 64000 Large
Catalyst 9800-CL 1000/3000/6000 10000/32000/64000 Small/Medium/Large
Cisco DNA Center Scale
Parameters DN2-HW-APL DN2-HW-APL-L DN2-HW-APL-XL
Number of Devices 1000 2000 5000
(Switch/Stack, Router, WLC)
Number of Access Points 4000 6000 12000
Number of Endpoints (Concurrent) 25,000 40,000 100,000
Number of Endpoints (Unique/Transient) 75,000 120,000 250,000
over 14 days
Number of Endpoints (Wired/Wireless) Any Any Wired: 40,000

ratio of total Wireless: 60,000
Number of Sites 500 1000 2000
Number of WLC 500 1000 2000
Number of Ports 48,000 192,000 480,000
API Rate-Limit 50 APIs/min 50 APIs/min 50 APIs/min
Cisco DNA Center Scale – Cisco SD-Access Focus
Parameters DN2-HW-APL DN2-HW-APL-L DN2-HW-APL-XL
Number of Fabric Domains 10 20 20
Number of Fabric Sites 500 1000 2000
Number of Virtual Networks 64/Site 64/site 256/site

(per Fabric Site)
Number of Fabric Devices 500/site 600/site 1000/site
(per Fabric Site)
Number of Scalable Groups 4000 4000 4000
Number of Access Contracts 500 500 500
Number of Group-Based Policies 25000 25000 25000
Number if IP Pools 100/site 300/site 600/site
* If any DNAC scale item (e.g. Endpoints) gets max out in single site, then it cannot be scaled more by adding another site.
** Cisco DNA Center Release 1.3.1.0 supports tracking upto only 1.2 million separate interfaces on the fabric devices.
Interfaces include physical and virtual interfaces, like switched virtual interface, loopback, Dot1Q, tunnel and so on.
Cisco DNA Center Design- Where to Locate it ?
Local DC or Services Block Remote DC (Over MAN/WAN)
ISE + AD/Other ISE + AD/Other
Cisco DNA Cisco DNA

Center DNS/DHCP Center DNS/DHCP
Internet Internet
DC
Metro
NOTE: Cisco DNA Center requires access to

Internet
Cisco DNA Center Design- Three Node High Availability
1 or 3 appliance HA Cluster
- Odd number to achieve quorum
of distributed system
- Scale does not change
Seen as 1 logical Cisco DNA Center

instance
- Virtual (Cluster) IP
Cisco DNA Center apps on Maglev - Rare need to access individual node
cluster (e.g. SSH)
Virtual IP 2 nodes active/sharing + 1 redundant

- Some services run multiple copies
spread across nodes (e.g. databases)
- Other services run single copy and
migrate from failed to redundant node
Cisco DNA Center Design- Three Node High Availability
Users can choose to deploy Cisco DNA Center as a single node or 3-node cluster.
 3-node cluster deployment is for redundancy and to mitigate the split-brain problem.
1. Bring up 1st Cisco DNA Center node

 Complete the installation (Virtual IP, Intra-Cluster link) and let the services come up...
2. Bring up the 2nd Cisco DNA Center node

 Let the installation complete
3. Bring up the 3rd Cisco DNA Center
node
Things to Remember:
• 2-node DNAC cluster cannot withstand a node failure
• A one node crash will lead to a stall of the other node
Cisco ISE Scale
 Policy Service Node (PSN)

– Makes policy decisions
Can run in a single host
– RADIUS server and provides endpoint/user services
 Policy Administration Node (PAN)
– Interface to configure policies and manage ISE deployment
– Replication hub for all database config changes
 Monitoring and Troubleshooting Node (MnT)
– Interface to reporting and logging
– Destination for syslog from other ISE nodes and NADs
PXG  pxGrid Controller
– Facilitates sharing of information between network elements
Cisco ISE Scale
All Personas on a Single Node: PAN, PSN, MnT
ISE Node • Maximum endpoints

(Platform dependent)
PAN
 5,000 for 3415
 7,500 for 3515
MnT  10,000 for 3495
 20,000 for 3595
PSN
PXG
Cisco Commerce Workspace tool

Cisco ISE Scale
• Maximum endpoints – 20,000 (platform dependent—same as standalone)
• Redundant sizing – 20,000 (platform dependent—same as standalone)
ISE Node ISE Node
Primary PAN PAN Secondary

Admin Admin
Primary MnT MnT Secondary

Monitoring Monitoring
PSN PSN
Primary Secondary
PxGrid PXG PXG PxGrid
Controller Controller
Scaling Strategy for Cisco SD-Access in a site
Summary
Vertical Scaling
• Higher Capacity control plane node can provide higher number of end points in a fabric
site.
• Corresponding higher capacity WLC controller needs to be used if wireless end points are
present in the fabric.
• Higher core Cisco DNAC appliance can provide higher scale for automation and assurance
in fabric.
• Larger ISE appliance/VM can provide policy and authentication for higher number of end
points in a fabric site.
Deploying Cisco
SD-Access in
small
branches/sites
Strategy for Cisco SD-Access in a small site
Design for a very small site
Cisco DNAC
DC 1 NCP + NDP
Cluster
ISP
ISE
1 PAN + PXG
+ PSN Internet
DDI
1 DHCP + DNS
+ IPAM IP Reduces cost to deploy
SDA for “mini” sites
Site
CP EB
FE + FB + CP + wireless on C9K
FE WLC
Can be a stack of C9300

FABRIC IN A BOX
Catalyst Supported SKU
9300 All
9500 12Q, 24Q, 16X, 24X, 40X, 48X
Collocated Border and Control Plane Fabric in a Box with Wireless

Embedded Wireless Embedded Wireless
“C9k Switch” “C9k Switch”
Border + CP Border + CP + Fabric Edge
Distributed
Enterprise
Campus
Branch
c c
Fabric Edge
c WAN Transit
SD-Access and IP Transit
Remote Site1 Remote Site 2 Remote Site N
Key Decision Points

B E C
Site B1 B E C Site B2
B E C
Site BN
• Multiple fabric roles collapsed
in a single device.
• Easy to roll out in small sites.
• Embedded Wireless supports

WAN/ up to 100 APs.
MAN
• Fabric in a box supports up to
DNAC
DC 5-7 NCP +
NDP
Cluster
ISE
4000 end points.

2 PAN 2 PXG
5-10 PSN
DDI
1 DHCP 1
DNS
1 IPAM
AB AB EB EB
Site HQ
CP CP
HQ Campus
Scaling the fabric
across multiple
sites
Scaling Strategy across Multiple Sites
General Scaling Strategy for Network Designs
Basic Goal is for fewer, larger Fabric Sites Some Needs require split into Multiple Sites
S S
Large Transit
Medium M
Small L
S
S
Higher scale due to more number of sites

(Control plane per site)
Wireless Client Roaming (< 20ms Latency)
Direct Internet Access (@ Remote Sites)
Survivable Remote Sites (Local CP/Borders) © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Why multiple sites ?
Advantages:
 Smaller or isolated Failure Domains
 Helps scale number of Endpoints
 Cisco DNA Center provides Automation and Single View of entire system
 VNs and SGTs gets pushed to all sites (consistent policy)
 Local breakout at each Site for Direct Internet Access (DIA)
Multi Site Design
Transit
C C
B B B B
Fabric Fabric
Site 1 Site 2
Scaling Strategy for Fabric across Multiple Sites
Control Plane Scale
Transit B
B
B B WAN/Metro
B
C C C C
• Each site has its own Control Plane Node

C
• This will help scale the number of end points in the network CP
Control Plane nodes scale
ASR1K or
Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Catalyst Nexus
Scale ISR4K CSR1KV
3850(XS) 9300 9400 9500 9500H 9600 6800 N7700
Control-Plane Not 200K (16GB)

3K 16K 80K (XL) 80K 150K 150K 50K 200K
(LISP) Entries Supported 100K (8GB)
Transit Connectivity
Why IP Based Transit?
Cloud
Data Center • Customers already using existing WAN
or have adopted SD-WAN
• Unable to carry VXLAN header in WAN
• Higher latencies because sites are in
LTE
different regions
MPLS INTERNET HQ Typical use cases

o Internet Handoff
o P2P IPSEC encryption
Remote Branch 1
o Policy Based Routing
o WAN Accelerators
Remote Branch 2 Remote Branch 3
o Traffic engineering
o Mobile Backhaul LTE
Design for a multi site with IP Transit
Remote Branch 1 Remote Branch 2 Remote Branch N Key Decision Points
Site BN
• Tends to be many remote
Site B1 Site B2
B E C
branch offices connected
via traditional IP WAN/MPLS or
SD-WAN
• Requires direct Internet

access
IP
Transit WAN
• Requires site-to-site
encryption
• Requires traffic engineering

and policy-based routing
Site HQ
• 2 Control Plane Nodes
HQ Campus • 2-4 Borders (Multiple Exits)
Transit Connectivity
Why SD-Access Transit?
Cloud
Data Center • Customers have multiple sites connect
via “Dark Fiber” links or DWDM links
• WAN can transport VXLAN header
• Sites are in same Metropolitan area
Metro
HQ
Typical use cases
Metro Metro
o Consistent policy and end-to-end
segmentation using VRFs and SGTs
Campus 1 o Smaller and Isolated fault domains
Campus 2 Campus 3 o Resiliency and Scalability
Design for a multi site with Cisco SD-Access Transit
Remote Building 1 Remote Building 2 Remote Building N Key Decision Points
• Tends to be like a Metro area
Site B1 Site B2 Site BN with multiple buildings or sites
• Requires direct Internet

access at multiple sites
• Requires local resiliency

SDA
MAN and smaller fault domains
Transit
T T
DNAC
• 2 Transit CP
DC 5-7 NCP +
NDP
Cluster
ISE
• 2-4 Site Borders

2 PAN 2 PXG
5-10 PSN
DDI
(Multiple Exits)
1 DHCP 1
DNS
1 IPAM
AB AB EB EB
Site HQ
CP CP
HQ Campus
Would you like to know more?
Check out the following session:
BRKCRS-2815
Cisco SD-Access – Connecting Multiple Sites in a
Single Fabric Domain
This session covers:
• How multiple Fabrics communicate
• Various Multi-Site design approaches
Wireless Controller Scale
Transit B
B
B B WAN/Metro
B
C C C C
• Each site has a WLC associated with its Control Plane

• This will help scale the number of end points in the network WLC
Wireless Controller Scale
Number of end
Platform Number of AP’s SDA Design
points
3504 150 3000 Small
5520 1500 20,000 Small or Medium
8540 6000 40,000 Medium or Large
Catalyst 9800L 250 5000 Small
Catalyst 9800-40 2000 32000 Medium
Catalyst 9800-80 6000 64000 Large
Catalyst 9800-CL 1000/3000/6000 10000/32000/64000 Small/Medium/Large
PSN PSN
Fabric Fabric Fabric Fabric

Site1 Site2 Site4 Site5
Transit Transit
Fabric Fabric
Site3 Site6
Site 1 to 3 Site 4 to 6 Site N
• Respective Cisco DNA Center cluster

PSN
manages a set of given fabric sites.
• Each Cisco DNA Center cluster integrates

with its Respective Cisco ISE Nodes for
policy.
Fabric Fabric
Site2
Site1 Transit • Policy needs to maintained across the
different Cisco DNA Center and ISE Clusters.
Fabric
Site3
PSN PSN
PSN PSN
PSN
PSN
PSN
PSN
PSN
Cisco ISE Scale
Admin + MnT on Same Appliance; Policy Service on Dedicated Appliance
Distributed Personas : PAN

MnT
PAN
MnT
• 2 x Admin+Monitor
• Max 5 PSNs
PSN
• Max endpoints – Platform dependent
PSN
PSN
PSN
PSN
Cisco ISE Scale
Standalone ISE + HA Distributed ISE + HA
Admin (P) Admin (S) Policy Services
MnT (P) MnT (S) Cluster
Admin (P) PXG PXG
Admin
MnT (P) (S) PSN PSN PSN
PSN MnT (S)
PSN
AD / LDAP AD / LDAP
(External ID or (External ID or
Attribute Store) Attribute Store)
ASA VPN ASA VPN

w/ CoA w/ CoA
Small Site Large Site
WLC WLC
802.1X 802.1X
Switch Switch
802.1X 802.1X
AP AP Small Site
AP
Small Site
WLC WLC
Switch
802.1X 802.1X
802.1X
Switch AP
802.1X
AP
Cisco ISE Scale
Admin (P) Admin (S) Policy Services
MnT (P) MnT (S) Cluster Distributed
Policy Services
PSN PSN
PSN PSN
HA Inline AD/LDAP
Posture Nodes (External ID/ AD/LDAP
Attribute Store) (External ID/
IPN
Data DC B Attribute Store)
IPN
Center A
WLC
Non-CoA 802.1X
ASA VPN
Switch
802.1X AP
WLC
802.1X Switch
AP 802.1X •Dedicated Management Appliances
•Primary Admin / Secondary MnT
Site B
•Primary MnT / Secondary Admin
Site A
•Dedicated Policy Service Nodes—Up to 5 PSNs
Switch Switch
802.1X 802.1X
AP AP
Cisco ISE Scale
• 2 x Admin and 2 x Monitoring

• Max PSNs (Platform dependent … 50 Max)
PAN PAN
• Max endpoints (Platform dependent … 500K Max)
MnT MnT
Dedicated Appliance for Each Persona: Administration, Monitoring, Policy Service
PSN PSN PSN PSN PSN PSN PSN PSN
Cisco ISE Scale
• Only two PSN’s are allowed per Fabric device in Cisco SD-Access.
• Policy Service nodes can be configured in a cluster behind a load balancer (LB).
• Access Devices send RADIUS and TACACS+ AAA requests to LB virtual IP.
PSNs (RADIUS
PSN PSN PSN PSN PSN PSN PSN PSN PSN Servers)
Load
Virtual IP Balancers
Fabric
Devices
Cisco ISE Scale
DC 1 DC 2
PAN PAN
MnT PXG MnT PXG
PSN PSN PSN PSN PSN PSN C
B
Load
Balancers Metro
B B B
C
B
C C C C
• PSN’s are behind a dedicated Load Balancer

• Cisco DNA Center site settings point to Load Balancer IP
Scaling Strategy for Across Multiple Sites
Summary
Horizontal Scaling
• Use Multi-Site design to Scale the fabric infrastructure
• Use multiple clusters of Cisco DNA Center appliances to achieve higher automation and
assurance scale for fabric.
• Use multiple ISE PSN nodes and with load balancing can achieve higher scale for authentications
and policy.
High Level Design
for a Cisco SD-
Access across
Geographies
High Level Design Overview
Across Geographies
• Deploy Fabric Domains per Geo Locations like US , Europe , Asia Pac etc.
• An Individual fabric domain consists of one or more sites
• A site is a fabric on its own with its own control plane and border nodes
• Different sites in Geo regions will be connected using Cisco SD-Access multi-site ( SDA
and / or IP Transit)
Across Geographies
SD-Access
sites Australia
SD-Access
India Site
(Bangalore and
SD-WAN(Viptela)
Chennai)
SD-Access SD-Access
Sites USA Sites X
Across Geographies
Hyderabad
Transit
Site 3
Bangalore
Hyderabad
SD-WAN
and Internet
Conclusion
Session Summary
Cisco DNA Center B B

Simple Workflows
C
DESIGN PROVISION POLICY ASSURANCE
Cisco SD-
Access
Fabric
SD-Access Resources
cs.co/sda-resources
cs.co/sda-community
• Search from your Browser

• Indexed by Search Engines
• Discuss with experts & friends
• Supported by SDA TMEs
• 24-hour First Response
• Questions are marked Answered
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco SD-Access Resources
cisco.com/go/dna
cisco.com/go/sdaccess cisco.com/go/dnacenter
cisco.com/go/cvd
• SD-Access At-A-Glance • Cisco DNA Center At-A-Glance
• SD-Access Ordering Guide • Cisco DNA ROI Calculator
• SD-Access Solution Data Sheet • Cisco DNA Center Data Sheet
• SD-Access Solution White Paper • SD-Access Design Guide • Cisco DNA Center 'How To' Video Resources
• SD-Access Deployment Guide
• SD-Access Segmentation Guide
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the
Walk-In Labs
Cisco Showcase
Meet the Engineer

Related sessions
1:1 meetings
Thank you

BRKCRS 2825

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKCRS 2825

Uploaded by

Copyright:

Available Formats

Cisco SD-Access

Scaling the fabric to multiple sites

• Cisco SDA Network Design and Partner Enablement

• This session assumes that there is a basic understanding

• To enable the audience to imagine a fully functional

Campus  Fabric Edge Nodes – A Fabric device

Control-Plane Node runs a Host Tracking Database to map location information

• A simple Host Database that maps Endpoint IDs to C

• Receives Endpoint ID map registrations from Edge

• Resolves lookup requests from Edge and/or Border

Control-Plane Node runs a Host Tracking Database to map location information

• Each fabric site can support up to six control plane C

• For a wired only network we can support a B B

• All the control planes nodes in a given state work in

• Responsible for Identifying and Authenticating C

• Provide an Anycast L3 Gateway for the connected

• Performs encapsulation / de-encapsulation of data

There are 3 Types of Border Node! C

• Outside World/External Border Used for

• Anywhere/External + Internal Border Used

Rest of Company/Internal Border advertises Endpoints to outside, and known

• Connects to any “known” IP subnets available from C

• Imports and registers (known) IP subnets from

• Hand-off requires mapping the context (VRF & SGT)

Outside World/External Border is a “Gateway of Last Resort” for any unknown

• Connects to any “unknown” IP subnets, outside of C

• Does NOT import any routes! It is a “default” exit, if

• Hand-off requires mapping the context (VRF & SGT)

• Connects to any “unknown” IP subnets, outside of

• Imports and registers (known) IP subnets from

• Exports all internal IP Pools outside (as aggregate)

• The above two borders are cumulative in a given

• Each fabric site can support hundreds (up to fabric site

Vertical Scaling Horizontal Scaling

Vertical Scaling Horizontal Scaling

• Using multiple ISE nodes and with load

Vertical Scaling Horizontal Scaling

• No Synchronization between CP nodes

LISP Map Server: 5.2.2.2

2.1.1.1 2.1.2.1 3.1.1.1 3.1.2.1

Edge Edge Edge Edge

< 100 IP Pools/Groups

• Looking at <1000 dynamic

• Can choose a Co-located or a

• Tends to be WLC + SD-

• Looking at < 10,000 dynamic

• Tends to be Many Buildings

• Can choose a Co-located or a

• Looking at < 25,000 dynamic

Vertical Scaling Horizontal Scaling

Fabric Control Plane CRN®

Catalyst 9300 Catalyst 9400 Catalyst 9500 Catalyst 9600

• Catalyst 9300 • Catalyst 9400 • Catalyst 9500 • Catalyst 9600

Fabric Control Plane

Catalyst 3K Catalyst 6K ISR 4K & ENCS ASR1K

• Catalyst 3650/3850 • Catalyst 6500/6800 • ISR 4430/4450 • ASR 1000-X

* 9200 = 1 DEFAULT_VN + 1 INFRA_VN + 3 User-Configured VNs

Fabric Enabled Wireless

AireOS WLC Catalyst 9800 Wifi 6, 11ac Wave 2 Wave 1*AP

• AIR-CT3504 • Catalyst 9800-40/80 • Catalyst 9100 • AIR-CAP1700, 2700

Stateful redundancy with WLC SSO

5520 1500 20,000 Small or Medium