BRKSPG 3489

#CiscoLiveLA
Case study of SP customers

running ACI based SDN for
telecom datacenter
Abhishek Mande – Principle SE, APJ
Sonu Khandelwal – TME, DC
BRKSPG-3489
#CiscoLiveLA
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRSPG-3489

by the speaker until December 10, 2018.
#CiscoLiveLA BRKSPG-3489 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Evolution of Telco Datacenter

• Customer Profiles & Technical Requirements
• How ACI solve the challenge
• Looking Ahead
• Conclusion
Session Objective
• This is a ”Case Study” session, highlighting how large

Telecom Service Providers used ACI to meet their
requirements.
Initial assumption:
The audience already has knowledge of ACI concepts
(Tenant, VRF, BD, EPG, L3Out, etc.)
Agenda

• Looking Ahead
• Conclusion
#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

© 2018 and/or its affiliates.
and/or All rights
its affiliates. reserved.
All rights reserved. Cisco Public
Cisco Public 6
Landscape of a Telco Datacenter
• Mobility: User plane, Gi-LAN, • Mobility: IMS, vEPC,

• SD-WAN
Caching vPCRF, Gi-LAN, Caching
• Wireline: BNG, vBNG • Wireline: BNG, vBNG Control Plane
• Mobility:
BBU,VBBU
• CUPS, Caching,
Gi-LAN, Iot
Public Cloud
Provider
C-RAN Hub CO/Agg/MSO/HE Regional DC Central DC

/ Pre-Agg.
Cell Site
100 – 1000
40 – 100
Sites 10< Sites
Sites
Agenda

• Applications
• Datacenter Fabric

• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 8
Customer Profiles
ü 9 Top operators in APJ

ü Largest deployment of 40 fabrics serving
300mn+ subscribers
ü Smallest deployment of 2 fabrics
ü Fabric for 3G, 4G services (Physical & Virtual)
ü Mix of Cisco and Non-Cisco 3G, 4G services
ü Readiness of 5G in some developed countries
Telco Applications Requirement
IMS SGW & PGW Gi-LAN CDN & OTT Caching

• TCP Optimizers
• Distributed userplane • Caching managed by
• Deep Packet
• Faster Convergence and control plane Inspection (DPI) OTT Provider
• Consistent low latency • Virtualized deployment • Self created Media
• CG-NAT
• Multiple Hypervisor across multiple racks • URL Filtering Content
• IPv4 & IPv6 connectivity • Failover & Redundancy • Service Chaining with • Driving 25G & 100G
• Routing with fabric interfaces
multiple nodes
Charging Gateway PCRF Voice over Wifi DNS, AAA, DHCP
• Faster Convergence
• IPv4 & IPv6 connectivity • IPv4 & IPv6 connectivity
• Consistent low latency • IPv4 & IPv6 connectivity
• Multiple hypervisor
• Multiple hypervisor
High Level Network Architecture
Distributed Telco DC
IT DC
PGW, SGW, MME,WLC,ISG
Other LTE /Wi-Fi backend systems OSS & BSS, CDN
EPC IMS CDN
CSR CSR
CSR DC-WAN
Aggregation
CSR
(IP/MPLS) Domestic
IBR
Pre-Aggregation Peering
WiFi MAP WiFi RAP (IP/MPLS)
Core & Super Core
(IP/MPLS)
WiFi MAP
IGW
Internet
CSR
WiFi RAP
OLT OLT
OLT
Residential
CPE FTTx
OLT OLT
Enterprise
Customer /
STB Med ISP
Datacenter Fabric Requirement
• Fabric Automation • North bound API

• Distributed DC • Simplified Operation
Fabric Operations • Troubleshooting
• Security & segmentation
Software
Defined
Network
(SDN)
• Consistent Policy
• Service Chaining
• Multi-hypervisor Connectivity Availability • Faster Convergence
• Carrier Grade Availability
Datacenter Fabric Protocol & Scale Requirement
Physical Fabric • 20-150 Leaf Per Physical Fabric
• Multi-Speed Interfaces on Same Leaf & Spine
Traffic Throughput • 1.5-5 Tbps
• Linerate
Protocol • BGPv4, BGPv6, Static route
• BFD
• IPv6 Multicast
• SNMP
• NTP
• TACACS
• Multi-Node PBR, PBR tracking, Symmetric load-balancing, resilient hashing
Carrier Grade Scale • 500 Vlans
• 50 VRFs
• 1K Mac, ARP & ND per switch
• 10-50K end hosts
• 10-20K IPv4 & IPv6 LPM addresses
Agenda

• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 14
Case-Study#1
Application to Fabric Connectivity
Deployment with physical SGW & PGW SAE GW
(SGW & PGW)
SAE GW
(SGW & PGW)
DNS-1 DNS-N OSS & BSS, NTP, Syslog,

TACACS+ DPI (L1 Device)
Spine Out of Band
N9508 Network
N9732C-EX Service Leaf
N93180YC-EX
40G/100G Links
Service Leaf
Server Leaf Service Leaf Service Leaf N93180YC-EX Border Leaf
N93180YC-EX N93180YC-EX N93180YC-EX N93180LC-EX N93180LC-EX
TCP TCP
Optimizer-1 Optimizer-N
Active LB Standby LB
CG-NAT ASR9K
IMS, MME, PCRF, Voice over
wifi, OTT Applications and Service
Active/Active
Other Telco Appliances Firewall Cluster Internet
IP/MPLS
Evolution of Case Study#1
Application to Fabric Connectivity 2nd stage
Next-gen switches in newer fabrics SAE GW

(SGW & PGW)
SAE GW
(SGW & PGW)
DNS-1 DNS-N OSS & BSS, NTP, Syslog,

TACACS+ DPI (L1 Device)
Spine Out of Band
N9508 Network
N97336-FX Service Leaf
N93180YC-FX
40G/100G Links
Service Leaf
Server Leaf Service Leaf Service Leaf N93180YC-EX Border Leaf
N93180YC-FX N93180YC-FX N93180YC-FX N9336C-FX2 N9336C-FX2
TCP TCP
Active LB Standby LB
CG-NAT ASR9K
wifi, OTT Applications and Service
Active/Active
Other Telco Appliances Firewall Cluster Internet
IP/MPLS
Evolution of Case Study#1
Application to Fabric Connectivity 3rd stage
vEPC deployment with separate Gi-LAN Fabric

• Separated ACI for L4-L7 Services.
• Isolated change domain
• Gi-LAN fabric for wireless and wireline customers
vEPC Fabric Gi-LAN Fabric
ASR9K
ASR9K
TCP Optimizers, CG-NAT, DPI
wifi, OTT Applications and
Other Telco Appliances Internet
IP/MPLS
Case-Study#2
vEPC deployment with Cisco Ultra EPC and Cisco VIM
IP/MPLS Internet
ASR9K
Spine
N9364C
Leaf Leaf Leaf Leaf

N93180YC-FX N93180YC-FX N93180YC-FX N93180YC-FX
TCP TCP
vEPC control functions (Control plane)
PGW, SGW, VoLTE, PCRF and

vEPC service functions (data plane) other telco services running as
virtual instances on Cisco VIM
Case-Study#3

5G Central DC architecture
Spine
N9508
N97336-FX
Leaf Leaf Leaf Border Leaf Border Leaf

N93180YC-FX N93180YC-FX N93180YC-FX N9336C-FX2 N9336C-FX2
Small Cell vEPC OSS/ BSS IMS

Controller 20 Rack 2 Rack 23 Rack NCS5508 NCS5508
2 Rack Cisco VIM (Bare-Metal) Cisco VIM
Cisco VIM
IP/MPLS Internet
Agenda
• Automation
• Services Connectivity
• Migration & Operational Simplification
• OSS & BSS Integration
• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 20
Fully Automated Provisioning of ACI Fabric
75% reduction in time spent bringing up network
• Fabric Provisioning
• Validation
• Inventory
Input simple
details like Fabric ✔
Connect APICs Subnet , APIC
Connect switches to Leaf Pair Login to APIC
Out of Band Fabric is up and
in Spine Leaf and register
Power on APIC Management IP running
topology switches
and switches & Login
credential on
APIC CIMC
Topology View
Automatic Policy deployment
Gateway- 100.1.1.1
Automatic deployment of Tenant, VRF, EPG, BD

(Gateway IP) when End Point is detected
100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1
100.1.1.100 100.1.1.101 100.1.1.102

Vlan 100 Vlan 100 Vlan 100
Host A Host B Host C
23
#CiscoLiveLA BRKSPG-3489 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• Automation
• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 24
Evolved Packet Core (EPC)
deployment
#CiscoLiveLA
SAE GW Connectivity to ACI Leaf
Advertise subscriber pool through static route
1.1.1.0/8 -> SAE GW-1 IPv4 address
2000::1/64-> SAE GW-1 IPv6 address
BGP RR 2.2.2.0/8 -> SAE GW-2 IPv4 address

2001::1/64-> SAE GW-2 IPv6 address
BGP in Fabric
eBGP connection from ACI 1.1.1.0/8

BorderLeaf to ASR9K 2000::/64
2.2.2.0/8
ASR9K 2001::/64
CG-NAT
Active Standby Standby Active Service
SAE GW-1 SAE GW-2
(SGW & PGW) (SGW & PGW)
IP/MPLS Internet
1.1.1.0/8 & 2000::/64 2.2.2.0/8 & 2001::/64
Subscriber Pool Subscriber Pool
SAE GW Connectivity to ACI Leaf
Handling Failure
BGP RR Static route over SVI should be

removed when both active &
standby link goes down
BGP in Fabric
No BFD
No Dynamic eBGP connection from ACI

Routing Support VLAN100 VLAN101 BorderLeaf to ASR9K 1.1.1.0/8
100.1.1.1.1 2000::/64
100.1.1.1.1
ASR9K 2.2.2.0/8
2001::/64
Active Standby Standby Active
SAE GW-1 SAE GW-2
IP/MPLS Internet
1.1.1.0/8 & 2000::/64 2.2.2.0/8 & 2001::/64
Subscriber Pool Subscriber Pool
Handling Failure of SAE GW
Delivered through APP in two weeks
cTrac
• Static route over SVI should be removed when all
interfaces in SVI goes down
• APP Infrastructure is available for any customer or
partner to develop Apps
• Multiple Free Apps including cTrac for static route
monitoring available at https://aciappcenter.cisco.com/
Visually monitor externally
• Easy way to integrate with Eco system partners routed interface states
And next hop add/delete
Monitoring and
Feature also delivered in ACI 3.1(1) Troubleshooting
Virtual EPC Deployment
#CiscoLiveLA
vEPC deployment with ACI
Ability to support multi-rack VNF cluster deployment
Flexible route peering

VNID
Resiliency
VNF-1 VNF-2 VNF-63 VNF-64
On demand expansion of VNFs anywhere in Fabric

Rack-1 Rack-n
Evenly traffic distribution among VMs with multi-rack cluster
Distributed vEPC design with ACI
• PGW functionality is divided across service

functions (SF) and control functions (CF).
• CF advertises subscriber pool information through

BGP with the common next-hop of user-plane VNFs
VNID
• All the use-plane VNFs are sharing a common IP
• ACI leaf switches are configured with static route or

dynamic routing protocol for provide reachability to SF-1 SF-2 SF-64
common IP of SF
Common IP for SFs CF-1
• 64-way ECMP from ACI Leaf to SF 2.2.2.2 Subscriber pool
1.0.0.0/8
• BFD is used between SF and ACI Leaf for resiliency 2000::/64
Rack-1 Rack-n
SF= Service Function (Data Plane) BGP with Control VM
CF= Control Function (Control Plane) Static route or dynamic
routing protocol with BFD
Gi-LAN services
#CiscoLiveLA
Integration with TCP
Optimizers
#CiscoLiveLA
Flow 1
TCP Optimizer Integration with ACI Flow 2
Automatic Load-balancing and Symmetry of traffic flow
SAE GW SAE GW ASR9K

(SGW & PGW) (SGW & PGW) TCP TCP TCP TCP
OPT 1 OPT 2 OPT 3 OPT N
Subscriber Pool
N-TCP Optimizers in a group Internet
ü Symmetric PBR ensure return traffic choses same TCP optimize
ü Automatic load-balancing of traffic across different TCP optimizers based on forwarding table hash (Source
IP, Destination IP, Source Port, Destination Port)
TCP Optimizer Integration with ACI
Simplified Configuration
PGW EPG Contract (PBR) Internet EPG

Classified based on TCP = Any or Classified based on
subscriber Pool UDP =443 Internet Prefixes
NO
YES
SAE GW SAE GW ASR9K

Subscriber Pool
Tracking TCP Optimizer Liveliness
Automatic Load-Balancing to remaining

ICMP & TCP Tracking of TCP Optimizers after failure
Inside & Outside Interface
Inside Outside
SAE GW SAE GW 1.1.1.1 2.2.2.1 ASR9K
Subscriber Pool

Removes whole TCP Optimizer if either
Inside or Outside interface goes down
Bypassing TCP Optimizers to avoid congestion
Traffic is directly send to internet

when more than defined number
of TCP Optimizers fails
Inside Outside
SAE GW SAE GW 1.1.1.1 2.2.2.1 ASR9K
Subscriber Pool
Flow 1
Non-Resilient Hashing
All flows get re-hashed on a PBR node failure, this can cause traffic drop for flows that lands on a PBR node
that does not have a session information.
TCP OPT2 flows

shifted to TCP OPT3
SAE GW SAE GW ASR9K

Subscriber Pool
38
Flow 2
Resilient Hashing
Only the flows that were going through failed node gets re-hashed.
SAE GW SAE GW ASR9K

Subscriber Pool
39
BRKSPG-3489
#CiscoLiveLA © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Simplified expansion
New TCP optimizers can be added anywhere in fabric
SAE GW SAE GW
(SGW & PGW) (SGW & PGW) TCP TCP TCP TCP TCP ASR9K
OPT 1 OPT 2 OPT 3 OPT 4 OPT 31
Subscriber Pool
N-TCP Optimizers in a group
Internet
BRKSPG-3489 40
Multi-Node Service
Chaining
#CiscoLiveLA
Multi-Node Service chaining in Telco DC
SAE GW SAE GW
(SGW & PGW) (SGW & PGW) ASR9K
Group of TCP CG-NAT Deep Packet
Subscriber Pool
Optimizers Inspection (DPI)
Gi-LAN
Internet
BRKSPG-3489 42
Logical ACI Construct for service chaining
PGW EPG Contract

Internet EPG
Classified based Service Graph Template Classified based

on subscriber on Internet
Pool prefixes
10.0.0.0/8 Group of TCP CG-NAT DPI

2000::/8 Optimizers
Simplified Configuration
WAN Connectivity
#CLUS
Option#1
WAN Connectivity from ACI Fabric from Border Leaf
Multiple Links to reach to

Multiple BorderLeaf for high
BorderLeaf
speed internet connectivity
eBGP connection from ACI

BorderLeaf to ASR9K
SAE GW SAE GW
ASR9K
IP/MPLS Internet
#CLUS BRKSPG-3489 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Option#2
External Connectivity from ACI Fabric from Spine
IP/MPLS Internet
ü Traffic forwarding from Spine
ü Supported with ASR9000,
Gi-LAN VRF ASR9K Subscriber VRF Nexus7000 and ASR1000
IMS VRF Internet VRF ü Automatic VRF creation on DC
Edge Router
VXLAN EVPN
Subscriber VRF Subscriber VRF

Internet VRF Internet VRF IMS VRF Gi-LAN VRF
TCP TCP
PGW, SGW, VoLTE, PCRF and other telco services running as virtual
instances or on Bare-Metal
#CLUS BRKSPG-3489 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Agenda
• Automation
• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 48
Migration
#CiscoLiveLA
Migration Process
7. Troubleshooting
1. Collected 3. Created simple python 5. Uploaded XML using ACI Operations
Existing scripts to develop XML configuration using tool
configuration POSTMAN tool
based ACI configuration
2. Mapped Existing 4. Built ACI Fabric 6. Migrated 2500+

Configuration to ACI Physical ports, 30
Constructs such as Tenant, VRF, 300+ Vlans,
VRF, EPG, BD, L3out etc. 200+ Static Routes,
50+ BGP neighbors
Any Operating Model
Migration Process
POST Using POSTMAN CLI GUI API
Automation for ACI Fabric in Production
Self developed Provisioning tool for pushing Configuration
Provisioning Tool
API Calls to
provisioning Tenant,
VRF, BD, EPG etc.
Operations
Simplification
#CiscoLiveLA
Operations Tools
Topology Dashboard Troubleshooting Wizard End Point Tracker
Faults Capacity DashBoard Link Statistics
Health Score Card Traffic Map Upgrade/Downgrade
Troubleshooting with Health Score Card
Drill Down from Dashboard
Troubleshooting with Health Score Card
Capacity Dashboard
Easy Upgrade Process for whole Fabric
Simple Return Merchandise Authorization (RMA)
Decommission the old switch by removing the controller
Register the new switch by provide same Node ID and Node Name
Old Leaf New Leaf
Agenda
• Automation
• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 64
ACI Fabric Convergence
Controller Failure - No Loss
Fabric Failure – 200 msec
Convergence happens within ASIC
Access Failure - within 200 msec

External Connectivity Failure - within 200 msec
ACI Fabric to vPC connected host
failure ACI Fabric to external connectivity failure
ASR9K
Active/Active
Server
IP/MPLS Internet
BRKSPG-3489
#CiscoLiveLA © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Agenda
• Automation
• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 66
OSS & BSS Integration
Fault Management Fabric 3
Fabric 1
OOB Network Tool that captures traps

from all the fabric
Fabric 2
Telecom Network
Fabric 4 Management platform
based on SNMP
ACI Integration with Splunk
Dashboard
Splunk APP
Syslog
for ACI
APIC SDK
https://splunkbase.splunk.com/app/1896/
https://splunkbase.splunk.com/app/1897/
Agenda

• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 69
ACI Multi-Pod
Single Telco DC Campus with multiple server halls
IP Network
Pod ‘A’ Pod ‘n’
MP-BGP - EVPN
Availability Zone
§ Managed by a single APIC Cluster § End-to-end policy enforcement

§ Single Management and Policy Domain § Control plane fault isolation
Management of Multiple Sites with ACI Multi-Site Solution
IP Network
Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
§ Separate ACI Fabrics with independent APIC clusters

§ ACI Multi-Site pushes cross-fabric configuration to multiple APIC clusters providing scoping of all
configuration changes
§ End-to-end policy definition and enforcement
71
Architecture for Distributed Datacenters
ACI Remote Physical Leaf
Remote Location contains Nexus 9300
connected to IP Network and fully
managed by APIC cluster of Main DC
IP Network
Bare-
Main Datacenter vSwitch
Hypervisor Metal
Remote Datacenter
All local traffic is switched directly between
endpoints, both virtual and bare metal
Agenda

• Looking Ahead
• Conclusion

and/or All rights
Cisco Public 73
Conclusion
• Automation
• Massive Scale
• Distributed DC
• Time to Market
Requirement • Scale
• Operations Solution
ACI • Simplified Operation
• Service chaining
• Integration with Tools
Looking Ahead
Consistent Policy & Management across

Geography
Network Insight & Assurance
Complete your online session evaluation
Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app
Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at www.CiscoLive.com/Online.
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings
Thank you
#CiscoLiveLA
#CiscoLiveLA

BRKSPG 3489

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKSPG 3489

Uploaded by

Copyright:

Available Formats

#CiscoLiveLA

Case study of SP customers

Webex Teams will be moderated cs.co/ciscolivebot#BRSPG-3489

• Evolution of Telco Datacenter

• This is a ”Case Study” session, highlighting how large

• Evolution of Telco Datacenter

#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

• Mobility: User plane, Gi-LAN, • Mobility: IMS, vEPC,

C-RAN Hub CO/Agg/MSO/HE Regional DC Central DC

• Evolution of Telco Datacenter

• How ACI solve the challenge

#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

ü 9 Top operators in APJ

IMS SGW & PGW Gi-LAN CDN & OTT Caching

Charging Gateway PCRF Voice over Wifi DNS, AAA, DHCP

• Fabric Automation • North bound API

• Evolution of Telco Datacenter

#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

DNS-1 DNS-N OSS & BSS, NTP, Syslog,

Next-gen switches in newer fabrics SAE GW

DNS-1 DNS-N OSS & BSS, NTP, Syslog,

vEPC deployment with separate Gi-LAN Fabric

Leaf Leaf Leaf Leaf

vEPC control functions (Control plane)

PGW, SGW, VoLTE, PCRF and

Application to Fabric Connectivity

Leaf Leaf Leaf Border Leaf Border Leaf

Small Cell vEPC OSS/ BSS IMS

#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

Automatic deployment of Tenant, VRF, EPG, BD

100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1

100.1.1.100 100.1.1.101 100.1.1.102

#CiscoLiveLA BRKSPG-3489 © 2018 CiscoCisco

BGP RR 2.2.2.0/8 -> SAE GW-2 IPv4 address

eBGP connection from ACI 1.1.1.0/8

BGP RR Static route over SVI should be

No Dynamic eBGP connection from ACI

Ability to support multi-rack VNF cluster deployment

Flexible route peering

On demand expansion of VNFs anywhere in Fabric

Evenly traffic distribution among VMs with multi-rack cluster

• PGW functionality is divided across service

• CF advertises subscriber pool information through

• All the use-plane VNFs are sharing a common IP

• ACI leaf switches are configured with static route or

SAE GW SAE GW ASR9K

PGW EPG Contract (PBR) Internet EPG

SAE GW SAE GW ASR9K

N-TCP Optimizers in a group Internet

Automatic Load-Balancing to remaining

N-TCP Optimizers in a group Internet

Traffic is directly send to internet

N-TCP Optimizers in a group Internet

TCP OPT2 flows

SAE GW SAE GW ASR9K

SAE GW SAE GW ASR9K

New TCP optimizers can be added anywhere in fabric

PGW EPG Contract

Classified based Service Graph Template Classified based

10.0.0.0/8 Group of TCP CG-NAT DPI

Multiple Links to reach to

eBGP connection from ACI

Subscriber VRF Subscriber VRF