BRKCLD-3440 (2019)

#CLMEL
Multicloud
Networking – Design
and Deployment
Shannon McFarland – CCIE#5245
Distinguished Engineer
Cloud CTO
@eyepv6
BRKCLD-3440
#CLMEL
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKCLD-3440
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Multicloud Networking
Overview
• Native IPsec VPN Services
• Multicloud with Cisco SD-WAN
• DMVPN
• Automation
• Conclusion
#CLMEL BRKCLD-3440 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Disclaimer
• You won’t learn security, routing, HA, performance best practices
• There are a gazillion ways to accomplish the same thing for ALL of this
• Be smart – Know IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff

• Dead Peer Detection
• IPsec SA lifetimes
• IPsec SA replay window-size
• Perfect Forward Secrecy (PFS)
• BGP timers, Local Preference, MED, inbound soft reset (check if cloud provider supports dynamic inbound soft reset)
• BGP graceful restart - Note: Each cloud provider uses BGP graceful restart with default timers (120 sec) – My configs do not show that due to
slide space but know that it is enabled on each On Premises router
router bgp 65002

bgp log-neighbor-changes
bgp graceful-restart restart-time 120
bgp graceful-restart stalepath-time 360
bgp graceful-restart
• IGP timers, configuration best practices

• HSRP timers, tracking
Multicloud Networking
Overview
Hybrid vs Multicloud Networking
• Hybrid Cloud Networking = Network transport from on-premises to a single public cloud provider
• Multicloud Networking = Network transport from on-premises to multiple public cloud providers and/or between multiple
public cloud providers
• The technologies used can be identical for every connection or they can be per-provider, per-region, per-project, etc..
• Common network transport ingredients for hybrid and multicloud:

• Encryption (IPsec/IKEv2/IKEv2, SSL, PKI)
• Routing (Static, BGP and with supported public cloud-hosted routers: OSPF, EIGRP)
• Tunnelling (IPsec tunnel mode, GRE, mGRE, MPLS, segment routing, etc..)
• Common network endpoint options:

• Native VPN (IPsec over Internet) using public cloud provider services that connect to on-premises router/firewall
• Commercial/Open Source VPN platform hosted on the public cloud provider connecting to an on-premises router/firewall
• Colocation/Direct Peering: Service from public cloud provider to on-premises via a 3rd party colo facility
• Google Cloud Platform Dedicated Internconnect/Direct Peering/Carrier Peering: https://cloud.google.com/interconnect/
• Amazon Web Services Direct Connect/PrivateLink: https://aws.amazon.com/directconnect/
• Microsoft Azure ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
Why Would You Use Multiple Cloud Providers?
• Cloud provider high availability
• M&A may dictate public cloud provider preference (for a time)
• Regional cloud provider access
• Feature disparity between providers, regions and/or services
• Per-project service requirements
Extending On Premises
Private Cloud to a Public
Cloud
Internet Over-the-Top (OTT)
• Enterprise CSP-
users/applications connect Amazon Amazon

published
service
to Cloud Service Provider

ECR S3 endpoints
(CSP) public endpoints Internet Enterprise

Data
Centre
Edge
and/or public IPs of Gateway
AZ: us-west-2b
EIP
Internet
NAT GW 2 172.16.1.0/24
applications
Public Subnet 2
VPC
Campus
pod Router
172.16.4.0/24 Enterprise
No ‘traditional’ IPsec VPN

Enterprise
Application Private Subnet 2 Site
•
Region us-west-2
• TLS/SSL capable
• Can be at odds with
Enterprise InfoSec policies
Cloud Service Provider - Native IPsec VPN Service
Cisco ASR,
CSR, ISR
Default Network IPsec/IKEv2 Private Network
10.138.0.0/20
Google
BGP
Cloud
VPN
Google Cloud Router
IPsec VPN - Cisco SD-WAN Example
Per-VPC Cisco vEdge
VPC Subnet(s) Private Network(s)
vEdge vEdge
VPC On-Premises
Router Cloud IPsec
Transit VPC: Cisco vEdge + CSP VPN
Transit VPC

IPsec vEdge
VPC VPN vEdge
Cloud
IPsec On-Premises
Router Gateway
Transit VPC: Cisco vEdge + Per-VPC vEdge

Transit VPC

vEdge vEdge vEdge
VPC IPsec On-Premises
Router Cloud Cloud
IPsec VPN - Cisco CSR 1000v Example
Per-VPC Cisco CSR 1000v
CSRs Cisco
VPC ASR/CSR/ISR On-Premises
Router DMVPN/IPsec
Transit VPC: Cisco CSR + CSP VPN
Transit VPC

IPsec CSRs Cisco
VPC VPN DMVPN/IPsec ASR/CSR/ISR On-Premises
Router Gateway
Transit VPC: Cisco CSR + Per-VPC CSR

Transit VPC

CSRs CSRs Cisco
VPC DMVPN/IPsec ASR/CSR/ISR On-Premises
Router
Multicloud with Transit VPC
Transit VPC
VNet Subnet
Azure VPN vEdge

GW
Private Network(s)
Cisco
Transit VPC vEdge
VPC Subnet
On-Premises
vEdge
AWS VPN
GW
SD-WAN
Transit VPC
VPC Subnet
Google vEdge
Cloud VPN
AWS – Transit Gateway (TGW)
Dev Prod Dev Prod Dev Prod
This replaces this

Dev Prod Dev Prod Dev Prod
Transit Gateway
AWS Direct
VPN
Connect
WAN
Transit VPC Transit VPC

CSR Cisco ASR/
VPC VPN IPsec CSR/ISR On-Premises
Router Gateway
Colocation - With or Without VPN
Cisco Routers or Firewalls + Some Combo of Colocation/peering
VLANs
DX Cisco
VPC VPN Cisco ASR On-Premises
Endpoint ASR/CSR/ASA
Router Gateway 1000
IPsec
Cisco SD-WAN + Some Combo of Colocation/peering
VLANs
DX vEdge
VPC VPN vEdge On-Premises
Router Gateway Endpoint
IPsec
IPsec
VPN over the Internet vs Direct
Connect/ExpressRoute/Dedicated Interconnect
VPN over the Internet Direct/Express/Dedicated
Throughput Winner
QoS Winner
Latency Winner
Inline Services Winner
Managed Services Winner
Cost Winner
Time to Provision Winner
Flexibility Winner
Location Availability Winner
Cisco Cloud Services Router (CSR) 1000V
Cisco IOS XE Software in a Virtual Appliance Form-Factor
Software
CSR 1000V
• Familiar IOS XE software with ASR1000 and ISR4000
App App Infrastructure Agnostic

• Runs on x86 platforms
OS OS • Supported Hypervisors: VMware ESXi, Linux KVM, Citrix Xen, Microsoft
Hyper-V, Cisco NFVIS and CSP2100
• Supported Cloud Platforms: Amazon AWS, Microsoft Azure, and Google
Virtual Switch Cloud Platform
Hypervisor Performance Elasticity

• Available licenses range from 10 Mbps to 10 Gbps
CPU footprint ranges from 1vCPU to 8vCPU
Server •
License Options
• Term based 1 year, 3 year or 5 year
https://www.youtube.com/playlist?list=PLCi • Smart License enabled
TBLSYkcoTUS6b4MFthdvhDrseo6MeN
Programmability
• NetConf/Yang, RESTConf, Guest Shell and SSH/Telnet
Reference
Public Cloud Provider Native VPN Services

The Big Three
• Google Cloud Platform (GCP):
• VPN: https://cloud.google.com/compute/docs/vpn/overview
• Dedicated Interconnect: https://cloud.google.com/interconnect/
• Amazon Web Services (AWS):

• VPN: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html
• Direct Connect: https://aws.amazon.com/directconnect/
• Microsoft Azure:
• VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/
• ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
• OpenStack public cloud goodness: https://www.openstack.org/passport
Let’s Backup
Starting Simple
Public Cloud Provider Native IPsec VPN Service
IPsec/IKEv2 eBGP<>IGP Redistribution
Tunnel Mode
BGP AS65003
VPC Network Private Network
10.138.0.0/20 Google
172.16.0.0/24
Cloud
VPN
BGP/OSPF/EIGRP
BGP AS65000
On-Premises
Google Cloud Router
Add More On-Premises Stuff On-Premises Tenant 1

CSR1000v
Private Network
192.168.100.0/24
BGP AS65002
BGP/OSPF/EIGRP
VPC Network
10.138.0.0/20 Google
Cloud CSR1000v
VPN
Private Network
BGP AS65000 BGP AS65003 172.16.0.0/24
Google Cloud Router
BGP/OSPF/EIGRP
On-Premises Tenant 2
Routes this side should see:
172.16.0.0/24
192.168.100.0/24 Routes this side should see:
10.138.0.0/20
On-Premises Physical/Virtual
Physical Router
ASR 1000
Private Network
192.168.yyy.0/24
VPC Network
10.138.0.0/20 Google
Cloud
VPN
Private Network
ASA Firewall 172.16.yyy.0/24
Google Cloud Router
Physical Firewall
Add More Public Cloud
Providers to the Mix
Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
Google
Cloud VPN
VPC Network
10.138.0.0/20
Private Network
Google Cloud Router 172.16.0.0/24
BGP/OSPF/EIGRP
VPC Network On Premises Private Cloud
172.31.0.0/16
VPC VPN
Router Gateway
Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
Google
Cloud VPN
VPC Network
10.138.0.0/20
Private Network
Google Cloud Router 172.16.0.0/24
BGP/OSPF/EIGRP
VPC Network On Premises Private Cloud
172.31.0.0/16 As the number of these
VPC VPN connections increase and/or
Router Gateway
change frequently... You can see
where this is going
Moving Away From Native VPN Services
What Conditions Cause a Change in Design?
• If On Premises routers/firewalls are behind NAT – Check for provider
support of NAT-T
• You need to extend your On Premises IGP (OSPF/EIGRP) into the public
cloud
• Operational consistency
• You need different IPsec/IKE configurations than what the provider offers
• You need SSL-based VPNs
• You need MPLS VPN
• QoS, specific network monitoring (IP SLA, NetFlow), Enterprise toolsets for
configuration and monitoring
Options
Cisco SD-WAN
vManage vBond vSmart
VNet Network
10.50.0.0/16
vEdge/cEdge
Private Network
172.16.0.0/24
vEdge/cEdge
VPC Network SD-WAN

On Premises Private Cloud
172.31.0.0/16
vEdge/cEdge
Cisco SD-WAN:
https://www.cisco.com/c/en/us/solutions/en
terprise-networks/sd-wan/index.html
DMVPN – Enable Dynamic Multicloud Networking
Cisco DMVPN - A Brownfield Way to Bolt on Multicloud
VNet Network
Cisco
10.50.0.0/16 Cisco Spoke
CSR1000v
CSR1000v
Hub FHRP Private Network

172.16.0.0/24
Spoke
VPC Network DMVPN
172.31.0.0/16 Cisco
CSR1000v
IGP Support: OSPF, EIGRP, iBGP
QoS Policies
IP SLA, NetFlow Cisco DMVPN:
NAT-T (Transparency) https://www.cisco.com/c/en/us/products
MPLS /security/dynamic-multipoint-vpn-
etc... dmvpn/index.html
A Note On MTU
• All three providers recommend a different size interface MTU for the IPsec tunnel
interface:
• Google recommends 1460 on the tunnel:
https://cloud.google.com/vpn/docs/concepts/advanced#mtu
• AWS recommends 1399 on the tunnel:
https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html
• Azure recommends 1400 on the tunnel: https://docs.microsoft.com/en-us/azure/vpn-
gateway/vpn-gateway-about-vpn-devices
• In addition to MTU, you need to set and test your TCP MSS values
• In my testing, an IP MTU of 1400 and TCP MSS of 1360 worked for all sites but this
may need to change based on your applications and if you are adding other encaps
like MPLS
Google Cloud Platform
– Native VPN
Reference
Google Cloud Platform – VPN Gateway

• GCP Cloud VPN overview
• https://cloud.google.com/vpn/docs/concepts/overview
• GCP Cloud VPN documentation
• https://cloud.google.com/vpn/docs/how-to/creating-vpns
• GCP Advanced VPN documentation
• https://cloud.google.com/vpn/docs/concepts/advanced
Topology for GCP to On Premises CSR – IPsec VPN
BGP Routing
BGP<>OSPF Redistribution
IPsec/IKEv2
Tunnel Mode Cisco
CSR1000v
Default Network 35.xxx.xxx.x 169.254.0.2 .1 Private Network
10.138.0.0/20 192.xxx.xxx.x 192.168.100.0/24
Google
Cloud OSPF 10 Area 0
VPN
169.254.0.1 Hypervisor
BGP AS65000 BGP AS65002
Google Cloud Router

192.168.100.0/24
10.138.0.0/20
gcloud – Create the VPN GW, External IP and Forwarding Rules
Create a VPN gateway
# gcloud compute target-vpn-gateways create csr-gcp-vm-gw --region us-west1 --network default
Create an external IP to use for the VPN

# gcloud compute addresses create gcp-to-csr --region us-west1
Capture the external IP address

# gcloud compute addresses list --filter="gcp-to-csr”
NAME REGION ADDRESS STATUS
gcp-to-csr us-west1 35.xxx.xxx.x RESERVED
Create a forwarding rule for ESP, UDP500 and UDP4500 – These are used by IKE/IPsec
# gcloud compute forwarding-rules create csr-gcp-vm-rule-esp \
--region us-west1 \
--address 35.xxx.xxx.x \
--ip-protocol ESP \
--target-vpn-gateway csr-gcp-vm-gw
# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \

--region us-west1 \
--ip-protocol UDP --ports 500 \
# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \

--region us-west1 \
--ip-protocol UDP --ports 4500 \
gcloud – Create Cloud Router, VPN Tunnel and BGP session
Create the Cloud router that is used for BGP (an existing router can be used)
# gcloud compute routers create csr-gcp-vm-bgp-rtr \
--region us-west1 \
--asn=65000 \
--network default
Create a VPN tunnel and link it to the router created in the previous step
# gcloud compute vpn-tunnels create csr-gcp-vm-gw-tunnel-1 \
--region us-west1 \
--peer-address 192.xxx.xxx.x --shared-secret <pre-shared-password-goes-here> \
--ike-version 2 \
--target-vpn-gateway csr-gcp-vm-gw \
--router csr-gcp-vm-bgp-rtr
Add a new interface to the router and set the BGP session IP address for the GCP side of the connection
# gcloud compute routers add-interface csr-gcp-vm-bgp-rtr \
--interface-name if-csr-gcp-vm-bgp-rtr-01 \
--ip-address 169.254.0.1 \
--mask-length 30 \
--vpn-tunnel csr-gcp-vm-gw-tunnel-1 \
--region us-west1
Create a new BGP peer – This peer will be the Cisco CSR at the On Premises cloud
# gcloud compute routers add-bgp-peer csr-gcp-vm-bgp-rtr \
--interface if-csr-gcp-vm-bgp-rtr-01 \
--peer-asn 65002 \
--peer-name csr-gcp-vm-bgp-peer \
--peer-ip-address 169.254.0.2 \
--region us-west1
... Output summarised
Cisco CSR Route Information

csr-gcp-01# show ip route
Default Network Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
10.138.0.0/20 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
169.254.0.2 ia - IS-IS inter area, * - candidate default, U - per-user static route
.1 o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
Area 0
a - application route
Google + - replicated route, % - next hop override, p - overrides from PfR
Cloud BGP 192.168.100.0/24
VPN S* 0.0.0.0/0 [1/0] via 192.xxx.xxx.x
10.0.0.0/20 is subnetted, 1 subnets
169.254.0.1 B 10.138.0.0 [20/100] via 169.254.0.1, 00:16:59
169.254.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 169.254.0.0/30 is directly connected, Tunnel0
L 169.254.0.2/32 is directly connected, Tunnel0
192.xxx.xxx.x/24 is variably subnetted, 2 subnets, 2 masks
Google Cloud Router C 192.xxx.xxx.x/26 is directly connected, GigabitEthernet1
L 192.xxx.xxx.x/32 is directly connected, GigabitEthernet1
C 192.168.100.0/24 is directly connected, GigabitEthernet2
L 192.168.100.1/32 is directly connected, GigabitEthernet2
Google VPN –
Dual/Redundant On
Premises Cisco CSRs
On Premises Cloud 1
Reference Topology for vSphere Hosted Cisco CSR
Dual Cisco CSR Design OSPF 10 Area 0

169.254.0.2 .20
192.yyy.yyy.y .2 VM ESXi Host 1
192.168.100.0/24
Private Network
Compute
Engine
2 1
BGP AS65002
Default Network .2
35.yyy.yyy.y HSRP – VIP = .1
10.138.0.0/20 35.xxx.xxx.x
Google
Cloud 169.254.0.10
VPN .3
169.254.0.1 ESXi Host 2
169.254.0.9 192.xxx.xxx.x
Google Cloud Router
vSphere Distributed vSwitch (DVS)

Routes this side should see: with a Distributed PortGroup for the
192.168.100.0/24 Private Network

10.138.0.0/20
Pre-Failure State (1)
GCE Instance traceroutes via 169.254.0.2 GCP BGP Path
[shmcfarl@instance-3 ~]$ traceroute 192.168.100.20
traceroute to 192.168.100.20 (192.168.100.20), 30 hops max, 60 byte packets
1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms
2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms
On Premises VM traceroutes via HSRP Active CSR (192.168.100.2)

[root@k8s-m-01 ~]# traceroute 10.138.0.2
1 192.168.100.2 (192.168.100.2) 0.545 ms 0.468 ms 0.415 ms
2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms
HSRP Active CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-01#show ip route
. . .
B 10.138.0.0/20 [20/100] via 169.254.0.1, 00:03:41
HSRP Standby CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-02#show ip route
. . .
B 10.138.0.0/20 [20/100] via 169.254.0.9, 00:08:47
HSRP Active
csr-gcp-01#show stand
GigabitEthernet2 - Group 0 (version 2)
State is Active
HSRP Standby
csr-gcp-02#show stand
GigabitEthernet2 - Group 0 (version 2)
State is Standby
https://cloud.google.com/router/docs/concepts/overview
First Google Cloud Router BGP State
# gcloud compute routers get-status csr-gcp-vm-bgp-rtr
kind: compute#routerStatusResponse • Determining best path
result:
bestRoutes: • If Cloud Router receives multiple routes for the
- creationTimestamp: '2017-09-19T14:48:49.137-07:00'
destRange: 192.168.100.0/24
same destination, GCP uses route metrics and,
kind: compute#route in some cases, AS path length to determine the
nextHopIp: 169.254.0.10 best path. To help you configure your On
priority: 0
Premises routers, the following list describes
destRange: 192.168.100.0/24 the algorithm that GCP uses for egress traffic.
kind: compute#route
nextHopIp: 169.254.0.2 • If you have multiple BGP sessions on a single
priority: 0 Cloud Router, GCP uses the route with the
bestRoutesForRouter:
- creationTimestamp: '2017-09-19T14:48:49.137-07:00' shortest AS path length.
destRange: 192.168.100.0/24
kind: compute#route • If routes have the same AS path length, GCP uses
nextHopIp: 169.254.0.2 the route with the lower MED value.
priority: 0
bgpPeerStatus: • If routes have equal costs (same AS path length and
- advertisedRoutes: metric), GCP uses ECMP to balance traffic across
- destRange: 10.138.0.0/20 multiple paths.
kind: compute#route
nextHopIp: 169.254.0.1 • If you use multiple Cloud Routers, GCP uses
priority: 100
ipAddress: 169.254.0.1 only the MED value to determine the best path.
name: csr-gcp-vm-bgp-peer The AS path length doesn't influence the path
numLearnedRoutes: 1 selection between multiple Cloud Routers.
peerIpAddress: 169.254.0.2
state: Established
status: UP
• If a static and dynamic route have the same
uptime: 1 minutes, 48 seconds prefix and metric, GCP uses the static route.
uptimeSeconds: '108'
network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default
https://cloud.google.com/router/docs/concepts/overview
Second Google Cloud Router BGP State
# gcloud compute routers get-status csr-gcp-vm-bgp-rtr-02
kind: compute#routerStatusResponse • Determining best path
result:
bestRoutes: • If Cloud Router receives multiple routes for the
destRange: 192.168.100.0/24
same destination, GCP uses route metrics and, in
kind: compute#route some cases, AS path length to determine the
nextHopIp: 169.254.0.10 best path. To help you configure your On
priority: 0
- creationTimestamp: '2017-09-19T14:48:49.137-07:00' Premises routers, the following list describes the
destRange: 192.168.100.0/24 algorithm that GCP uses for egress traffic.
kind: compute#route
nextHopIp: 169.254.0.2 • If you have multiple BGP sessions on a single
priority: 0
bestRoutesForRouter:
Cloud Router, GCP uses the route with the
- creationTimestamp: '2017-09-19T14:43:36.121-07:00' shortest AS path length.
destRange: 192.168.100.0/24
kind: compute#route • If routes have the same AS path length, GCP uses the
nextHopIp: 169.254.0.10 route with the lower MED value.
priority: 0
bgpPeerStatus: • If routes have equal costs (same AS path length and
- advertisedRoutes: metric), GCP uses ECMP to balance traffic across
- destRange: 10.138.0.0/20 multiple paths.
kind: compute#route
nextHopIp: 169.254.0.9 • If you use multiple Cloud Routers, GCP uses only
priority: 100
ipAddress: 169.254.0.9 the MED value to determine the best path. The
name: csr-gcp-vm-bgp-peer-02 AS path length doesn't influence the path
numLearnedRoutes: 1
peerIpAddress: 169.254.0.10
selection between multiple Cloud Routers.
state: Established
status: UP
• If a static and dynamic route have the same prefix
uptime: 6 minutes, 50 seconds and metric, GCP uses the static route.
uptimeSeconds: '410'
network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default
Failure Scenario 1 – HSRP Primary CSR VM Reload

HSRP Debug on HSRP Standby
csr-gcp-02#
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby: i/Resign rcvd (110/192.168.100.2)
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Active router is local, was 192.168.100.2
*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 no longer active for group 0 (Standby)
*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 Was active or standby - start passive holddown
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby router is unknown, was local
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby -> Active
*Sep 19 21:59:17.396: %HSRP-5-STATECHANGE: GigabitEthernet2 Grp 0 state Standby -> Active
*Sep 19 21:59:17.396: HSRP: Peer not present
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Redundancy "hsrp-Gi2-0" state Standby -> Active
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Added 192.168.100.1 to ARP (0000.0c9f.f000)
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Activating MAC 0000.0c9f.f000
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Adding 0000.0c9f.f000 to MAC address filter
*Sep 19 21:59:17.396: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" standby, local -> unknown
*Sep 19 21:59:17.398: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Standby -> Active
*Sep 19 21:59:20.379: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Active -> Active
*Sep 19 21:59:57.361: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.100.2 on GigabitEthernet2 from FULL to DOWN, Neighbor Down: Dead timer expired
On Premises VM traceroutes via HSRP Newly Active CSR (192.168.100.3)

[root@k8s-m-01 ~]# traceroute 10.138.0.2
1 192.168.100.3 (192.168.100.3) 0.545 ms 0.468 ms 0.415 ms
2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms
Failure Scenario 2 – Shut HSRP Primary LAN Interface
(BGP session is still active)
Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path
1 169.254.0.2 (169.254.0.2) 24.223 ms 24.430 ms 24.716 ms
2 192.168.100.20 (192.168.100.20) 24.180 ms 24.595 ms 24.422 ms
Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path

1 169.254.0.10 (169.254.0.10) 32.756 ms 42.796 ms 25.635 ms
2 192.168.100.20 (192.168.100.20) 66.674 ms 72.234 ms 74.331 ms
Failure Scenario 3 – Shut IPsec Tunnel on HSRP Primary
CSR – With/Without HSRP Interface Tracking
Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path
1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms
2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms
Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path BUT traffic is re-routed to the HSRP Primary
(192.168.100.2) before going to the end host
1 169.254.0.10 (169.254.0.10) 24.863 ms 42.763 ms 32.908 ms
2 192.168.100.2 (192.168.100.2) 54.069 ms 86.788 ms 70.963 ms
On Premises LAN re-route to HSRP Active
3 192.168.100.20 (192.168.100.20) 174.753 ms * 134.706 ms on router with failed IPsec Tunnel
LAN Re-Route Issue Resolved – Use Track
track 10 interface Tunnel0 line-protocol csr-gcp-01#show stand
! GigabitEthernet2 - Group 0 (version 2) Tunnel failed and
interface GigabitEthernet2 State is Standby track changed
description Private Network On Premises . . .
ip address 192.168.100.2 255.255.255.0 Priority 100 (configured 110) HSRP state
standby version 2 Track object 10 state Down decrement 10
standby 0 ip 192.168.100.1
standby 0 priority 110 [shmcfarl@instance-3 ~]$ traceroute 192.168.100.20
standby 0 preempt traceroute to 192.168.100.20 (192.168.100.20), 30 hops max, 60 byte packets
standby 0 authentication md5 key-string 7 01300F175804575D720D 1 169.254.0.10 (169.254.0.10) 43.113 ms 25.269 ms 33.033 ms
standby 0 track 10 decrement 10 2 192.168.100.20 (192.168.100.20) 72.879 ms 111.849 ms 53.904 ms
interface Tunnel0 ... Output summarized
Reference Cisco CSR Config – Primary ip address 169.254.0.2 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
crypto ikev2 proposal PHASE1-PROP tunnel source GigabitEthernet1
encryption aes-cbc-256 tunnel mode ipsec ipv4
integrity sha1 tunnel destination 35.yyy.yyy.y
group 14 tunnel protection ipsec profile CSR-GCP
! !
crypto ikev2 policy IKE-POL interface GigabitEthernet1
proposal PHASE1-PROP ip address 192.yyy.yyy.y 255.255.255.192
! !
crypto ikev2 keyring KEY interface GigabitEthernet2
peer GCP-PEER description Private Network On Premises
address 35.yyy.yyy.y ip address 192.168.100.2 255.255.255.0
hostname csr-gcp-dmz-sjc standby version 2
pre-shared-key local <PSK_PASSWORD_GOES_HERE> standby 0 ip 192.168.100.1
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> standby 0 priority 110
! standby 0 preempt
crypto ikev2 profile IKEV2-SETUP standby 0 authentication md5 key-string 7 <HSRP_KEY>
match identity remote address 0.0.0.0 standby 0 track 10 decrement 10
authentication local pre-share !
authentication remote pre-share router ospf 10
keyring local KEY redistribute bgp 65002 subnets
lifetime 36000 network 192.168.100.0 0.0.0.255 area 0
! !
crypto ikev2 dpd 10 2 periodic router bgp 65002
! bgp log-neighbor-changes
track 10 interface Tunnel0 line-protocol neighbor 169.254.0.1 remote-as 65000
! neighbor 169.254.0.1 timers 20 60 60
crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac !
mode tunnel address-family ipv4
! redistribute ospf 10
crypto ipsec profile CSR-GCP neighbor 169.254.0.1 activate
set transform-set CSR-GCP-SET neighbor 169.254.0.1 soft-reconfiguration inbound
set pfs group14 !
#CLMEL ip route BRKCLD-3440
0.0.0.0 © 2019 Cisco and/or
0.0.0.0 its affiliates. All rights reserved. Cisco Public
192.yyy.yyy.y 46
set ikev2-profile IKEV2-SETUP
interface Tunnel0
ip address 169.254.0.10 255.255.255.252
Reference Cisco CSR Config – Secondary

ip mtu 1400
crypto ikev2 proposal PHASE1-PROP tunnel source GigabitEthernet1
encryption aes-cbc-256 tunnel mode ipsec ipv4
integrity sha1 tunnel destination 35.xxx.xxx.x
group 14 tunnel protection ipsec profile CSR-GCP
! !
crypto ikev2 policy IKE-POL interface GigabitEthernet1
proposal PHASE1-PROP ip address 192.xxx.xxx.x 255.255.255.192
! !
crypto ikev2 keyring KEY interface GigabitEthernet2
peer GCP-PEER description Private Network On Premises
address 35.xxx.xxx.x ip address 192.168.100.3 255.255.255.0
hostname csr-vpn-gw-02 standby version 2
pre-shared-key local <PSK_PASSWORD_GOES_HERE> standby 0 ip 192.168.100.1
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> standby 0 priority 105
! standby 0 preempt
crypto ikev2 profile IKEV2-SETUP standby 0 authentication md5 key-string 7 <HSRP_KEY>
match identity remote address 0.0.0.0 !
authentication local pre-share router ospf 10
authentication remote pre-share redistribute bgp 65002 subnets
keyring local KEY network 192.168.100.0 0.0.0.255 area 0
lifetime 36000 !
! router bgp 65002
crypto ikev2 dpd 10 2 periodic bgp log-neighbor-changes
! neighbor 169.254.0.9 remote-as 65000
crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac neighbor 169.254.0.9 timers 20 60 60
mode tunnel !
! address-family ipv4
crypto ipsec profile CSR-GCP redistribute ospf 10
set transform-set CSR-GCP-SET neighbor 169.254.0.9 activate
set pfs group14 neighbor 169.254.0.9 soft-reconfiguration inbound
set ikev2-profile IKEV2-SETUP !
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
Amazon Web Services –
Native VPN
AWS – VPN Gateway
• AWS VPN Overview
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html
• AWS VPN Setup
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/SetUpVPNConnections.
html
• AWS does support NAT-T:
https://aws.amazon.com/blogs/aws/ec2-vpc-vpn-update-nat-traversal-additional-
encryption-options-and-more/
• Example templates for Cisco IOS:
http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Cisco.html
Topology for AWS to On Premises CSR – IPsec
VPN
BGP Routing BGP <> OSPF Redistribution
IPsec/IKEv2
Cisco
Tunnel Mode CSR1000v
VPC Network 52.xxx.xxx.x 169.254.11.178 .1 Private Network

172.31.0.0/16 192.168.200.0/24
169.254.11.177 192.xxx.xxx.x
VPC VPN
Router Gateway OSPF 10 Area 0
BGP AS64512
Hypervisor
BGP AS65002

192.168.200.0/24
172.31.0.0/16
AWS CLI: Create VPC, VPN GW,
Customer GW and VPN Connection
Create a new AWS VPC (or use an existing one)
# aws ec2 create-vpc --cidr-block 172.31.0.0/16
Create VPN Gateway and set the AWS BGP ASN

# aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn 64512
Attach VPN Gateway to the VPC

# aws ec2 attach-vpn-gateway --vpc-id vpc-ce2124aa --vpn-gateway-id vgw-64277e21
Create a new customer gateway with the On Premises BGP ASN and the On Premises router IP address (do this for each connection)
# aws ec2 create-customer-gateway --bgp-asn 65002 --public-ip 192.xxx.xxx.x --type ipsec.1
Create a new VPN connection

# aws ec2 create-vpn-connection --customer-gateway-id cgw-d6055d93 --type ipsec.1 --vpn-gateway-id vgw-64277e21
Note: Lots of output will come from the above VPN creation command.
This information can be used to build the On Premises CSR config. The best method for getting the configuration is
shown on the next slide.
Enable route propagation for the VPC

# aws ec2 enable-vgw-route-propagation --gateway-id vgw-64277e21 --route-table-id rtb-515e8e36
Permit SSH and ICMP

# aws ec2 authorize-security-group-ingress --group-name default --protocol tcp --port 22 --cidr 0.0.0.0/0
# aws ec2 authorize-security-group-ingress --group-name default --protocol icmp --port -1 --cidr 0.0.0.0/0
Optional: Download Router Configuration
• VPC Dashboard > VPN Connections
Reference Cisco CSR Config - Primary ... Output summarised
crypto isakmp policy 200
encryption aes 128 interface Tunnel1
authentication pre-share ip address 169.254.11.178 255.255.255.252
group 2 ip virtual-reassembly
lifetime 28800 ip mtu 1400
hash sha tunnel source 192.xxx.xxx.x
! tunnel destination 52.xxx.xxx.x
crypto keyring keyring-vpn-cec15996-0 tunnel mode ipsec ipv4
local-address 192.xxx.xxx.x tunnel protection ipsec profile ipsec-vpn-cec15996-0
pre-shared-key address 52.xxx.xxx.x key ip tcp adjust-mss 1379
<PSK_PASSWORD_GOES_HERE> !
! router ospf 10
crypto isakmp profile isakmp-vpn-cec15996-0 redistribute bgp 65002 subnets
local-address 192.xxx.xxx.x network 192.168.200.0 0.0.0.255 area 0
match identity address 52.xxx.xxx.x !
keyring keyring-vpn-cec15996-0 router bgp 65002
crypto ipsec transform-set ipsec-prop-vpn-cec15996-0 esp-aes neighbor 169.254.11.177 activate
128 esp-sha-hmac neighbor 169.254.11.177 timers 10 30 30
mode tunnel !
crypto ipsec profile ipsec-vpn-cec15996-0 redistribute ospf 10
set pfs group2 neighbor 169.254.11.177 remote-as 64512
set security-association lifetime seconds 3600 neighbor 169.254.11.177 activate
set transform-set ipsec-prop-vpn-cec15996-0 neighbor 169.254.11.177 soft-reconfiguration inbound
!
crypto ipsec df-bit clear
!
crypto isakmp keepalive 10 10 on-demand
!
crypto ipsec fragmentation before-encryption
Verify Routing and Reachability

On the on-premises CSR check the route for the the AWS VPC network 172.31.0.0/16
csr-mc-01#show ip route | i 172.31.0.0
B 172.31.0.0/16 [20/100] via 169.254.11.177, 00:13:35
On AWS check for the route for the on-premises network (192.168.200.0/24)
# aws ec2 describe-route-tables | grep 192.168.200.0
ROUTES 192.168.200.0/24 vgw-64277e21 EnableVgwRoutePropagation active
Connect to an AWS instance and ping to the on-premises private network

ubuntu@ip-172-31-0-121:~$ ping 192.168.200.30
PING 192.168.200.30 (192.168.200.30) 56(84) bytes of data.
64 bytes from 192.168.200.30: icmp_seq=1 ttl=63 time=4.95 ms
Cisco
VM
BGP .30
VPC Network CSR1000v
172.31.0.0/16 169.254.11.178 .1 Private Network
.121 .1 169.254.11.177 192.168.200.0/24
Hypervisor
On Premises Cloud 1
Topology for Dual Cisco CSR on AWS vSphere Hosted Cisco CSR
OSPF 10 Area 0
BGP AS65002
169.254.11.178
.2 ESXi Host 1
192.168.200.0/24
Private Network
VPC Network 169.254.11.177 HSRP – VIP = .1
169.254.10.213
172.31.0.0/16
VPC VPN
Router Gateway
BGP AS64512 .3 ESXi Host 2
169.254.10.214
BGP AS65002
vSphere Distributed vSwitch (DVS)

Routes this side should see: with a Distributed PortGroup for the
192.168.200.0/24 Private Network

172.31.0.0/16
Microsoft Azure – Native
VPN
Microsoft Azure – VPN Gateway
• Azure VPN Overview
• https://azure.microsoft.com/en-us/services/vpn-gateway/
• https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways
• In order to use BGP you must use Route-Based VPN and SKUs VpnGw1, VpnGw2,
VpnGw3, Standard or HighPerformance SKUs : https://docs.microsoft.com/en-
us/azure/vpn-gateway/vpn-gateway-bgp-overview
Azure to On Premises CSR – IPsec VPN
BGP Routing BGP <> OSPF Redistribution
IPsec/IKEv2
Cisco
Vnet Subnet 40.xxx.xxx.x 10.11.255.1 .1 Private Network
10.10.0.0/16 192.168.200.0/24
VPN 10.10.255.30 192.xxx.xxx.x
Gateway OSPF 10 Area 0
BGP AS64512 Hypervisor
BGP AS65002
Azure CLI: Create Resource Group, Networks,
Subnets
Create a new Azure Resource Group (rg)
# az group create --name azure-vpn-rg --location westus
# az configure --defaults location=westus
# az configure --defaults group=azure-vpn-rg
Create a new virtual network (vnet) and a new ‘outside’ subnet

# az network vnet create \
--name vnet1 \
--address-prefix 10.10.0.0/16 \
--subnet-name outside \
--subnet-prefix 10.10.0.0/24
Create a ’inside’ subnet

# az network vnet subnet create \
--vnet-name vnet1 \
--name inside \
--address-prefix 10.10.1.0/24
Create a new subnet that is used for the IPsec/BGP interface on the Azure side
--vnet-name vnet1 \
--name gatewaysubnet \
Azure CLI: Create a Public IP, VPN/Vnet Gateway and
Local Gateway
Create a new public IP address (Using Azure VPN service, the allocation must be ‘dynamic’)
# az network public-ip create \
--name azure-vpn-gw-eip \
--allocation-method dynamic
Create Vnet gateway using ‘RouteBased’ (BGP) and a supported sku (see earlier links for requirements). THIS TAKES AWHILE
# az network vnet-gateway create \
--name vpn-gw \
--public-ip-address azure-vpn-gw-eip \
--vnet vnet1 \
--gateway-type Vpn \
--sku VpnGw1 \
--vpn-type RouteBased \
--asn 65010
Once the Vnet gateway is up, get the Azure-side BGP Peering address (Needed for On Premises configuration)
# az network vnet-gateway list | grep bgpPeeringAddress
"bgpPeeringAddress": "10.10.255.30",
Create the local gateway (On Premises target). Local prefix/BGP peer should be the On Premises CSR tunnel info. Can’t be in Azure vne
# az network local-gateway create \
--gateway-ip-address 192.xxx.xxx.x \
--name azure-lng \
--local-address-prefixes 10.11.255.1/32 \
--asn 65002 \
--bgp-peering-address 10.11.255.1
Azure CLI: Vnet GW, Local GW, VPN Connection
Copy the full path from the “id” line (under the ‘gatewayType: Vpn’ line) that is shown in the vnet-gateway output
# az network vnet-gateway show --name vpn-gw
"gatewayType": "Vpn",
"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw",
Copy the full path from the “id” line that is shown in the vnet-gateway output
# az network local-gateway show --name azure-ln
"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng"
Create the VPN connection using information from above

# az network vpn-connection create \
--name azure-to-csr \
--vnet-gateway1 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw \
--enable-bgp \
--shared-key ”<YOUR_PRE_SHARED_KEY>" \
--local-gateway2 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng
Optional: Create a new test VM on Azure and associate it with the ‘inside’ subnet
# az vm create \
--name AzTestVm \
--authentication-type ssh \
--ssh-key-value "$(< ~/.ssh/id_rsa.pub)" \
--image Canonical:UbuntuServer:16.04-LTS:latest \
--size Standard_DS1_v2 \
--vnet-name vnet1 \
--subnet inside \
--public-ip-address-allocation dynamic
On Premises Cisco CSR IPsec/Routing Config ... Output summarised
crypto ikev2 proposal PHASE1-PROP interface Tunnel2
encryption aes-cbc-256 ip address 10.11.255.1 255.255.255.255
integrity sha1 ip mtu 1400
group 2 ip tcp adjust-mss 1360
! tunnel source GigabitEthernet1
crypto ikev2 policy IKE-POL tunnel mode ipsec ipv4
proposal PHASE1-PROP tunnel destination 40.xxx.xxx.x
! tunnel protection ipsec profile CSR-AZURE
crypto ikev2 keyring KEY !
peer AZURE-PEER interface GigabitEthernet1
address 40.xxx.xxx.x description Internet
pre-shared-key local <PSK_PASSWORD_GOES_HERE> ip address 192.xxx.xxx.x 255.255.255.0
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> no ip redirects
! no ip unreachables
crypto ikev2 profile IKEV2-SETUP no ip proxy-arp
match identity remote address 0.0.0.0 negotiation auto
authentication local pre-share !
authentication remote pre-share router ospf 10
keyring local KEY router-id 10.1.0.2
lifetime 36000 redistribute bgp 65002 subnets
! network 192.168.200.0 0.0.0.255 area 0
crypto ikev2 dpd 10 2 periodic !
! router bgp 65002
crypto ipsec security-association replay window-size 1024 bgp log-neighbor-changes
crypto ipsec transform-set CSR-AZURE-SET esp-aes 256 esp-sha-hmac neighbor 10.10.255.30 ebgp-multihop 255
mode tunnel !
crypto ipsec profile CSR-AZURE redistribute ospf 10
set transform-set CSR-AZURE-SET neighbor 10.10.255.30 activate
set pfs group14 neighbor 10.10.255.30 soft-reconfiguration inbound
set ikev2-profile IKEV2-SETUP !
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
ip route 10.10.255.30 255.255.255.255 Tunnel2

On the on-premises CSR check the route for the Azure Vnet route of 10.10.0.0/16
B 10.10.0.0/16 [20/0] via 10.10.255.30, 00:51:26
On Azure check for the route for the on-premises network (192.168.200.0/24)
PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName AzTestVmVMNic -ResourceGroupName azure-vpn-rg | Format-Table
Name State Source AddressPrefix NextHopType NextHopIpAddress
---- ----- ------ ------------- ----------- ----------------
Active VirtualNetworkGateway {192.168.200.0/24} VirtualNetworkGateway {40.xxx.xxx.x}
Connect to an Azure instance and ping to the on-premises private network

shmcfarl@AzTestVm:~$ ping 192.168.200.30
Cisco
Inside Subnet CSR1000v
.30
VM
10.10.1.0/24 40.xxx.xxx.x 10.11.255.1
.1 Private Network
.4 10.10.255.30 192.xxx.xxx.x 192.168.200.0/24
VPN
Gateway
Hypervisor
Multicloud with Cisco
SD-WAN
Reference
Cisco SD-WAN Architecture

Orchestration Plane
The Power of Abstraction - vBond
- Orchestrates control
vManage and mgmt. plane
- First point of auth
APIs Management Plane
- vManage
3rd Party
vAnalytics - UI
Automation
- Policies, templates
vBond - Monitoring
Control Plane
vSmart Controllers - vSmart
- Fabric discovery
4G
- Control plane policies
MPLS
INET
vEdge Routers
Data Plane
- vEdge
Cloud Data Centre Campus Branch SOHO
Cisco SD-WAN
VNet Network
10.10.1.0/16
vEdge/cEdge
Private Network
10.1.1.0/24
vEdge/cEdge
VPC Network SD-WAN

On-Premises
172.3.0.0/24
vEdge/cEdge
Cisco SD-WAN:
https://www.cisco.com/c/en/us/solutions/en
terprise-networks/sd-wan/index.html
Cisco SD-WAN
Public Cloud Support
• Cisco SD-WAN (vEdge) on AWS: https://sdwan-
docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07
Deploy_the_vEdge_Routers/01Create_vEdge_Cloud_VM_Instance_on_AWS
• AWS Marketplace: https://aws.amazon.com/marketplace/pp/B07BZ53FJT
• Cisco SD-WAN on Microsoft Azure: https://sdwan-
docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07
Deploy_the_vEdge_Routers/02Create_vEdge_Cloud_VM_Instance_on_Azure
• Microsoft Azure Marketplace: https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/cisco.cisco_cloud_vedge_4_nics?tab=Overview
• Brand New SD-WAN Design/Deployment Guides:
https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-wan-
edge.html
Cisco SD-WAN and AWS Options
SD-WAN + Internet + Host VPC
vEdge vEdge
VPC On-Premises
Router Cloud IPsec
SD-WAN + Transit VPC Transit VPC

vEdge vEdge
VPC VPN IPsec On-Premises
Router Gateway Cloud
SD-WAN + Some Combo of Colocation/peering
VLANs
DX vEdge
VPC VPN vEdge On-Premises
Router Gateway Endpoint Cloud
IPsec
IPsec
Cisco SD-WAN – Transit VPC
Cloud onRamp for IaaS - AWS vManage vBond vSmart
Transit VPC
VPC Network Private Network

VPC VPN vEdge
IPsec vEdge
Router Gateway Cloud
On-Premises
• AWS: https://sdwan-
docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_On
Ramp_with_AWS
• Azure: https://sdwan-
docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_On
Ramp_with_Azure
AWS with Cisco SD-WAN
Cloud onRamp for IaaS - AWS
GatewayVpc (192.168.0.0/16)
Transit Subnet 0
192.168.30.31
EIP
HostVpc (172.16.0.0/16)
vpn 512
VPN
Tunnel EIP
vpn 1 vpn 0
192.168.85.0 192.168.59.199
EIP
172.16.0.0/24
PublicSubnet Transit Subnet 2 Transit Subnet 1
Private Network
VPC VPN GW Transit Subnet 2 Transit Subnet 1 IGW 10.1.1.0/24
172.16.3.0/24
VPC
Router
vedge
Router (VGW) vpn 1 vpn 0
PrivateSubnet 192.168.176.185 192.168.139.23
On-Premises
EIP
192.168.126.106
VPN EIP
Tunnel
vpn 512
EIP
Transit Subnet 0
Cloud onRamp for IaaS - AWS IPsec VPN
GatewayVpc (192.168.0.0/16)
Transit Subnet 0
192.168.30.31
EIP
HostVpc (172.16.0.0/16)
vpn 512
VPN
Tunnel EIP
vpn 1 vpn 0
192.168.85.0 192.168.59.199
EIP
172.16.0.0/24
Transit Subnet 2 Transit Subnet 1
PublicSubnet Private Network
172.16.3.0/24
VPC
Router
vedge
PrivateSubnet 192.168.176.185 192.168.139.23
On-Premises
EIP
192.168.126.106
VPN EIP
Tunnel
vpn 512
EIP
Transit Subnet 0
IPsec VPN
Cloud onRamp for IaaS - AWS IPsec VPN
GatewayVpc (192.168.0.0/16)
Transit Subnet 0
192.168.30.31
EIP
HostVpc (172.16.0.0/16)
vpn 512
VPN
Tunnel
vpn 1 vpn 0
EIP
IPsec
172.16.0.0/24
EIP
192.168.85.0 192.168.59.199 VPN
PublicSubnet Transit Subnet 2 Transit Subnet 1
Private Network
172.16.3.0/24
VPC
Router
vedge
PrivateSubnet 192.168.176.185 192.168.139.23
On-Premises
EIP
192.168.126.106
VPN EIP
Tunnel
vpn 512
EIP
Transit Subnet 0
IPsec VPN
vManage
Dashboard View (Yeah, I know, no HA on the control plane )

Host VPCs are ‘mapped’
(connected via VPN) to the
Transit VPCs
Transit VPCs – Two vEdge-

Cloud EC2 Instances – These
connect to the on-premises
via SD-WAN setup
AWS – VPC/Subnet View
VPC View
Subnet View
AWS – Host VPC –to- Transit VPC Mapping
VPN Gateway (VPG) View
Customer Gateway Endpoints (EIPs of each Transit vEdge Cloud)
VPN Connections (only one of the two is shown below)
AWS – Host VPC –to- Transit VPC Mapping - IPsec
vEdge-Cloud – Transit VPC HostVpc (172.16.0.0/16)
interface ipsec8
ip address 169.254.10.14/30 Transit VPC
VPN
tunnel-source 192.168.59.199 Tunnel vEdge
tunnel-destination 52.xx.xx.xx
vpn 0
ike EIP 192.168.59.199
version 1 172.16.0.0/24
mode main PublicSubnet
rekey 28800 169.254.10.14/30
cipher-suite aes128-cbc-sha1
VPC VPN GW
group 2 172.16.3.0/24
Router (VGW)
authentication-type PrivateSubnet
pre-shared-key
pre-shared-secret <PSK_HERE> VPN
EIP 169.254.10.13/30
! Tunnel
!
!
ipsec
rekey 3600
replay-window 512
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy group-16
!
AWS – Host VPC –to- Transit VPC Mapping - BGP
vEdge-Cloud – Transit VPC HostVpc (172.16.0.0/16)
vpn 1 Transit VPC
VPN
router
Tunnel vEdge
bgp 9988
timers vpn 0
holdtime 30 EIP 192.168.59.199
172.16.0.0/24
!
address-family ipv4-unicast PublicSubnet 169.254.10.14/30
network 0.0.0.0/0
redistribute omp VPC VPN GW
172.16.3.0/24
Router (VGW)
!
neighbor 169.254.10.13 PrivateSubnet
no shutdown
VPN
EIP 169.254.10.13/30
remote-as 64512
Tunnel
update-source ipsec8
vedge-aws-01# show ip route

OUTPUT OMITTED...
PROTOCOL NEXTHOP NEXTHOP NEXTHOP
VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS
-------------------------------------------------------------------------------------------------------------------------------------
1 172.16.0.0/16 bgp e ipsec8 169.254.10.13 - - - - F,S
Transit VPC –to- On-Premises - IPsec
Transit VPC vEdge - IPsec
vedge-aws-01# show ipsec outbound-connections
OUTPUT SUMMARIZED...
SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION

IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED
--------------------------------------------------------------------------------------------------------------------------------------
192.168.59.199 12406 <ON_PREMISES_vEDGE_PUBLIC_IP> 12346 270 1441 1.1.1.4 public-internet AH_SHA1_HMAC
On-Premises vEdge - IPsec

vedge-01# show ipsec outbound-connections
SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION

IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED
--------------------------------------------------------------------------------------------------------------------------------------
<ON_PREMISES_vEDGE_PUBLIC_IP> 12346 <TRANSIT-vEDGE-EIP> 12346 258 1441 2.2.2.5 default AH_SHA1_HMAC
<ON_PREMISES_vEDGE_PUBLIC_IP> 12346 <TRANSIT-vEDGE-EIP> 12406 258 1441 2.2.2.6 default AH_SHA1_HMAC
Transit VPC
vEdge EIP
VPC CIDR VPN GW vpn 0 IPsec VPN Private Network
(VGW) 192.168.59.199
172.16.0.0/16
10.1.1.0/24
EIP vEdge
VPN
Tunnel
IPsec VPN
Transit VPC –to- On-Premises - BGP
Transit VPC vEdge - BGP
vedge-aws-01# show ip route
--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 omp - - - - 1.1.1.4 public-internet ipsec F,S
1 169.254.8.40/30 connected - ipsec7 - - - - - F,S
1 169.254.10.12/30 connected - ipsec8 - - - - - F,S
1 172.16.0.0/16 bgp e ipsec8 169.254.10.13 - - - - F,S
On-Premises vEdge - IPsec
vedge-01# show ip route
--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
1 172.16.0.0/16 omp - - - - 2.2.2.5 default ipsec F,S
1 172.16.0.0/16 omp - - - - 2.2.2.6 default ipsec F,S
Transit VPC
vEdge EIP
VPC CIDR VPN GW vpn 0 IPsec VPN Private Network
(VGW) 192.168.59.199
172.16.0.0/16
10.1.1.0/24
vEdge
VPN
EIP
IPsec VPN
Tunnel
DMVPN – Dynamic
Multipoint VPN
Merging in Multicloud to
an Existing Branch/WAN
Deployment
DMVPN (Dynamic Multipoint VPN)
• Cisco DMVPN
• https://www.cisco.com/c/en/us/products/security/dynamic-multipoint-vpn-dmvpn/index.html
• Cisco Live DMVPN
• https://www.ciscolive.com/global/on-demand-library/?search=dmvpn#/
• Cisco IWAN CVD
• https://www.cisco.com/c/en/us/solutions/design-zone/networking-design-guides/branch-
wan-edge.html
• DMVPN is a Cisco innovation for building GRE/mGRE + IPsec VPN connections in a
dynamic and scalable manner
Terminology and Features
Core Network Overlay Addresses
192.168.128.0/17
192.168.101.0/24 192.168.102.0/24
Tunnel Address
Hub1 Hub 2
Tunnel: 10.0.0.101 Tunnel: 10.0.0.102

Physical: 172.16.101.1 Physical: 172.16.102.1
NBMA Address
Tunnel: 10.0.0.1 Tunnel: 10.0.0.2

Physical: 172.16.1.1 Physical: 172.16.2.1
Spoke 1 GRE/IPsec Spoke 2
Tunnels
192.168.1.0/24 192.168.2.0/24
On Demand
Spoke Tunnels
DMVPN Components
• Next Hop Resolution Protocol (NHRP)
• Creates a distributed (NHRP) mapping database of all the spoke’s tunnel to real (public
interface) addresses
• Multipoint GRE Tunnel Interface (mGRE)

• Single GRE interface to support multiple GRE/IPsec tunnels
• Simplifies size and complexity of configuration
• IPsec tunnel protection

• Dynamically creates and applies encryption policies
• Routing
• Dynamic advertisement of branch networks; almost all routing protocols (EIGRP, RIP, OSPF,
BGP, ODR) are supported
Spoke-to-hub tunnels
DMVPN Implementation
Spoke-to-spoke tunnels
2547oDMVPN tunnels
Hub and spoke Spoke-to-spoke

(Phase 1) (Phase 2) VRF-lite
Server Load Balancing Hierarchical (Phase 3) 2547oDMVPN
Google Cloud Platform–
Cisco CSR and DMVPN
GCP to On Premises CSR – IPsec VPN
Example 1
BGP <> OSPF Redistribution
Compute
Engine IPsec/IKEv2
.30
Cisco
VM
2 1
Tunnel Mode
CSR1000v
Default Network 35.xxx.xxx.x 169.254.0.2 .1 Private Network
10.138.0.0/20 192.xxx.xxx.x 192.168.200.0/24
Google
Cloud OSPF 10 Area 0
VPN
169.254.0.1 Hypervisor
Google Cloud Router
GCP CSR to On Premises CSR – IPsec VPN
Example 2
IPsec/IKEv2
.30
VM
Default Network Cisco
inside-network Tunnel Mode CSR1000v
10.138.0.0/20
10.0.1.0/24
Compute .1 35.xxx.xxx.x .1 Private Network
Engine
2 1
.3 .2 .100 192.xxx.xxx.x 192.168.200.0/24
Cisco
CSR1000v OSPF 10 Area 0
Hypervisor
OSPF
GCP CSR to On Premises CSR – DMVPN
Default Network .30

inside-network
10.138.0.0/20
Cisco
CSR1000v
VM
10.0.1.0/24 35.xxx.xxx.x 192.xxx.xxx.x
Compute .1 .1 Private Network
Engine
2 1
.3 .2 .100
Spoke Hub 192.168.200.0/24
Cisco CSR Tunnel: CSR Tunnel:
CSR1000v
10.1.0.1 10.1.0.2 OSPF 10 Area 0
Hypervisor
DMVPN
OSPF
192.168.200.0/24
10.0.1.0/24
gcloud – Create the GCP External IP, Inside VPC
Network and Route
Create a new external IP reservation that will be used for the GCP CSR NATed connection (or use an existing one)
# gcloud compute addresses create csr-to-csr-ext-ip --region us-west1
Capture the external IP address

# gcloud compute addresses list --filter="csr-to-csr-ext-ip"
NAME REGION ADDRESS STATUS
csr-to-csr-ext-ip us-west1 35.xxx.xxx.x RESERVED
Create a new GCP inside network that will be attached to the ‘inside’ interface of the CSR
# gcloud compute networks create inside-network --subnet-mode=custom
Create a new GCP inside subnet - Associate it with the inside network
# gcloud compute networks subnets create inside-subnet \
--network=inside-network \
--range=10.0.1.0/24
Create a new GCP route from the CSR inside network to the On Premises private network which routes through the IPsec VPN
# gcloud compute routes create inside-to-csr-private \
--destination-range=192.168.200.0/24 \
--next-hop-address=10.0.1.2
gcloud – Create GCP Firewall Rules
Create a new GCP firewall rule to allow traffic into the inside CSR network from the default network
# gcloud compute firewall-rules create allow-default-to-csr-inside \
--direction=INGRESS \
--action=ALLOW \
--rules=all \
--source-ranges=0.0.0.0/0
Create a new GCP firewall rule to allow traffic between the default network and the On Premises CSR public IP for IKE, IPsec
# gcloud compute firewall-rules create csr-csr-vpn \
--direction=INGRESS \
--network=default \
--action=ALLOW \
--rules=udp:500,udp:4500,esp \
--source-ranges=192.xxx.xxx.x
gcloud – Create CSR and Test Instances
Create a new GCE CSR instance and set fixed IPv4 addresses to each of the two interfaces
# gcloud compute instances create "csr-gcp-01" \
--zone "us-west1-a" \
--machine-type "n1-standard-4" \
--network-interface subnet="default",private-network-ip="10.138.0.100",address="35.xxx.xxx.x" \
--can-ip-forward \
--network-interface subnet="inside-subnet",private-network-ip="10.0.1.2",no-address \
--image ”name_of_csr_image" \
--boot-disk-size "10" \
--boot-disk-type "pd-standard" \
--boot-disk-device-name "csr-gcp-01"
Create a new GCE test instance that will be used to validate the VPN and routing
# gcloud compute instances create "csr-inside-vm" \
--zone "us-west1-a" \
--machine-type "g1-small" \
--subnet "inside-subnet" \
--private-network-ip "10.0.1.3" \
--image "debian-9-stretch-v20170918" \
--image-project "debian-cloud" \
--boot-disk-size "10" \
--boot-disk-type "pd-standard" \
--boot-disk-device-name "csr-inside-vm"
Connect to the GCP CSR – Enable Interfaces

Connect to the new GCP-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# gcloud compute ssh cisco-user@csr-gcp-01
csr1kv-gcp#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
csr1kv-gcp(config)#interface gigabitEthernet 2
csr1kv-gcp(config-if)#ip address dhcp
csr1kv-gcp(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and with the correct IP
addresses:
csr1kv-gcp#show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 10.138.0.100 YES TFTP up up
GigabitEthernet2 10.0.1.2 YES DHCP up up
GCP Cisco CSR DMVPN Config ... Output summarised
Spoke interface Tunnel0
crypto ikev2 proposal AES/GCM/256 description DMVPN
encryption aes-gcm-256 ip address 10.1.0.1 255.255.255.0
prf sha512 no ip redirects
group 19 ip mtu 1400
! ip nhrp authentication <NHRP_PASSWORD>
crypto ikev2 policy AES/GCM/256 ip nhrp network-id 100
match fvrf any ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
proposal AES/GCM/256 ip tcp adjust-mss 1360
! ip ospf authentication-key 7 <OSPF_PASSWORD>
crypto ikev2 keyring DMVPN-KEYRING ip ospf network point-to-multipoint
peer ANY ip ospf hello-interval 10
address 0.0.0.0 0.0.0.0 tunnel source GigabitEthernet1
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel mode gre multipoint
! tunnel key 100
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
description PSK Profile !
match identity remote address 0.0.0.0 interface GigabitEthernet1
identity local address 35.xxx.xxx.x description Internet
authentication remote pre-share ip address 10.138.0.100 255.255.255.0
authentication local pre-share no ip redirects
keyring local DMVPN-KEYRING no ip unreachables
dpd 40 5 on-demand no ip proxy-arp
! negotiation auto
crypto ipsec security-association replay window-size 1024 !
! router ospf 10
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router-id 10.1.0.1
mode transport network 10.0.1.0 0.0.0.255 area 1
! network 10.1.0.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE !
set transform-set AES256/GCM/TRANSFORM ip route 0.0.0.0 0.0.0.0 138.0.0.1
set ikev2-profile DMVPN-IKEv2-PROFILE
On Premises Cisco CSR DMVPN Config ... Output summarised
Hub interface Tunnel0
crypto ikev2 proposal AES/GCM/256 description DMVPN
encryption aes-gcm-256 ip address 10.1.0.2 255.255.255.0
prf sha512 no ip redirects
group 19 ip mtu 1400
! ip nhrp authentication <NHRP_PASSWORD>
crypto ikev2 policy AES/GCM/256 ip nhrp map multicast dynamic
match fvrf any ip nhrp network-id 100
proposal AES/GCM/256 ip nhrp redirect
! ip tcp adjust-mss 1360
crypto ikev2 keyring DMVPN-KEYRING ip ospf authentication-key 7 <OSPF_PASSWORD>
peer ANY ip ospf network point-to-multipoint
address 0.0.0.0 0.0.0.0 ip ospf hello-interval 10
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel source GigabitEthernet1
! tunnel mode gre multipoint
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel key 100
description PSK Profile tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
identity local address 192.xxx.xxx.x interface GigabitEthernet1
authentication remote pre-share description Internet
authentication local pre-share ip address 192.xxx.xxx.x 255.255.255.0
keyring local DMVPN-KEYRING no ip redirects
dpd 40 5 on-demand no ip unreachables
! no ip proxy-arp
crypto ipsec security-association replay window-size 1024 negotiation auto
! !
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router ospf 10
mode transport router-id 10.1.0.2
! network 10.1.0.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE network 192.168.200.0 0.0.0.255 area 0
set transform-set AES256/GCM/TRANSFORM !
set ikev2-profile DMVPN-IKEv2-PROFILE ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

On the GCP CSR, check for the private network route from the on-premises
side(192.168.200.0/24)
csr1kv-gcp#show ip route | i 192.168.200.0
. . .
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:09:51, Tunnel0
On the on-premises CSR, check for the VPC inside network route (10.0.1.0/24)
. . .
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 00:40:08, Tunnel0
Check the DMVPN Next-Hop Resolution Protocol (NHRP) Status

csr1kv-gcp#show ip nhrp csr-mc-01#show ip nhrp
10.1.0.2/32 via 10.1.0.2 10.1.0.1/32 via 10.1.0.1
Tunnel0 created 5d14h, never expire Tunnel0 created 00:40:25, expire 00:08:20
Type: static, Flags: Type: dynamic, Flags: registered used nhop
NBMA address: 192.xxx.xxx.x NBMA address: 35.xxx.xxx.x
(Claimed NBMA address: 10.138.0.100)
Connect to the GCP test instance that was created earlier and ping to the on-premises private network
# gcloud compute ssh "csr-inside-vm“
shmcfarl@csr-inside-vm:~$ ping 192.168.200.30

Amazon Web Services –
Cisco CSR and DMVPN
AWS with Cisco CSR 1000v Support
• Amazon Web Services Marketplace + Cisco CSR:
• https://aws.amazon.com/marketplace/search/results?x=0&y=0&searchTerms=csr&page=1&ref_=nav_search_
box
• Cisco CSR for AWS Deployment

• DMVPN
https://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.
html
• Deployment https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws.html
• Cisco Live Session for AWS with Cisco CSR:

• https://www.ciscolive.com/global/on-demand-library/?search=brkarc-
2023#/session/1486155288098001AhER
• Transit VPC with CSR: http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKARC-2749.pdf
AWS to On Premises CSR – IPsec VPN
Example 1 BGP <> OSPF Redistribution
IPsec/IKEv2
Tunnel Mode Cisco
CSR1000v
VPC Network 52.xxx.xxx.x 169.254.11.178 .1 Private Network
172.31.0.0/16 192.168.200.0/24
169.254.11.177 192.xxx.xxx.x
VPC VPN
Router Gateway OSPF 10 Area 0
BGP AS65002
AWS CSR to On Premises CSR – IPsec VPN
Example 2 Public-side Network
172.16.1.0/24
IPsec/IKEv2
Cisco
Cisco Tunnel Mode CSR1000v
CSR1000v
VPC Network 52.xxx.xxx.x .1 Private Network
172.16.2.0/24 192.xxx.xxx.x 192.168.200.0/24
VPC
Router OSPF 10 Area 0
Hypervisor
OSPF
AWS CSR to On Premises CSR – DMVPN
Public-side Network
172.16.1.0/24
Cisco Cisco
VPC
CSR1000v CSR1000v
Router
VPC Network 52.xxx.xxx.x 192.xxx.xxx.x .1 Private Network
172.16.2.0/24 Spoke Hub 192.168.200.0/24
CSR Tunnel: CSR Tunnel:
10.1.0.2
OSPF 10 Area 0
10.1.0.4
Hypervisor
DMVPN
OSPF
192.168.200.0/24
172.16.2.0/16
AWS CLI: Create VPC, Subnets and Internet GW
Create a new AWS VPC (vpc)
# aws ec2 create-vpc --cidr-block 172.16.0.0/16
Create a new subnet in the VPC (this one will be used for the CSR’s ’outside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.1.0/24
Create another new subnet in the VPC (this one will be used for the CSR’s ‘inside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.2.0/24
Create a new AWS Internet Gateway (igw)

# aws ec2 create-internet-gateway
Attach the Internet gateway to the VPC

# aws ec2 attach-internet-gateway --vpc-id vpc-66a0a102 --internet-gateway-id igw-591fba3d
AWS CLI: Create Route Tables
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘outside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-aaa37dcd --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Associate the new routable with the ‘outside’ VPC subnet

# aws ec2 associate-route-table --subnet-id subnet-0c15b86b --route-table-id rtb-aaa37dcd
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘inside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table for the ‘inside’ subnet and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 192.168.200.0/24 --network-interface-id eni-af67db80
Associate the new route table with the ‘inside’ VPC subnet
# aws ec2 associate-route-table --subnet-id subnet-c617baa1 --route-table-id rtb-3741e750
Reference
AWS CLI: Create a Security Group/Rules
Create a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)
# aws ec2 create-security-group --group-name csr --description csr-rules --vpc-id vpc-66a0a102
Create a new security group rule for SSH to the CSR

# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 --protocol tcp --port 22 --cidr 0.0.0.0/0
Create a new security group rule for ICMP from the other CSRs (On Premises and GCP CSR [optional: Just showing the format for your use])
# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \
--ip-permissions '[{"IpProtocol": "icmp", "FromPort": -1, "ToPort": -1, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]
Create a new security group rule for ESP (IP 50) from the other CSRs
--ip-permissions '[{"IpProtocol": "50", "IpRanges": [{"CidrIp": ”192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE/NAT-T from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 4500, "ToPort": 4500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]’
Optional: You may want to create a security group just for the ’inside’ subnet that has
different rules than the one for the ‘outside’ subnet
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 192.168.200.0/24
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 172.16.2.0/24
AWS CLI: Run a new CSR Instance Using Previous Parameters
csr-create.json
{
"ImageId": "ami-99e5d0f9",
"InstanceType": "t2.medium",
"KeyName": "mc-aws-key", Create a CSR instance using the JSON file shown to the left
"NetworkInterfaces": [ # aws ec2 run-instances --cli-input-json file://csr-create.json
{
"DeviceIndex": 0, Create a tag/name and associate it with the CSR (Optional)
"Description": "Primary network interface", # aws ec2 create-tags --resources i-0f2a0ee857e9c2540 \
"Groups": [ --tags Key=Name,Value=csr-aws-01
"sg-65c39b03"
], Create a new External IP (EIP) allocation (or use an existing one)
"PrivateIpAddresses": [ # aws ec2 allocate-address
{ eipalloc-ab35cb96 vpc 52.xxx.xxx.x
"Primary": true,
"PrivateIpAddress": "172.16.1.10" Associate the EIP with the ’outside’ interface of the CSR (GigabitEthernet 1)
} # aws ec2 associate-address --allocation-id eipalloc-ab35cb96 \
], --network-interface-id eni-dd5bd6f2
"SubnetId": "subnet-0c15b86b"
}, Modify the ’inside’ subnet to disable source/destination checking
{ # aws ec2 modify-network-interface-attribute \
"DeviceIndex": 1, --network-interface-id eni-af67db80 \
"PrivateIpAddresses": [ --source-dest-check "{\"Value\": false}"
{
"Primary": true,
"PrivateIpAddress": "172.16.2.10" A note about NAT: If you plan to use the CSR for NAT
} operation, you must disable source/destination checking
],
"SubnetId": "subnet-c617baa1" on the outside CSR interface/subnet
}
] http://docs.aws.amazon.com/AmazonVPC/latest/UserG
} uide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCh
eck #CLMEL BRKCLD-3440 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Connect to the AWS CSR – Enable Interfaces
Connect to the new AWS-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# ssh -i "mc-aws-key.pem" ec2-user@52.xxx.xxx.x
csr-aws-01#configure terminal
csr-aws-01(config)#interface gigabitEthernet 2
csr-aws-01(config-if)#ip address dhcp
csr-aws-01(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
csr-aws-01#show ip interface brief
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Note: This can all be automated (along with the DMVPN configs) by creating AWS CloudFormation templates
AWS Cisco CSR DMVPN Config ... Output summarised
Spoke
crypto ikev2 proposal AES/GCM/256 interface Tunnel0
encryption aes-gcm-256 description DMVPN
prf sha512 ip address 10.1.0.4 255.255.255.0
group 19 no ip redirects
! ip mtu 1400
crypto ikev2 policy AES/GCM/256 ip nhrp authentication <NHRP_PASSWORD>
match fvrf any ip nhrp network-id 100
proposal AES/GCM/256 ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
! ip tcp adjust-mss 1360
crypto ikev2 keyring DMVPN-KEYRING ip ospf authentication-key 7 <OSPF_PASSWORD>
peer ANY ip ospf network point-to-multipoint
address 0.0.0.0 0.0.0.0 ip ospf hello-interval 10
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel source GigabitEthernet1
! tunnel mode gre multipoint
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel key 100
description PSK Profile tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
authentication local pre-share ip address dhcp
! no ip proxy-arp
! !
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router ospf 10
mode transport router-id 10.1.0.4
! network 172.16.2.0 0.0.0.255 area 2
crypto ipsec profile DMVPN-IPSEC-PROFILE network 10.1.0.0 0.0.0.255 area 0
set transform-set AES256/GCM/TRANSFORM !
set ikev2-profile DMVPN-IKEv2-PROFILE ip route 0.0.0.0 0.0.0.0 172.16.1.1
On Premises Cisco CSR DMVPN Config
Hub – Nothing ever changes on the hub for each example
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256 interface Tunnel0
prf sha512 description DMVPN
group 19 ip address 10.1.0.2 255.255.255.0
! no ip redirects
crypto ikev2 policy AES/GCM/256 ip mtu 1400
match fvrf any ip nhrp authentication <NHRP_PASSWORD>
ip nhrp map multicast dynamic
proposal AES/GCM/256
ip nhrp network-id 100
! ip nhrp redirect
crypto ikev2 keyring DMVPN-KEYRING ip tcp adjust-mss 1360
peer ANY ip ospf authentication-key 7 <OSPF_PASSWORD>
address 0.0.0.0 0.0.0.0 ip ospf network point-to-multipoint
pre-shared-key <PSK_PASSWORD_GOES_HERE> ip ospf hello-interval 10
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel mode gre multipoint
tunnel key 100
description PSK Profile
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
! no ip proxy-arp
negotiation auto
crypto ipsec security-association replay window-size 1024
!
! router ospf 10
! network 192.168.200.0 0.0.0.255 area 0
set transform-set AES256/GCM/TRANSFORM ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
set ikev2-profile DMVPN-IKEv2-PROFILE #CLMEL BRKCLD-3440 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107

On the on-premises CSR check the route for the AWS VPC network 172.16.2.0/24
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 00:11:41, Tunnel0
On AWS check for the route for the on-premises network (192.168.200.0/24)
csr-aws-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 6d17h, Tunnel0
Connect to an AWS instances and ping to the on-premises private network

[ec2-user@ip-172-16-2-192 ~]$ ping 192.168.200.30
Cisco
VPC Network
Cisco
CSR1000v CSR1000v
.30
VM
172.16.2.0/24
.1 Private Network
.192 .10 192.168.200.0/24
Spoke Hub
10.1.0.4 10.1.0.2
Hypervisor
OSPF
For Reference
Amazon Web
Services –
Marketplace-based
Launch Walk-thru
AWS Marketplace CSR Launch – Console (1)
AWS Launch CSR as an Instance – Console (1)
4
3
Microsoft Azure – Cisco
CSR and DMVPN
Azure to On Premises CSR – IPsec VPN
Example 1 BGP <> OSPF Redistribution
IPsec/IKEv2
Cisco
Vnet Subnet 40.xxx.xxx.x 169.254.11.178 Private Network
.1
10.10.0.0/16 192.168.200.0/24
VPN 169.254.11.177 192.xxx.xxx.x
Gateway OSPF 10 Area 0
BGP AS65002
Azure CSR to On Premises CSR – IPsec VPN
Example 2 Outside Subnet
10.10.0.0/24
IPsec/IKEv2
Cisco
Cisco Tunnel Mode CSR1000v
CSR1000v
Inside Subnet 40.xxx.xxx.x .1 Private Network
10.10.1.0/24 192.xxx.xxx.x 192.168.200.0/24
OSPF 10 Area 0
Hypervisor
OSPF
Azure CSR to On Premises CSR – DMVPN
Outside Subnet
10.10.0.0/24
Cisco
Cisco CSR1000v
CSR1000v
Inside Subnet 40.xxx.xxx.x 192.xxx.xxx.x .1 Private Network
10.10.1.0/24 Spoke Hub 192.168.200.0/24
10.1.0.2
OSPF 10 Area 0
10.1.0.6
Hypervisor
DMVPN
OSPF
192.168.200.0/24
10.10.1.0/24
Microsoft Azure with Cisco CSR 1000v
• Microsoft Azure Marketplace

• https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/cisco.cisco-csr-basic-template
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-
csr-1000v
• Cisco CSR 1000v with Azure Deployment
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b
_csr1000config-azure.html
Azure CLI: Create Resource Group, Networks,
Subnets
Create a new Azure Resource Group (rg)
# az group create --name multicloud-rg --location westus
Create a new public (external IP) IPv4 address to be used for the CSR’s ‘outside’ interface
# az network public-ip create --resource-group multicloud-rg --name csr-azure-01-eip --allocation-method static
Create a new virtual network (vnet) and a subnet to be used for the CSR’s ‘outside interface
# az network vnet create \
--resource-group multicloud-rg \
--name mc-csr-vnet \
--address-prefix 10.10.0.0/16 \
--subnet-name csr-outside \
--subnet-prefix 10.10.0.0/24
Create a new subnet for the CSR’s ‘inside’ interface and associate it with the vnet created above
--vnet-name mc-csr-vnet \
--name csr-inside \
Azure CLI: Create Route Tables
Create a new route table (rt) that will be used for the CSR’s ’outside’ subnet
# az network route-table create \
--name csr-outside-rt
Create a new route table that will used for the CSR’s ‘inside’ subnet
# az network route-table create \
--name csr-inside-rt
Create a new route table entry for the ‘inside’ subnet to reach the On Premises network (192.168.200.0) via the CSR’s IP (10.10.1.4)
# az network route-table route create \
--name csr-to-On Premises-route \
--route-table-name csr-inside-rt \
--address-prefix 192.168.200.0/24 \
--next-hop-type VirtualAppliance \
--next-hop-ip-address 10.10.1.4
Associate the ‘outside’ route table with the ‘outside’ subnet

# az network vnet subnet update \
--name csr-outside \
--route-table csr-outside-rt
Associate the ‘inside’ route table with the ‘inside’ subnet

# az network vnet subnet update \
--name csr-inside \
--route-table csr-inside-rt
Azure CLI: Create Network Security Group (NSG) Reference
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
# az network nsg create \
--name csr-nsg-outside
Create a new NSG rule to allow inbound SSH access to the CSR (can make it specific to an IP/Prefix)
# az network nsg rule create \
--nsg-name csr-nsg-outside \
--name SSHRule \
--priority 100 \
--source-address-prefixes 'Internet' \
--source-port-ranges '*' \
--destination-address-prefixes '*' \
--destination-port-ranges 22 \
--access Allow \
--protocol Tcp \
--direction inbound
Create a new NSG rule to allow inbound UDP 500 (IKE) traffic to the CSR (can make it specific to an IP/Prefix)
--name UDP-500 \
--priority 101 \
--access Allow \
--protocol Udp \
--direction inbound
Azure CLI: Create NSG Rule and NICs Reference
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
--name UDP-4500 \
--priority 102 \
--access Allow \
--protocol Udp \
--direction inbound
Create a new NIC to be used by the CSR’s ‘outside’ interface. Associate the NIC with the NSG, Subnet, Public IP & enable forwarding
# az network nic create \
--name csr-nic-g1 \
--subnet csr-outside \
--network-security-group csr-nsg-outside \
--ip-forwarding true \
--public-ip-address csr-azure-01-eip
Create a new NIC to be used by the CSR’s ‘inside’ interface. Associate the NIC with the NSG, Subnet and enable forwarding
# az network nic create \
--name csr-nic-g2 \
--subnet csr-inside \
--ip-forwarding true
Azure CLI: Run a new CSR Instance Using
Previous Parameters
Create a CSR VM using a Azure Marketplace image. Associate the VM with two the NICs created earlier.
# Note: The VM can be created with a large number of options to include SSH keys, image (BYOL, # of NICs), and size
# az vm create \
--name csr-azure-01 \
--admin-username csr-azure \
--admin-password <PASSWORD> \
--authentication-type password \
--image cisco:cisco-csr-1000v:16_6:16.6.120170804 \
--nics csr-nic-g1 csr-nic-g2 \
--size Standard_D2_v2
Connect to the Azure CSR – Enable Interfaces
Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# ssh csr-azure@40.xxx.xxx.x
csr-azure-01#configure terminal
csr-azure-01(config)#interface gigabitEthernet 2
csr-azure-01(config-if)#ip address dhcp
csr-azure-01(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
csr-azure-01#show ip interface brief
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager
Azure Cisco CSR DMVPN Config ... Output summarised
Spoke
interface Tunnel0
encryption aes-gcm-256
description DMVPN
prf sha512
ip address 10.1.0.6 255.255.255.0
group 19
no ip redirects
!
ip mtu 1400
crypto ikev2 policy AES/GCM/256
ip nhrp authentication <NHRP_PASSWORD>
match fvrf any
proposal AES/GCM/256
ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
!
crypto ikev2 keyring DMVPN-KEYRING
ip ospf authentication-key 7 <OSPF_PASSWORD>
peer ANY
ip ospf network point-to-multipoint
address 0.0.0.0 0.0.0.0
ip ospf hello-interval 10
pre-shared-key <PSK_PASSWORD_GOES_HERE>
tunnel source GigabitEthernet1
!
tunnel mode gre multipoint
crypto ikev2 profile DMVPN-IKEv2-PROFILE
tunnel key 100
description PSK Profile
match identity remote address 0.0.0.0
!
identity local address 40.xxx.xxx.x
interface GigabitEthernet1
authentication remote pre-share
description Internet
authentication local pre-share
ip address dhcp
keyring local DMVPN-KEYRING
no ip redirects
dpd 40 5 on-demand
no ip unreachables
!
no ip proxy-arp
crypto ipsec security-association replay window-size 1024
negotiation auto
!
!
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256
router ospf 10
mode transport
router-id 10.1.0.6
!
network 10.1.0.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE
network 10.10.1.0 0.0.0.255 area 3
set transform-set AES256/GCM/TRANSFORM
!
ip route 0.0.0.0 0.0.0.0 10.10.0.1
On Premises Cisco CSR DMVPN Config... Output summarised
Hub - Nothing ever changes on the hub for each example
encryption aes-gcm-256 interface Tunnel0
prf sha512 description DMVPN
group 19 ip address 10.1.0.2 255.255.255.0
! no ip redirects
crypto ikev2 policy AES/GCM/256 ip mtu 1400
match fvrf any ip nhrp authentication <NHRP_PASSWORD>
proposal AES/GCM/256 ip nhrp map multicast dynamic
!
ip nhrp redirect
crypto ikev2 keyring DMVPN-KEYRING ip tcp adjust-mss 1360
peer ANY ip ospf authentication-key 7 <OSPF_PASSWORD>
address 0.0.0.0 0.0.0.0 ip ospf network point-to-multipoint
pre-shared-key <PSK_PASSWORD_GOES_HERE> ip ospf hello-interval 10
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel mode gre multipoint
description PSK Profile tunnel key 100
match identity remote address 0.0.0.0
!
! no ip proxy-arp
!
!
router ospf 10
! network 192.168.200.0 0.0.0.255 area 0
set transform-set AES256/GCM/TRANSFORM ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

On the on-premises CSR check the route for the Azure Vnet 10.10.1.0/24
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:19:15, Tunnel0
On Azure check for the route for the on-premises network (192.168.200.0/24)
csr-azure-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:17:57, Tunnel0
Connect to an Azure instance and ping to the on-premises private network

shmcfarl@AzTestVm:~$ping 192.168.200.30
Cisco .30
Inside Subnet Cisco
CSR1000v CSR1000v VM
10.10.1.0/24
.1 Private Network
.5 .4 192.168.200.0/24
Spoke Hub
10.1.0.6 10.1.0.2
Hypervisor
OSPF
For Reference
Azure – Marketplace-
based Launch Walk-
thru
Azure Marketplace/Resource Search
Azure Marketplace
– There are multiple CSR types to pick from
Azure Marketplace
Deployment Flow
1 2 3
Linking DMVPN Sites
DMVPN – Enable Dynamic Multicloud Networking
Cisco DMVPN
VNet Network Spoke
10.10.1.0/24
Cisco
CSR1000v
Cisco
CSR1000v
Private Network
Spoke Hub
192.168.200.0/24
VPC Network
172.16.2.0/24 BGP/OSPF/EIGRP
Cisco
CSR1000v
DMVPN
VPC Network
10.0.1.0/24 Spoke
Cisco
CSR1000v
General Guidelines for DMVPN Between Clouds
• Set the VPC routes for each site
gcloud compute routes create inside-to-aws \
--network=csr-inside-network \
gcloud compute routes create inside-to-azure \

--network=csr-inside-network \
• Set the firewall/security groups/network security groups for each site/protocol

Create a very specific per-site route (AWS example allowing UDP500 from each cloud provider public IP)
aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \
--ip-permissions '[{"IpProtocol": " ", "FromPort": , "ToPort": , "IpRanges": [{"CidrIp": " .x.x.x/32"},
{"CidrIp": " .x.x.x/32"}, {"CidrIp": " .x.x.x/32"}]}]’
Alternatively, you can open it up (Azure example)

az network nsg rule create \
--name UDP-4500 \
--priority 102 \
--access Allow \
--protocol Udp \
--direction inbound
Routing Example – All Sites
• For spoke-to-spoke direct routing with DMVPN/NHRP:
• ‘ip nhrp redirect’ on the hubs IA - OSPF inter area
• ‘ip nhrp shortcut’ on the spokes % - next hop override
Hub On Premises CSR Spoke – Amazon Web Services CSR
csr-mc-01#show ip route ospf
csr-aws-01#show ip route ospf
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 02:40:45, Tunnel0
O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.1/32 [110/1000] via 10.1.0.1, 02:40:45, Tunnel0
O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.4/32 [110/1000] via 10.1.0.4, 01:18:49, Tunnel0
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.6/32 [110/1000] via 10.1.0.6, 00:56:19, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:59:01, Tunnel0
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:55:34, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:14, Tunnel0
172.16.0.0/24 is subnetted, 1 subnets
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 01:21:32, Tunnel0
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 01:18:49, Tunnel0
Spoke – Google Cloud Platform CSR Spoke – Azure CSR

csr1kv-gcp#show ip route ospf csr-azure-01#show ip route ospf
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 02:43:14, Tunnel0 O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 01:21:14, Tunnel0 O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:58:47, Tunnel0 O 10.1.0.2/32 [110/1000] via 10.1.0.2, 00:58:44, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:00, Tunnel0 O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
172.16.0.0/24 is subnetted, 1 subnets 172.16.0.0/24 is subnetted, 1 subnets
O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 01:21:14, Tunnel0 O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 02:43:14, Tunnel0 O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:58:44, Tunnel0
NHRP Example – Hub/Spoke Spoke – Azure CSR
csr-azure-01#show ip nhrp
Hub On Premises CSR 10.0.1.0/24 via 10.1.0.1
Tunnel0 created 00:06:26, expire 00:03:32
csr-mc-01#show ip nhrp
Type: dynamic, Flags: router rib nho
10.1.0.1/32 via 10.1.0.1
NBMA address: 35.xxx.xxx.x
Type: dynamic, Flags: registered used nhop
10.1.0.1/32 via 10.1.0.1
Type: dynamic, Flags: router nhop rib nho
10.1.0.4/32 via 10.1.0.4
10.1.0.2/32 via 10.1.0.2
Tunnel0 created 00:21:28, never expire
Type: static, Flags:
10.1.0.6/32 via 10.1.0.6
10.1.0.4/32 via 10.1.0.4
Type: dynamic, Flags: router nhop rib nho
csr-mc-01#show ip nhrp multicast
10.10.1.0/24 via 10.1.0.6
I/F NBMA address
Tunnel0 35.xxx.xxx.x Flags: dynamic (Enabled)
Type: dynamic, Flags: router unique local
NBMA address: 10.10.0.4
(no-socket)
172.16.2.0/24 via 10.1.0.4
Spoke – Azure VM Tunnel0 created 00:07:19, expire 00:02:40
Type: dynamic, Flags: router rib nho
shmcfarl@AzureTestVm:~$ traceroute 10.0.1.3 NBMA address: 52.xxx.xxx.x
traceroute to 10.0.1.3 (10.0.1.3), 30 hops max, 60 byte packets (Claimed NBMA address: 172.16.1.10)
1 10.10.1.4 (10.10.1.4) 1.220 ms 1.192 ms 1.328 ms csr-azure-01#show ip nhrp multicast
2 10.0.1.3 (10.0.1.3) 25.794 ms * 25.782 ms I/F NBMA address
Tunnel0 192.xxx.xxx.x Flags: nhs (Enabled)
Split-Tunnelling/Routing
Options
Split-Tunnel/Routing Options
• All three public cloud providers allow for either split-tunnelling or forced/direct routing
• Split-tunnelling:
• Public cloud resources (instances/VMs, container clusters) will use the default VPC gateway for non-On
Premises routes
• Public cloud resources will use the On Premises-specific routes advertised by the CSR
• Forced/Direct routing – All public cloud resources will use the VPN connection as their default
route for ALL traffic (forces traffic through the On Premises site)
External/NAT
Cisco
Google Cloud VPN
Compute
10.0.0.1 CSR1000v
Engine Routing 35.xxx.xxx.x
2 1
10.0.0.5
VPC Subnetwork
GW 192.xxx.xxx.x
BGP
Google Cloud Router

CSR High Availability
Public Cloud Provider – CSR High-Availability
• Common challenge with all public cloud provider is that there is not true layer 2
support on a VPC subnet – this prevents FHRPs from working properly
• Must setup a monitoring/tracking feature to watch for CSR interface/instance failure
and adjust the VPC route table to point to 2nd CSR inside interface
• AWS CSR High-Availability:
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws/b_csraws_c
hapter_0100.pdf
• Azure CSR High-Availability:

• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-
azure/b_csr1000config-azure_chapter_0110.html
Automation Challenges
Automating the Multicloud Network
• Challenges:
• Different toolsets for different jobs (Ansible, Python, Bash scripts, Terraform, etc..)
• Different toolsets for different clouds (Heat for OpenStack, CloudFormation for AWS, Deployment Manager for GCP,
Azure Automation)
• Different toolsets for different vendor products (Cisco NSO, CloudCentre, Prime, YANG development kit, etc..)
• There is no silver bullet - Start simple:

• Use what your team knows – Perform a gap analysis on what you have against what you need
• Initially, automate the things that hurt a lot to do by hand and that change frequently – I use free tools but that doesn’t
mean the process is free 
• Native Tools: It’s safe to use the cloud provider’s native automation toolset (e.g., AWS CloudFormation) when that is the
only provider you need to deal with
• Abstracted Tools: When you are dealing with multiple providers to include on-premises providers (e.g., VMware vSphere
or Microsoft Azure Stack), it makes life easier to abstract away from native cloud provider tool sets and use something
like Terraform and/or combo of tools
• Full Stack Tools: When you want to stop pulling your hair out and you want to build full ‘stacks’ in nearly any environment,
move to something that can treat the environment as a whole – Cisco CloudCentre:
https://www.cisco.com/c/en/us/products/cloud-systems-management/cloudcenter/index.html
Amazon CloudFormation
• https://aws.amazon.com/cloudformation/
• Template-based (JSON/YAML) – Build a stack(s) from a template file
• Sometimes you need to run more than one stack (in order) to get what you need
• Example template: https://github.com/shmcfarl/multicloud/tree/master/aws/cloudformation
Google Cloud Platform – Deployment Manager
• https://cloud.google.com/deployment-manager/
• Configuration files (YAML), Templates (Python/Jinja2), Schema files (JSON)
• Sometimes you need to run more than one stack (in order) to get what you need
• Example templates: https://github.com/shmcfarl/multicloud/tree/master/gcp/deployment-
manager
Microsoft Azure Automation/Resource Manager
• https://azure.microsoft.com/en-us/services/automation/
• Runbooks (create graphically, PowerShell, Python)

• Read and select these carefully: https://docs.microsoft.com/en-
us/azure/automation/automation-runbook-types
• Resource Manager: https://docs.microsoft.com/en-us/azure/azure-resource-

manager/resource-group-overview
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v
• Example template:
https://github.com/shmcfarl/multicloud/blob/master/azure/resource-manager/az-
arm-csr-cleaned.json
Call APIs Directly
• Google Cloud Platform: https://cloud.google.com/compute/docs/reference/latest/
• Amazon Web Services:

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welcome.html
• Microsoft Azure: https://docs.microsoft.com/en-us/rest/api/
Google VPN – Creating
Google VPN, Router,
IPsec, BGP via REST
APIs
Google Cloud API – Creating GCP Cloud
VPN/Routers
• Assumptions/environment:
• Understand how to authenticate to GCP APIs:
https://cloud.google.com/docs/authentication/
• In this example, the Paw application was used to craft GET, POST and PATCH calls
• Some configurations have been sanitised for security purposes
• Have On Premises Cloud infrastructure deployed and a CSR/ASR configured (can
be done after GCP side is deployed)
• In this example, the configuration will be deployed against the OpenStack use
case discussed in the earlier slides
• In this example, the default network created by GCP will be used
• Note: gcloud has VERY long delays in commands if you have IPv6 enabled on your
local machine – set to “link-local” mode on your Mac
Reference Topology for GCP API Example
OSPF<>BGP Redistribution
IPsec/IKEv2
Tunnel Mode
169.254.0.6
Default Network 35.yyy.yyy.y .11 Private Network
10.138.0.0/20 192.yyy.yyy.y 172.16.0.0/24
Google
Cloud
VPN OSPF 10 Area 0
169.254.0.5
BGP AS65000 BGP AS65003 On Premises Cloud

Google Cloud Router Routes this side should see:
10.138.0.0/20

172.16.0.0/24
GCP API (1) – Create VPN GW and External IP
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways HTTP/1.1
Authorization: Bearer XXXX POST: Create VPN
Content-Type: application/json; charset=utf-8
Host: www.googleapis.com
Connection: close
Gateway
Content-Length: 138
{
"name": "csr-gcp-os-aio-gw",
"network": "projects/<gcp_project_number>/global/networks/default",
"region": "projects/<gcp_project_number>/regions/us-west1"
}
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses HTTP/1.1

Authorization: Bearer XXXX
POST: Create External IP
Connection: close Address
Content-Length: 29
{
"name": "gcp-to-os-dmz"
}
GET: Get the External IP

GET /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses/gcp-to-os-dmz HTTP/1.1
Connection: close Address
RESPONSE - SUMMARIZED:
"name": "gcp-to-os-dmz",
"description": "",
"address": ”35.yyy.yyy.y",
"status": "RESERVED",
GCP API (2) – Create Forwarding Rules
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1
POST: Create
Connection: close Forwarding rule for ESP
Content-Length: 257
{
"name": "csr-gcp-os-aio-rule-esp",
"IPProtocol": "ESP",
"IPAddress": "35.yyy.yyy.y",
"region": "projects/<gcp_project_number>/regions/us-west1",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}

POST: Create
Connection: close Forwarding rule for UDP
500
Content-Length: 278
{
"name": "csr-gcp-os-aio-rule-udp500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "500"
}

POST: Create
Connection: close Forwarding rule for UDP
4500
Content-Length: 280
{
"name": "csr-gcp-os-aio-rule-udp4500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "4500"
}
GCP API (3) – Create Cloud Router and BGP Session
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/routers HTTP/1.1
Connection: close
Content-Length: 574
{
"name": "csr-gcp-os-bgp-rtr",
"bgp": {
"asn": "65000"
},
"interfaces": [
{
"name": "if-csr-gcp-os-bgp-rtr-02",
"linkedVpnTunnel": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels/csr-gcp-os-aio-gw-tunnel-1",
"ipRange": "169.254.0.5/30"
}
],
"bgpPeers": [
{
"name": "csr-gcp-os-bgp-peer",
"interfaceName": "if-csr-gcp-os-bgp-rtr-02",
"ipAddress": "169.254.0.5",
"peerIpAddress": "169.254.0.6",
"peerAsn": "65003"
}
],
"network": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/global/networks/default"
}
POST: Create Cloud Router, BGP session and link to the Cloud VPN tunnel
GCP API (5) – Create Cloud VPN Tunnel

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels HTTP/1.1
Connection: close
Content-Length: 417
{
"name": "csr-gcp-os-aio-gw-tunnel-1",
"sharedSecret": " <pre-shared-password-goes-here> ",
"router": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/routers/csr-gcp-os-bgp-rtr",
"peerIp": "192.yyy.yyy.y",
"ikeVersion": "2",
"targetVpnGateway": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}
POST: Create a Cloud VPN tunnel and associated it with the Cloud router
Summary
• Cisco Multicloud Solutions: https://www.cisco.com/c/en/us/solutions/cloud/multicloud-portfolio.html
• Public cloud native IPsec VPN support is good, but it is always point-to-point, does not have consistent support and lacks
network-rich features - It may be good enough for your initial use case(s)
• If you have deployed or want to deploy SD-WAN, adding in your public cloud sites into your overall SD-WAN design can
reap many operational and cost benefits
• If you have an existing WAN/Branch deployment of DMVPN, adding spokes at public cloud site(s) can help optimize traffic
flow (no hair-pinning), enable rich network features at the public cloud site and allow for a consistent technical and
operation experience
• Multicloud between multiple public cloud providers and on-premises look like distinctly separate hybrid cloud deployments
but..
• You have to take into consideration:

• Team knowledge of public cloud operations, tools, automation
• Cross cloud tools and automation
• Diversity of network designs, protocols, security
• Multi-region designs
• Availability zones within and across providers
Q&A
#CLMEL
Complete Your Online Session Evaluation
• Give us your feedback and receive
a complimentary Cisco Live 2019
Power Bank after completing the
overall event evaluation and 5
session evaluations.
• All evaluations can be completed
via the Cisco Live Melbourne
Mobile App.
• Don’t forget: Cisco Live sessions
will be available for viewing on
demand after the event at:
https://ciscolive.cisco.com/on-demand-
library/
Thank you
#CLMEL
Reference
Application Deployment
GKE, Cloud VPN, Cloud
Router and an On
Premises CSR
Deployment with
Dynamic Routing (IP
Alias)
Google Container Engine (GKE) – Dynamic
Routing
• Prior to the IP alias feature, GKE clusters did not advertise their IP ranges via the GCP
Cloud Router (BGP) service: https://cloud.google.com/container-engine/docs/ip-aliases
• IP alias and self-directed alias ranges, cluster IP ranges and service IP ranges can all be
enabled via REST, gcloud and the GKE console
# gcloud beta container clusters create gke-cls-istio \

> --enable-ip-alias \
> --create-subnetwork name=gke-istio-subnetwork
GKE – Dynamic Routing with On Premises CSR
External/
Google Container
NAT
Cluster (GKE)
Google Private Network

10.0.0.2 Cloud Cisco
Pods cbr0
eth0 VPN CSR1000v 192.168.100.0/24
10.56.0.0/24 10.0.0.1
Routing 35.xxx.xxx.x .1
10.0.0.3 VPC
Pods cbr0
eth0 Subnetwork
10.56.1.0/24 GW 192.xxx.xxx.x
Pods cbr0
10.0.0.4 BGP 192.168.100.20
eth0
10.56.2.0/24
Hypervisor
Default Network: Google Cloud
- Subnetwork: Router
- Nodes: 10.0.0.0/22
- Container Range: 10.56.0.0/14
- Services Range: 10.0.16.0/20
Google Container Engine - Setup Google Container
Cluster (GKE)
Create a basic GKE cluster with IP alias enabled 10.0.0.2

# gcloud beta container clusters create gke-cls-istio \ Pods cbr0
eth0
> --enable-ip-alias \ 10.56.0.0/24
> --create-subnetwork name=gke-istio-subnetwork
10.0.0.3
Pods cbr0
eth0
Get a list of the nodes 10.56.1.0/24
# kubectl get nodes
NAME STATUS AGE VERSION 10.0.0.4
Pods cbr0
gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6 eth0
10.56.2.0/24
gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6
gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6
Check the IP ranges of the new subnetwork “gke-istio-subnetwork”

Default Network:
# gcloud compute networks subnets describe gke-istio-subnetwork | grep ipCidrRange - Subnetwork:
ipCidrRange: 10.0.0.0/22 - Nodes: 10.0.0.0/22
- ipCidrRange: 10.56.0.0/14 - Container Range: 10.56.0.0/14
- ipCidrRange: 10.0.16.0/20 - Services Range: 10.0.16.0/20
Google Container Engine – Node/Pod IP Verification
NAME STATUS AGE VERSION Google Container

gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6
Cluster (GKE)
gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6
gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6
Using the node list from above, check the IPs assignments of each node
# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-zgdq | grep 'InternalIP\|PodCIDR'
InternalIP: 10.0.0.2 10.0.0.2
Pods cbr0
PodCIDR: 10.56.0.0/24 eth0
10.56.0.0/24
# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-6lsc | grep 'InternalIP\|PodCIDR' 10.0.0.3

InternalIP: 10.0.0.3 Pods cbr0
eth0
PodCIDR: 10.56.1.0/24 10.56.1.0/24
10.0.0.4
# kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-x04p | grep 'InternalIP\|PodCIDR' Pods cbr0
eth0
InternalIP: 10.0.0.4 10.56.2.0/24
PodCIDR: 10.56.2.0/24
GKE/GCP and On Premises CSR Dynamic
Routing
Get the advertised route list from the GCP Cloud Router
# gcloud compute routers get-status csr-gcp-vm-bgp-rtr
. . .
result:
. . .
bgpPeerStatus: Google
- advertisedRoutes: Cloud
- destRange: 10.0.16.0/20
VPN
kind: compute#route 169.254.0.2
nextHopIp: 169.254.0.1
priority: 100
- destRange: 10.56.0.0/14
kind: compute#route Cisco CSR
priority: 100
- destRange: 10.0.0.0/22 169.254.0.1 BGP
kind: compute#route
priority: 100
Check the BGP routes on the On Premises CSR
csr-gcp-01#show ip route bgp Google Cloud
. . . Router
B 10.0.0.0/22 [20/100] via 169.254.0.1, 00:00:04
B 10.0.16.0/20 [20/100] via 169.254.0.1, 00:00:04
B 10.56.0.0/14 [20/100] via 169.254.0.1, 00:00:04
GKE and CSR Routing/Access Verification
From a VM at the On Premises network (192.168.100.0/24), ping a GKE nodes IP and the cbr0 interface on that node
[root@k8s-m-01 ~]# ip a
. . .
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 Google Container
link/ether 00:50:56:bc:4b:91 brd ff:ff:ff:ff:ff:ff Cluster (GKE)
inet 192.168.100.20/24 brd 192.168.100.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::50de:b58f:8dc8:2fd5/64 scope link
valid_lft forever preferred_lft forever
10.0.0.2
Pods cbr0
[root@k8s-m-01 ~]# ping 10.0.0.2 eth0
10.56.0.0/24
64 bytes from 10.0.0.2: icmp_seq=1 ttl=63 time=25.4 ms 10.0.0.3
64 bytes from 10.0.0.2: icmp_seq=2 ttl=63 time=24.3 ms Pods cbr0
eth0
10.56.1.0/24
10.0.0.4
[root@k8s-m-01 ~]# ping 10.56.0.1 Pods cbr0
eth0
PING 10.56.0.1 (10.56.0.1) 56(84) bytes of data. 10.56.2.0/24
GKE Pod Routing/Access Verification
Deploy an nginx pod
# kubectl run my-nginx --image=nginx --port=80
deployment "my-nginx" created
# kubectl get pods

NAME READY STATUS RESTARTS AGE
my-nginx-4293833666-1jbjl 1/1 Running 0 14s
Find the IP addres of the pod

# kubectl describe pods my-nginx-4293833666-1jbjl | grep IP:
IP: 10.56.0.5
Ping the IP address of the pod from the On Premises VM

[root@k8s-m-01 ~]# ping 10.56.0.5
curl the nginx pod

[root@k8s-m-01 ~]# curl -o /dev/null -s -w "%{http_code}\n" http://10.56.0.5
200
Google Container Engine
• Deploy Pods
Deploy NGINX as a test

# kubectl run my-nginx --image=nginx --replicas=3 --port=80
deployment "my-nginx" created
Check to make sure the pods are running

# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-nginx-858393261-7x8mp 1/1 Running 0 6s
my-nginx-858393261-rt9sp 1/1 Running 0 6s
my-nginx-858393261-vhq6f 1/1 Running 0 6s
Get the IPv4 address for each pod

# kubectl describe pods my-nginx-858393261-7x8mp | grep IP:
IP: 10.28.2.18
# kubectl describe pods my-nginx-858393261-rt9sp | grep IP:
IP: 10.28.3.36
# kubectl describe pods my-nginx-858393261-vhq6f | grep IP:
IP: 10.28.1.29
#CLMEL TECSEC-2723 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 178
Complete Your Online Session Evaluation
• Give us your feedback and receive a
complimentary Cisco Live 2019 Power
Bank after completing the overall event
evaluation and 5 session evaluations.
• All evaluations can be completed via the
Cisco Live Melbourne Mobile App.
• Don’t forget: Cisco Live sessions will be
available for viewing on demand after
the event at:
https://ciscolive.cisco.com/on-demand-library/
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you
#CLMEL
#CLMEL

BRKCLD-3440 (2019)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKCLD-3440 (2019)

Uploaded by

Copyright:

Available Formats

#CLMEL

• Be smart – Know IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff

router bgp 65002

• IGP timers, configuration best practices

• Common network transport ingredients for hybrid and multicloud:

• Common network endpoint options:

• M&A may dictate public cloud provider preference (for a time)

• Regional cloud provider access

• Feature disparity between providers, regions and/or services

• Per-project service requirements

users/applications connect Amazon Amazon

to Cloud Service Provider

(CSP) public endpoints Internet Enterprise

No ‘traditional’ IPsec VPN

Google Cloud Router

VPC Subnet(s) Private Network(s)

Transit VPC: Cisco vEdge + Per-VPC vEdge

VPC Subnet(s) Private Network(s)

VPC Subnet(s) Private Network(s)

Transit VPC: Cisco CSR + Per-VPC CSR

VPC Subnet(s) Private Network(s)

Azure VPN vEdge

This replaces this

Transit VPC Transit VPC

VPC Subnet(s) Private Network(s)

Cisco SD-WAN + Some Combo of Colocation/peering

Inline Services Winner

Managed Services Winner

Time to Provision Winner

Location Availability Winner

App App Infrastructure Agnostic

Hypervisor Performance Elasticity

Public Cloud Provider Native VPN Services

• Amazon Web Services (AWS):

• OpenStack public cloud goodness: https://www.openstack.org/passport

Public Cloud Provider Native IPsec VPN Service

VPC Network SD-WAN

Hub FHRP Private Network

Google Cloud Platform – VPN Gateway

Routes this side should see:

Create an external IP to use for the VPN

Capture the external IP address

# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \

# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \

Cisco CSR Route Information

Dual Cisco CSR Design OSPF 10 Area 0

vSphere Distributed vSwitch (DVS)

Routes this side should see:

On Premises VM traceroutes via HSRP Active CSR (192.168.100.2)

HSRP Active CSR Route to GCP Default Network (10.138.0.0)

HSRP Standby CSR Route to GCP Default Network (10.138.0.0)

Failure Scenario 1 – HSRP Primary CSR VM Reload

On Premises VM traceroutes via HSRP Newly Active CSR (192.168.100.3)

Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path

Reference Cisco CSR Config – Secondary

VPC Network 52.xxx.xxx.x 169.254.11.178 .1 Private Network

Routes this side should see:

Create VPN Gateway and set the AWS BGP ASN

Attach VPN Gateway to the VPC

Create a new VPN connection

Enable route propagation for the VPC

Permit SSH and ICMP

Verify Routing and Reachability

Connect to an AWS instance and ping to the on-premises private network