DGTL-BRKCLD-3440-Multicloud Networking - Design and Deployment

#CiscoLive
Multicloud Networking –
Design and Deployment
Shannon McFarland – CCIE#5245
Distinguished Engineer
Cloud CTO
@eyepv6
DGTL-BRKCLD-3440
#CiscoLive
Agenda
• Level Set: Multicloud Networking Overview
• Getting Started: Cloud Provider VPN Services
• Beyond the Basics:
• Using Cisco SD-WAN for Cloud Connectivity
• Common Design Challenges
• Automation of Cloud Deployments
• Conclusion
#CiscoLive DGTL-BRKCLD-3440 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Disclaimer
• This is a 3000 level talk - It is ”advanced” only because of the large number of combined technologies and skills
involved
• You will need to know the basics of:

• How public cloud networking is defined
• IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff
• Dead Peer Detection
• IPsec SA lifetimes
• IPsec SA replay window-size
• Perfect Forward Secrecy (PFS)
• BGP timers, Local Preference, MED, inbound soft reset (check if cloud provider supports dynamic inbound soft reset)
• IGP timers, configuration best practices
• HSRP timers, tracking
• You won’t learn security, routing, HA, performance best practices
• The concepts introduced here are NOT the only way to get this stuff done
Multicloud Networking
Overview
Hybrid vs Multicloud Networking
• Hybrid Cloud Networking = Network transport from on-premises to a single public cloud provider
• Multicloud Networking = Network transport from on-premises to multiple public cloud providers and/or between multiple
public cloud providers
• The technologies used can be identical for every connection or they can be per-provider, per-region, per-project, etc..
• Common network transport ingredients for hybrid and multicloud:

• Encryption (IPsec/IKEv1/IKEv2, SSL, PKI)
• Routing (Static, BGP and with supported public cloud-hosted routers: OSPF, EIGRP)
• Tunneling (IPsec tunnel mode, GRE, mGRE, MPLS, segment routing, etc..)
• Common network endpoint options:

• Native VPN (IPsec over Internet) using public cloud provider services that connect to on-premises router/firewall
• Commercial/Open Source VPN platform hosted on the public cloud provider connecting to an on-premises router/firewall
• Colocation/Direct Peering: Service from public cloud provider to on-premises via a 3rd party colo facility
• Google Cloud Platform Dedicated Interconnect/Direct Peering/Carrier Peering: https://cloud.google.com/interconnect/
• Amazon Web Services Direct Connect/PrivateLink: https://aws.amazon.com/directconnect/
• Microsoft Azure ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
Why Would You Use Multiple Cloud Providers?
• Cloud provider high availability
• M&A may dictate public cloud provider preference (for a time)
• Regional cloud provider access
• Feature disparity between providers, regions and/or services
• Per-project service requirements
Extending On-
Premises Private
Cloud to a Public
Cloud
Internet Over-the-Top (OTT)
• Enterprise CSP-
users/applications connect Amazon Amazon

published
service
to Cloud Service Provider ECR S3 endpoints
(CSP) public endpoints Internet Enterprise

Data
Center
and/or public IPs of AZ: us-west-2b Gateway Edge

EIP
Internet
NAT GW 2 172.16.1.0/24
applications
Public Subnet 2
VPC
pod Router Campus
172.16.4.0/24 Enterprise
Enterprise
No ‘traditional’ IPsec VPN Application Private Subnet 2 Site

•
Region us-west-2
• TLS/SSL capable
• Can be at odds with
Enterprise InfoSec policies
Cloud Service Provider - Native IPsec VPN
Service
Cisco ASR,
CSR, ISR
Default Network IPsec/IKEv2 Private Network
10.138.0.0/20
Google
BGP
Cloud
VPN
Google Cloud Router
Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
Google
Cloud VPN
VPC Network
10.138.0.0/20
Private Network
Google Cloud Router 172.16.0.0/24
BGP/OSPF/EIGRP
VPC Network On-Premises Private Cloud
172.31.0.0/16 As the number of these
VPC VPN
connections increase and/or
Router Gateway
change frequently... You can see
where this is going
Stepping into Multicloud Networking
Multiple Native IPsec VPN Services
Google
Cloud VPN
VPC Network
10.138.0.0/20
Private Network
Google Cloud Router 172.16.0.0/24
BGP/OSPF/EIGRP
VPC Network On-Premises Private Cloud
172.31.0.0/16 As the number of these
VPC VPN
connections increase and/or
Router Gateway
change frequently... You can see
where this is going
Moving Away From Native VPN Services
What Conditions Cause a Change in Design?
• Operational consistency
• You need to extend your on-premises IGP (OSPF/EIGRP) into the public
cloud
• If on-premises routers/firewalls are behind NAT – Check for provider
support of NAT-T
• You need different IPsec/IKE configurations than what the provider offers
• You need SSL-based VPNs
• You need MPLS VPN
• QoS, specific network monitoring (IP SLA, NetFlow), Enterprise toolsets for
configuration and monitoring
IPsec VPN Performance - There are MANY
Factors Involved
• MTU, Internet vs DX/ER/Interconnect, ingress/egress, ECMP, CEF algorithm, etc..
• AWS Site-to-Site VPN: https://aws.amazon.com/vpn/faqs/#AWS_Site-to-
Site_VPN_connectivity
• Each S2S VPN has two tunnels, each can do 1.25 Gbps
• Transit GW: https://aws.amazon.com/transit-gateway/faqs/ - ECMP up to 50 Gbps
• Azure Site-to-Site VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-

gateway-about-vpngateways
• Depending on Gateway SKU and VPN type - Can go up to 10 Gbps aggregate throughput per
GW https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-
vpngateways#gwsku
• GCP Site-to-Site VPN: https://cloud.google.com/vpn/docs/concepts/topologies

• Each Cloud VPN tunnel supports up to 3 Gbps:
https://cloud.google.com/vpn/docs/concepts/overview#network-bandwidth
Let’s Backup
Starting Simple
• Public Cloud Provider Native IPsec VPN Service
IPsec/IKEv2 eBGP<>IGP Redistribution

Tunnel Mode
BGP AS65003
VPC Network Private Network
10.138.0.0/20 Google
172.16.0.0/24
Cloud
VPN
BGP/OSPF/EIGRP
BGP AS65000
On-Premises
Google Cloud Router
Add More On-Premises Stuff On-Premises Tenant 1
Public Cloud Provider Native IPsec VPN Service
Router
Private Network
192.168.100.0/24
BGP AS65002
BGP/OSPF/EIGRP
VPC Network
10.138.0.0/20 Google
Cloud Router
VPN
Private Network
BGP AS65000 BGP AS65003 172.16.0.0/24
Google Cloud Router
BGP/OSPF/EIGRP
Routes this side should see: On-Premises Tenant 2
172.16.0.0/24
Routes this side should see:
192.168.100.0/24
10.138.0.0/20
On-Premises Physical/Virtual
Public Cloud Provider Native IPsec VPN Service
Physical Router
ASR 1000
Private Network
192.168.yyy.0/24
VPC Network
10.138.0.0/20 Google
Cloud
VPN
Private Network
ASA Firewall
172.16.yyy.0/24
Google Cloud Router
Physical Firewall
VPN over the Internet vs Direct
Connect/ExpressRoute/Dedicated Interconnect
VPN over the Internet Direct/Express/Dedicated
Throughput Winner
QoS Winner
Latency Winner
Inline Services Winner
Managed Services Winner
Cost Winner
Time to Provision Winner
Flexibility Winner
Location Availability Winner
Cisco Cloud Services Router (CSR) 1000V
Cisco IOS XE Software in a Virtual Appliance Form-Factor
Software
CSR 1000V
• Familiar IOS XE software with ASR1000 and ISR4000
App App Infrastructure Agnostic

• Runs on x86 platforms
OS OS • Supported Hypervisors: VMware ESXi, Linux KVM, Citrix Xen, Microsoft
Hyper-V, Cisco NFVIS and CSP2100
• Supported Cloud Platforms: Amazon AWS, Microsoft Azure, and Google
Virtual Switch Cloud Platform
Hypervisor Performance Elasticity

• Available licenses range from 10 Mbps to 10 Gbps
CPU footprint ranges from 1vCPU to 8vCPU
Server •
License Options
• Term based 1 year, 3 year or 5 year
https://www.youtube.com/playlist?list=PLCi • Smart License enabled
TBLSYkcoTUS6b4MFthdvhDrseo6MeN
Programmability
• NetConf/Yang, RESTConf, Guest Shell and SSH/Telnet
Reference
Public Cloud Provider Native VPN Services

The Big Three
• Google Cloud Platform (GCP):
• VPN: https://cloud.google.com/compute/docs/vpn/overview
• Dedicated Interconnect: https://cloud.google.com/interconnect/
• Amazon Web Services (AWS):
• VPN: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html
• Direct Connect: https://aws.amazon.com/directconnect/
• Microsoft Azure:
• VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/
• ExpressRoute: https://azure.microsoft.com/en-us/services/expressroute/
• OpenStack public cloud goodness: https://www.openstack.org/passport
Options
Use Cloud Provider Services and Stitch Together
APIs/Automation to Manage it
• AWS/Azure/GCP:
• VPC/VNet Peering
• Shared VPC/VNet
• AWS Transit GW - AWS Accelerated VPN (Global Accelerator + Transit GW)
• AWS CloudHub
• Colocation Partners:
• Equinix Cloud Exchange Fabric
• etc.
Cisco SD-WAN
vManage vBond vSmart
VNet Network
10.50.0.0/16
vEdge/cEdge
Private Network
172.16.0.0/24
vEdge/cEdge
VPC Network SD-WAN

On-Premises Private Cloud
172.31.0.0/16
vEdge/cEdge
Cisco SD-WAN:
https://www.cisco.com/c/en/us/solutions/en
terprise-networks/sd-wan/index.html
IPsec VPN - Cisco SD-WAN Example
Per-VPC Cisco vEdge
VPC Subnet(s) Private Network(s)
vEdge vEdge
VPC On-Premises
Router
Cloud IPsec
Transit VPC: Cisco vEdge + CSP VPN
Transit VPC

IPsec vEdge
VPC VPN vEdge
Cloud IPsec On-Premises
Router Gateway
Transit VPC: Cisco vEdge + Per-VPC vEdge

Transit VPC

vEdge vEdge vEdge
VPC IPsec On-Premises
Router Cloud Cloud
Multicloud with Transit VPC
Transit VPC
VNet Subnet
Azure VPN vEdge

GW
Private Network(s)
Cisco
Transit VPC vEdge
VPC Subnet
On-Premises
vEdge
AWS VPN
GW
SD-WAN
Transit VPC
VPC Subnet
Google vEdge
Cloud VPN
Colocation - With or Without VPN
Cisco Routers or Firewalls + Some Combo of Colocation/peering
VLANs
DX Cisco
VPC VPN Cisco ASR
Endpoint 1000
ASR/CSR/ASA On-Premises
Router Gateway
IPsec
Cisco SD-WAN + Some Combo of Colocation/peering
VLANs
DX vEdge
VPC VPN vEdge On-Premises
Router Gateway Endpoint
IPsec
IPsec
DMVPN – Enable Dynamic Multicloud Networking
Cisco DMVPN - A Brownfield Way to Bolt on Multicloud
VNet Network
Cisco
10.50.0.0/16 Cisco Spoke
CSR1000v
CSR1000v
Hub
Private Network
172.16.0.0/24
Spoke
VPC Network DMVPN
172.31.0.0/16 Cisco
CSR1000v
Cisco DMVPN:
https://www.cisco.com/c/en/us/products
/security/dynamic-multipoint-vpn-
dmvpn/index.html
IPsec VPN - Cisco CSR 1000v Example
Per-VPC Cisco CSR 1000v
CSRs Cisco
VPC ASR/CSR/ISR On-Premises
Router DMVPN/IPsec
Transit VPC: Cisco CSR + CSP VPN
Transit VPC

IPsec CSRs Cisco
VPC VPN DMVPN/IPsec ASR/CSR/ISR On-Premises
Router Gateway
Transit VPC: Cisco CSR + Per-VPC CSR

Transit VPC

CSRs CSRs Cisco
VPC DMVPN/IPsec ASR/CSR/ISR On-Premises
Router
AWS – Transit Gateway (TGW)
Dev Prod Dev Prod Dev Prod
Dev Prod Dev Prod Dev Prod
Transit Gateway
TGW ‘can’ replace transit
VPC - More on this later
AWS Direct
VPN
Connect
WAN
Transit VPC Transit VPC

CSR Cisco ASR/
VPC VPN IPsec CSR/ISR On-Premises
Router Gateway
A Note On MTU
• All three providers recommend a different size interface MTU for the IPsec tunnel
interface:
• Google recommendation: https://cloud.google.com/vpn/docs/concepts/advanced#mtu
• AWS recommendation:
https://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Introduction.html
• Azure recommendation: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-
about-vpn-devices
• In addition to MTU, you need to set and test your TCP MSS values
Google Cloud Platform
– Native VPN
Reference
Google Cloud Platform – VPN Gateway

• GCP Cloud VPN overview
• https://cloud.google.com/vpn/docs/concepts/overview
• GCP Cloud VPN documentation
• https://cloud.google.com/vpn/docs/how-to/creating-vpns
• GCP Advanced VPN documentation
• https://cloud.google.com/vpn/docs/concepts/advanced
Topology for GCP to On-Premises Cisco Routers – IPsec VPN
BGP Routing
BGP<>OSPF Redistribution
IPsec/IKEv2
Tunnel Mode Cisco
ASR/CSR/ISR Private Network
Default Network 35.xxx.xxx.x 169.254.0.2 .1 192.168.100.0/24
10.138.0.0/20 192.xxx.xxx.x OSPF 10 Area 0
Google
Cloud
VPN
169.254.0.1
BGP AS65002 10.138.0.0/20
BGP AS65000
Google Cloud Router

192.168.100.0/24
gcloud – Create the VPN GW, External IP and Forwarding Rules
Create a VPN gateway
# gcloud compute target-vpn-gateways create csr-gcp-vm-gw --region us-west1 --network default
Create an external IP to use for the VPN

# gcloud compute addresses create gcp-to-csr --region us-west1
Capture the external IP address

# gcloud compute addresses list --filter="gcp-to-csr”
NAME REGION ADDRESS STATUS
gcp-to-csr us-west1 35.xxx.xxx.x RESERVED
Create a forwarding rule for ESP, UDP500 and UDP4500 – These are used by IKE/IPsec
# gcloud compute forwarding-rules create csr-gcp-vm-rule-esp \
--region us-west1 \
--address 35.xxx.xxx.x \
--ip-protocol ESP \
--target-vpn-gateway csr-gcp-vm-gw
# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \

--region us-west1 \
--ip-protocol UDP --ports 500 \
# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \

--region us-west1 \
--ip-protocol UDP --ports 4500 \
gcloud – Create Cloud Router, VPN Tunnel and BGP session
Create the Cloud router that is used for BGP (an existing router can be used)
# gcloud compute routers create csr-gcp-vm-bgp-rtr \
--region us-west1 \
--asn=65000 \
--network default
Create a VPN tunnel and link it to the router created in the previous step
# gcloud compute vpn-tunnels create csr-gcp-vm-gw-tunnel-1 \
--region us-west1 \
--peer-address 192.xxx.xxx.x --shared-secret <pre-shared-password-goes-here> \
--ike-version 2 \
--target-vpn-gateway csr-gcp-vm-gw \
--router csr-gcp-vm-bgp-rtr
Add a new interface to the router and set the BGP session IP address for the GCP side of the connection
# gcloud compute routers add-interface csr-gcp-vm-bgp-rtr \
--interface-name if-csr-gcp-vm-bgp-rtr-01 \
--ip-address 169.254.0.1 \
--mask-length 30 \
--vpn-tunnel csr-gcp-vm-gw-tunnel-1 \
--region us-west1
Create a new BGP peer – This peer will be the Cisco CSR at the On-Premises cloud
# gcloud compute routers add-bgp-peer csr-gcp-vm-bgp-rtr \
--interface if-csr-gcp-vm-bgp-rtr-01 \
--peer-asn 65002 \
--peer-name csr-gcp-vm-bgp-peer \
--peer-ip-address 169.254.0.2 \
--region us-west1
... Output summarized
Cisco CSR Route Information

csr-gcp-01# show ip route
Default Network Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
10.138.0.0/20 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
169.254.0.2
.1 ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
Google Area 0 a - application route

+ - replicated route, % - next hop override, p - overrides from PfR
Cloud BGP 192.168.100.0/24
VPN S* 0.0.0.0/0 [1/0] via 192.xxx.xxx.x
10.0.0.0/20 is subnetted, 1 subnets
169.254.0.1 B 10.138.0.0 [20/100] via 169.254.0.1, 00:16:59
169.254.0.0/16 is variably subnetted, 2 subnets, 2 masks
C 169.254.0.0/30 is directly connected, Tunnel0
L 169.254.0.2/32 is directly connected, Tunnel0
192.xxx.xxx.x/24 is variably subnetted, 2 subnets, 2 masks
Google Cloud Router
C 192.xxx.xxx.x/26 is directly connected, GigabitEthernet1
L 192.xxx.xxx.x/32 is directly connected, GigabitEthernet1
C 192.168.100.0/24 is directly connected, GigabitEthernet2
L 192.168.100.1/32 is directly connected, GigabitEthernet2
Google VPN –
Dual/Redundant On-
Premises Cisco CSRs
Reference Topology for On-Premises
Dual Cisco Router Design OSPF 10 Area 0
169.254.0.2 .20
192.yyy.yyy.y .2 VM
192.168.100.0/24
Private Network
Compute Engine
2 1
BGP AS65002
Default Network .2
35.yyy.yyy.y
HSRP – VIP = .1
10.138.0.0/20 35.xxx.xxx.x
Google
Cloud 169.254.0.10
VPN
169.254.0.1 .3
169.254.0.9 192.xxx.xxx.x
BGP AS65000 BGP AS65002
Google Cloud Router
10.138.0.0/20
192.168.100.0/24
Pre-Failure State (1)
GCE Instance traceroutes via 169.254.0.2 GCP BGP Path
[shmcfarl@instance-3 ~]$ traceroute 192.168.100.20
traceroute to 192.168.100.20 (192.168.100.20), 30 hops max, 60 byte packets
1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms
2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms
On-Premises VM traceroutes via HSRP Active CSR (192.168.100.2)

[root@k8s-m-01 ~]# traceroute 10.138.0.2
1 192.168.100.2 (192.168.100.2) 0.545 ms 0.468 ms 0.415 ms
2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms
HSRP Active CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-01#show ip route
. . .
B 10.138.0.0/20 [20/100] via 169.254.0.1, 00:03:41
HSRP Standby CSR Route to GCP Default Network (10.138.0.0)

csr-gcp-02#show ip route
. . .
B 10.138.0.0/20 [20/100] via 169.254.0.9, 00:08:47
HSRP Active
csr-gcp-01#show stand
GigabitEthernet2 - Group 0 (version 2)
State is Active
HSRP Standby
csr-gcp-02#show stand
GigabitEthernet2 - Group 0 (version 2)
State is Standby
https://cloud.google.com/router/docs/concepts/overview
First Google Cloud Router BGP State
# gcloud compute routers get-status csr-gcp-vm-bgp-rtr
kind: compute#routerStatusResponse • Determining best path
result:
bestRoutes: • If Cloud Router receives multiple routes for the
- creationTimestamp: '2017-09-19T14:48:49.137-07:00'
destRange: 192.168.100.0/24
same destination, GCP uses route metrics and,
kind: compute#route in some cases, AS path length to determine the
nextHopIp: 169.254.0.10 best path. To help you configure your On-
priority: 0
Premises routers, the following list describes
destRange: 192.168.100.0/24 the algorithm that GCP uses for egress traffic.
kind: compute#route
nextHopIp: 169.254.0.2 • If you have multiple BGP sessions on a single
priority: 0 Cloud Router, GCP uses the route with the
bestRoutesForRouter:
- creationTimestamp: '2017-09-19T14:48:49.137-07:00' shortest AS path length.
destRange: 192.168.100.0/24
kind: compute#route • If routes have the same AS path length, GCP uses
nextHopIp: 169.254.0.2 the route with the lower MED value.
priority: 0
bgpPeerStatus: • If routes have equal costs (same AS path length and
- advertisedRoutes: metric), GCP uses ECMP to balance traffic across
- destRange: 10.138.0.0/20 multiple paths.
kind: compute#route
nextHopIp: 169.254.0.1 • If you use multiple Cloud Routers, GCP uses
priority: 100
ipAddress: 169.254.0.1 only the MED value to determine the best path.
name: csr-gcp-vm-bgp-peer The AS path length doesn't influence the path
numLearnedRoutes: 1 selection between multiple Cloud Routers.
peerIpAddress: 169.254.0.2
state: Established
• If a static and dynamic route have the same
status: UP
uptime: 1 minutes, 48 seconds prefix and metric, GCP uses the static route.
uptimeSeconds: '108'
network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default
https://cloud.google.com/router/docs/concepts/overview
Second Google Cloud Router BGP State
# gcloud compute routers get-status csr-gcp-vm-bgp-rtr-02
kind: compute#routerStatusResponse • Determining best path
result:
bestRoutes: • If Cloud Router receives multiple routes for the
destRange: 192.168.100.0/24 same destination, GCP uses route metrics and, in
kind: compute#route some cases, AS path length to determine the
nextHopIp: 169.254.0.10 best path. To help you configure your On-
priority: 0
- creationTimestamp: '2017-09-19T14:48:49.137-07:00' Premises routers, the following list describes the
destRange: 192.168.100.0/24 algorithm that GCP uses for egress traffic.
kind: compute#route
nextHopIp: 169.254.0.2 • If you have multiple BGP sessions on a single
priority: 0 Cloud Router, GCP uses the route with the
bestRoutesForRouter:
- creationTimestamp: '2017-09-19T14:43:36.121-07:00' shortest AS path length.
destRange: 192.168.100.0/24
kind: compute#route • If routes have the same AS path length, GCP uses the
nextHopIp: 169.254.0.10 route with the lower MED value.
priority: 0
bgpPeerStatus: • If routes have equal costs (same AS path length and
- advertisedRoutes: metric), GCP uses ECMP to balance traffic across
- destRange: 10.138.0.0/20 multiple paths.
kind: compute#route
nextHopIp: 169.254.0.9 • If you use multiple Cloud Routers, GCP uses only
priority: 100
ipAddress: 169.254.0.9 the MED value to determine the best path. The
name: csr-gcp-vm-bgp-peer-02 AS path length doesn't influence the path
numLearnedRoutes: 1
peerIpAddress: 169.254.0.10
selection between multiple Cloud Routers.
state: Established
status: UP
• If a static and dynamic route have the same prefix
uptime: 6 minutes, 50 seconds and metric, GCP uses the static route.
uptimeSeconds: '410'
network: https://www.googleapis.com/compute/v1/projects/public-175116/global/networks/default
Failure Scenario 1 – HSRP Primary CSR VM Reload

HSRP Debug on HSRP Standby
csr-gcp-02#
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby: i/Resign rcvd (110/192.168.100.2)
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Active router is local, was 192.168.100.2
*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 no longer active for group 0 (Standby)
*Sep 19 21:59:17.396: HSRP: Gi2 Nbr 192.168.100.2 Was active or standby - start passive holddown
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby router is unknown, was local
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby -> Active
*Sep 19 21:59:17.396: %HSRP-5-STATECHANGE: GigabitEthernet2 Grp 0 state Standby -> Active
*Sep 19 21:59:17.396: HSRP: Peer not present
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Redundancy "hsrp-Gi2-0" state Standby -> Active
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Added 192.168.100.1 to ARP (0000.0c9f.f000)
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Activating MAC 0000.0c9f.f000
*Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Adding 0000.0c9f.f000 to MAC address filter
*Sep 19 21:59:17.396: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" standby, local -> unknown
*Sep 19 21:59:17.398: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Standby -> Active
*Sep 19 21:59:20.379: HSRP: Gi2 IP Redundancy "hsrp-Gi2-0" update, Active -> Active
*Sep 19 21:59:57.361: %OSPF-5-ADJCHG: Process 10, Nbr 192.168.100.2 on GigabitEthernet2 from FULL to DOWN, Neighbor Down: Dead timer expired
On-Premises VM traceroutes via HSRP Newly Active CSR (192.168.100.3)

[root@k8s-m-01 ~]# traceroute 10.138.0.2
1 192.168.100.3 (192.168.100.3) 0.545 ms 0.468 ms 0.415 ms
2 10.138.0.2 (10.138.0.2) 96.261 ms 58.105 ms 77.147 ms
Failure Scenario 2 – Shut HSRP Primary LAN Interface
(BGP session is still active)
Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path

1 169.254.0.2 (169.254.0.2) 24.223 ms 24.430 ms 24.716 ms
2 192.168.100.20 (192.168.100.20) 24.180 ms 24.595 ms 24.422 ms
Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path

1 169.254.0.10 (169.254.0.10) 32.756 ms 42.796 ms 25.635 ms
2 192.168.100.20 (192.168.100.20) 66.674 ms 72.234 ms 74.331 ms
Failure Scenario 3 – Shut IPsec Tunnel on HSRP Primary
CSR – With/Without HSRP Interface Tracking
Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path
1 169.254.0.2 (169.254.0.2) 24.728 ms 24.780 ms 24.766 ms
2 192.168.100.20 (192.168.100.20) 24.480 ms 24.495 ms 24.482 ms
Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path BUT traffic is re-routed to the HSRP Primary
(192.168.100.2) before going to the end host
1 169.254.0.10 (169.254.0.10) 24.863 ms 42.763 ms 32.908 ms
2 192.168.100.2 (192.168.100.2) 54.069 ms 86.788 ms 70.963 ms
On-Premises LAN re-route to HSRP Active
3 192.168.100.20 (192.168.100.20) 174.753 ms * 134.706 ms on router with failed IPsec Tunnel
LAN Re-Route Issue Resolved – Use Track
track 10 interface Tunnel0 line-protocol csr-gcp-01#show stand
! GigabitEthernet2 - Group 0 (version 2) Tunnel failed and
interface GigabitEthernet2
description Private Network On-Premises
State is Standby
. . .
track changed
ip address 192.168.100.2 255.255.255.0 Priority 100 (configured 110) HSRP state
standby version 2 Track object 10 state Down decrement 10
standby 0 ip 192.168.100.1
standby 0 priority 110 [shmcfarl@instance-3 ~]$ traceroute 192.168.100.20
standby 0 preempt traceroute to 192.168.100.20 (192.168.100.20), 30 hops max, 60 byte packets
standby 0 authentication md5 key-string 7 01300F175804575D720D 1 169.254.0.10 (169.254.0.10) 43.113 ms 25.269 ms 33.033 ms
standby 0 track 10 decrement 10 2 192.168.100.20 (192.168.100.20) 72.879 ms 111.849 ms 53.904 ms
Reference Cisco CSR Config – interface Tunnel0
ip address 169.254.0.2 255.255.255.252
Primary ip mtu 1400

ip tcp adjust-mss 1360
crypto ikev2 proposal PHASE1-PROP tunnel source GigabitEthernet1
encryption aes-cbc-256 tunnel mode ipsec ipv4
integrity sha1 tunnel destination 35.yyy.yyy.y
group 14 tunnel protection ipsec profile CSR-GCP
! !
crypto ikev2 policy IKE-POL interface GigabitEthernet1
proposal PHASE1-PROP ip address 192.yyy.yyy.y 255.255.255.192
! !
crypto ikev2 keyring KEY interface GigabitEthernet2
peer GCP-PEER description Private Network On-Premises
address 35.yyy.yyy.y ip address 192.168.100.2 255.255.255.0
hostname csr-gcp-dmz-sjc standby version 2
pre-shared-key local <PSK_PASSWORD_GOES_HERE> standby 0 ip 192.168.100.1
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> standby 0 priority 110
! standby 0 preempt
crypto ikev2 profile IKEV2-SETUP standby 0 authentication md5 key-string 7 <HSRP_KEY>
match identity remote address 0.0.0.0 standby 0 track 10 decrement 10
authentication local pre-share !
authentication remote pre-share router ospf 10
keyring local KEY redistribute bgp 65002 subnets
lifetime 36000 network 192.168.100.0 0.0.0.255 area 0
! !
crypto ikev2 dpd 10 2 periodic router bgp 65002
! bgp log-neighbor-changes
track 10 interface Tunnel0 line-protocol neighbor 169.254.0.1 remote-as 65000
! neighbor 169.254.0.1 timers 20 60 60
crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac !
mode tunnel address-family ipv4
! redistribute ospf 10
crypto ipsec profile CSR-GCP neighbor 169.254.0.1 activate
set transform-set CSR-GCP-SET neighbor 169.254.0.1 soft-reconfiguration inbound
set pfs group14 !
#CiscoLiveip route 0.0.0.0 © 2020 Cisco and/or
0.0.0.0 its affiliates. All rights reserved. Cisco Public
192.yyy.yyy.y 46
set ikev2-profile IKEV2-SETUP
Reference Cisco CSR Config –
interface Tunnel0
ip address 169.254.0.10 255.255.255.252
Secondary ip mtu 1400

ip tcp adjust-mss 1360
crypto ikev2 proposal PHASE1-PROP tunnel source GigabitEthernet1
encryption aes-cbc-256 tunnel mode ipsec ipv4
integrity sha1 tunnel destination 35.xxx.xxx.x
group 14 tunnel protection ipsec profile CSR-GCP
! !
crypto ikev2 policy IKE-POL interface GigabitEthernet1
proposal PHASE1-PROP ip address 192.xxx.xxx.x 255.255.255.192
! !
crypto ikev2 keyring KEY interface GigabitEthernet2
peer GCP-PEER description Private Network On-Premises
address 35.xxx.xxx.x ip address 192.168.100.3 255.255.255.0
hostname csr-vpn-gw-02 standby version 2
pre-shared-key local <PSK_PASSWORD_GOES_HERE> standby 0 ip 192.168.100.1
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> standby 0 priority 105
! standby 0 preempt
crypto ikev2 profile IKEV2-SETUP standby 0 authentication md5 key-string 7 <HSRP_KEY>
match identity remote address 0.0.0.0 !
authentication local pre-share router ospf 10
authentication remote pre-share redistribute bgp 65002 subnets
keyring local KEY network 192.168.100.0 0.0.0.255 area 0
lifetime 36000 !
! router bgp 65002
crypto ikev2 dpd 10 2 periodic bgp log-neighbor-changes
! neighbor 169.254.0.9 remote-as 65000
crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac neighbor 169.254.0.9 timers 20 60 60
mode tunnel !
! address-family ipv4
crypto ipsec profile CSR-GCP redistribute ospf 10
set transform-set CSR-GCP-SET neighbor 169.254.0.9 activate
set pfs group14 neighbor 169.254.0.9 soft-reconfiguration inbound
set ikev2-profile IKEV2-SETUP !
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
Amazon Web Services
– Native VPN
AWS – VPN Gateway
• AWS VPN Overview
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpn-connections.html
• AWS VPN Setup
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/SetUpVPNConnections.
html
• AWS does support NAT-T:
https://aws.amazon.com/blogs/aws/ec2-vpc-vpn-update-nat-traversal-additional-
encryption-options-and-more/
• Example templates for Cisco IOS:
http://docs.aws.amazon.com/AmazonVPC/latest/NetworkAdminGuide/Cisco.html
Topology for AWS to On-Premises Cisco Router – IPsec VPN
BGP Routing
BGP <> OSPF Redistribution
IPsec/IKEv2
Tunnel Mode Cisco
ASR/CSR/ISR
VPC Network 52.xxx.xxx.x 169.254.11.178 .1 Private Network
172.31.0.0/16 169.254.11.177 192.xxx.xxx.x 192.168.200.0/24
VPC VPN
Router Gateway OSPF 10 Area 0
BGP AS64512
BGP AS65002 172.31.0.0/16

192.168.200.0/24
AWS CLI: Create VPC, VPN GW,
Customer GW and VPN Connection
Create a new AWS VPC (or use an existing one)
# aws ec2 create-vpc --cidr-block 172.31.0.0/16
Create VPN Gateway and set the AWS BGP ASN

# aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn 64512
Attach VPN Gateway to the VPC

# aws ec2 attach-vpn-gateway --vpc-id vpc-ce2124aa --vpn-gateway-id vgw-64277e21
Create a new customer gateway with the On-Premises BGP ASN and the On-Premises router IP address (do this for each connection)
# aws ec2 create-customer-gateway --bgp-asn 65002 --public-ip 192.xxx.xxx.x --type ipsec.1
Create a new VPN connection

# aws ec2 create-vpn-connection --customer-gateway-id cgw-d6055d93 --type ipsec.1 --vpn-gateway-id vgw-64277e21
Note: Lots of output will come from the above VPN creation command.
This information can be used to build the On-Premises CSR config. The best method for getting the configuration is
shown on the next slide.
Enable route propagation for the VPC

# aws ec2 enable-vgw-route-propagation --gateway-id vgw-64277e21 --route-table-id rtb-515e8e36
Permit SSH and ICMP

# aws ec2 authorize-security-group-ingress --group-name default --protocol tcp --port 22 --cidr 0.0.0.0/0
# aws ec2 authorize-security-group-ingress --group-name default --protocol icmp --port -1 --cidr 0.0.0.0/0
Optional: Download Router Configuration
• VPC Dashboard > VPN Connections
Reference Cisco CSR Config - Primary
crypto isakmp policy 200
encryption aes 128 interface Tunnel1
authentication pre-share ip address 169.254.11.178 255.255.255.252
group 2 ip virtual-reassembly
lifetime 28800 ip mtu 1400
hash sha tunnel source 192.xxx.xxx.x
! tunnel destination 52.xxx.xxx.x
crypto keyring keyring-vpn-cec15996-0 tunnel mode ipsec ipv4
local-address 192.xxx.xxx.x tunnel protection ipsec profile ipsec-vpn-cec15996-0
pre-shared-key address 52.xxx.xxx.x key ip tcp adjust-mss 1379
<PSK_PASSWORD_GOES_HERE> !
! router ospf 10
crypto isakmp profile isakmp-vpn-cec15996-0 redistribute bgp 65002 subnets
local-address 192.xxx.xxx.x network 192.168.200.0 0.0.0.255 area 0
match identity address 52.xxx.xxx.x !
keyring keyring-vpn-cec15996-0 router bgp 65002
crypto ipsec transform-set ipsec-prop-vpn-cec15996-0 esp-aes neighbor 169.254.11.177 activate
128 esp-sha-hmac neighbor 169.254.11.177 timers 10 30 30
mode tunnel !
crypto ipsec profile ipsec-vpn-cec15996-0 redistribute ospf 10
set pfs group2 neighbor 169.254.11.177 remote-as 64512
set security-association lifetime seconds 3600 neighbor 169.254.11.177 activate
set transform-set ipsec-prop-vpn-cec15996-0 neighbor 169.254.11.177 soft-reconfiguration inbound
!
crypto ipsec df-bit clear
!
crypto isakmp keepalive 10 10 on-demand
!
crypto ipsec fragmentation before-encryption
Verify Routing and Reachability

On the on-premises CSR check the route for the the AWS VPC network 172.31.0.0/16
csr-mc-01#show ip route | i 172.31.0.0
B 172.31.0.0/16 [20/100] via 169.254.11.177, 00:13:35
On AWS check for the route for the on-premises network (192.168.200.0/24)
# aws ec2 describe-route-tables | grep 192.168.200.0
ROUTES 192.168.200.0/24 vgw-64277e21 EnableVgwRoutePropagation active
Connect to an AWS instance and ping to the on-premises private network

ubuntu@ip-172-31-0-121:~$ ping 192.168.200.30
PING 192.168.200.30 (192.168.200.30) 56(84) bytes of data.
64 bytes from 192.168.200.30: icmp_seq=1 ttl=63 time=4.95 ms
Cisco .30
VPC Network BGP ASR/CSR/ISR VM
172.31.0.0/16 169.254.11.178 .1 Private Network
.121 .1 169.254.11.177 192.168.200.0/24
Topology for Dual Cisco On-Premises
Routers on AWS BGP AS65002

OSPF 10 Area 0
169.254.11.178
.2
192.168.200.0/24
Private Network
VPC Network 169.254.11.177 HSRP – VIP = .1
169.254.10.213
172.31.0.0/16
VPC VPN
Router Gateway
BGP AS64512 .3
169.254.10.214
BGP AS65002
Routes this side should see: Routes this side should see:
192.168.200.0/24 172.31.0.0/16
Microsoft Azure –
Native VPN
Microsoft Azure – VPN Gateway
• Azure VPN Overview
• https://azure.microsoft.com/en-us/services/vpn-gateway/
• https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-
about-vpngateways
• In order to use BGP you must use Route-Based VPN and SKUs
VpnGw1, VpnGw2, VpnGw3, Standard or HighPerformance SKUs :
https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-
gateway-bgp-overview
Azure to On-Premises Cisco Router – IPsec VPN
BGP Routing
IPsec/IKEv2
Tunnel Mode Cisco
ASR/CSR/ISR
Vnet Subnet 40.xxx.xxx.x 10.11.255.1 .1 Private Network
10.10.0.0/16 192.168.200.0/24
VPN 10.10.255.30 192.xxx.xxx.x
Gateway OSPF 10 Area 0
BGP AS64512
BGP AS65002
Azure CLI: Create Resource Group, Networks,
Subnets
Create a new Azure Resource Group (rg)
# az group create --name azure-vpn-rg --location westus
# az configure --defaults location=westus
# az configure --defaults group=azure-vpn-rg
Create a new virtual network (vnet) and a new ‘outside’ subnet

# az network vnet create \
--name vnet1 \
--address-prefix 10.10.0.0/16 \
--subnet-name outside \
--subnet-prefix 10.10.0.0/24
Create a ’inside’ subnet

# az network vnet subnet create \
--vnet-name vnet1 \
--name inside \
--address-prefix 10.10.1.0/24
Create a new subnet that is used for the IPsec/BGP interface on the Azure side
--vnet-name vnet1 \
--name gatewaysubnet \
Azure CLI: Create a Public IP, VPN/Vnet Gateway and
Local Gateway
Create a new public IP address (Using Azure VPN service, the allocation must be ‘dynamic’)
# az network public-ip create \
--name azure-vpn-gw-eip \
--allocation-method dynamic
Create Vnet gateway using ‘RouteBased’ (BGP) and a supported sku (see earlier links for requirements). THIS TAKES AWHILE
# az network vnet-gateway create \
--name vpn-gw \
--public-ip-address azure-vpn-gw-eip \
--vnet vnet1 \
--gateway-type Vpn \
--sku VpnGw1 \
--vpn-type RouteBased \
--asn 65010
Once the Vnet gateway is up, get the Azure-side BGP Peering address (Needed for On-Premises configuration)
# az network vnet-gateway list | grep bgpPeeringAddress
"bgpPeeringAddress": "10.10.255.30",
Create the local gateway (On-Premises target). Local prefix/BGP peer should be the On-Premises CSR tunnel info.
Can’t be in Azure vnet range
# az network local-gateway create \
--gateway-ip-address 192.xxx.xxx.x \
--name azure-lng \
--local-address-prefixes 10.11.255.1/32 \
--asn 65002 \
--bgp-peering-address 10.11.255.1
Azure CLI: Vnet GW, Local GW, VPN Connection
Copy the full path from the “id” line (under the ‘gatewayType: Vpn’ line) that is shown in the vnet-gateway output
# az network vnet-gateway show --name vpn-gw
"gatewayType": "Vpn",
"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw",
Copy the full path from the “id” line that is shown in the vnet-gateway output
# az network local-gateway show --name azure-ln
"id": "/subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng"
Create the VPN connection using information from above

# az network vpn-connection create \
--name azure-to-csr \
--vnet-gateway1 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/virtualNetworkGateways/vpn-gw \
--enable-bgp \
--shared-key ”<YOUR_PRE_SHARED_KEY>" \
--local-gateway2 /subscriptions/<YOUR_ID>/resourceGroups/azure-vpn-rg/providers/Microsoft.Network/localNetworkGateways/azure-lng
Optional: Create a new test VM on Azure and associate it with the ‘inside’ subnet
# az vm create \
--name AzTestVm \
--authentication-type ssh \
--ssh-key-value "$(< ~/.ssh/id_rsa.pub)" \
--image Canonical:UbuntuServer:16.04-LTS:latest \
--size Standard_DS1_v2 \
--vnet-name vnet1 \
--subnet inside \
--public-ip-address-allocation dynamic
On-Premises Cisco CSR IPsec/Routing Config ... Output summarized
interface Tunnel2
crypto ikev2 proposal PHASE1-PROP
encryption aes-cbc-256 ip address 10.11.255.1 255.255.255.255
integrity sha1 ip mtu 1400
group 2 ip tcp adjust-mss 1360
! tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
crypto ikev2 policy IKE-POL
proposal PHASE1-PROP tunnel destination 40.xxx.xxx.x
! tunnel protection ipsec profile CSR-AZURE
crypto ikev2 keyring KEY !
peer AZURE-PEER interface GigabitEthernet1
description Internet
address 40.xxx.xxx.x
pre-shared-key local <PSK_PASSWORD_GOES_HERE> ip address 192.xxx.xxx.x 255.255.255.0
pre-shared-key remote <PSK_PASSWORD_GOES_HERE> no ip redirects
! no ip unreachables
crypto ikev2 profile IKEV2-SETUP no ip proxy-arp
negotiation auto
match identity remote address 0.0.0.0
authentication local pre-share !
authentication remote pre-share router ospf 10
keyring local KEY router-id 10.1.0.2
lifetime 36000 redistribute bgp 65002 subnets
network 192.168.200.0 0.0.0.255 area 0
!
crypto ikev2 dpd 10 2 periodic !
! router bgp 65002
crypto ipsec security-association replay window-size 1024 bgp log-neighbor-changes
neighbor 10.10.255.30 ebgp-multihop 255
crypto ipsec transform-set CSR-AZURE-SET esp-aes 256 esp-sha-hmac
mode tunnel !
crypto ipsec profile CSR-AZURE redistribute ospf 10
set transform-set CSR-AZURE-SET neighbor 10.10.255.30 activate
neighbor 10.10.255.30 soft-reconfiguration inbound
set pfs group14
set ikev2-profile IKEV2-SETUP !
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
ip route 10.10.255.30 255.255.255.255 Tunnel2

On the on-premises CSR check the route for the Azure Vnet route of 10.10.0.0/16
B 10.10.0.0/16 [20/0] via 10.10.255.30, 00:51:26
On Azure check for the route for the on-premises network (192.168.200.0/24)
PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName AzTestVmVMNic -ResourceGroupName azure-vpn-rg | Format-Table
Name State Source AddressPrefix NextHopType NextHopIpAddress
---- ----- ------ ------------- ----------- ----------------
Active VirtualNetworkGateway {192.168.200.0/24} VirtualNetworkGateway {40.xxx.xxx.x}
Connect to an Azure instance and ping to the on-premises private network

shmcfarl@AzTestVm:~$ ping 192.168.200.30
Cisco
Inside Subnet ASR/CSR/ISR .30 VM
10.10.1.0/24 40.xxx.xxx.x 10.11.255.1
.1 Private Network
.4 10.10.255.30 192.xxx.xxx.x 192.168.200.0/24
VPN
Gateway
Multicloud with Cisco
SD-WAN
Cisco SD-WAN Architecture
vManage Management Plane

- vManage
APIs - UI
Orchestration Plane - Policies, templates
- vBond 3rd Party - Monitoring
vAnalytics
- Orchestrates control Automation
and mgmt. plane
- First point of auth vBond Control Plane
- vSmart
vSmart Controllers - Fabric discovery
- Control plane policies
MPLS 4G
INET
vEdge/cEdge Routers
Data Plane
- vEdge/cEdge
Cloud Data Center Campus Branch SOHO
Cisco SD-WAN
VNet Network
10.10.1.0/16
vEdge/cEdge
Private Network
10.1.1.0/24
vEdge/cEdge
VPC Network SD-WAN

On-Premises
172.3.0.0/24
vEdge/cEdge
Cisco SD-WAN:
https://www.cisco.com/c/en/us/solutions/en
terprise-networks/sd-wan/index.html
Cisco SD-WAN
Public Cloud Support
• Cisco SD-WAN (vEdge) on AWS: https://sdwan-
docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07
Deploy_the_vEdge_Routers/01Create_vEdge_Cloud_VM_Instance_on_AWS
• AWS Marketplace: https://aws.amazon.com/marketplace/pp/B07BZ53FJT
• Cisco SD-WAN on Microsoft Azure: https://sdwan-
docs.cisco.com/Product_Documentation/Getting_Started/Viptela_Overlay_Network_Bringup/07
Deploy_the_vEdge_Routers/02Create_vEdge_Cloud_VM_Instance_on_Azure
• Microsoft Azure Marketplace: https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/cisco.cisco_cloud_vedge_4_nics?tab=Overview
• Cisco SD-WAN Design/Deployment Guides: https://www.cisco.com/c/en/us/solutions/design-
zone/networking-design-guides/branch-wan-edge.html
• Cisco SD-WAN Cloud OnRamp for Colocation:
https://www.cisco.com/c/en/us/td/docs/routers/sdwan-cloud-onramp-for-colocation/solution-
user-guide/cisco-sdwan-cloud-onramp-colocation-solution-guide-19_1.html
We Can Do This The Easy Way or Hard Way
I’ll explain the hard stuff, but..
1) Design it
2) Deploy the Control Plane
3) Deploy the On-Premises Data Plane (to include connections to
workloads)
4) Create/Gather your Public Cloud Credentials/Roles
5) Deploy the Transit VNet/VPCs via Cloud onRamp
We will talk about
6) Map the application/host VNet/VPCs to the Transit these two
7) Deploy Policy(s) that Meets Your Requirements
8) Have a Nice Day! J
Cisco SD-WAN Cloud
onRamp for IaaS -
AWS
Cisco SD-WAN – Transit VPC
Cloud onRamp for IaaS - AWS
Transit VPC
VPC Network Private Network

VPC VPN vEdge
IPsec vEdge
Router Gateway Cloud
On-Premises
• AWS: https://sdwan-
docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_On
Ramp_with_AWS
• Azure: https://sdwan-
docs.cisco.com/Product_Documentation/vManage_Help/Release_18.3/Configuration/Cloud_On
Ramp_with_Azure
Cloud OnRamp IaaS for AWS
2
1
Cisco SD-WAN CoR for AWS
High-Level View GatewayVpc
Management VPN
172.18.2.22
vpn 512
Service VPN
HostVpc (172.16.0.0/16)
vpn 1
192.168.254.5
VPN Cisco
Transport VPN Container
172.18.2.40
Tunnel
vpn 0
pod
Platform
Enterprise
EKS EIP Edge Data CCP Control Plane
Worker Center pod pod pod
Node
pod
VM VM VM VM VM VM
172.18.2.148
EKS VPC VPN GW HYPERVISOR
Worker Router (VGW)
Node CCP
vpn 0
pod
Network
EIP Transport VPN 10.1.1.0/24
EKS VPN
Worker Tunnel vpn 1
Node 192.168.254.6
Service VPN
172.18.2.120
vpn 512
Amazon Elastic
Container Service
for Kubernetes
(EKS) Management VPN
CoR-Centric View GatewayVpc
Management VPN
172.18.2.22
vpn 512
Service VPN
HostVpc (172.16.0.0/16)
vpn 1
192.168.254.5
VPN Cisco
172.18.2.40
Tunnel 34.211.x.x AS 9988:169.254.14.130
vpn 0
AS 65512
pod BGP Instance

169.254.14.129 AS 9988:169.254.12.226 Platform
Enterprise
EKS BGP Instance EIP 54.27.x.x Edge Data CCP Control Plane
Worker 169.254.13.237 Center pod pod pod
Node 52.35.x.x
pod
VM VM VM VM VM VM
172.18.2.148
EKS VPC VPN GW HYPERVISOR
Worker Router (VGW) 52.33.x.x
Node 35.163.y.x CCP
vpn 0
pod
Network
BGP Instance AS 9988: 169.254.13.238
AS 65512
169.254.12.225 EIP AS 9988:169.254.15.110 Transport VPN 10.1.1.0/24

EKS VPN 54.190.y.x
Worker
BGP Instance Tunnel vpn 1
169.254.15.109
Node 192.168.254.6
Service VPN
172.18.2.120
vpn 512
Amazon Elastic
Container Service
for Kubernetes
(EKS) Management VPN
vManage
Cloud onRamp for IaaS - AWS
Host VPCs are ‘mapped’
(connected via VPN) to the
Transit VPC
Transit VPCs – Two vEdge-

Cloud Instances – These
connect to the on-premises
via SD-WAN setup
AWS – Host VPC –to- Transit VPC Mapping - IPsec
vEdge-Cloud – Transit VPC GatewayVpc
interface ipsec1 Management VPN
ip address 169.254.14.130/30
172.18.2.22
tunnel-source 172.18.2.40 Source NATed to 52.27.x.x
vpn 512
tunnel-destination 34.211.x.x
ike
version 1 HostVpc (172.16.0.0/16) Service VPN
mode main vpn 1
rekey 28800 VPN 192.168.254.5
cipher-suite aes128-cbc-sha1 Tunnel 34.211.x.x
AS 65512
pod BGP Instance
Transport VPN
172.18.2.40
AS 9988:169.254.14.130
vpn 0
group 2 169.254.13.129
authentication-type EKS EIP
pre-shared-key Worker 52.27.x.x
Node
pre-shared-secret <PSK_KEY> pod
!
! EKS VPC VPN GW
! Worker Router (VGW)
Node
ipsec
pod
rekey 3600
replay-window 512 EKS
cipher-suite aes256-cbc-sha1 Worker
Node
perfect-forward-secrecy group-16
AWS – Host VPC –to- Transit VPC Mapping - BGP
vEdge-Cloud – Transit VPC HostVpc (172.16.0.0/16)
GatewayVpc
Management VPN
vpn 1
172.18.2.22
router VPN
Tunnel 34.211.x.x
vpn 512
AS 65512
bgp 9988 pod BGP Instance
169.254.14.129
timers
holdtime 30
EKS EIP Service VPN
Worker
! Node vpn 1
address-family ipv4-unicast pod 192.168.254.5
network 0.0.0.0/0 Transport VPN
172.18.2.40
EKS VPN GW AS 9988:169.254.14.130
vpn 0
VPC
! Worker Router (VGW)
Node
neighbor 169.254.14.129 pod
no shutdown 52.27.x.x
remote-as 64512 EKS
Worker
update-source ipsec1 Node
transit-aws-03# show ip route

OUTPUT OMITTED...
PROTOCOL NEXTHOP NEXTHOP NEXTHOP
VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS
-------------------------------------------------------------------------------------------------------------------------------------
1 172.16.0.0/16 bgp e ipsec1 169.254.14.129 - - - - F,S
Transit VPC –to- On-Premises - IPsec
Transit VPC vEdge - IPsec
transit-aws-03## show ipsec outbound-connections
OUTPUT SUMMARIZED...
SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION

IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED
--------------------------------------------------------------------------------------------------------------------------------------
172.18.2.40 12426 <ON_PREMISES_vEDGE_PUBLIC_IP> 12426 290 1441 1.1.1.4 public-internet AH_SHA1_HMAC
On-Premises vEdge - IPsec
vedge-01# show ipsec outbound-connections

--------------------------------------------------------------------------------------------------------------------------------------
<ON_PREMISES_vEDGE_PUBLIC_IP> 12426 <TRANSIT-vEDGE-EIP> 12366 257 1441 60.1.1.2 default AH_SHA1_HMAC
System IP (TLOC): Cisco

60.1.1.1 Container
Transport VPN Platform
Data Center
vpn 0
CCP Control Plane

System IP (TLOC): pod pod pod
1.1.1.4
Infra.
VM VM VM VM VM VM
HYPERVISOR
vedge
vpn 0
CCP
Transport VPN Cluster
System IP (TLOC): Network
60.1.1.2 10.1.1.0/24
Transit VPC -to- On-Premises - BGP/OMP
Transit VPC vEdge - BGP/OMP
--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 omp - - - - 1.1.1.4 public-internet ipsec F,S
1 172.16.0.0/16 bgp e ipsec1 169.254.14.129 - - - - F,S
On-Premises vEdge - Connected/OMP

vedge-01# show ip route
--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
1 172.16.0.0/16 omp - - - - 60.1.1.1 default ipsec F,S
Cisco
HostVpc 172.16.0.0/16 GatewayVpc Container
Platform
VPN Service VPN
Tunnel 34.211.x.x System IP (TLOC):
CCP Control Plane
AS 65512
BGP Instance pod pod pod
Data Center
169.254.14.129 vpn 1 1.1.1.4
192.168.254.5
Infra.
EIP VM VM VM VM VM VM
Transport VPN
172.18.2.40
AS 9988:169.254.14.130
vpn 0
pod HYPERVISOR
vedge
EKS VPC VPN GW
Worker (VGW) 52.27.x.x
Router
Node
10.1.1.0/24
Traffic Flow Example
HostVpc 172.16.0.0/16 GatewayVpc

VPN Service VPN 3
Tunnel 34.211.x.x Kubernetes
AS 65512
BGP Instance
169.254.14.129 vpn 1
5 192.168.254.5 Worker/Pod
EIP pod 10.51.1.20
Pod:172.16.3.206 Transport VPN VM
172.18.2.40
AS 9988:169.254.14.130
vpn 0
pod 10.1.1.137
172.16.3.219 vedge 2 10.1.1.104
EKS VPN GW 1
4 VPC 52.27.x.x
Worker Router (VGW)
Node
/ # traceroute 172.16.3.206 -n
1 1 10.1.1.104 0.003 ms 0.002 ms 0.002 ms
2 2 10.1.1.137 0.153 ms 0.202 ms 0.173 ms
3 3 192.168.254.5 2.453 ms 2.535 ms 2.566 ms
4 4 172.16.3.219 5.123 ms 4.123 ms 4.206 ms
5 5 172.16.3.206 5.743 ms 5.723 ms 5.206 ms
Internet Exit Routing Considerations (1)
• By default, Cloud onRamp reconfigures the VPC route tables so that all traffic
traverses the transit vEdges - Great for Enterprise InfoSec policies
AWS Route Table Before CoR:
||+---------+------------------------------------------+-----------------------------------+||
||| Routes |||
||+-------------------------+---------------------------+----------------------+-----------+||
||| DestinationCidrBlock | GatewayId | Origin | State |||
||+-------------------------+---------------------------+----------------------+-----------+||
||| 172.16.0.0/24 | local | CreateRouteTable | active |||
||| 0.0.0.0/0 | igw-03da938263fb0b4a0 | CreateRoute | active |||
||+-------------------------+---------------------------+----------------------+-----------+||
AWS Route Table After CoR:

||+----------------------------------------------------------------------------------------+||
||| Routes |||
||+----------------------+-------------------------+-----------------------------+---------+||
||| DestinationCidrBlock | GatewayId | Origin | State |||
||+----------------------+-------------------------+-----------------------------+---------+||
||| 172.16.0.0/16 | local | CreateRouteTable | active |||
||| 0.0.0.0/0 | vgw-04a174cc628b9463a | EnableVgwRoutePropagation | active |||
||+----------------------+-------------------------+-----------------------------+---------+||
• If you want to have specific traffic or all non-on-premises traffic leave the transit
vEdges directly:
• https://sdwan-
docs.cisco.com/Product_Documentation/Software_Features/Release_18.4/07Policy_Applicati
ons/04Using_a_vEdge_Router_as_a_NAT_Device/Configuring_Local_Internet_Exit
• Perform NAT on a WAN/Transport VPN (e.g., VPN 0) for specific or all destinations not found in the transit
routing table
• Create a Data Policy to do NAT per-VPN for specific or all destinations not found in the transit routing table
HostVpc
Transit vEdge VPN

Tunnel
Service VPN
vpn 0 vpn 1
interface ge0/0 192.168.254.5
EIP
nat Transport VPN
172.18.2.40
vpn 0
! pod
vpn 1
EKS VPC VPN GW
ip route 0.0.0.0/0 null0 Worker (VGW)
Router
ip route x.x.x.x/32 vpn 0 Node
NAT INET
------------------------------------------------------------------------------------------------------------------------------------
1 x.x.x.x/32 nat - ge0/0 - 0 - - - F,S
Cisco SD-WAN Cloud
onRamp for IaaS -
Azure
Cloud OnRamp IaaS for Azure
1 3
Cisco SD-WAN CoR for Azure
GatewayVNet
az-tz-vnet
Management VPN
vpn 512
10.0.0.4
Host VNet (10.16.0.0/16) Service VPN
Vnet0
vpn 1
Gateway Subnet 192.168.254.1 Cisco
10.16.250.0/28
Container
BGP Instance 1 AS 65512:192.168.250.1 Transport VPN
AS 65512
10.0.32.5
vpn 0
10.16.250.4 AS 65512:192.168.250.5 Platform
VPN GW Enterprise Data
BGP Instance 2 Edge Center
CCP Control Plane
pod pod pod
10.16.250.5
168.61.x.x 40.85.x.x
168.61.x.x
ACI 0.4 container- 40.85.x.x VM VM VM VM VM VM
subnet
10.0.32.4
10.16.241.0/24 VPN CCP HYPERVISOR
vpn 0
appcontainer Tunnels
AS 65512:192.168.250.9 Cluster
AS 65512:192.168.250.13 Transport VPN Network
10.1.1.0/24
Azure Container Instance (ACI) vpn 1
192.168.254.2
Service VPN
vpn 512
10.0.0.5
Management VPN
Azure – Host VNet –to- Transit VNet Mapping - IPsec
vEdge-Cloud – Transit VNet
interface ipsec9 GatewayVNet
az-tz-vnet
ip address 192.168.250.1/30 Management VPN
tunnel-source 10.0.32.5 Source NATed to 40.85.x.x
vpn 512
10.0.0.4
ike Host VNet (10.16.0.0/16) Service VPN
version 2 Vnet0
rekey 28800 vpn 1
Gateway Subnet 192.168.254.1
cipher-suite aes128-cbc-sha1 10.16.250.0/28
group 2 BGP Instance 1 AS 64600:192.168.250.1 Transport VPN
AS 65512
10.0.32.5
vpn 0
authentication-type 10.16.250.4
VPN GW
pre-shared-key
pre-shared-secret <PSK_HERE> 168.61.x.x 40.85.x.x
!
ACI 0.4 container-
! subnet
! 10.16.241.0/24 VPN
ipsec
rekey 3600
replay-window 512
cipher-suite aes256-cbc-sha1
perfect-forward-secrecy none
Azure – Host Vnet –to- Transit VNet Mapping - BGP
GatewayVNet
vEdge-Cloud – Transit VNet az-tz-vnet
Management VPN
vpn 512
10.0.0.4
vpn 1
router
bgp 64600 Vnet0
timers vpn 1
holdtime 30 Gateway Subnet 192.168.254.1
! 10.16.250.0/28
10.0.32.5
AS 65512
vpn 0
address-family ipv4-unicast 10.16.250.4
network 0.0.0.0/0 VPN GW
!
neighbor 10.16.250.4 168.61.x.x 40.85.x.x
no shutdown ACI 0.4 container-
remote-as 65512 subnet
10.16.241.0/24 VPN
update-source ipsec9 appcontainer Tunnels
ebgp-multihop 2
transit-az-01# show ip route

OUTPUT OMITTED... Received
-------------------------------------------------------------------------------------------------------------------------------------
1 10.16.0.0/16 bgp i - 10.16.250.4 - - - - F,S,R
# az network vnet-gateway list-advertised-routes --name COR_Vnet0_Virtual_Network_Gateway --peer 192.168.250.1 --output yaml

value:
- asPath: '65512'
localAddress: 10.16.250.4
network: 10.16.0.0/16
nextHop: 10.16.250.4
Advertised
origin: Igp
sourcePeer: null #CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Transit VNet –to- On-Premises - IPsec
Transit VNet vEdge - IPsec
transit-az-01## show ipsec outbound-connections

--------------------------------------------------------------------------------------------------------------------------------------
On-Premises vEdge - IPsec
vedge-01# show ipsec outbound-connections

--------------------------------------------------------------------------------------------------------------------------------------
System IP (TLOC):
40.1.1.1
Transport VPN
10.0.32.5
vpn 0
System IP (TLOC):
1.1.1.4
Container Network
10.1.1.0/24
10.0.32.4
vedge
vpn 0
Transport VPN
System IP (TLOC): On-Premises
40.1.1.2
Transit VNet –to- On-Premises - BGP
Transit VNet vEdge - BGP
--------------------------------------------------------------------------------------------------------------------------------------
1 10.16.0.0/16 bgp i - 10.16.250.4 - - - - F,S,R
On-Premises vEdge - BGP

--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
Host VNet (10.16.0.0/16)

Vnet0 GatewayVNet
Gateway Subnet
10.16.250.0/28 Service VPN System IP (TLOC):
BGP Instance 1 vpn 1 System IP (TLOC): 1.1.1.4
AS 65512
10.16.250.4
192.168.254.1 40.1.1.1
VPN GW Container Network
AS 64600:192.168.250.1 Transport VPN 10.1.1.0/24
10.0.32.5
vpn 0
168.61.x.x
vedge
ACI 0.4 container- 40.85.x.x
subnet
appcontainer
10.16.241.0/24 VPN
Tunnels On-Premises
Cisco SD-WAN CoR for Azure
Traffic Flow Example az-tz-vnet GatewayVNet
Management VPN
vpn 512
10.0.0.4
Vnet0
vpn 1 vManage vBond vSmart
Gateway Subnet 192.168.254.1
10.16.250.0/28
10.0.32.5
AS 65512
vpn 0
10.16.250.4 AS 64600:192.168.250.5
VPN GW
BGP Instance 2
10.16.250.5 pod Container
168.61.x.x 40.85.x.x
0.4
168.61.x.x
40.85.x.x
10.1.1.137 Network
ACI container- VM
subnet vedge
10.0.32.4
10.16.241.0/24 VPN 10.1.1.106 10.1.1.0/24
vpn 0
AS 64600:192.168.250.9
AS 64600:192.168.250.13 Transport VPN
vpn 1 On-Premises
192.168.254.2
Service VPN
vpn 512
10.0.0.5
Management VPN
/ # traceroute 10.16.241.4 -n
1 10.1.1.106 0.006 ms 0.004 ms 0.028 ms
2 10.1.1.137 0.153 ms 0.202 ms 0.173 ms
3 192.168.254.1 2.453 ms 2.535 ms 2.566 ms
4 168.61.17.148 4.272 ms 3.467 ms 3.287 ms
#CiscoLive
5 10.16.241.4 5.123 ©ms2020 4.123
Cisco and/or
msits affiliates.
4.206 All rights
ms reserved. Cisco Public 89
• By default, Cloud onRamp reconfigures the VPC/VNet route tables so that all traffic
traverses the transit vEdges - Great for Enterprise InfoSec policies
Azure Route Table Before CoR:
PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName corvm444 -ResourceGroupName CtoRG | Format-Table

---- ----- ------ ------------- ----------- ----------------
Active Default {10.16.0.0/16} VnetLocal {}
Active Default {0.0.0.0/0} Internet {}
Active Default {10.0.0.0/8} None {}
Azure Route Table After CoR:

PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName corvm444 -ResourceGroupName CtoRG | Format-Table

---- ----- ------ ------------- ----------- ----------------
Active Default {10.16.0.0/16} VnetLocal {}
Active VirtualNetworkGateway {192.168.250.1/32} VirtualNetworkGateway {10.16.250.4}
• If you want to have specific traffic or all non-on-premises traffic leave the transit
vEdges directly:
• https://sdwan-
docs.cisco.com/Product_Documentation/Software_Features/Release_18.4/07Policy_Applicati
ons/04Using_a_vEdge_Router_as_a_NAT_Device/Configuring_Local_Internet_Exit
• Perform NAT on a WAN/Transport VPN (e.g., VPN 0) for specific or all destinations not found in the transit
routing table
• Create a Data Policy to do NAT per-VPN for specific or all destinations not found in the transit routing table
Service VPN
Transit vEdge
vpn 0 vpn 1
VPN GW
192.168.254.1
interface ge0/0
nat Transport VPN
10.0.32.5
vpn 0
! VPN
vpn 1 0.4 Vnet0-subnet Tunnel
ip route 0.0.0.0/0 null0 10.16.241.0/24
NAT
INET
ip route x.x.x.x/32 vpn 0 Container

------------------------------------------------------------------------------------------------------------------------------------
1 x.x.x.x/32 nat - ge0/0 - 0 - - - F,S
Link Them Altogether
Cisco SD-WAN and Multicloud
10.16.241.4 Transit VNet

pod
Container
Azure VPN vEdge
GW
VM
Internet
vEdge 10.1.1.106
Transit VPC
Pod:172.16.3.206
pod
SD-WAN
EKS
Worker vEdge
Node AWS VPN
GW
SD-WAN Multicloud Routing
Transit VNet vEdge - BGP
--------------------------------------------------------------------------------------------------------------------------------------
1 10.16.0.0/16 bgp i - 10.16.250.4 - - - - F,S,R
Transit VPC vEdge - BGP

--------------------------------------------------------------------------------------------------------------------------------------
1 172.16.0.0/16 bgp e ipsec1 169.254.14.129 - - - - F,S
On-Premises vEdge - BGP

--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
Common Design
Challenges
Multi-Region VPC with
Hybrid Cloud
Dealing with VPC/VNet Peering - Transitive Limitations
Azure Example
Support for VPC/VNet Transitive Routing:
• VPC/VNet and Peering (not transitive) AWS
Azure
GCP
Azure
Note: Azure Gateway Transit Support
VNet-A
Azure
VNet Peering* VNet-Hub Internet
Azure vEdge
VNet-B On Premises
2nd Note: AWS Transit GW supports
VPC-to-VPC without VPC Peering
• VPC/VNet peering with appliances (transitive support)

Azure
VNet-A vEdge/CSR
Internet
Azure vEdge
VNet-B Azure VNet- On Premises
Transit-VPC
*Azure VNet Peering limit = 500

Combining Native Cloud VPN
Services with On-Prem
Designs
AWS Transit Gateway
Service + Cisco SD-WAN
CoR
Linking Cisco SD-WAN to AWS Transit Gateway
Transit VPC Option Transit VPC

CSR/vEdge
Cisco Stuff
VPC VPN IPsec On-Premises
Router Gateway
Transit Gateway Option

Elastic Network AWS Transit VPN Tunnel Cisco Stuff
Interface Gateway IPsec On-Premises
https://aws.amazon.com/transit-gateway/
AWS Transit Gateway - Connection Options
• AWS Transit Gateway (TGW) uses ”attachments” to
connect to:
• VPCs - Attaches an Elastic Network Interface (ENI) to each
VPC subnet
• VPN - Creates a VPN tunnel with one or more attachments
to a customer gateway (CGW)
• AWS TGW can be used with:

• Site-to-Site VPN connection to on-premises
routers/firewalls
• Site-to-Site VPN connection to a new/existing transit VPC
deployment (to include Cisco SD-WAN Cloud onRamp)
• AWS Direct Connect Gateway
• AWS/Cisco Partnership Design:

https://aws.amazon.com/blogs/apn/exploring-architectures-with-cisco-sd-wan-
and-aws-transit-gateway/?_lrsc=9e764f01-0249-4319-ba87-
274cffc80f04&dtid=osolin001080
Cisco SD-WAN Connecting to AWS Transit
Gateway
VPC (172.31.0.0/16)
VPN Cisco
172.18.2.40
Tunnel 52.34.x.x AS 9988:169.254.15.166
vpn 0
AS 64512
BGP Instance
169.254.15.165 AS 9988:169.254.12.178 Platform
ENI Enterprise
VPC BGP Instance EIP Edge Data
54.27.x.x
CCP Control Plane
Subnet 1 169.254.12.177 Center pod pod pod
52.38.x.x
VPC
Subnet 2 VM VM VM VM VM VM
172.18.2.148
VPC HYPERVISOR
Subnet 3 AWS Transit 52.33.x.x
54.68.y.x CCP
VPC Gateway
vpn 0
Subnet 4 Network
BGP Instance AS 9988: 169.254.14.174
AS 64512
169.254.14.173 EIP AS 9988: 169.254.14.162 Transport VPN 10.1.1.0/24

VPN 54.191.y.x
BGP Instance Tunnel
169.254.14.161
AWS Transit Gateway Setup (1)
NOTE: If you use vManage and templates

for your configuration, YOU need to set
your own PSK here - Known issue with
PSK length.
Use the contents of the files to build your router configs

172.31.0.0/16 Elastic Network AWS Transit VPN Tunnel 10.1.1.0/24
Interface vEdge vEdge On-Premises
Gateway
IPsec
AWS TGW –to- vEdge Mapping - IPsec
vEdge-Cloud
interface ipsec4
ip address 169.254.15.166/30
tunnel-source 172.18.2.40
Source NATed to 52.27.x.x
ike
version 1 VPC (172.31.0.0/16)
mode main
rekey 28800
VPN
cipher-suite aes128-cbc-sha1 Tunnel 52.34.x.x
AS 64512
BGP Instance
group 2 Transport VPN
172.18.2.40
169.254.15.165 AS 9988:169.254.15.166
vpn 0
authentication-type
VPC EIP
pre-shared-key Subnet 1
pre-shared-secret <PSK_HERE> 54.27.x.x
VPC
! Subnet 2
! VPC
! Subnet 3 AWS Transit
ipsec VPC Gateway
rekey 3600 Subnet 4
replay-window 128
perfect-forward-secrecy group-2 Contents of this file used to build config
!
tcp-mss-adjust 1379
AWS TGW –to- vEdge Mapping - BGP
vEdge-Cloud
vpn 1
router
bgp 9988 VPC (172.31.0.0/16)
propagate-aspath
timers VPN
Tunnel 52.34.x.x
AS 64512
keepalive 10 BGP Instance
Transport VPN
172.18.2.40
169.254.15.165 AS 9988:169.254.15.166
vpn 0
holdtime 30
VPC EIP
! Subnet 1
address-family ipv4-unicast 54.27.x.x
VPC
maximum-paths paths 4 Subnet 2
redistribute omp VPC

Subnet 3 AWS Transit
!
VPC Gateway
neighbor 169.254.15.165 Subnet 4
no shutdown
remote-as 64512
OUTPUT OMITTED...
-------------------------------------------------------------------------------------------------------------------------------------
1 172.31.0.0/16 bgp e ipsec3 169.254.12.177 - - - - F,S
1 172.31.0.0/16 bgp e ipsec4 169.254.15.165 - - - - F,S
#CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
AWS TGW IPsec Connections to On-Prem & TGW
vEdge Cloud - IPsec to On-Premises vEdge
transit-aws-03## show ipsec outbound-connections

--------------------------------------------------------------------------------------------------------------------------------------
vEdge Cloud - IPsec to TGW VPN Tunnels
transit-aws-03# show ipsec ike outbound-connections
SOURCE SOURCE DEST DEST CIPHER EXT

IP PORT IP PORT SPI SUITE KEY HASH TUNNEL MTU SEQ
--------------------------------------------------------------------------------------------------------------------------
172.18.2.40 4500 52.34.x.x 4500 28958975 aes256-cbc-sha1 ******** 1418 no
172.18.2.40 4500 52.38.x.x 4500 2844174670 aes256-cbc-sha1 ******** 1418 no
VPC (172.31.0.0/16)
VPN
Tunnel 52.34.x.x
AS 64512
BGP Instance
169.254.15.165
VPC BGP Instance EIP
Subnet 1 169.254.12.177
52.38.x.x Transport VPN Enterprise
172.18.2.40
vpn 0
VPC Edge
Subnet 2
10.1.1.0/24
VPC
54.27.x.x
VPC Gateway
Subnet 4
AWS TGW Routing to On-Prem & TGW
vEdge Cloud - OMP/BGP
--------------------------------------------------------------------------------------------------------------------------------------
1 172.31.0.0/16 bgp e ipsec3 169.254.12.177 - - - - F,S
1 172.31.0.0/16 bgp e ipsec4 169.254.15.165 - - - - F,S

--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
VPC (172.31.0.0/16)
VPN
Tunnel 52.34.x.x “redistribute omp”
AS 65512
BGP Instance
169.254.15.165
VPC BGP Instance EIP
Subnet 1 169.254.12.177
52.38.x.x Enterprise
Transport VPN
172.18.2.40
vpn 0
VPC Edge
Subnet 2 BGP 10.1.1.0/24
VPC
54.27.x.x
OMP
VPC Gateway
Subnet 4
vEdge Cloud Example Configuration (only one IPsec/BGP connection shown)
omp
no shutdown interface ipsec4
graceful-restart ip address 169.254.15.166/30
advertise bgp tunnel-source 172.18.2.40
advertise connected tunnel-destination 52.34.x.x
ike
advertise static
! version 1
security mode main
ipsec rekey 28800
authentication-type sha1-hmac ah-sha1-hmac cipher-suite aes128-cbc-sha1
group 2
!
vpn 1 authentication-type
router pre-shared-key
bgp 9988 pre-shared-secret <PSK_HERE>
propagate-aspath !
!
timers
keepalive 10 ipsec
holdtime 30 rekey 3600
! replay-window 128
address-family ipv4-unicast cipher-suite aes256-cbc-sha1
maximum-paths paths 4
redistribute omp !
! tcp-mss-adjust 1379
neighbor 169.254.15.165 no shutdown
no shutdown
remote-as 64512
!
!
!
Combining Native
Cloud VPN Services
with On-Prem Designs
AWS Transit Gateway
Service + Cisco SD-
WAN
AWS Transit Gateway - Connection Options
• AWS Transit Gateway (TGW) uses ”attachments” to
connect to:
• VPCs - Attaches an Elastic Network Interface (ENI) to each
VPC subnet
• VPN - Creates a VPN tunnel with one or more attachments
to a customer gateway (CGW)
• AWS TGW can be used with:

• Site-to-Site VPN connection to on-premises
routers/firewalls
• Site-to-Site VPN connection to a new/existing transit VPC
deployment (to include Cisco SD-WAN Cloud onRamp)
• AWS Direct Connect Gateway
• AWS/Cisco Partnership Design:

https://aws.amazon.com/blogs/apn/exploring-architectures-with-cisco-sd-wan-
and-aws-transit-gateway/?_lrsc=9e764f01-0249-4319-ba87-
274cffc80f04&dtid=osolin001080
Cisco SD-WAN Connecting to AWS Transit
Gateway

VPC (172.31.0.0/16)
Cisco
AS 64512
BGP Instance
ENI
169.254.68.41 Container
VPC BGP Instance Platform
VPN AS 9988:169.254.68.42/30 Enterprise
Subnet 1 169.254.252.185
Tunnel AS 9988:169.254.252.186/30 Edge Data
44.228.x.x
CCP Control Plane
VPC Center pod pod pod
Subnet 2 192.x.x.x
VPC 52.11.x.x VM VM VM VM VM VM
Subnet 3 AWS Transit EIP

HYPERVISOR
VPC Gateway
Subnet 4 CCP
Network
10.1.1.0/24
NOTE: If you use vManage and templates

for your configuration, YOU need to set
your own PSK here - Known issue with
PSK length.
Use the contents of the files to build your router configs
Private Network(s)
VPC Subnet(s)
10.1.1.0/24
172.31.0.0/16 Elastic Network AWS Transit VPN Tunnel vEdge On-Premises
Interface Gateway IPsec
AWS TGW –to- vEdge Mapping - IPsec
vEdge-Cloud
interface ipsec3
ip address 169.254.68.42/30
tunnel-source 192.x.x.x
ike
version 1 VPC (172.31.0.0/16)
mode main
rekey 28800 ge0/1
VPN
cipher-suite aes128-cbc-sha1 Tunnel
10.1.1.0/24
BGP Instance 44.228.x.x
AS 64512
group 2 169.254.68.41 AS 9988:169.254.68.42/30
vpn 0
192.x.x.x
authentication-type
VPC EIP
pre-shared-key Subnet 1
pre-shared-secret <PSK_HERE> 192.x.x.x
VPC
! Subnet 2 Transport VPN
! VPC
! Subnet 3 AWS Transit
ipsec VPC Gateway
rekey 3600 Subnet 4
replay-window 128
perfect-forward-secrecy group-2 Contents of this file used to build config
!
tcp-mss-adjust 1379
AWS TGW –to- vEdge Mapping - BGP
vEdge-Cloud
vpn 1
router
bgp 9988 VPC (172.31.0.0/16)
propagate-aspath
timers ge0/1
VPN 10.1.1.0/24
keepalive 10 BGP Instance Tunnel
44.228.x.x
AS 64512
169.254.68.41 AS 9988:169.254.68.42/30
vpn 0
holdtime 30
192.x.x.x
! VPC EIP
Subnet 1
address-family ipv4-unicast 192.x.x.x
VPC
maximum-paths paths 4 Subnet 2 Transport VPN
redistribute connected VPC
redistribute omp Subnet 3 AWS Transit
! VPC Gateway
Subnet 4
neighbor 169.254.68.41
no shutdown
remote-as 64512
vEdge-01# show ip route
OUTPUT OMITTED...
-------------------------------------------------------------------------------------------------------------------------------------
1 172.31.0.0/16 bgp e ipsec3 169.254.68.41 - - - - F,S
1 172.31.0.0/16 bgp e ipsec4 169.254.252.185 - - - - F,S
#CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
AWS TGW IPsec Connections to On-Prem & TGW
vEdge Cloud - IPsec to TGW VPN Tunnels
vEdge-01# show ipsec ike outbound-connections
SOURCE SOURCE DEST DEST CIPHER EXT

IP PORT IP PORT SPI SUITE KEY HASH TUNNEL MTU SEQ
--------------------------------------------------------------------------------------------------------------------------
192.x.x.x 4500 44.228.x.x 4500 660717429 aes256-cbc-sha1 ******** 1418 no
192.x.x.x 4500 52.11.x.x 4500 4040796161 aes256-cbc-sha1 ******** 1418 no
VPC (172.31.0.0/16)
ge0/1
VPN 10.1.1.0/24
BGP Instance Tunnel 44.228.x.x
AS 64512
169.254.68.41 AS 9988:169.254.68.42/30
vpn 0
192.x.x.x
VPC EIP
Subnet 1
192.x.x.x
VPC
Subnet 2 Transport VPN
VPC
VPC Gateway
Subnet 4
AWS TGW Routing to On-Prem & TGW
--------------------------------------------------------------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 - - - - - F,S
1 172.31.0.0/16 bgp - - 169.254.68.41 - - - - F,S
1 172.31.0.0/16 bgp - - 169.254.252.185 - - - - F,S
VPC (172.31.0.0/16)
ge0/1
VPN 10.1.1.0/24
BGP Instance Tunnel AS 9988:169.254.68.42
44.228.x.x
AS 64512
169.254.68.41 AS 9988:169.254.252.186
vpn 0
192.x.x.x
BGP Instance
VPC 169.254.252.185
Subnet 1 52.11.x.x
EIP 192.x.x.x
VPC
Subnet 2 Transport VPN
VPC
VPC Gateway
Subnet 4
AWS TGW - You aren’t done yet
• You will have to figure out how you want to update your per-VPC
route tables
• There are many options to include:
• Update VPC route table to point to specific routes (10.1.1.0/24) to aim at
the TGW as the target for the route
• Update VPC route table to point to a wild card or default (0.0.0.0/0) to
aim at the TGW as the target
• Be careful when you have a route to an IGW
vEdge Cloud Example Configuration (only one IPsec/BGP connection shown)
omp
no shutdown interface ipsec3
graceful-restart ip address 169.254.68.42/30
advertise bgp tunnel-source 192.x.x.x
advertise connected tunnel-destination 44.228.x.x
ike
advertise static
! version 1
security mode main
ipsec rekey 28800
authentication-type sha1-hmac ah-sha1-hmac cipher-suite aes128-cbc-sha1
group 2
!
vpn 1 authentication-type
router pre-shared-key
bgp 9988 pre-shared-secret <PSK_HERE>
propagate-aspath !
!
timers
keepalive 10 ipsec
holdtime 30 rekey 3600
! replay-window 128
address-family ipv4-unicast cipher-suite aes256-cbc-sha1
maximum-paths paths 4
redistribute omp !
! tcp-mss-adjust 1379
neighbor 169.254.68.41 no shutdown
no shutdown
remote-as 64512
!
!
!
AWS Transit Gateway
+ Multi-VPC + Multi-
Region + SD-WAN
AWS Transit Gateway Multi-VPC
VPC Subnet(s) us-west-2
172.32.0.0/16
Private Network(s)
VPC Subnet(s)
10.1.1.0/24
Interface Gateway IPsec
AWS Transit Gateway Multi-Region + Multi-VPC - Setup
1
us-west-2
2 us-east-2
us-east-2
AWS Transit
Gateway
VPC Subnet(s)
172.30.0.0/16 Elastic Network
Connection
Interface
Peering
172.32.0.0/16
VPC Subnet(s)
172.31.0.0/16 Elastic Network AWS Transit
Interface
VPN Tunnel
Gateway
AWS Transit Gateway Multi-Region + Multi-VPC - Route Tables
us-east-2 -Create Peering Routes
us-west-2 - Create Peering Routes
us-east-2
AWS Transit
Gateway
VPC Subnet(s)
172.30.0.0/16 ENI
Connection
Peering
172.32.0.0/16 Private Network(s)

VPC Subnet(s) 10.1.1.0/24
Interface Gateway
IPsec
#CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
AWS Transit Gateway Multi-Region + Multi-VPC - Routes
us-west-2

PROTOCOL NEXTHOP NEXTHOP
VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR
---------------------------------------------------------------------------------
1 10.1.1.0/24 connected - ge0/1 -
us-east-2 1 172.30.0.0/16 bgp e ipsec4 169.254.252.185
AWS Transit 1 172.30.0.0/16 bgp e ipsec3 169.254.68.41
Gateway
1 172.31.0.0/16 bgp e ipsec3 169.254.68.41
1 172.31.0.0/16 bgp e ipsec4 169.254.252.185
VPC Subnet(s) 1 172.32.0.0/16 bgp e ipsec3 169.254.68.41
1 172.32.0.0/16 bgp e ipsec4 169.254.252.185
172.30.0.0/16 ENI
Connection
Peering
172.32.0.0/16 Private Network(s)

VPC Subnet(s) 10.1.1.0/24
Interface Gateway
IPsec
*Examples only!
AWS TGW - VPC Routes This is your mess to figure out J
• You will have to figure out how you want to update your per-VPC
route tables
us-east-2 172.30.0.0/16
us-west-2 172.31.0.0/16
us-west-2 172.32.0.0/16
Using ECMP to
Increase Throughput
https://aws.amazon.com/blogs/networking-and-content-
delivery/scaling-vpn-throughput-using-aws-transit-gateway/
Equal Cost Multi-path (ECMP)

VPC (172.31.0.0/16) AS 65002:
169.254.209.174/30
169.254.206.66/30
AS 64512
BGP Instance
169.254.209.173
ENI 169.254.55.74/30
VPC BGP Instance 52.34.x.x
VPN 169.254.175.198/30
Subnet 1 169.254.206.65
Tunnel
VPC 52.39.x.x
Subnet 2
VPC
34.215.x.x 10.1.1.0/24
Subnet 3 AWS Transit Cisco ASR1k, ISR,
VPC Gateway 44.233.x.x CSR
Subnet 4
AS 64512
BGP Instance EIP
169.254.55.73
BGP Instance
169.254.175.197
ASR1K_1#sh ip route bgp

...output summarized
B 172.31.0.0/16 [20/100] via 169.254.209.173, 00:00:29
[20/100] via 169.254.206.65, 00:00:29
[20/100] via 169.254.175.197, 00:00:29
[20/100] via 169.254.55.73, 00:00:29
ECMP Configuration Considerations ... Output summarized
ip cef load-sharing algorithm include-ports destination
!
router bgp 65002
• ECMP-related configs with Cisco routers and public cloud bgp log-neighbor-changes
neighbor 169.254.55.73 remote-as 64512
routers will vary wildly based on the application use case, neighbor 169.254.55.73 timers 10 30 30
number of links, cloud BGP support and other stuff neighbor 169.254.175.197 remote-as 64512
neighbor 169.254.175.197 timers 10 30 30
• Basics: neighbor 169.254.206.65 remote-as 64512
neighbor 169.254.206.65 timers 10 30 30
• Adjust BGP for maximum paths neighbor 169.254.209.173 remote-as 64512
neighbor 169.254.209.173 timers 10 30 30
• Test multiple ‘ip cef load-sharing’ algorithms and validate those
!
against your application use cases (i.e., ‘include-ports address-family ipv4
destination’ when you have large port variance) redistribute ospf 10
neighbor 169.254.55.73 activate
• Your BGP configs are heavily dependent on what the CSP neighbor 169.254.55.73 default-originate
supports on their side (they all give you a template to download) neighbor 169.254.55.73 soft-reconfiguration inbound
AS 64512
BGP Instance
169.254.209.173 neighbor 169.254.175.197 default-originate
BGP Instance 52.34.x.x neighbor 169.254.175.197 soft-reconfiguration inbound
VPN
169.254.206.65
Tunnel neighbor 169.254.206.65 activate
52.39.x.x neighbor 169.254.206.65 default-originate
neighbor 169.254.206.65 soft-reconfiguration inbound
34.215.x.x 10.1.1.0/24
AWS Transit Cisco ASR1k, ISR, neighbor 169.254.209.173 default-originate
Gateway 44.233.x.x CSR neighbor 169.254.209.173 soft-reconfiguration inbound
AS 64512
BGP Instance EIP maximum-paths 4

169.254.55.73
exit-address-family
BGP Instance
169.254.175.197
Dealing with VPC CIDR
Ranges that Overlap with
On-Prem CIDR Ranges
Dancing with the NAT
Devil
What is the issue and what to do?
• Someone picks a CIDR range for a new VPC/VNet and it overlaps with a CIDR range used
somewhere else:
• In another VPC/VNet
• An On-Prem CIDR
• First off, don’t do this

• Use an IPAM tool (it could be Microsoft Excel J) to track CIDR ranges in use
• Use anything to avoid using NAT - Get creative, if possible, use stuff like AWS PrivateLink
(https://aws.amazon.com/privatelink/), Azure Private Link (https://azure.microsoft.com/en-us/services/private-link/),
GCP Private Google Access (https://cloud.google.com/vpc/docs/private-access-options)
• If you must do this then do it carefully:

• Try to deploy whatever you are using for NAT as close to the collision spot as possible
• If you are going to do a lot of NAT to deal with a lot of overlap, then forget what I said above and aggregate your NAT on
a highly scalable device
• There are MANY limitations with trying to use the native NAT services at the Cloud provider (e.g. AWS NAT GW or NAT
Instance)
• They require an External IP and typically can’t NAT to a VPN service
• You can’t route traffic through the NAT service from another VPC/VNet (e.g. VPC/VNet peering)
• They are source NATing from the VPC/VNet subnet side
NAT Designs N
NAT at On-Prem Cisco GW(s) A
T
VPC CIDR 172.16.0.0/16
172.16.0.0/16 VPN Cisco
VPC VPN Router/Firewall On-Premises
Router Gateway
N
NAT at CSP Cisco GW(s) A
T
VPC CIDR 172.16.0.0/16
172.16.0.0/16 VPN Cisco
VPC Cisco Router/ Router/Firewall On-Premises
Router Firewall
N
A
NAT at Transit VPC Cisco GW(s) T
VPC CIDR 172.16.0.0/16

172.16.0.0/16 VPN Cisco
Cisco On-Premises
VPC VPN Router/Firewall
Router Gateway Routers
Transit VPC
N
NAT at Colocation Cisco GW(s) A
T
VPC Subnet(s) VLANs 172.16.0.0/16
172.16.0.0/16 Cisco
VPC DX Cisco
VPN
Endpoint Router/Firewall
Router/Firewall On-Premises
Router Gateway
VPN
DMVPN – Dynamic
Multipoint VPN
Merging in Multicloud to
an Existing Branch/WAN
Deployment
DMVPN (Dynamic Multipoint VPN)
• Cisco DMVPN
• https://www.cisco.com/c/en/us/products/security/dynamic-multipoint-
vpn-dmvpn/index.html
• Cisco Live DMVPN
• https://www.ciscolive.com/global/on-demand-library/?search=dmvpn#/
• Cisco IWAN CVD
• https://www.cisco.com/c/en/us/solutions/design-zone/networking-
design-guides/branch-wan-edge.html
• DMVPN is a Cisco innovation for building GRE/mGRE + IPsec VPN
connections in a dynamic and scalable manner
Terminology and Features
Core Network Overlay Addresses
192.168.128.0/17
192.168.101.0/24 192.168.102.0/24
Tunnel Address
Hub1 Hub 2
Tunnel: 10.0.0.101 Tunnel: 10.0.0.102

Physical: 172.16.101.1 Physical: 172.16.102.1
NBMA Address
Tunnel: 10.0.0.1 Tunnel: 10.0.0.2

Physical: 172.16.1.1 Physical: 172.16.2.1
Spoke 1 GRE/IPsec Spoke 2
Tunnels
192.168.1.0/24 192.168.2.0/24
On Demand
Spoke Tunnels
DMVPN Components
• Next Hop Resolution Protocol (NHRP)
• Creates a distributed (NHRP) mapping database of all the spoke’s tunnel to real (public
interface) addresses
• Multipoint GRE Tunnel Interface (mGRE)
• Single GRE interface to support multiple GRE/IPsec tunnels
• Simplifies size and complexity of configuration
• IPsec tunnel protection

• Dynamically creates and applies encryption policies
• Routing
• Dynamic advertisement of branch networks; almost all routing protocols (EIGRP, RIP, OSPF,
BGP, ODR) are supported
Spoke-to-hub tunnels
DMVPN Implementation
Spoke-to-spoke tunnels
2547oDMVPN tunnels
Hub and spoke Spoke-to-spoke

(Phase 1) (Phase 2) VRF-lite
Server Load Balancing Hierarchical (Phase 3) 2547oDMVPN
Google Cloud
Platform– Cisco CSR
and DMVPN
GCP with Cisco CSR 1000v Support
• Cisco CSR 1000v Deployment on GCP:
https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/
gcp/b_csrgcp/b_csrgcp_chapter_00.html
• GCP Marketplace - Cisco CSR 1000v:
https://console.cloud.google.com/marketplace/details/cisco-
public/cisco-csr-1000v
GCP to On-Premises CSR – IPsec VPN
Example 1
Compute Engine
IPsec/IKEv2
.30
2 1
Tunnel Mode Cisco
CSR1000v
VM
Default Network 35.xxx.xxx.x 169.254.0.2 .1 Private Network
10.138.0.0/20 192.xxx.xxx.x 192.168.200.0/24
Google
Cloud OSPF 10 Area 0
VPN
169.254.0.1 Hypervisor
BGP AS65000 BGP AS65002

Google Cloud Router
GCP CSR to On-Premises CSR – IPsec VPN
Example 2
IPsec/IKEv2
Default Network .30
inside-network 10.138.0.0/20
Tunnel Mode Cisco
CSR1000v
VM
10.0.1.0/24
Compute Engine .1 35.xxx.xxx.x .1 Private Network
2 1
.3 .2 .100 192.xxx.xxx.x 192.168.200.0/24
Cisco
CSR1000v OSPF 10 Area 0
Hypervisor
OSPF
https://www.cisco.com/c/en/us/td/docs/routers/csr1000/
software/gcp/b_csrgcp/b_csrgcp_chapter_00.html
GCP CSR to On-Premises CSR – DMVPN
Default Network .30

inside-network 10.138.0.0/20
Cisco
CSR1000v
VM
10.0.1.0/24 35.xxx.xxx.x 192.xxx.xxx.x
Compute Engine .1 .1 Private Network
.3 .2 .100
2 1
Cisco
Spoke Hub 192.168.200.0/24
CSR Tunnel: CSR Tunnel:
CSR1000v
10.1.0.1 10.1.0.2 OSPF 10 Area 0
Hypervisor
Routes this side should see: DMVPN

192.168.200.0/24 OSPF
https://www.cisco.com/c/en/us/td/docs/routers/csr1000/
software/gcp/b_csrgcp/b_csrgcp_chapter_00.html Routes this side should see:
10.0.1.0/24
gcloud – Create the GCP External IP, Inside VPC
Network and Route
Create a new external IP reservation that will be used for the GCP CSR NATed connection (or use an existing one)
# gcloud compute addresses create csr-to-csr-ext-ip --region us-west1
Capture the external IP address

# gcloud compute addresses list --filter="csr-to-csr-ext-ip"
NAME REGION ADDRESS STATUS
csr-to-csr-ext-ip us-west1 35.xxx.xxx.x RESERVED
Create a new GCP inside network that will be attached to the ‘inside’ interface of the CSR
# gcloud compute networks create inside-network --subnet-mode=custom
Create a new GCP inside subnet - Associate it with the inside network
# gcloud compute networks subnets create inside-subnet \
--network=inside-network \
--range=10.0.1.0/24
Create a new GCP route from the CSR inside network to the On-Premises private network which routes through the IPsec VPN
# gcloud compute routes create inside-to-csr-private \
--destination-range=192.168.200.0/24 \
--next-hop-address=10.0.1.2
gcloud – Create GCP Firewall Rules
Create a new GCP firewall rule to allow traffic into the inside CSR network from the default network
# gcloud compute firewall-rules create allow-default-to-csr-inside \
--direction=INGRESS \
--action=ALLOW \
--rules=all \
--source-ranges=0.0.0.0/0
Create a new GCP firewall rule to allow traffic between the default network and the On-Premises CSR public IP for IKE, IPsec
# gcloud compute firewall-rules create csr-csr-vpn \
--direction=INGRESS \
--network=default \
--action=ALLOW \
--rules=udp:500,udp:4500,esp \
--source-ranges=192.xxx.xxx.x
gcloud – Create CSR and Test Instances
Create a new GCE CSR instance and set fixed IPv4 addresses to each of the two interfaces
# gcloud compute instances create "csr-gcp-01" \
--zone "us-west1-a" \
--machine-type "n1-standard-4" \
--network-interface subnet="default",private-network-ip="10.138.0.100",address="35.xxx.xxx.x" \
--can-ip-forward \
--network-interface subnet="inside-subnet",private-network-ip="10.0.1.2",no-address \
--image ”name_of_csr_image" \
--boot-disk-size "10" \
--boot-disk-type "pd-standard" \
--boot-disk-device-name "csr-gcp-01"
Create a new GCE test instance that will be used to validate the VPN and routing
# gcloud compute instances create "csr-inside-vm" \
--zone "us-west1-a" \
--machine-type "g1-small" \
--subnet "inside-subnet" \
--private-network-ip "10.0.1.3" \
--image "debian-9-stretch-v20170918" \
--image-project "debian-cloud" \
--boot-disk-size "10" \
--boot-disk-type "pd-standard" \
--boot-disk-device-name "csr-inside-vm"
Connect to the GCP CSR – Enable Interfaces

Connect to the new GCP-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# gcloud compute ssh cisco-user@csr-gcp-01
csr1kv-gcp#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
csr1kv-gcp(config)#interface gigabitEthernet 2
csr1kv-gcp(config-if)#ip address dhcp
csr1kv-gcp(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and with the correct IP
addresses:
csr1kv-gcp#show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 10.138.0.100 YES TFTP up up
GigabitEthernet2 10.0.1.2 YES DHCP up up
GCP Cisco CSR DMVPN Config ... Output summarized
• Spoke interface Tunnel0
crypto ikev2 proposal AES/GCM/256 description DMVPN
encryption aes-gcm-256 ip address 10.1.0.1 255.255.255.0
prf sha512 no ip redirects
group 19 ip mtu 1400
! ip nhrp authentication <NHRP_PASSWORD>
crypto ikev2 policy AES/GCM/256 ip nhrp network-id 100
match fvrf any ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
proposal AES/GCM/256 ip tcp adjust-mss 1360
! ip ospf authentication-key 7 <OSPF_PASSWORD>
crypto ikev2 keyring DMVPN-KEYRING ip ospf network point-to-multipoint
peer ANY ip ospf hello-interval 10
address 0.0.0.0 0.0.0.0 tunnel source GigabitEthernet1
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel mode gre multipoint
! tunnel key 100
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
description PSK Profile !
match identity remote address 0.0.0.0 interface GigabitEthernet1
identity local address 35.xxx.xxx.x description Internet
authentication remote pre-share ip address 10.138.0.100 255.255.255.0
authentication local pre-share no ip redirects
keyring local DMVPN-KEYRING no ip unreachables
dpd 40 5 on-demand no ip proxy-arp
! negotiation auto
crypto ipsec security-association replay window-size 1024 !
! router ospf 10
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router-id 10.1.0.1
mode transport network 10.0.1.0 0.0.0.255 area 1
! network 10.1.0.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE !
set transform-set AES256/GCM/TRANSFORM ip route 0.0.0.0 0.0.0.0 138.0.0.1
set ikev2-profile DMVPN-IKEv2-PROFILE
On-Premises Cisco CSR DMVPN Config ... Output summarized
• Hub interface Tunnel0
crypto ikev2 proposal AES/GCM/256 description DMVPN
encryption aes-gcm-256 ip address 10.1.0.2 255.255.255.0
prf sha512 no ip redirects
group 19 ip mtu 1400
! ip nhrp authentication <NHRP_PASSWORD>
crypto ikev2 policy AES/GCM/256 ip nhrp map multicast dynamic
match fvrf any ip nhrp network-id 100
proposal AES/GCM/256 ip nhrp redirect
! ip tcp adjust-mss 1360
crypto ikev2 keyring DMVPN-KEYRING ip ospf authentication-key 7 <OSPF_PASSWORD>
peer ANY ip ospf network point-to-multipoint
address 0.0.0.0 0.0.0.0 ip ospf hello-interval 10
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel source GigabitEthernet1
! tunnel mode gre multipoint
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel key 100
description PSK Profile tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
identity local address 192.xxx.xxx.x interface GigabitEthernet1
authentication remote pre-share description Internet
authentication local pre-share ip address 192.xxx.xxx.x 255.255.255.0
keyring local DMVPN-KEYRING no ip redirects
dpd 40 5 on-demand no ip unreachables
! no ip proxy-arp
crypto ipsec security-association replay window-size 1024 negotiation auto
! !
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router ospf 10
mode transport router-id 10.1.0.2
! network 10.1.0.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE network 192.168.200.0 0.0.0.255 area 0
set transform-set AES256/GCM/TRANSFORM !
set ikev2-profile DMVPN-IKEv2-PROFILE ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

On the GCP CSR, check for the private network route from the on-premises
side(192.168.200.0/24)
csr1kv-gcp#show ip route | i 192.168.200.0
. . .
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:09:51, Tunnel0
On the on-premises CSR, check for the VPC inside network route (10.0.1.0/24)
. . .
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 00:40:08, Tunnel0
Check the DMVPN Next-Hop Resolution Protocol (NHRP) Status

csr1kv-gcp#show ip nhrp csr-mc-01#show ip nhrp
10.1.0.2/32 via 10.1.0.2 10.1.0.1/32 via 10.1.0.1
Tunnel0 created 5d14h, never expire Tunnel0 created 00:40:25, expire 00:08:20
Type: static, Flags: Type: dynamic, Flags: registered used nhop
NBMA address: 192.xxx.xxx.x NBMA address: 35.xxx.xxx.x
(Claimed NBMA address: 10.138.0.100)
Connect to the GCP test instance that was created earlier and ping to the on-premises private network
# gcloud compute ssh "csr-inside-vm“
shmcfarl@csr-inside-vm:~$ ping 192.168.200.30

Amazon Web Services
– Cisco CSR and
DMVPN
AWS with Cisco CSR 1000v Support
• Amazon Web Services Marketplace + Cisco CSR:
• https://aws.amazon.com/marketplace/search/results?x=0&y=0&searchTerms=csr&page=1&ref_=nav_search_
box
• Cisco CSR for AWS Deployment

• DMVPN
https://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/Intercloud/CSR/AWS/CSRAWS/CSRAWS_3.
html
• Deployment https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws.html
• Cisco Live Session for AWS with Cisco CSR:

• https://www.ciscolive.com/global/on-demand-library/?search=brkarc-
2023#/session/1486155288098001AhER
• Transit VPC with CSR: http://d2zmdbbm9feqrf.cloudfront.net/2017/usa/pdf/BRKARC-2749.pdf
AWS to On-Premises CSR – IPsec VPN
• Example 1

IPsec/IKEv2
Tunnel Mode Cisco
CSR1000v
VPC Network 52.xxx.xxx.x 169.254.11.178 .1 Private Network
172.31.0.0/16 169.254.11.177 192.xxx.xxx.x 192.168.200.0/24
VPC VPN
Router Gateway OSPF 10 Area 0
BGP AS64512 Hypervisor
BGP AS65002
AWS CSR to On-Premises CSR – IPsec VPN
• Example 2
Public-side Network
172.16.1.0/24
IPsec/IKEv2
Cisco
Cisco Tunnel Mode CSR1000v
CSR1000v
VPC Network 52.xxx.xxx.x .1 Private Network
172.16.2.0/24 192.xxx.xxx.x 192.168.200.0/24
VPC
Router OSPF 10 Area 0
Hypervisor
OSPF
AWS CSR to On-Premises CSR – DMVPN
Public-side Network
172.16.1.0/24
Cisco Cisco
VPC
CSR1000v CSR1000v
Router
VPC Network 52.xxx.xxx.x 192.xxx.xxx.x .1 Private Network
172.16.2.0/24 Spoke Hub 192.168.200.0/24
10.1.0.4 10.1.0.2
OSPF 10 Area 0
Hypervisor
DMVPN
OSPF
192.168.200.0/24
172.16.2.0/16
AWS CLI: Create VPC, Subnets and Internet GW
Create a new AWS VPC (vpc)
# aws ec2 create-vpc --cidr-block 172.16.0.0/16
Create a new subnet in the VPC (this one will be used for the CSR’s ’outside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.1.0/24
Create another new subnet in the VPC (this one will be used for the CSR’s ‘inside’ interface)
# aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block 172.16.2.0/24
Create a new AWS Internet Gateway (igw)

# aws ec2 create-internet-gateway
Attach the Internet gateway to the VPC

# aws ec2 attach-internet-gateway --vpc-id vpc-66a0a102 --internet-gateway-id igw-591fba3d
AWS CLI: Create Route Tables
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘outside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-aaa37dcd --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Associate the new routable with the ‘outside’ VPC subnet

# aws ec2 associate-route-table --subnet-id subnet-0c15b86b --route-table-id rtb-aaa37dcd
Create a new route table in the VPC (rtb) that will be used by the CSR’s ‘inside’ subnet
# aws ec2 create-route-table --vpc-id vpc-66a0a102
Create a new default route in the route table for the ‘inside’ subnet and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 0.0.0.0/0 --gateway-id igw-591fba3d
Create a new default route in the route table and point it to the Internet gateway
# aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block 192.168.200.0/24 --network-interface-id eni-af67db80
Associate the new route table with the ‘inside’ VPC subnet
# aws ec2 associate-route-table --subnet-id subnet-c617baa1 --route-table-id rtb-3741e750
Reference
AWS CLI: Create a Security Group/Rules

Create a new security group for the ‘outside’ facing interface (Optional: You can just use an existing group)
# aws ec2 create-security-group --group-name csr --description csr-rules --vpc-id vpc-66a0a102
Create a new security group rule for SSH to the CSR

# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 --protocol tcp --port 22 --cidr 0.0.0.0/0
Create a new security group rule for ICMP from the other CSRs (On-Premises and GCP CSR [optional: Just showing the format for your use])
# aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \
--ip-permissions '[{"IpProtocol": "icmp", "FromPort": -1, "ToPort": -1, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]
Create a new security group rule for ESP (IP 50) from the other CSRs
--ip-permissions '[{"IpProtocol": "50", "IpRanges": [{"CidrIp": ”192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]'
Create a new security group rule for IKE/NAT-T from the other CSRs
--ip-permissions '[{"IpProtocol": "17", "FromPort": 4500, "ToPort": 4500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]’
Optional: You may want to create a security group just for the ’inside’ subnet that has
different rules than the one for the ‘outside’ subnet
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 192.168.200.0/24
# aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr 172.16.2.0/24
AWS CLI: Run a new CSR Instance Using Previous Parameters
csr-create.json
{
"ImageId": "ami-99e5d0f9",
"InstanceType": "t2.medium",
"KeyName": "mc-aws-key", Create a CSR instance using the JSON file shown to the left
"NetworkInterfaces": [ # aws ec2 run-instances --cli-input-json file://csr-create.json
{
"DeviceIndex": 0, Create a tag/name and associate it with the CSR (Optional)
"Description": "Primary network interface", # aws ec2 create-tags --resources i-0f2a0ee857e9c2540 \
"Groups": [ --tags Key=Name,Value=csr-aws-01
"sg-65c39b03"
], Create a new External IP (EIP) allocation (or use an existing one)
"PrivateIpAddresses": [ # aws ec2 allocate-address
{ eipalloc-ab35cb96 vpc 52.xxx.xxx.x
"Primary": true,
"PrivateIpAddress": "172.16.1.10" Associate the EIP with the ’outside’ interface of the CSR (GigabitEthernet 1)
} # aws ec2 associate-address --allocation-id eipalloc-ab35cb96 \
], --network-interface-id eni-dd5bd6f2
"SubnetId": "subnet-0c15b86b"
}, Modify the ’inside’ subnet to disable source/destination checking
{ # aws ec2 modify-network-interface-attribute \
"DeviceIndex": 1, --network-interface-id eni-af67db80 \
"PrivateIpAddresses": [ --source-dest-check "{\"Value\": false}"
{
"Primary": true,
"PrivateIpAddress": "172.16.2.10" A note about NAT: If you plan to use the CSR for NAT
} operation, you must disable source/destination checking
],
"SubnetId": "subnet-c617baa1"
on the outside CSR interface/subnet
}
] http://docs.aws.amazon.com/AmazonVPC/latest/UserG
} uide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCh
eck #CiscoLive DGTL-BRKCLD-3440 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
Connect to the AWS CSR – Enable Interfaces
Connect to the new AWS-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# ssh -i "mc-aws-key.pem" ec2-user@52.xxx.xxx.x
csr-aws-01#configure terminal
csr-aws-01(config)#interface gigabitEthernet 2
csr-aws-01(config-if)#ip address dhcp
csr-aws-01(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
csr-aws-01#show ip interface brief
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Note: This can all be automated (along with the DMVPN configs) by creating AWS CloudFormation templates
AWS Cisco CSR DMVPN Config ... Output summarized
Spoke
crypto ikev2 proposal AES/GCM/256 interface Tunnel0
encryption aes-gcm-256 description DMVPN
prf sha512 ip address 10.1.0.4 255.255.255.0
group 19 no ip redirects
! ip mtu 1400
crypto ikev2 policy AES/GCM/256 ip nhrp authentication <NHRP_PASSWORD>
match fvrf any ip nhrp network-id 100
proposal AES/GCM/256 ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
! ip tcp adjust-mss 1360
crypto ikev2 keyring DMVPN-KEYRING ip ospf authentication-key 7 <OSPF_PASSWORD>
peer ANY ip ospf network point-to-multipoint
address 0.0.0.0 0.0.0.0 ip ospf hello-interval 10
pre-shared-key <PSK_PASSWORD_GOES_HERE> tunnel source GigabitEthernet1
! tunnel mode gre multipoint
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel key 100
description PSK Profile tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
authentication local pre-share ip address dhcp
! no ip proxy-arp
crypto ipsec security-association replay window-size 1024 negotiation auto
! !
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router ospf 10
mode transport router-id 10.1.0.4
! network 172.16.2.0 0.0.0.255 area 2
set transform-set AES256/GCM/TRANSFORM !
set ikev2-profile DMVPN-IKEv2-PROFILE ip route 0.0.0.0 0.0.0.0 172.16.1.1
On-Premises Cisco CSR DMVPN Config
Hub – Nothing ever changes on the hub for each example
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256 interface Tunnel0
prf sha512 description DMVPN
group 19 ip address 10.1.0.2 255.255.255.0
! no ip redirects
crypto ikev2 policy AES/GCM/256 ip mtu 1400
match fvrf any ip nhrp authentication <NHRP_PASSWORD>
ip nhrp map multicast dynamic
proposal AES/GCM/256
ip nhrp network-id 100
! ip nhrp redirect
crypto ikev2 keyring DMVPN-KEYRING ip tcp adjust-mss 1360
peer ANY ip ospf authentication-key 7 <OSPF_PASSWORD>
address 0.0.0.0 0.0.0.0 ip ospf network point-to-multipoint
pre-shared-key <PSK_PASSWORD_GOES_HERE> ip ospf hello-interval 10
crypto ikev2 profile DMVPN-IKEv2-PROFILE tunnel mode gre multipoint
tunnel key 100
description PSK Profile
tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
authentication local pre-share ip address 192.xxx.xxx.x 255.255.255.0
! no ip proxy-arp
negotiation auto
crypto ipsec security-association replay window-size 1024
!
! router ospf 10
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 router-id 10.1.0.2
mode transport network 10.1.0.0 0.0.0.255 area 0
! network 192.168.200.0 0.0.0.255 area 0
crypto ipsec profile DMVPN-IPSEC-PROFILE !
set transform-set AES256/GCM/TRANSFORM ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x
set ikev2-profile DMVPN-IKEv2-PROFILE #CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 166

On the on-premises CSR check the route for the AWS VPC network 172.16.2.0/24
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 00:11:41, Tunnel0
On AWS check for the route for the on-premises network (192.168.200.0/24)
csr-aws-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 6d17h, Tunnel0
Connect to an AWS instances and ping to the on-premises private network

[ec2-user@ip-172-16-2-192 ~]$ ping 192.168.200.30
VPC Network
Cisco Cisco .30 VM
CSR1000v CSR1000v
172.16.2.0/24
.1 Private Network
.192 .10 192.168.200.0/24
Spoke Hub
10.1.0.4 10.1.0.2
Hypervisor
OSPF
For Reference
Amazon Web Services

– Marketplace-based
Launch Walk-thru
AWS Marketplace CSR Launch – Console (1)
AWS Launch CSR as an Instance – Console (1)
4
3
Microsoft Azure –
Cisco CSR and
DMVPN
Azure to On-Premises CSR – IPsec VPN
Example 1
IPsec/IKEv2
Cisco
Tunnel Mode CSR1000v
VNet Subnet 40.xxx.xxx.x 169.254.11.178 Private Network
.1
10.10.0.0/16 169.254.11.177 192.xxx.xxx.x 192.168.200.0/24
VPN
Gateway OSPF 10 Area 0
BGP AS64512 Hypervisor
BGP AS65002
Azure CSR to On-Premises CSR – IPsec VPN
Example 2
Outside Subnet
10.10.0.0/24
IPsec/IKEv2
Cisco
Cisco Tunnel Mode CSR1000v
CSR1000v
Inside Subnet 40.xxx.xxx.x .1 Private Network
10.10.1.0/24 192.xxx.xxx.x 192.168.200.0/24
OSPF 10 Area 0
Hypervisor
OSPF
Azure CSR to On-Premises CSR – DMVPN
Outside Subnet
10.10.0.0/24
Cisco
Cisco CSR1000v
CSR1000v
Inside Subnet 40.xxx.xxx.x 192.xxx.xxx.x .1 Private Network
10.10.1.0/24 Spoke Hub 192.168.200.0/24
10.1.0.6 10.1.0.2
OSPF 10 Area 0
Hypervisor
DMVPN
OSPF
192.168.200.0/24
10.10.1.0/24
Microsoft Azure with Cisco CSR 1000v
• Microsoft Azure Marketplace
• https://azuremarketplace.microsoft.com/en-
us/marketplace/apps/cisco.cisco-csr-basic-template
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-
csr-1000v
• Cisco CSR 1000v with Azure Deployment
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b
_csr1000config-azure.html
Azure CLI: Create Resource Group, Networks,
Subnets
Create a new Azure Resource Group (rg)
# az group create --name multicloud-rg --location westus
Create a new public (external IP) IPv4 address to be used for the CSR’s ‘outside’ interface
# az network public-ip create --resource-group multicloud-rg --name csr-azure-01-eip --allocation-method static
Create a new virtual network (vnet) and a subnet to be used for the CSR’s ‘outside interface
# az network vnet create \
--resource-group multicloud-rg \
--name mc-csr-vnet \
--address-prefix 10.10.0.0/16 \
--subnet-name csr-outside \
--subnet-prefix 10.10.0.0/24
Create a new subnet for the CSR’s ‘inside’ interface and associate it with the vnet created above
--vnet-name mc-csr-vnet \
--name csr-inside \
Azure CLI: Create Route Tables
Create a new route table (rt) that will be used for the CSR’s ’outside’ subnet
# az network route-table create \
--name csr-outside-rt
Create a new route table that will used for the CSR’s ‘inside’ subnet
# az network route-table create \
--name csr-inside-rt
Create a new route table entry for the ‘inside’ subnet to reach the On-Premises network (192.168.200.0) via the CSR’s IP (10.10.1.4)
# az network route-table route create \
--name csr-to-on-premises-route \
--route-table-name csr-inside-rt \
--address-prefix 192.168.200.0/24 \
--next-hop-type VirtualAppliance \
--next-hop-ip-address 10.10.1.4
Associate the ‘outside’ route table with the ‘outside’ subnet

# az network vnet subnet update \
--name csr-outside \
--route-table csr-outside-rt
Associate the ‘inside’ route table with the ‘inside’ subnet

# az network vnet subnet update \
--name csr-inside \
--route-table csr-inside-rt
Azure CLI: Create Network Security Group (NSG) Reference
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
# az network nsg create \
--name csr-nsg-outside
Create a new NSG rule to allow inbound SSH access to the CSR (can make it specific to an IP/Prefix)
# az network nsg rule create \
--nsg-name csr-nsg-outside \
--name SSHRule \
--priority 100 \
--source-address-prefixes 'Internet' \
--source-port-ranges '*' \
--destination-address-prefixes '*' \
--destination-port-ranges 22 \
--access Allow \
--protocol Tcp \
--direction inbound
Create a new NSG rule to allow inbound UDP 500 (IKE) traffic to the CSR (can make it specific to an IP/Prefix)
--name UDP-500 \
--priority 101 \
--access Allow \
--protocol Udp \
--direction inbound
Azure CLI: Create NSG Rule and NICs Reference
Create a new Network Security Group (NSG) to be used for the ‘outside’ CSR interface
--name UDP-4500 \
--priority 102 \
--access Allow \
--protocol Udp \
--direction inbound
Create a new NIC to be used by the CSR’s ‘outside’ interface. Associate the NIC with the NSG, Subnet, Public IP & enable forwarding
# az network nic create \
--name csr-nic-g1 \
--subnet csr-outside \
--network-security-group csr-nsg-outside \
--ip-forwarding true \
--public-ip-address csr-azure-01-eip
Create a new NIC to be used by the CSR’s ‘inside’ interface. Associate the NIC with the NSG, Subnet and enable forwarding
# az network nic create \
--name csr-nic-g2 \
--subnet csr-inside \
--ip-forwarding true
Azure CLI: Run a new CSR Instance Using
Previous Parameters
Create a CSR VM using a Azure Marketplace image. Associate the VM with two the NICs created earlier.
# Note: The VM can be created with a large number of options to include SSH keys, image (BYOL, # of NICs), and size
# az vm create \
--name csr-azure-01 \
--admin-username csr-azure \
--admin-password <PASSWORD> \
--authentication-type password \
--image cisco:cisco-csr-1000v:16_6:16.6.120170804 \
--nics csr-nic-g1 csr-nic-g2 \
--size Standard_D2_v2
Connect to the Azure CSR – Enable Interfaces
Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP
# ssh csr-azure@40.xxx.xxx.x
csr-azure-01#configure terminal
csr-azure-01(config)#interface gigabitEthernet 2
csr-azure-01(config-if)#ip address dhcp
csr-azure-01(config-if)#no shutdown
Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct
IP addresses:
csr-azure-01#show ip interface brief
VirtualPortGroup0 192.168.35.1 YES TFTP up up
Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager
Azure Cisco CSR DMVPN Config ... Output summarized
Spoke
crypto ikev2 proposal AES/GCM/256
encryption aes-gcm-256 interface Tunnel0
prf sha512 description DMVPN
group 19 ip address 10.1.0.6 255.255.255.0
! no ip redirects
crypto ikev2 policy AES/GCM/256 ip mtu 1400
match fvrf any ip nhrp authentication <NHRP_PASSWORD>
proposal AES/GCM/256 ip nhrp network-id 100
! ip nhrp nhs 10.1.0.2 nbma 192.xxx.xxx.x multicast
description PSK Profile tunnel key 100
match identity remote address 0.0.0.0 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
identity local address 40.xxx.xxx.x !
authentication remote pre-share interface GigabitEthernet1
authentication local pre-share description Internet
keyring local DMVPN-KEYRING ip address dhcp
dpd 40 5 on-demand no ip redirects
crypto ipsec security-association replay window-size 1024 no ip proxy-arp
! negotiation auto
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 !
mode transport router ospf 10
! router-id 10.1.0.6
set transform-set AES256/GCM/TRANSFORM network 10.10.1.0 0.0.0.255 area 3
set ikev2-profile DMVPN-IKEv2-PROFILE !
ip route 0.0.0.0 0.0.0.0 10.10.0.1
On-Premises Cisco CSR DMVPN Config ... Output summarized
Hub - Nothing ever changes on the hub for each example
crypto ikev2 proposal AES/GCM/256 interface Tunnel0
encryption aes-gcm-256 description DMVPN
prf sha512 ip address 10.1.0.2 255.255.255.0
group 19 no ip redirects
! ip mtu 1400
crypto ikev2 policy AES/GCM/256 ip nhrp authentication <NHRP_PASSWORD>
match fvrf any ip nhrp map multicast dynamic
proposal AES/GCM/256 ip nhrp network-id 100
! ip nhrp redirect
description PSK Profile tunnel key 100
match identity remote address 0.0.0.0 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE
identity local address 192.xxx.xxx.x !
authentication remote pre-share interface GigabitEthernet1
authentication local pre-share description Internet
keyring local DMVPN-KEYRING ip address 192.xxx.xxx.x 255.255.255.0
dpd 40 5 on-demand no ip redirects
crypto ipsec security-association replay window-size 1024 no ip proxy-arp
! negotiation auto
crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 !
mode transport router ospf 10
! router-id 10.1.0.2
set transform-set AES256/GCM/TRANSFORM network 192.168.200.0 0.0.0.255 area 0
set ikev2-profile DMVPN-IKEv2-PROFILE !
ip route 0.0.0.0 0.0.0.0 192.xxx.xxx.x

On the on-premises CSR check the route for the Azure VNet 10.10.1.0/24
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:19:15, Tunnel0
On Azure check for the route for the on-premises network (192.168.200.0/24)
csr-azure-01#show ip route | i 192.168.200.0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:17:57, Tunnel0
Connect to an Azure instance and ping to the on-premises private network

shmcfarl@AzTestVm:~$ping 192.168.200.30
.30
Inside Subnet Cisco
CSR1000v
Cisco
CSR1000v VM
10.10.1.0/24
.1 Private Network
.5 .4 192.168.200.0/24
Spoke Hub
10.1.0.6 10.1.0.2
Hypervisor
OSPF
For Reference
Azure – Marketplace-
based Launch Walk-
thru
Azure Marketplace/Resource Search
Azure Marketplace - There are multiple CSR
types to pick from
Azure Marketplace
Deployment Flow
1 2 3
Linking DMVPN Sites
DMVPN – Enable Dynamic Multicloud Networking
Cisco DMVPN
VNet Network Spoke
10.10.1.0/24
Cisco
CSR1000v
Cisco
CSR1000v
Private Network
Spoke Hub 192.168.200.0/24
VPC Network
172.16.2.0/24 BGP/OSPF/EIGRP
Cisco
CSR1000v
DMVPN
VPC Network
10.0.1.0/24 Spoke
Cisco
CSR1000v
General Guidelines for DMVPN Between Clouds
• Set the VPC routes for each site
gcloud compute routes create inside-to-aws \
--network=csr-inside-network \
gcloud compute routes create inside-to-azure \

--network=csr-inside-network \
• Set the firewall/security groups/network security groups for each site/protocol

Create a very specific per-site route (AWS example allowing UDP500 from each cloud provider public IP)
aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \
--ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"},
{"CidrIp": "35.x.x.x/32"}, {"CidrIp": "40.x.x.x/32"}]}]’
Alternatively, you can open it up (Azure example)

az network nsg rule create \
--name UDP-4500 \
--priority 102 \
--access Allow \
--protocol Udp \
--direction inbound
Routing Example – All Sites
• For spoke-to-spoke direct routing with DMVPN/NHRP:
• ‘ip nhrp redirect’ on the hubs IA - OSPF inter area
• ‘ip nhrp shortcut’ on the spokes % - next hop override
Hub On-Premises CSR Spoke – Amazon Web Services CSR
csr-mc-01#show ip route ospf
csr-aws-01#show ip route ospf
O IA 10.0.1.0/24 [110/1001] via 10.1.0.1, 02:40:45, Tunnel0
O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.1/32 [110/1000] via 10.1.0.1, 02:40:45, Tunnel0
O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.4/32 [110/1000] via 10.1.0.4, 01:18:49, Tunnel0
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 01:21:32, Tunnel0
O 10.1.0.6/32 [110/1000] via 10.1.0.6, 00:56:19, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:59:01, Tunnel0
O IA 10.10.1.0/24 [110/1001] via 10.1.0.6, 00:55:34, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:14, Tunnel0
172.16.0.0/24 is subnetted, 1 subnets
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 01:21:32, Tunnel0
O IA 172.16.2.0 [110/1001] via 10.1.0.4, 01:18:49, Tunnel0
Spoke – Google Cloud Platform CSR Spoke – Azure CSR

csr1kv-gcp#show ip route ospf csr-azure-01#show ip route ospf
10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks 10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
O 10.1.0.2/32 [110/1000] via 10.1.0.2, 02:43:14, Tunnel0 O IA% 10.0.1.0/24 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 01:21:14, Tunnel0 O % 10.1.0.1/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
O % 10.1.0.6/32 [110/2000] via 10.1.0.2, 00:58:47, Tunnel0 O 10.1.0.2/32 [110/1000] via 10.1.0.2, 00:58:44, Tunnel0
O IA% 10.10.1.0/24 [110/2001] via 10.1.0.2, 00:58:00, Tunnel0 O % 10.1.0.4/32 [110/2000] via 10.1.0.2, 00:58:44, Tunnel0
172.16.0.0/24 is subnetted, 1 subnets 172.16.0.0/24 is subnetted, 1 subnets
O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 01:21:14, Tunnel0 O IA% 172.16.2.0 [110/2001] via 10.1.0.2, 00:58:44, Tunnel0
O 192.168.200.0/24 [110/1001] via 10.1.0.2, 02:43:14, Tunnel0 O 192.168.200.0/24 [110/1001] via 10.1.0.2, 00:58:44, Tunnel0
NHRP Example – Hub/Spoke Spoke – Azure CSR
Hub On-Premises CSR csr-azure-01#show ip nhrp
10.0.1.0/24 via 10.1.0.1
csr-mc-01#show ip nhrp Tunnel0 created 00:06:26, expire 00:03:32
10.1.0.1/32 via 10.1.0.1 Type: dynamic, Flags: router rib nho
Tunnel0 created 02:02:42, expire 00:08:17 NBMA address: 35.xxx.xxx.x
Type: dynamic, Flags: registered used nhop (Claimed NBMA address: 10.138.0.100)
NBMA address: 35.xxx.xxx.x 10.1.0.1/32 via 10.1.0.1
(Claimed NBMA address: 10.138.0.100) Tunnel0 created 00:06:26, expire 00:03:32
10.1.0.4/32 via 10.1.0.4 Type: dynamic, Flags: router nhop rib nho
Type: dynamic, Flags: registered used nhop (Claimed NBMA address: 10.138.0.100)
NBMA address: 52.xxx.xxx.x 10.1.0.2/32 via 10.1.0.2
(Claimed NBMA address: 172.16.1.10) Tunnel0 created 00:21:28, never expire
10.1.0.6/32 via 10.1.0.6 Type: static, Flags:
Type: dynamic, Flags: registered used nhop 10.1.0.4/32 via 10.1.0.4
NBMA address: 40.xxx.xxx.x Tunnel0 created 00:12:29, expire 00:02:40
(Claimed NBMA address: 10.10.0.4) Type: dynamic, Flags: router nhop rib nho
NBMA address: 52.xxx.xxx.x
csr-mc-01#show ip nhrp multicast (Claimed NBMA address: 172.16.1.10)
I/F NBMA address 10.10.1.0/24 via 10.1.0.6
Tunnel0 35.xxx.xxx.x Flags: dynamic (Enabled) Tunnel0 created 00:08:30, expire 00:03:33
Tunnel0 52.xxx.xxx.x Flags: dynamic (Enabled) Type: dynamic, Flags: router unique local
Tunnel0 40.xxx.xxx.x Flags: dynamic (Enabled) NBMA address: 10.10.0.4
(no-socket)
Spoke – Azure VM 172.16.2.0/24 via 10.1.0.4
Tunnel0 created 00:07:19, expire 00:02:40
shmcfarl@AzureTestVm:~$ traceroute 10.0.1.3 Type: dynamic, Flags: router rib nho
traceroute to 10.0.1.3 (10.0.1.3), 30 hops max, 60 byte packets NBMA address: 52.xxx.xxx.x
1 10.10.1.4 (10.10.1.4) 1.220 ms 1.192 ms 1.328 ms (Claimed NBMA address: 172.16.1.10)
2 10.0.1.3 (10.0.1.3) 25.794 ms * 25.782 ms csr-azure-01#show ip nhrp multicast
I/F NBMA address
Tunnel0 192.xxx.xxx.x Flags: nhs (Enabled)
Split-
Tunneling/Routing
Options
Split-Tunnel/Routing Options
• All three public cloud providers allow for either split-tunneling or forced/direct routing
• Split-tunneling:
• Public cloud resources (instances/VMs, container clusters) will use the default VPC gateway for non-On-
Premises routes
• Public cloud resources will use the On-Premises-specific routes advertised by the CSR
• Forced/Direct routing – All public cloud resources will use the VPN connection as their default
route for ALL traffic (forces traffic through the On-Premises site)
External/NAT
Cisco
Google Cloud VPN
Compute Engine CSR1000v
10.0.0.1
Routing 35.xxx.xxx.x
2 1
10.0.0.5
VPC Subnetwork
GW 192.xxx.xxx.x
BGP
Google Cloud Router

CSR High Availability
Public Cloud Provider – CSR High-Availability
• Common challenge with all public cloud provider is that there is not true layer 2
support on a VPC subnet – this prevents FHRPs from working properly
• Must setup a monitoring/tracking feature to watch for CSR interface/instance failure
and adjust the VPC route table to point to 2nd CSR inside interface
• AWS CSR High-Availability:
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/aws/b_csraws/b_csraws_c
hapter_0100.pdf
• Azure CSR High-Availability:
• https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/azu/b_csr1000config-
azure/b_csr1000config-azure_chapter_0110.html
Automation
Challenges
“You can’t possibly be
successful automating
something you don’t understand
the design for”
- Dumb, bald, loud guy
Automating the Multicloud Network
• Challenges:
• Different toolsets for different jobs (Ansible, Python, Bash scripts, Terraform, etc..)
• Different toolsets for different clouds (Heat for OpenStack, CloudFormation for AWS, Deployment Manager for GCP, Azure
Automation)
• Different toolsets for different vendor products (Cisco NSO, CloudCenter Suite, YANG development kit, etc..)
• There is no silver bullet - Start simple:

• Use what your team knows – Perform a gap analysis on what you have against what you need
• Initially, automate the things that hurt a lot to do by hand and that change frequently – I use free tools but that doesn’t mean
the process is free J
• Native Tools: It’s safe to use the cloud provider’s native automation toolset (e.g., AWS CloudFormation) when that is the only
provider you need to deal with
• Abstracted Tools: When you are dealing with multiple providers to include on-premises providers (e.g., VMware vSphere or
Microsoft Azure Stack), it makes life easier to abstract away from native cloud provider tool sets and use something like
Terraform and/or combo of tools
• Full Stack Tools: When you want to stop pulling your hair out and you want to build full ‘stacks’ in nearly any environment,
move to something that can treat the environment as a whole
• Cisco CloudCenter Suite: https://www.cisco.com/c/en/us/products/cloud-systems-management/cloudcenter/index.html
• Cisco Managed Services Accelerator: https://www.cisco.com/c/en/us/products/cloud-systems-management/managed-services-accelerator/index.html
Amazon CloudFormation
• https://aws.amazon.com/cloudformation/
• Template-based (JSON/YAML) – Build a stack(s) from a template file
• Sometimes you need to run more than one stack (in order) to get what you need
• Example template: https://github.com/shmcfarl/multicloud/tree/master/aws/cloudformation
Google Cloud Platform – Deployment Manager
• https://cloud.google.com/deployment-manager/
• Configuration files (YAML), Templates (Python/Jinja2), Schema files
(JSON)
• Sometimes you need to run more than one stack (in order) to get
what you need
• Example templates:
https://github.com/shmcfarl/multicloud/tree/master/gcp/deployment-
manager
Microsoft Azure Automation/Resource Manager
• https://azure.microsoft.com/en-us/services/automation/
• Runbooks (create graphically, PowerShell, Python)
• Read and select these carefully: https://docs.microsoft.com/en-
us/azure/automation/automation-runbook-types
• Resource Manager: https://docs.microsoft.com/en-us/azure/azure-resource-

manager/resource-group-overview
• https://github.com/Azure/azure-quickstart-templates/tree/master/cisco-csr-1000v
• Example template:
https://github.com/shmcfarl/multicloud/blob/master/azure/resource-manager/az-
arm-csr-cleaned.json
Call APIs Directly
• Cisco SD-WAN APIs: https://developer.cisco.com/sdwan/
• Google Cloud Platform:
https://cloud.google.com/compute/docs/reference/latest/
• Amazon Web Services:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/Welc
ome.html
• Microsoft Azure: https://docs.microsoft.com/en-us/rest/api/
HashiCorp Terraform - Abstracted Model
https://github.com/terraform-providers
https://github.com/terraform-providers/terraform-provider-aws
Providers
https://github.com/terraform-providers/terraform-provider-
azurerm
https://github.com/terraform-providers/terraform-provider-
kubernetes
https://github.com/terraform-providers/terraform-provider-google
https://github.com/CiscoDevNet/terraform-sdwan
Google VPN –
Creating Google VPN,
Router, IPsec, BGP via
REST APIs
Google Cloud API – Creating GCP Cloud
VPN/Routers
• Assumptions/environment:
• Understand how to authenticate to GCP APIs:
https://cloud.google.com/docs/authentication/
• In this example, the Paw application was used to craft GET, POST and PATCH calls
• Some configurations have been sanitized for security purposes
• Have On-Premises Cloud infrastructure deployed and a CSR/ASR configured (can
be done after GCP side is deployed)
• In this example, the configuration will be deployed against the OpenStack use case
discussed in the earlier slides
• In this example, the default network created by GCP will be used
• Note: gcloud has VERY long delays in commands if you have IPv6 enabled on your
local machine – set to “link-local” mode on your Mac
Reference Topology for GCP API Example
OSPF<>BGP Redistribution
IPsec/IKEv2
Tunnel Mode
169.254.0.6
Default Network 35.yyy.yyy.y .11 Private Network
10.138.0.0/20 192.yyy.yyy.y 172.16.0.0/24
Google
Cloud
VPN
169.254.0.5
OSPF 10 Area 0
BGP AS65000 BGP AS65003 On-Premises Cloud

Google Cloud Router Routes this side should see:
10.138.0.0/20

172.16.0.0/24
GCP API (1) – Create VPN GW and External IP
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways HTTP/1.1
Authorization: Bearer XXXX POST: Create VPN
Gateway
Content-Type: application/json; charset=utf-8
Host: www.googleapis.com
Connection: close
Content-Length: 138
{
"name": "csr-gcp-os-aio-gw",
"network": "projects/<gcp_project_number>/global/networks/default",
"region": "projects/<gcp_project_number>/regions/us-west1"
}
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses HTTP/1.1

Authorization: Bearer XXXX
POST: Create External IP
Connection: close
Address
Content-Length: 29
{
"name": "gcp-to-os-dmz"
}
GET: Get the External IP

GET /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses/gcp-to-os-dmz HTTP/1.1
Connection: close Address
RESPONSE - SUMMARIZED:
"name": "gcp-to-os-dmz",
"description": "",
"address": ”35.yyy.yyy.y",
"status": "RESERVED",
GCP API (2) – Create Forwarding Rules
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingRules HTTP/1.1
POST: Create
Connection: close Forwarding rule for ESP
Content-Length: 257
{
"name": "csr-gcp-os-aio-rule-esp",
"IPProtocol": "ESP",
"IPAddress": "35.yyy.yyy.y",
"region": "projects/<gcp_project_number>/regions/us-west1",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}

Authorization: Bearer XXXX POST: Create
Forwarding rule for UDP
Connection: close
500
Content-Length: 278
{
"name": "csr-gcp-os-aio-rule-udp500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "500"
}

Authorization: Bearer XXXX POST: Create
Forwarding rule for UDP
Connection: close
4500
Content-Length: 280
{
"name": "csr-gcp-os-aio-rule-udp4500",
"IPProtocol": "UDP",
"target": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw",
"portRange": "4500"
}
GCP API (3) – Create Cloud Router and BGP Session
POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/routers HTTP/1.1
Connection: close
Content-Length: 574
{
"name": "csr-gcp-os-bgp-rtr",
"bgp": {
"asn": "65000"
},
"interfaces": [
{
"name": "if-csr-gcp-os-bgp-rtr-02",
"linkedVpnTunnel": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels/csr-gcp-os-aio-gw-tunnel-1",
"ipRange": "169.254.0.5/30"
}
],
"bgpPeers": [
{
"name": "csr-gcp-os-bgp-peer",
"interfaceName": "if-csr-gcp-os-bgp-rtr-02",
"ipAddress": "169.254.0.5",
"peerIpAddress": "169.254.0.6",
"peerAsn": "65003"
}
],
"network": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/global/networks/default"
}
POST: Create Cloud Router, BGP session and link to the Cloud VPN tunnel
GCP API (5) – Create Cloud VPN Tunnel

POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/vpnTunnels HTTP/1.1
Connection: close
Content-Length: 417
{
"name": "csr-gcp-os-aio-gw-tunnel-1",
"sharedSecret": " <pre-shared-password-goes-here> ",
"router": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/routers/csr-gcp-os-bgp-rtr",
"peerIp": "192.yyy.yyy.y",
"ikeVersion": "2",
"targetVpnGateway": "https://www.googleapis.com/compute/v1/projects/<gcp_project_number>/regions/us-west1/targetVpnGateways/csr-gcp-os-aio-gw"
}
POST: Create a Cloud VPN tunnel and associated it with the Cloud router
Summary
• Cisco Multicloud Solutions: https://www.cisco.com/c/en/us/solutions/cloud/multicloud-portfolio.html
• Public cloud native IPsec VPN support is good, but it is always point-to-point, does not have consistent support
and lacks network-rich features - It may be good enough for your initial use case(s)
• If you have deployed or want to deploy SD-WAN, adding in your public cloud sites into your overall SD-WAN
design can reap many operational and cost benefits
• Go learn about ACI Anywhere (Cisco Cloud ACI): https://aws.amazon.com/blogs/apn/extending-on-premises-

cisco-cloud-aci-network-security-segmentation-to-aws/
• Multicloud between multiple public cloud providers and on-premises look like distinctly separate hybrid cloud
deployments but..
• Automate everything
• You must take into consideration:
• Team knowledge of public cloud operations, tools, automation
• Cross cloud tools and automation
• Diversity of network designs, protocols, security
• Multi-region designs
• Availability zones within and across providers
Thank you
#CiscoLive
#CiscoLive

DGTL-BRKCLD-3440-Multicloud Networking - Design and Deployment

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DGTL-BRKCLD-3440-Multicloud Networking - Design and Deployment

Uploaded by

Copyright:

Available Formats

#CiscoLive

• You will need to know the basics of:

• You won’t learn security, routing, HA, performance best practices

• Common network transport ingredients for hybrid and multicloud:

• Common network endpoint options:

users/applications connect Amazon Amazon

to Cloud Service Provider ECR S3 endpoints

(CSP) public endpoints Internet Enterprise

and/or public IPs of AZ: us-west-2b Gateway Edge

No ‘traditional’ IPsec VPN Application Private Subnet 2 Site

Google Cloud Router

• Azure Site-to-Site VPN: https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-

• GCP Site-to-Site VPN: https://cloud.google.com/vpn/docs/concepts/topologies

IPsec/IKEv2 eBGP<>IGP Redistribution

Inline Services Winner

Managed Services Winner

Time to Provision Winner

Location Availability Winner

App App Infrastructure Agnostic

Hypervisor Performance Elasticity

Public Cloud Provider Native VPN Services

• OpenStack public cloud goodness: https://www.openstack.org/passport

VPC Network SD-WAN

VPC Subnet(s) Private Network(s)

Transit VPC: Cisco vEdge + Per-VPC vEdge

VPC Subnet(s) Private Network(s)

Azure VPN vEdge

Cisco SD-WAN + Some Combo of Colocation/peering

VPC Subnet(s) Private Network(s)

Transit VPC: Cisco CSR + Per-VPC CSR

VPC Subnet(s) Private Network(s)

Dev Prod Dev Prod Dev Prod

Dev Prod Dev Prod Dev Prod

Transit VPC Transit VPC

VPC Subnet(s) Private Network(s)

Google Cloud Platform – VPN Gateway

Routes this side should see:

Create an external IP to use for the VPN

Capture the external IP address

# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \

# gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \

Cisco CSR Route Information

Google Area 0 a - application route

On-Premises VM traceroutes via HSRP Active CSR (192.168.100.2)

HSRP Active CSR Route to GCP Default Network (10.138.0.0)

HSRP Standby CSR Route to GCP Default Network (10.138.0.0)

Failure Scenario 1 – HSRP Primary CSR VM Reload

On-Premises VM traceroutes via HSRP Newly Active CSR (192.168.100.3)

Pre-Failure GCE Instance traceroutes via 169.254.0.2 GCP BGP Path

Post-Failure GCE Instance traceroutes via 169.254.0.10 GCP BGP Path

Primary ip mtu 1400

Secondary ip mtu 1400

Routes this side should see:

Create VPN Gateway and set the AWS BGP ASN

Attach VPN Gateway to the VPC

Create a new VPN connection

Enable route propagation for the VPC

Permit SSH and ICMP

Verify Routing and Reachability

Connect to an AWS instance and ping to the on-premises private network

Routers on AWS BGP AS65002

Create a new virtual network (vnet) and a new ‘outside’ subnet

Create a ’inside’ subnet