Automated VPC

Connection Using a
Transitive Hub in AWS
Jason Bomar, Chief Network Architect, CCIE
9316, jbomar@la-networks.com jbomar@la-
networks.com
Calvin Chow, SE Cisco Systems, CCIE 11848
cchow@cisco.com

Session ID CCSCLD-2003

#CLUS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#CCSCLD-2003

by the speaker until June 18, 2018.

#CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Agenda
• The Customer Case
• AWS Networking Overview
• Transitive Routing in the AWS Cloud
• Architecting our AWS solution
• First Attempt – Solving Problems
• Our Second Attempt - Network Automation
• Summary/Review
• Appendix – Links, Etc.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
The Customer Case
Introduction
Overview of the client scenario
• Large enterprise that started with a few large VPC’s, but eventually
moved to many smaller VPC’s.
• Reduced ‘Blast Radius’.
• Ability to bill back to departments and projects.

• Currently at ~100 client/spoke VPC’s with plans to scale this

number up significantly.
• Started in single AWS Region, but have moved to additional
regions since.
• Desire to automate the process.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Introduction
Quick High Level Drawing
The vision
VPC VPC VPC VPC • Many ‘small’ VPC’s – 1
Dept/Proj
A
Dept/Proj
B
Dept/Proj
C
Dept/Proj
D
per department per
project.
VGW VGW VGW VGW

• All Internet traffic

comes back to on-
prem.
VPC VPC VPC VPC
Dept/Proj Dept/Proj Dept/Proj Dept/Proj
E F G H

VGW VGW VGW VGW

• Wrapped in a service
catalog experience
where the workflow is
automated.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
AWS Networking and
Transitive Routing
Overview
AWS Networking Overview
You Can’t Tell the Players Without a Program
Amazon VPC
• Availability Zone – A geographically close, but highly
IGW independent buildings.
• VPC – Virtual Private Cloud, a virtual DC in AWS.
CSR CSR

• VGW – Private Virtual Gateway, an AWS routing

AZ-A AZ-B service for the VPC to connect to your network.
VGW
• IGW – Internet Gateway, an AWS routing service
for the Internet.
• Elastic IP – Similar to a static NAT.
DX DX
• Security Group – Similar to a FW ACL, stateful and host
specific.
• CSR – Cloud Services Router, a virtual Cisco router.
Customer Customer
Gateway Gateway

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Transitive Routing in AWS
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24

VPC VPC VPC

10.1.1.0/24 10.1.2.0/24 10.1.3.0/24

VGW VGW VGW

X
#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Our First Attempt at
Building a Transitive
VPC in AWS
Our First Attempt
CPE to Transit VPC

VPC IGW
Create a VPC – this will become your transitive hub, add a
VGW, IGW other standard networking constructs.
CSR CSR

AZ-A
VGW
AZ-B
Connect your network to AWS – we used Direct Connect
G G (DX), but VPN is fine.
R R
E E After the VPC is up, we add in a pair of CSR1000V’s. These
will mostly be used for BGP/IPSec, and VRF’s as well.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Transitive Routing in AWS
Transit VPC to Client VPC

VPC
10.1.1.0/24
VPC
10.1.2.0/24
Build a spoke/client VPC and assign it a VGW
and CIDR block.
VGW VGW

G G
Each Client VPC builds two tunnels to each
R R
VPC CSR – Four total tunnels per Client VPC.
E E
IGW

Once the client VPC builds its VPN’s, you can

CSR CSR

AZ-A AZ-B

G
VGW
G download a template for the CSR.
R R
E E At this point, you could spin up an EC2
instance in the Client VPC and ping it from
your on-prem network!

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Our First Attempt – challenges!
Scalability of Route Tables

VPC VPC
Challenge:
10.1.1.0/24 10.1.2.0/24 You can only have a maximum of 100 routes in
VGW VGW the routing table on the Client VPC.
G G Solution:
R R
VPC
E
We chose to use the CIDR summary from AWS.
E
IGW

You can download these in JSON format (link at

AZ-A
CSR CSR

AZ-B
end) and you can use a Python script to sort it by
G
VGW
G region and output to a CSV (see Appendix).
R R
E E Remember that the VGW route table and the VPC
route table are separate, but both count!

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Our First Attempt – challenges!
Tunnel interfaces can get duplicate IP’s

VPC VPC
Challenge:
10.1.1.0/24 10.1.2.0/24 Once you get to ~20 tunnels you will get
VGW VGW duplicate IP addresses on the tunnel interfaces
G G about 50% of the time.
R R
VPC
E
IGW
E Solution:
Obviously this screams “VRF” – and that is
what we did. We named the VRF based on the
CSR CSR

AZ-A AZ-B

VPN name from AWS, and we set the RD to

VGW
G G
R R BGP ASN:Tunnel#’s (e.g. 65000:9192 for ASN
E E 65000 and tunnels 91 and 92).

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Our First Attempt – challenges!
Scalability of Security Groups

VPC VPC
Challenge:
10.1.1.0/24 10.1.2.0/24 When we deployed you had a maximum of 50
VGW VGW entries in a Security Group.
G G
Solution:
R R
VPC
E
IGW
E To deal with IPSec re-key compatibility issues, we
attached a UDP/500 and UDP/50 permit to each
CIDR block sending us traffic.
CSR CSR

AZ-A AZ-B
VGW
G G
R R
E E

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Transitive Routing in AWS
Overview of our first solution

VPC Main Challenges:

10.1.1.0/24 Tunnel IP: 169.254.x.y • We were getting a lot of duplicate
VGW
Elastic IP: 35.160.x.y IP’s on the tunnels.

Elastic IP: 35.160.x.y • We were running into scalability

VPC IGW Tunnel IP: 169.254.x.y limits on the AWS Routing Table.
CSR
CSR
172.16.16.0/23
CSR
CSR
• We were running into scalability
VGW
G1 IP: 172.16.16.142 limits on the Security Groups.
Tunnel 0 IP: 172.16.12.1/30
• From the client perspective,
Sub-Int IP: 172.16.12.6 everything else was automated,
Tunnel 0 IP: 172.16.12.2/30 then they would wait on us to
BGP ASN: 65000 configure everything manually!!

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Our Second Attempt at
Building a Transitive
VPC - Automation
Our Second Attempt - Overview
Using AWS Automation for Network Configuration

S3
AWS KMS

bucket Cisco
Configurator
VGW
Poller
IGW
VPC

10.1.1.0/24
CSR
172.16.16.0/23 CSR
CSR CSR

VGW
VGW

Region-spoke = true

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Our Second Attempt – Ordering CSR’s

• Go to the AWS
Marketplace.

• Pick a license –
either BYOL or get a
license from AWS.

• Be aware of what
region you are
ordering in.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Our Second Attempt – Ordering CSR’s

• Select the tab called

Manual Launch.
• Select an IOS
Version to run.
• Be aware of what
region you are
ordering in.
• Leave this tab open,
but do not ‘order’
the CSR.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Our Second Attempt – Launching Cloud
Formation
• In a separate
browser tab pull
up the solution
doc (link in
appendix).
• Click on the button
that says “Launch
Solution”.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Our Second Attempt – CloudFormation (CFN)

• You will be taken to the Cloud Formation Template page. Currently

it will always default to the North Virginia Region – so if you want
any other region, the first thing to do is select the appropriate
region.
• This solution relies heavily on Lambda functions, not all regions
support Lambda, so please ensure your desired region does.
• You will need Administrator access for this solution to work.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Our Second Attempt – CloudFormation (CFN)
• This screen is where
the technical info is
input:
• You need to pick the
throughput of the
CSR’s.
• Your key-pair for the
region.
• If you need a second
account you can add
it here.
• The TAG to place on
the spoke VGW’s.
• Review and Accept

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Our Second Attempt – CloudFormation (CFN)
• After you submit, you
will then see the Cloud
Formation Template
start to deploy.
• This step will take
perhaps 15 minutes.
• The CFN will try and
clean itself up if it fails.
If it fails, you can see
what did NOT get
cleaned up and do it
manually.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
You now have an automated transit VPC!
How do I use it?

VPC
• So now I have an automated
10.1.1.0/24 Transit VPC … but all my
VGW
Region-spoke = true
clients are on the Manual one.
X • Tag the Spoke, it builds 4 new
VPC IGW Manual
VPC IGW Automated tunnels
CSR
CSR
172.16.16.0/23 CSR
CSR CSR
CSR
172.16.16.0/23
CSR
CSR
• Bring down the old tunnels
VGW VGW
• Now you are on the
automated solution.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Summary
What Did We Learn?
Key Take Aways Tips and Tricks Traps and Pitfalls
• You can teach an old • Always check what • Review scalability
dog new tricks. Region you are limits before you
working in. deploy.
• How can we enable
the business to be • If you have an • Consider BYOL
nimble? AWS/Cloud team you Licenses for your
work with, find out if CSR’s.
• When you are doing they have an account
a similar task again that needs access. • Put nothing in the
and again, can you Transit VPC other
automate it? • Consider your than the CSR’s
naming standards
upfront.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Thank you

#CLUS
Complete your online session evaluation

Give us your feedback to be entered

into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings DEVNET-
3677

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Appendix
Helpful links for your use
• Overview of the automated solution -
https://docs.aws.amazon.com/solutions/latest/cisco-based-transit-
vpc/overview.html
• Be aware of scalability limits -
https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_
Appendix_Limits.html
• Supported instance types for CSR’s (avoid 8XL) -
https://www.cisco.com/c/en/us/td/docs/routers/csr1000/software/
aws/b_csraws/b_csraws_chapter_011.html

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Appendix
Helpful links for your use
• JSON file for public IP ranges - https://ip-
ranges.amazonaws.com/ip-ranges.json
• The BGP rekey issue we ran into (Fan Yang explains it part way
down) - https://supportforums.cisco.com/t5/cisco-cloud-service-
router-csr/csr-to-aws-vgw-vpn-rekey-is-delayed-transit-vpc/td-
p/3062071
• The Python Script for parsing the JSON and saving to CSV -
https://github.com/la-networks/AWS-IPrange-parser
• Deeper look into general AWS items -
https://www.youtube.com/watch?v=p8RzqWepAsA

#CLUS CCSCLD-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
#CLUS