Professional Documents
Culture Documents
BGP Multihoming Techniques: Nanog 41 14th - 16th October 2007 Albuquerque, New Mexico
BGP Multihoming Techniques: Nanog 41 14th - 16th October 2007 Albuquerque, New Mexico
Philip Smith
<pfs@cisco.com>
NANOG 41
14th - 16th October 2007
Albuquerque, New Mexico
NANOG 41
Presentation Slides
Available on
ftp://ftp-eng.cisco.com
/pfs/seminars/NANOG41-Multihoming.pdf
And on the NANOG website
NANOG 41
Preliminaries
Presentation has many configuration examples
Uses Cisco IOS CLI
NANOG 41
NANOG 41
Why Multihome?
NANOG 41
Why Multihome?
Redundancy
One connection to internet means the network is dependent on:
Local router (configuration, software, hardware)
WAN media (physical failure, carrier failure)
Upstream Service Provider (configuration, software,
hardware)
NANOG 41
Why Multihome?
Reliability
Business critical applications demand continuous availability
Lack of redundancy implies lack of reliability implies loss of
revenue
NANOG 41
Why Multihome?
Supplier Diversity
Many businesses demand supplier diversity as a matter of
course
Internet connection from two or more suppliers
With two or more diverse WAN paths
With two or more exit points
With two or more international connections
Two of everything
NANOG 41
Why Multihome?
Not really a reason, but oft quoted
Leverage:
Playing one ISP off against the other for:
Service Quality
Service Offerings
Availability
NANOG 41
Why Multihome?
Summary:
Multihoming is easy to demand as requirement for any service
provider or end-site network
But what does it really mean:
In real life?
For the network?
For the Internet?
And how do we do it?
NANOG 41
10
NANOG 41
11
NANOG 41
12
Multihoming Definition
More than one link external to the local network
two or more links to the same ISP
two or more links to different ISPs
NANOG 41
13
14
See www.iana.org/assignments/as-numbers
NANOG 41
15
Private-AS Application
Applications
An ISP with customers
multihomed on their backbone
(RFC2270)
65001
193.0.32.0/24
NANOG 41
65002
193.0.33.0/24
1880
193.1.34.0/24
65003
193.2.35.0/24
193.1.32.0/22 1880
16
Private-AS Removal
Private ASNs MUST be removed from all prefixes
announced to the public Internet
Include configuration to remove private ASNs in the eBGP
template
Cisco IOS
neighbor x.x.x.x remove-private-AS
NANOG 41
17
Configuring Policy
Three BASIC Principles for IOS configuration examples
throughout presentation:
prefix-lists to filter prefixes
filter-lists to filter ASNs
route-maps to apply policy
NANOG 41
18
Policy Tools
Local preference
outbound traffic flows
Metric (MED)
inbound traffic flows (local scope)
AS-PATH prepend
inbound traffic flows (Internet scope)
Communities
specific inter-provider peering
NANOG 41
19
NANOG 41
20
Originating Prefixes
The RIRs publish their minimum allocation sizes per /8 address block
AfriNIC:
www.afrinic.net/docs/policies/afpol-v4200407-000.htm
APNIC:
www.apnic.net/db/min-alloc.html
ARIN:
www.arin.net/reference/ip_blocks.html
LACNIC:
lacnic.net/en/registro/index.html
RIPE NCC:
www.ripe.net/ripe/docs/smallest-alloc-sizes.html
Note that AfriNIC only publishes its current minimum allocation size, not the
allocation size for its address blocks
IANA publishes the address space it has assigned to end-sites and allocated
to the RIRs:
www.iana.org/assignments/ipv4-address-space
NANOG 41
21
NANOG 41
22
Multihoming Scenarios
Stub network
Multi-homed stub network
Multi-homed network
Configuration Options
NANOG 41
23
Stub Network
AS101
AS100
24
AS65530
AS100
25
Multi-homed Network
Global Internet
AS200
AS300
AS100
NANOG 41
26
AS 200
1.1.1.1
Cisco IOS
router bgp 65534
neighbor 1.1.1.1 remote-as 200
neighbor 1.1.1.1 ebgp-multihop 2
!
ip route 1.1.1.1 255.255.255.255 serial 1/0
ip route 1.1.1.1 255.255.255.255 serial 1/1
ip route 1.1.1.1 255.255.255.255 serial 1/2
NANOG 41
AS 65534
27
R1
R3
AS 100
AS 200
R2
Desired Path
Used Path
NANOG 41
28
or
NANOG 41
29
AS 200
ISP
AS 201
NANOG 41
30
ISP
No magic solution
NANOG 41
AS 201
31
NANOG 41
32
NANOG 41
33
NANOG 41
34
NANOG 41
35
NANOG 41
36
NANOG 41
37
NANOG 41
38
Add loopback
configuration
!
interface serial 0/0
ip address 121.10.0.1 255.255.255.252
!
interface serial 0/1
ip address 121.10.0.5 255.255.255.252
!
router ospf 100
network 121.10.255.1 0.0.0.0 area 0
network 121.10.2.0 0.0.0.15 area 0
passive-interface default
no passive-interface Ethernet 0/0
!
ip route 121.10.24.0 255.255.252.0 serial 0/0
Customer
connections
39
B
A
a subset of routers, or
just on the upstream edge
NANOG 41
C
E
AS200
40
B
A
NANOG 41
AS200
41
NANOG 41
42
43
NANOG 41
44
NANOG 41
45
NANOG 41
46
NANOG 41
47
NANOG 41
48
NANOG 41
49
NANOG 41
50
NANOG 41
51
NANOG 41
52
Basic Multihoming
NANOG 41
53
Basic Multihoming
No frills multihoming
Will look at two cases:
Multihoming with the same ISP
Multihoming to different ISPs
NANOG 41
54
Basic Multihoming
This type is most commonplace at the edge of the
Internet
Networks here are usually concerned with inbound traffic flows
Outbound traffic flows being nearest exit is usually sufficient
NANOG 41
55
Basic Multihoming
NANOG 41
56
Basic Multihoming:
Multihoming to the same ISP
Use BGP for this type of multihoming
use a private AS (ASN > 64511)
There is no need or justification for a public ASN
Making the nets of the end-site visible gives no useful
information to the Internet
NANOG 41
57
NANOG 41
58
NANOG 41
59
AS 100
E
AS 65534
D
B
backup
NANOG 41
60
NANOG 41
61
NANOG 41
62
..next slide
NANOG 41
63
NANOG 41
64
NANOG 41
65
NANOG 41
66
NANOG 41
67
With Loadsharing
NANOG 41
68
NANOG 41
69
AS 100
E
AS 65534
D
B
Link two
NANOG 41
70
NANOG 41
71
72
Router C only allows in /19 and /20 prefixes from customer block
Router D configuration is identical
NANOG 41
73
NANOG 41
74
NANOG 41
75
NANOG 41
76
NANOG 41
77
NANOG 41
78
AS 100
E
A1 AS 65534
B1
A2 AS 65534
B2
A3 AS 65534
B3
79
NANOG 41
80
81
NANOG 41
82
Router C only allows in /19 and /20 prefixes from customer block
Router D configuration is almost identical
NANOG 41
83
84
Router E configuration:
router bgp 100
neighbor 122.102.10.17 remote-as 110
neighbor 122.102.10.17 prefix-list my-aggregate out
!
ip prefix-list my-aggregate permit 121.8.0.0/13
NANOG 41
85
Basic Multihoming
NANOG 41
86
NANOG 41
87
Inconsistent-AS?
Viewing the prefixes
originated by AS65534 in the
Internet shows they appear to
be originated by both AS210
and AS200
AS 65534
AS 200
IOS command is
AS 210
Internet
NANOG 41
88
NANOG 41
89
AS 100
AS 120
C
AS 130
NANOG 41
90
NANOG 41
91
NANOG 41
92
NANOG 41
93
NANOG 41
94
With Loadsharing
NANOG 41
95
Internet
AS 100
AS 120
C
Announce first
/20 and /19 block
Announce second
/20 and /19 block
AS 130
NANOG 41
96
NANOG 41
97
NANOG 41
98
NANOG 41
99
NANOG 41
100
NANOG 41
101
AS 100
AS 120
C
AS 130
NANOG 41
102
NANOG 41
103
NANOG 41
104
105
NANOG 41
106
NANOG 41
107
NANOG 41
108
NANOG 41
109
NANOG 41
110
3: BGP is complex
In the wrong hands, yes it can be! Keep it Simple!
NANOG 41
111
NANOG 41
112
NANOG 41
113
NANOG 41
114
NANOG 41
115
C
Local Peer
AS120
NANOG 41
AS 110
116
NANOG 41
117
Prefix filters
inbound
NANOG 41
118
AS Path filters
more trusting
NANOG 41
119
NANOG 41
120
NANOG 41
121
Aside:
Configuration Recommendations
Private Peers
The peering ISPs exchange prefixes they originate
Sometimes they exchange prefixes from neighbouring ASNs
too
NANOG 41
122
NANOG 41
123
NANOG 41
124
C
A
NANOG 41
AS 110
125
NANOG 41
126
next slide
NANOG 41
127
NANOG 41
120.5.10.2
120.5.10.2
120.5.10.2
120.5.10.3
120.5.10.3
120.5.10.3
120.5.10.4
120.5.10.4
120.5.10.4
120.5.10.5
120.5.10.5
120.5.10.5
remote-as 100
peer-group ixp-peers
prefix-list peer100
remote-as 101
peer-group ixp-peers
prefix-list peer101
remote-as 102
peer-group ixp-peers
prefix-list peer102
remote-as 103
peer-group ixp-peers
prefix-list peer103
in
in
in
in
128
NANOG 41
129
NANOG 41
130
NANOG 41
131
NANOG 41
132
Aside:
IXP Configuration Recommendations
IXP peers
The peering ISPs at the IXP exchange prefixes they originate
Sometimes they exchange prefixes from neighbouring ASNs
too
NANOG 41
133
NANOG 41
134
NANOG 41
135
Upstream ISP
AS140
C
Local Peer
AS120
NANOG 41
AS 110
136
Router A
Same routing configuration as in example with one upstream
and one local peer
NANOG 41
137
NANOG 41
138
NANOG 41
139
NANOG 41
140
Accept default from one upstream and some routes from the
other upstream
The way to go!
NANOG 41
141
...next slide
NANOG 41
142
NANOG 41
143
NANOG 41
144
NANOG 41
145
NANOG 41
146
NANOG 41
147
Router C Configuration
router bgp 110
network 121.10.0.0 mask 255.255.224.0
neighbor 122.102.10.1 remote-as 130
..next slide
NANOG 41
148
NANOG 41
149
NANOG 41
150
151
NANOG 41
152
NANOG 41
153
Aside:
Configuration Recommendation
When distributing internal default by iBGP or OSPF
Make sure that routers connecting to private peers or to IXPs do
NOT carry the default route
Otherwise they could point a default route to you and
unintentionally transit your backbone
Simple fix for Private Peer/IXP routers:
ip route 0.0.0.0 0.0.0.0 null0
NANOG 41
154
NANOG 41
155
NANOG 41
156
Diagram
ISP B
AS120
ISP A
AS110
ISP C
AS130
B
AS 100
157
NANOG 41
158
NANOG 41
159
NANOG 41
160
NANOG 41
3561 Savvis
3356 Level 3
7018 AT&T
701
UUNET
2914 Verio
209
Qwest
161
.*
162
NANOG 41
163
NANOG 41
164
Example:
Link to ISP B could be used just for Broadband/Dial customers so
number all such customers out of one contiguous subprefix
Link to ISP C could be used just for commercial leased line customers
so number all such customers out of one contiguous subprefix
NANOG 41
165
NANOG 41
166
NANOG 41
167
/21 to ISP B
dial customers
/22 to ISP C
biz customers
e.g. Single prepend
on ISP B link
168
Strategy:
Originate default route by OSPF on Router A (with metric 10)
link to ISP A
Originate default route by OSPF on Router B (with metric 30)
links to ISPs B & C
Plus on Router B:
Static default route to ISP B with distance 240
Static default route to ISP C with distance 245
When link goes down, static route is withdrawn
NANOG 41
169
NANOG 41
170
NANOG 41
171
Other considerations
Default route should not be propagated to devices
terminating non-transit peers and customers
No need to carry default in iBGP
Filter out default in iBGP mesh peerings
NANOG 41
172
NANOG 41
173
NANOG 41
174
NANOG 41
175
Communities
NANOG 41
176
Using Communities:
RFC1998
Informational RFC
Describes how to implement loadsharing and backup
on multiple inter-AS links
BGP communities used to determine local preference in
upstreams network
NANOG 41
177
RFC1998
Community values defined to have particular meanings:
NANOG 41
preferred route
ASx:90
ASx:80
ASx:70
178
RFC1998
Sample Customer Router Configuration
router bgp 130
neighbor x.x.x.x remote-as 100
neighbor x.x.x.x description Backup ISP
neighbor x.x.x.x route-map config-community out
neighbor x.x.x.x send-community
!
ip as-path access-list 20 permit ^$
ip as-path access-list 20 deny .*
!
route-map config-community permit 10
match as-path 20
set community 100:90
NANOG 41
179
RFC1998
Sample ISP Router Configuration
! Homed to another ISP
ip community-list 70 permit 100:70
! Homed to another ISP with equal ASPATH length
ip community-list 80 permit 100:80
! Customer backup routes
ip community-list 90 permit 100:90
!
route-map set-customer-local-pref permit 10
match community 70
set local-preference 70
!
..next slide
NANOG 41
180
RFC1998
route-map set-customer-local-pref permit 20
match community 80
set local-preference 80
!
route-map set-customer-local-pref permit 30
match community 90
set local-preference 90
!
route-map set-customer-local-pref permit 40
set local-preference 100
NANOG 41
181
RFC1998
Supporting RFC1998
Many ISPs do, more should
Check AS object in the Internet Routing Registry
If you do, insert comment in AS object in the IRR
Or make a note on your website
NANOG 41
182
NANOG 41
183
Background
RFC1998 is okay for simple multihomed customers
assumes that upstreams are interconnected
NANOG 41
184
NANOG 41
185
More info at
www.sprintlink.net/policy/bgp.html
NANOG 41
186
AS2764
ASN-CONNECT-NET
AAPT Limited
CNO2-AP
CNO2-AP
Community support definitions
Community Definition
-----------------------------------------------2764:2 Don't announce outside local POP
2764:4 Lower local preference by 15
2764:5 Lower local preference by 5
2764:6 Announce to customers and all peers
(incl int'l peers), but not transit
2764:7 Announce to customers only
2764:14 Announce to AANX
routing@connect.com.au
CONNECT-AU
nobody@connect.com.au 20050225
CCAIR
More at http://info.connect.com.au/docs/routing/general/multi-faq.shtml#q13
NANOG 41
187
NANOG 41
188
mnt-by:
source:
NANOG 41
189
AS5400
BT Ignite European Backbone
Community to
Not announce
Community to
AS prepend 5400
To peer:
5400:2000
5400:1500
5400:1501
5400:1502
5400:1503
5400:1504
5400:1506
5400:2500
5400:2501
5400:2502
5400:2503
5400:2504
5400:2506
All Transits
Sprint Transit (AS1239)
SAVVIS Transit (AS3561)
Level 3 Transit (AS3356)
AT&T Transit (AS7018)
GlobalCrossing Trans(AS3549)
5400:2001
5400:2002
5400:2004
And many
many more!
190
AS3356
Level 3 Communications
------------------------------------------------------customer traffic engineering communities - Suppression
------------------------------------------------------64960:XXX - announce to AS XXX if 65000:0
65000:0
- announce to customers but not to peers
65000:XXX - do not announce at peerings to AS XXX
------------------------------------------------------customer traffic engineering communities - Prepending
------------------------------------------------------65001:0
- prepend once to all peers
65001:XXX - prepend once at peerings to AS XXX
3356:70
3356:80
3356:90
3356:9999
LEVEL3-MNT
RIPE
set local
set local
set local
blackhole
preference to 70
preference to 80
preference to 90
(discard) traffic
And many
many more!
191
NANOG 41
192
Summary
NANOG 41
193
Summary
Multihoming is not hard, really
Keep It Simple & Stupid!
NANOG 41
194
End of Tutorial
NANOG 41
195