Internet Topology

COS 461: Computer Networks

Jennifer Rexford

1
Goals of Today’s Lecture
• Internet’s two-tiered topology
– Autonomous Systems, and connections between them
– Routers, and the links between them

• AS-level topology
– Autonomous System (AS) numbers
– Business relationships between ASes

• Router-level topology
– Points of Presence (PoPs)
– Backbone and enterprise network topologies

• Inferring network topologies

– By measuring paths from many vantage points
2
Internet Routing Architecture
• Divided into Autonomous Systems
– Distinct regions of administrative control
– Routers/links managed by a single “institution”
– Service provider, company, university, …

• Hierarchy of Autonomous Systems

– Large, tier-1 provider with a nationwide backbone
– Medium-sized regional provider with smaller backbone
– Small network run by a single company or university

• Interaction between Autonomous Systems

– Internal topology is not shared between ASes
– … but, neighboring ASes interact to coordinate routing 3
Autonomous System Numbers
AS Numbers are 16 bit values.
Currently just over 20,000 in use.
• Level 3: 1
• MIT: 3
• Harvard: 11
• Yale: 29
• Princeton: 88
• AT&T: 7018, 6341, 5074, …
• UUNET: 701, 702, 284, 12199, …
• Sprint: 1239, 1240, 6211, 6242, …
• …
4
AS Topology
• Node: Autonomous System
• Edge: Two ASes that connect to each other

4
3

2
7 6

1
5
What is an Edge, Really?
• Edge in the AS graph
– At least one connection between two ASes
– Some destinations reached from one AS via the other
d
d

AS 1 AS 1

Exchange Point

AS 2 AS 2 AS 3
6
Interdomain Paths
Path: 6, 5, 4, 3, 2, 1

2
7 6

1
Web server
Client 7
Business Relationships
• Neighboring ASes have business contracts
–How much traffic to carry
–Which destinations to reach
–How much money to pay

• Common business relationships

–Customer-provider
 E.g., Princeton is a customer of AT&T
 E.g., MIT is a customer of Level 3
–Peer-peer
 E.g., Princeton is a peer of Patriot Media
 E.g., AT&T is a peer of Sprint
8
Customer-Provider Relationship
• Customer needs to be reachable from everyone
– Provider tells all neighbors how to reach the customer

• Customer does not want to provide transit service

– Customer does not let its providers route through it
Traffic to the customer Traffic from the customer

d provider
advertisements

provider
traffic
customer
d customer
9
Peer-Peer Relationship
• Peers exchange traffic between customers
– AS exports only customer routes to a peer
– AS exports a peer’s routes only to its customers
– Often the relationship is settlement-free (i.e., no $$$)

Traffic to/from the peer and its customers

advertisements

peer peer
traffic

d 10
Princeton Example
• Internet: customer of AT&T and USLEC
• Research universities/labs: customer of Internet2
• Local residences: peer with Patriot Media
• Local non-profits: provider for several non-profits

AT&T USLEC Internet2

peer Patriot
11
AS Structure: Tier-1 Providers
• Tier-1 provider
– Has no upstream provider of its own
– Typically has a national or international backbone
– UUNET, Sprint, AT&T, Level 3, …
• Top of the Internet hierarchy of 12-20 ASes
– Full peer-peer connections between tier-1 providers

12
Efficient Early-Exit Routing
• Diverse peering locations
Customer B
– Both costs, and middle

• Comparable capacity at all

peering points
Provider B
– Can handle even load

• Consistent routes
– Same destinations advertised
multiple
at all points
peering
points Early-exit – Same AS path length for a
routing destination at all points

Provider A

13
Customer A
AS Structure: Other ASes
• Tier-2 providers
– Provide transit service to downstream customers
– … but, need at least one provider of their own
– Typically have national or regional scope
– E.g., Minnesota Regional Network
– Includes a few thousand of the ASes

• Stub ASes
– Do not provide transit service to others
– Connect to one or more upstream providers
– Includes vast majority (e.g., 85-90%) of the ASes

14
Characteristics of the AS Graph
• AS graph structure
– High variability in node degree (“power law”)
– A few very highly-connected ASes
– Many ASes have only a few connections

1 All ASes have 1 or more neighbors

0.1
CCDF

0.01 Very few have degree >= 100

0.001
1 10 100 1000 AS degree 15
 Characteristics of AS Paths
• AS path may be longer than shortest AS path
• Router path may be longer than shortest path
2 AS hops,
8 router hops

s d

16
3 AS hops, 7 router hops
Intra-AS Topology
• Node: router
• Edge: link

17
Hub-and-Spoke Topology
• Single hub node
–Common in enterprise networks
–Main location and satellite sites
–Simple design and trivial routing

• Problems
–Single point of failure
–Bandwidth limitations
–High delay between sites
–Costs to backhaul to hub

18
Princeton Example
• Hub-and-spoke
–Four hub routers and many spokes

• Hub routers
–Outside world (e.g., AT&T, USLEC, …)
–Dorms
–Academic and administrative buildings
–Servers

19
Simple Alternatives to Hub-and-Spoke
• Dual hub-and-spoke
– Higher reliability
– Higher cost
– Good building block

• Levels of hierarchy
– Reduce backhaul cost
– Aggregate the bandwidth
– Shorter site-to-site delay
…
20
Backbone Networks
• Backbone networks
–Multiple Points-of-Presence (PoPs)
–Lots of communication between PoPs
–Accommodate traffic demands and limit delay

21
Abilene Internet2 Backbone

22
Points-of-Presence (PoPs)
• Inter-PoP links
–Long distances Inter-PoP
–High bandwidth Intra-PoP

• Intra-PoP links
–Short cables between
racks or floors
–Aggregated bandwidth

• Links to other networks Other networks

–Wide range of media and
bandwidth
23
Where to Locate Nodes and Links
• Placing Points-of-Presence (PoPs)
–Large population of potential customers
–Other providers or exchange points
–Cost and availability of real-estate
–Mostly in major metropolitan areas

• Placing links between PoPs

–Already fiber in the ground
–Needed to limit propagation delay
–Needed to handle the traffic load

24
Customer Connecting to a Provider
Provider Provider

1 access link 2 access links

Provider Provider

2 access routers 2 access PoPs

25
Multi-Homing: Two or More Providers
• Motivations for multi-homing
–Extra reliability, survive single ISP failure
–Financial leverage through competition
–Better performance by selecting better path
–Gaming the 95th-percentile billing model

Provider 1 Provider 2

26
Shared Risks
• Co-location facilities (“co-lo hotels”)
– Places ISPs meet to connect to each other
– … and co-locate their routers, and share space & power
– E.g., 32 Avenue of the Americas in NYC

• Shared links
– Fiber is sometimes leased by one institution to another
– Multiple fibers run through the same conduits
– … and run through the same tunnels, bridges, etc.

• Difficult to identify and accounts for these risks

– Not visible in network-layer measurements
– E.g., traceroute does not tell you links in the same ditch
27
Learning the Internet Topology
• Internet does not have any central management
– No public record of the AS-level topology
– No public record of the intra-AS topologies

• Some public topologies are available

– Maps on public Web sites
– E.g., Abilene Internet2 backbone

• Otherwise, you have to infer the topology

– Measure many paths from many vantage points
– Extract the nodes and edges from the paths
– Infer the relationships between neighboring ASes

28
Inferring an Intra-AS Topology
• Run traceroute from many vantage points
– Learn the paths running through an AS
– Extract the hops within the AS of interest
1 169.229.62.1 inr-daedalus-0.CS.Berkeley.EDU
2 169.229.59.225 soda-cr-1-1-soda-br-6-2
3 128.32.255.169 vlan242.inr-202-doecev.Berkeley.EDU
4 128.32.0.249 gigE6-0-0.inr-666-doecev.Berkeley.EDU
5 128.32.0.66 qsv-juniper--ucb-gw.calren2.net
6 209.247.159.109 POS1-0.hsipaccess1.SanJose1.Level3.net
7 209.247.9.170 pos8-0.hsa2.Atlanta2.Level3.net
8 66.185.138.33 pop2-atm-P0-2.atdn.net
AOL 9 66.185.142.97 Pop1-atl-P3-0.atdn.net
10 66.185.136.17 pop1-atl-P4-0.atdn.net
11 64.236.16.52 www4.cnn.com 29
Challenges of Intra-AS Mapping
• Firewalls at the network edge
– Cannot typically map inside another stub AS
– … because the probe packets will be blocked by firewall
– So, typically used only to study service providers

• Identifying the hops within a particular AS

– Relies on addressing and DNS naming conventions
– Difficult to identify the boundaries between ASes

• Seeing enough of the edges

– Need to measure from a large number of vantage points
– And, hope that the topology and routing doesn’t change

30
Inferring the AS-Level Topology
• Collect AS paths from many vantage points
– Learn a large number of AS paths
– Extract the nodes and the edges from the path

• Example: AS path “1 7018 88” implies

– Nodes: 1, 7018, and 88
– Edges: (1, 7018) and (7018, 88)

• Ways to collect AS paths from many places

– Mapping traceroute data to the AS level
– Measurements of the interdomain routing protocol

31
Map Traceroute Hops to ASes
Traceroute output: (hop number, IP)
1 169.229.62.1 AS25
2 169.229.59.225 AS25 Berkeley
3 128.32.255.169 AS25
4 128.32.0.249 AS25
5 128.32.0.66 AS11423 Calren
6 209.247.159.109 AS3356
7 * AS3356
Level3
8 64.159.1.46 AS3356
9 209.247.9.170 AS3356
10 66.185.138.33 AS1668
11 * AS1668 AOL
12 66.185.136.17 AS1668
13 64.236.16.52 AS5662 CNN
32
Challenges of Inter-AS Mapping
• Mapping traceroute hops to ASes is hard
– Need an accurate registry of IP address ownership
– Whois data are notoriously out of date

• Collecting diverse interdomain data is hard

– Public repositories like RouteViews and RIPE-RIS
– Covers hundreds to thousands of vantage points
– Especially hard to see peer-peer edges

AT&T Sprint

??? Harvard
Harvard 33
d1 B-school d2
Inferring AS Relationships
• Key idea
– The business relationships determine the routing policies
– The routing policies determine the paths that are chosen
– So, look at the chosen paths and infer the policies

• Example: AS path “1 7018 88” implies

– AS 7018 allows AS 1 to reach AS 88
– AT&T allows Level 3 to reach Princeton
– Each “triple” tells something about transit service

• Collect and analyze AS path data

– Identify which ASes can transit through the other
– … and which other ASes they are able to reach this way
34
Paths You Should Never See (“Invalid”)

Customer-provider

Peer-peer

two peer edges

transit through a customer 35

Challenges of Relationship Inference
• Incomplete measurement data
– Hard to get a complete view of the AS graph
– Especially hard to see peer-peer edges low in hierarchy

• Real relationships are sometime more complex

– Peer is one part of the world, customer in another
– Other kinds of relationships (e.g., backup and sibling)
– Special relationships for certain destination prefixes

• Still, inference work has proven very useful

– Qualitative view of Internet topology and relationships
36
Conclusions
• Two-tiered Internet topology
–AS-level topology
–Intra-AS topology
• Inferring network topologies
–By measuring paths from many vantage points
• Next class
–Intradomain and interdomain routing

37