.Ietf88.Isoc - Peering Brazil.2013110503

Peering in Brazil
PTT.br
IETF 88th – ISOC Panel

Regional Interconnection Overview
Vancouver, BC, Canada
November 6, 2013
Eduardo Ascenço Reis <eascenco@nic.br>

PTT.br Engineering Team <eng@ptt.br>
PTTMetro/PTT.br - Peering in Brazil – ISOC Panel Regional Interconnection Overview – IETF 88th – PTT.br <eng@ptt.br> - NIC.br - http://ptt.br/ 1
Summary
The purpose of this presentation is to highlight peering in Brazil
considering the regional interconnection scenario.
CGI.br - The Brazilian Internet Steering Committee
CGI.br - The Brazilian Internet Steering Committee
The Brazilian Internet Steering Committee (CGI.br) was created by
Interministerial Ordinance 147, of May 31st, 1995, which was amended by
Presidential Decree 4,829 of September 3rd, 2003,
with the purpose of coordinating and integrating all Internet service initiatives in Brazil,
as well as promoting technical quality, innovation and the dissemination of
the services available.
NIC.br - The Brazilian Network Information Center
CGI.br – NIC.br – PTT.br - Financial Model
Brazilians
(persons or companies)
R$
Domains / IP / ASN
Registration
.br Authoritative Name Systems

DNS
.br
NIC.br divisions / activities

Registro.br
CERT.br
Cetic.br
Return Benefits Ceptro.br
Help Improve Internet PTT.br / IPv6.br/ SIMET.nic.br / etc
Reference Name
IXP - Internet eXchange Point
PTT – Ponto de Troca de Tráfego
Brazilian PTT History and Internet Key Points
Year Location (City/State) Event
1989 São Paulo / SP .br TLD Activation
1991 São Paulo / SP Brazil First Connection to Internet via FAPESP (Dawn of Brazil Internet)
1996 São Paulo / SP PTT ANSP/FAPESP Operation Start
1997 São Paulo / SP Registro.br Operation Start (Brazil NIR)
2000 Porto Alegre / RS PTT RSIX Operation Start (RNP UFRGS)
2000 São Paulo / SP PTT OPTiX-LA Operation Start (Optiglobe Inc, now Tivit)
2002 Curitiba / PR PTT PriX Operation Start (RNP UFPR)
2002 Brasilia / DF PTT FIX Operation Start (RNP)
2004 São Paulo / SP PTTMetro (CGI.br)
2004 Rio de Janeiro / RJ PTTMetro (CGI.br)
2004 Brasilia / DF PTT FIX move to PTTMetro (CGI.br)
2004 Barueri / SP PTT ANSP/FAPESP moved to NAP do Brasil (Terremark Latin America)
2005 Curitiba / PR PTT PRIX moved to PTTMetro (CGI.br)
From Peering in Brazil presentation @ LACNIC XI / Salvador, BA – Brazil - Interconnection Schemes and Strategies for ISPs Tutorial
PTT Close Proliferation Problem (e.g. São Paulo city in the past) - 1/3
City Area
PTT PTT
Optix-LA ANSP/FAPEP
PTT PTT
Diveo Telcomp
Customer Too many Connections

AS High Cost
City Area
PTT PTT
Optix-LA ANSP/FAPEP
PTT PTT
Diveo Telcomp
?
Customer
AS
City Area
PTT PTT
Optix-LA ANSP/FAPEP
PTT PTT
Diveo Telcomp
Customer ISP
AS NSP
IP Transit
PTTMetro – Brazilian Metropolitan IXP Project
PTTMetro (PTT- Internet Exchange Point) is the project of the

Brazilian Internet Steering Committee (CGI.br) that provides the
necessary infrastructure for the direct interconnection between the
diverse networks that operate in a metropolitan region.
http://cgi.br/
http://www.cgi.br/internacional/
http://ptt.br/
PTTMetro – PTT.br – Project # Operation
PTTMetro
CGI.br project started in 2004 and is still active.
PTT.br
Operation division responsible to deploy and maintain PTTMetro locations.
PTT.br has a partnership to support its operation with entities that host PIX
on all locations. Not complete list of PIX's hosts types:
● National Education and Research Network (RNP)
● Academic (e.g. Universities)
● Information Technology Government Companies (e.g. Procempa, Prodest, Prodepa)
● Internet Data Centers (IDC)
● Internet Service Providers (ISP)
● Internet Network Providers (NSP)
PTTMetro/PTT.br - Financial Model HighLights
PTTMetro / PTT.br costs are sustained by CGI.br / NIC.br, despite

Autonomous Systems (AS) participants needs for ports (number and capacity).
Each participant AS has only costs to reach one or more Interconnection Points (PIX)
of a specific PTT.br location in order to access its peering fabric.
PTTMetro/PTT.br - Regional Metro Ethernet Non-Blocking Switch/Peering Fabric
AS A
Router
IXP
PIX A
PIX
AS D Router PIX D PIX B Router AS B
Central
PIX C
Router
PIX: Interconnection Point
AS C
PTTMetro/PTT.br – High Availability Network Topology Model
PIX Central PIX Central

C1 C2
PIX
Remote
PIX: Interconnection Point
Dark Fiber Pair
PTTMetro/PTT.br – Operational Responsibilities
NIC.br
PTT.br is responsible for:
Installation, Management, Administration, Monitoring and Supporting all IX locations.
Always keeping Neutrality and High Quality as main targets.
Highlight: PTT.br is responsible to provide enough capacity to sustain bandwidth demand
on connection between PIX: (e.g. 1GE, 10GE, nx 10GE with DWDM).
PIX
Each entity that hosts a PTT.br PIX is responsible to provide continuous and high quality:
● IDC Infrastructure
Rack Space, Electrical Energy, Refrigeration, Physical Security,

Operational Hand, etc
● Dark Fiber Pair (preferably redundant) to Central PIX
PTTMetro/PTT.br – São Paulo – PIX Topology
PTT.br – Actual Domain of Operation
BRAZIL
PTT.br Locations after PTTMetro Project was Started in 2004
● Brasília
● Rio de Janeiro
● São Paulo
PTT.br Locations Currently in Operation
Each location is an independent IX
PTT.br Locations Currently in Operation – Two Recently Activated
PTTMetro/PTT.br – IXP Locations
All PTTMetro Locations are Isolated
The main purpose for PTTMetro locations is to allow the local traffic to
stay on the location.
The Brazilian Internet Steering Committee do not intend to interconnect PTTMetro

Locations and compete with telecommunications companies.
PTTMetro project stimulates and support inter locations transport services done
by participants companies.
PTTMetro have defined some transport modes in order to optimize common

resources and reduce costs that are already in production
(e.g. Campinas <=> São Paulo by Fasternet company)
NIC.br - Brazilian AS Distribution
NIC.br - Brazilian AS Distribution
NIC.br/PTT.br – IXP Locations and Brazilian AS Distribution
Brazilian Autonomous Systems (AS) ~2200
PTT.br IXP Locations
PTT.br – Participants Distribution per Location
Americana Florianopolis Porto Alegre

Belem Fortaleza Recife
Belo Horizonte Goiania Rio de Janeiro
Brasilia Lajeado Salvador
Campina Grande Londrina Sao Jose do Rio
Preto
Campinas Manaus Sao Jose dos
Campos
Caxias Maringa Sao Paulo
Curitiba Natal Vitoria
PTT.br – All Locations Aggregated Traffic
PTT.br – Day Sample Traffic per Location
PTT.br – All Locations Aggregated Exchanged Traffic Growth
50
45
40
35
30
Petabytes
25
20
15
10
0
Aug
Aug
Aug
Aug
Feb
Aug
Ago
Ago
Feb
Feb
Feb
Feb
Fev
/ 201
/ 20
/ 20
/ 20
/ 20
/ 20
/2
/2
/2
/2
/2
/2
/2
007
008
009
010
011
012
013
08
09
10
12
13
1
PTT.br – Three Years Traffic Curve Comparison
Week Day Comparison

PTT.br Sao Paulo
200
180
160
140
120
100
Gbps
80
60
40
20
0
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2012 2013
Week Day Comparison

PTT.br Sao Paulo
200
180
160
140
120
100
Gbps
80
60
40
20
0
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2012 2013
Comercial Hours Reference (8am to 6pm)
Week Day Comparison

PTT.br Sao Paulo
200
180
160
140
120
100
Gbps
80
60
40
20
0
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2012 2013
Comercial Hours Reference (8am to 6pm)
Sunday Comparison
PTT.br Sao Paulo
250
200
150
Gbps
100
50
0
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2012 2013
Sunday Comparison
PTT.br Sao Paulo
250
200
150
Gbps
100
50
0
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2012 2013
Week Day Relative Comparison

PTT.br Sao Paulo
7,00%
6,00%
5,00%
4,00%
Bytes
3,00%
2,00%
1,00%
0,00%
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2013
Relative Analyses: per hour proportional to full day traffic exchanged
Week Day Relative Comparison

PTT.br Sao Paulo
7,00%
6,00%
5,00%
4,00%
Bytes
3,00%
2,00%
1,00%
0,00%
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2013
Sunday Relative Comparison

PTT.br Sao Paulo
8,00%
7,00%
6,00%
5,00%
4,00%
Bytes
3,00%
2,00%
1,00%
0,00%
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2013
Sunday Relative Comparison

PTT.br Sao Paulo
8,00%
7,00%
6,00%
5,00%
4,00%
Bytes
3,00%
2,00%
1,00%
0,00%
06 08 10 12 14 16 18 20 22 00 02 04
05 07 09 11 13 15 17 19 21 23 01 03
Time
2011 2013
PTTMetro/PTT.br – Peering Agreements
Multi-Lateral Peering Agreements (MLPA)
IP (v4/v6) traffic exchange between all participants.

BGP sessions established between participants and IXP route servers.
Different MLPA VLANs for IPv4 and IPv6.
Bilateral Peering Agreements (BPA)
Shared VLAN Mode

IP traffic exchange only between participants.
Use the same MLPA VLANs.
Dedicated VLAN Mode

IP traffic exchange only between participants (e.g. Internet Transit).
Use a dedicate VLAN for each peering agreement.
PTTMetro/PTT.br – Logical Connection Structure per Location
MLPA IPv4 MLPA IPv6

AS A AS B AS C
(e.g. VLAN tag 10) (e.g. VLAN tag 20)
Dedicated
AS D VLAN AS E
(e.g. tag 2010)
Dedicated
AS F VLAN AS G
(e.g. tag 2020)
PTTMetro/PTT.br – Reference Model
Fundamental Definition
IP (v4|v6)
AS A AS B
PTTMetro/PTT.br – Basic Operational Model – Allowed Traffic
AS A MAC A MAC B AS B
IP (v4|v6)
PTT.br
Location
Peering Fabric
Allowed Frames with Ethertypes:

● 0x0800 - IPv4
● 0x0806 - ARP
● 0x86dd - IPv6
PTTMetro/PTT.br – Extended Operational Model – Multiple AS per Single Port
Use Different VLAN Tags for Logical Isolation (IEEE 802.1Q)
VLAN
A
AS A MAC A
VLAN
A
PTT.br
NSP Location
Transport Peering Fabric
Provider MLPA
VLAN C
VLAN
B
AS D MAC D
VLAN
B
PTTMetro/PTT.br – Economical Summary
PTTMetro - Economically Interests for Telecommunication Companies
● Reduction of Transit and Interconnection Costs
● Last/First Mile Provisioning
● IP Services Interconnection
Using dedicated VLANs to provide L2 logical isolation for:
Internet Transit (IPv4 and IPv6), Backup, Storage, VoIP, etc
● Inter Locations Transport
● Interconnection Points (PIX) Hosting
● Neutral Facilities for the Brazilian Telecommunications Agency Regulated Interconnections
PTTMetro/PTT.br – SIMET
SIMET.NIC.BR
Internet Traffic Measurement System
NIC.br AS14026 operating in most

of PTT.br locations
PTTMetro/PTT.br – SIMET
PTTMetro/PTT.br – Redundant Authoritative DNS Servers Service
Anycast Copies at 14 PTT.br IX Locations
Root: L.root-servers.net
.BR cctld: [b-f].dns.br
PTTMetro/PTT.br – Redundant Authoritative DNS Servers Service
PTT.br – Participação no Euro-IX
European Internet Exchange Association (Euro-IX)

https://www.euro-ix.net/
PTT.br is an Euro-IX member since 2010

with personal attendance on meetings since 2011.
PTT.br – Participação no LAC-IX
LAC-IX
Latino American and Caribean Internet Exchange Asociation (LAC-IX)
http://lac-ix.org/
PTT.br is a cofounder and member of LAC-IX.
PTTMetro/PTT.br – PTT Forum 7 – São Paulo, Brazil
http://ptt.br/pttforum/
The PTT Fórum 7 event will take place in São Paulo, Brazil on the 2nd and 3rd of
December 2013.
PTT Fórum normal audience is composed by Brazilian Autonomous Systems (AS) and
PTTMetro/PTT.br IX participants (some of them are international companies).
The event will be translated between Portuguese and English.
The event will be part of the Brazilian Internet Infrastructure week together with an
IPv6 Forum and GTER/GTS (Network Engineering, Operation and Security Working
Groups).
PTTMetro/PTT.br – Thanks & Contact
Thanks | Obrigado
http://ptt.br/
eng@ptt.br

.Ietf88.Isoc - Peering Brazil.2013110503

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

.Ietf88.Isoc - Peering Brazil.2013110503

Uploaded by

Copyright:

Available Formats

Peering in Brazil

IETF 88th – ISOC Panel

Eduardo Ascenço Reis <eascenco@nic.br>

The purpose of this presentation is to highlight peering in Brazil

considering the regional interconnection scenario.

The Brazilian Internet Steering Committee (CGI.br) was created by

Interministerial Ordinance 147, of May 31st, 1995, which was amended by

Presidential Decree 4,829 of September 3rd, 2003,

as well as promoting technical quality, innovation and the dissemination of

the services available.

.br Authoritative Name Systems

NIC.br divisions / activities

IXP - Internet eXchange Point

PTT – Ponto de Troca de Tráfego

Year Location (City/State) Event

1989 São Paulo / SP .br TLD Activation

1996 São Paulo / SP PTT ANSP/FAPESP Operation Start

1997 São Paulo / SP Registro.br Operation Start (Brazil NIR)

2000 Porto Alegre / RS PTT RSIX Operation Start (RNP UFRGS)

2002 Curitiba / PR PTT PriX Operation Start (RNP UFPR)

2002 Brasilia / DF PTT FIX Operation Start (RNP)

2004 São Paulo / SP PTTMetro (CGI.br)

2004 Rio de Janeiro / RJ PTTMetro (CGI.br)

2004 Brasilia / DF PTT FIX move to PTTMetro (CGI.br)

2005 Curitiba / PR PTT PRIX moved to PTTMetro (CGI.br)

Customer Too many Connections

PTTMetro (PTT- Internet Exchange Point) is the project of the

● Academic (e.g. Universities)

● Information Technology Government Companies (e.g. Procempa, Prodest, Prodepa)

● Internet Data Centers (IDC)

● Internet Service Providers (ISP)

● Internet Network Providers (NSP)

PTTMetro / PTT.br costs are sustained by CGI.br / NIC.br, despite

PIX Central PIX Central

PIX: Interconnection Point

Dark Fiber Pair

Rack Space, Electrical Energy, Refrigeration, Physical Security,

All PTTMetro Locations are Isolated

The Brazilian Internet Steering Committee do not intend to interconnect PTTMetro

PTTMetro have defined some transport modes in order to optimize common

Brazilian Autonomous Systems (AS) ~2200

PTT.br IXP Locations

Americana Florianopolis Porto Alegre

Week Day Comparison

2011 2012 2013

Week Day Comparison

2011 2012 2013

Comercial Hours Reference (8am to 6pm)

Week Day Comparison

2011 2012 2013

Comercial Hours Reference (8am to 6pm)

2011 2012 2013

2011 2012 2013

Week Day Relative Comparison

Relative Analyses: per hour proportional to full day traffic exchanged

Week Day Relative Comparison

Relative Analyses: per hour proportional to full day traffic exchanged

Sunday Relative Comparison

Relative Analyses: per hour proportional to full day traffic exchanged

Sunday Relative Comparison

Relative Analyses: per hour proportional to full day traffic exchanged

Multi-Lateral Peering Agreements (MLPA)