PS FTEL Metro HLD - 20162901

FTEL NEW METRO NETWORK HLD
Generated For : FTEL

Project Name : 22107 – NEW METRO
Generated On : November 9, y
Version : 0.8
FTEL New METRO Project
Document Control
Authors: Haluk Goktas
Change Authority: N/A
Change Forecast: low
Revision History:
Version Date Issued Status Author Reason for Change

0.1 30/11/2015 Draft Haluk Goktas Initial Issue
13/12/2015 Haluk Goktas Section 3.6,4,5,6 and 7
0.2 Draft
are added
21/12/2015 Haluk Goktas Section
3.3,3.6.4,3.8,6.3,7,8 and
0.3 Final
9.7 are added. Section 9.1
and 3.7.4 are revised
0.4 22/12/2015 Issue Haluk Goktas Minor typos are fixed
4/1/2016 Haluk Goktas Section 6.2, 6.3.4, 6.3.5,
0.5 Issue
7.3,9.7, 3..7.4.2 are added
11/1/2016 Haluk Goktas Section 2.1.1 is added and
also ipv6 address
0.6 Issue
allocation is provided per
region on section 9.2.1
21/1/2016 Haluk Goktas Section 3.1.5 is added for
0.7 Issue
ISIS aithentication
29/1/2016 Haluk Goktas Minor change on section
0.8 Issue 9.7 ( L3VPN Vlans are
added)
Reviewers:
Department Name Approval Date

Juniper AdvancedServices Hai Dang Le
Intellectual Property Rights

This document contains valuable trade secrets and confidential information of Juniper Networks and its affiliates,
subsidiaries and suppliers, and shall not be disclosed to any person, organization, or entity unless such disclosure is
subject to the provisions of a written non-disclosure and proprietary rights agreement or intellectual property license
agreement approved by Juniper Networks. The distribution of this document does not grant any license in or rights, in
whole or in part, to the content, the product(s), technology, or intellectual property described herein.
JUNIPER NETWORKS Copyright  2015 All rights reserved Page 2 of 64

1. INTRODUCTION............................................................................................................................................7
1.1 Purpose.............................................................................................................................................................. 7
1.2 Audience............................................................................................................................................................. 7
1.3 Scope.................................................................................................................................................................. 7
1.4 Comments.......................................................................................................................................................... 7
1.5 Alphabetical Index of Terms............................................................................................................................... 8
2. PHYSICAL TOPOLOGY................................................................................................................................11
2.1 UPE (USER PROVIDER EDGE) NETWORK ELEMENTS...............................................................................11
2.1.1. UPE-Site Swicth Topolofy Options............................................................................................................... 11
2.2 Distributed Switch NETWORK ELEMENTS...................................................................................................... 13
2.3 NPE (NETWORK PROVIDER EDGE) NETWORK ELEMENTS......................................................................13
3. PROTOCOL DESIGN.............................................................................................................................15
3.1 IGP Routing...................................................................................................................................................... 15
3.1.1. Hierarchy and Area Architecture.................................................................................................................. 15
3.1.2. Routes to be carried in ISIS.......................................................................................................................... 16
3.1.3. Link Metric.................................................................................................................................................... 16
3.1.4. MTU size...................................................................................................................................................... 18
3.2 BGP Routing..................................................................................................................................................... 18
3.3 BGP-LU............................................................................................................................................................ 19
3.3.1. BGP-LU Design Consideration on Existing PE routers................................................................................20
3.4 Non-Stop-Routing (NSR).................................................................................................................................. 20
3.5 GRES ( Graceful Routing Engine SwitchOver)................................................................................................. 21
3.6 Fast Failure Detection....................................................................................................................................... 22
3.6.1. LFS Advantages........................................................................................................................................... 22
3.6.2. LFS Through the DWDM Transponders....................................................................................................... 23
3.6.3. Distributed BFD Sessions............................................................................................................................. 23
3.6.4. OAM LFM (Link Fault Management)............................................................................................................ 23
3.7 MPLS................................................................................................................................................................ 25
3.7.1. MPLS Network Design Options.................................................................................................................... 25
3.7.2. LDP (Label Distribution Protocol)................................................................................................................. 25
3.7.3. RSVP (Resource Reservation Protocol........................................................................................................ 26
3.7.4. End-to-End RSVP Signaled LSP................................................................................................................. 27
3.8 LLDP (Link Layer Discovery Protocol).............................................................................................................. 30
4. TRAFFIC LOAD BALANCING...............................................................................................................32
4.1 Load Balancing of the MPLS Packets............................................................................................................... 32
5. CLASS OF SERVICE.............................................................................................................................33
5.1 Forwarding classes........................................................................................................................................... 33
5.2 Packet classification and pre-classification....................................................................................................... 33
5.3 Egress Scheduling............................................................................................................................................ 34
5.4 Egress Re-write rule......................................................................................................................................... 35
6. NETWORK SERVICES...........................................................................................................................36
6.1 Inline Jflow........................................................................................................................................................ 36
6.2 RPM (Real-Time Performance Monitoring)....................................................................................................... 38
6.2.1. Timestamping............................................................................................................................................... 38
6.2.2. Reporting of Results..................................................................................................................................... 40
6.3 Ethernet OAM (Operations, Administration and Maintenance) For Services....................................................42
6.3.1. Fault Detection (CCM).................................................................................................................................. 43
6.3.2. Fault Verification........................................................................................................................................... 43
6.3.3. Fault Isolation............................................................................................................................................... 43
6.3.4. OAM on JUNOS........................................................................................................................................... 44
6.3.5. Junos Space OAM Insight Overview............................................................................................................ 44
6.4 Inline Video Monitoring..................................................................................................................................... 46

7. SERVICES..............................................................................................................................................47
7.1 eVPN (Ethernet VPN)...................................................................................................................................... 47
7.2 Multicast............................................................................................................................................................ 47
7.2.1. GLOP........................................................................................................................................................... 47
7.2.2. Administratively Scoped IPv4 Multicast Space............................................................................................. 48
7.2.3. PIM-SM........................................................................................................................................................ 48
7.2.4. ANYCAST RP.............................................................................................................................................. 48
7.2.5. Next Generation Multicast............................................................................................................................ 48
7.3 SERVICES BY MULTISERVICE CARDS ( MS-MIC )......................................................................................49
7.3.1. Logical Topology.......................................................................................................................................... 50
7.3.2. CGNAT on MS-MIC...................................................................................................................................... 50
8. SYSTEM MANAGEMENT......................................................................................................................56
8.1 Time Zone......................................................................................................................................................... 56
8.2 Root Authentication........................................................................................................................................... 56
8.3 RADIUS............................................................................................................................................................ 56
8.4 Authentication Order......................................................................................................................................... 56
8.5 Multicast ICMP Response................................................................................................................................. 56
8.6 Redirects........................................................................................................................................................... 56
8.7 Routing Engine (RE) Synchronization............................................................................................................... 56
8.8 Default Address Selection................................................................................................................................. 56
8.9 Internet Options................................................................................................................................................ 56
8.10 Location............................................................................................................................................................ 57
8.11 NTP................................................................................................................................................................... 57
8.12 Ports................................................................................................................................................................. 57
8.13 Services............................................................................................................................................................ 57
8.14 Syslog............................................................................................................................................................... 57
8.15 SNMP............................................................................................................................................................... 57
9. PLANNING..............................................................................................................................................58
9.1 HOSTNAME..................................................................................................................................................... 58
9.2 IP ADDRESSING.............................................................................................................................................. 58
9.2.1. IPv6 IP Addressing....................................................................................................................................... 59
9.2.2. Loopback Ip Address.................................................................................................................................... 59
9.2.3. IP Address Mapping to Hostname................................................................................................................ 60
9.3 AS Numbers..................................................................................................................................................... 60
9.4 VRF Route Distinguishers................................................................................................................................. 60
9.5 VRF Route Targets........................................................................................................................................... 61
9.6 Link Description................................................................................................................................................ 61
9.7 VLAN STRUCTURE......................................................................................................................................... 61
9.8 SCALING.......................................................................................................................................................... 61
10. SIGN OFF..................................................................................................................................................64
List of Figures
Figure 1: Site Swicth-uPE Topology Option 1............................................................................................................... 12
Figure 2: New Metro Network Topology........................................................................................................................ 14
Figure 3: Provincial Metro Network Topology................................................................................................................ 14
Figure 4: NSAP Address Encoding.............................................................................................................................. 15
Figure 5: Traffic Flow- Scenario I................................................................................................................................. 17
Figure 6: Traffic Flow-Scenario II.................................................................................................................................. 18
Figure 7: BGP-LU Diagram........................................................................................................................................... 20

Figure 8: LFS Illustration Diagram................................................................................................................................. 22

Figure 9: LFM Illustration Diagram............................................................................................................................... 23
Figure 10: BFD/LFM Illustration Diagram..................................................................................................................... 24
Figure 11: IP/MPLS Network Design Options.............................................................................................................. 25
Figure 12: Dynamic bypass LSP Illustration Diagram................................................................................................... 28
Figure 13: Egress Protection Diagram.......................................................................................................................... 29
Figure 14: End-to-end Arbor DDOS design................................................................................................................. 37
Figure 15: RPM Client-Server....................................................................................................................................... 38
Figure 16: PFE-based Time-Stamping on Trio Chipset............................................................................................... 39
Figure 17: Service-PIC based Time-Stamping............................................................................................................ 39
Figure 18: Payload of a Time-stamped RPM Probe..................................................................................................... 39
Figure 19: MIB OIDS for each probes........................................................................................................................... 41
Figure 20: Prober Results from third-party MIB browser............................................................................................... 41
Figure 21: CFM diagram with multiple domains........................................................................................................... 42
Figure 22: Inline Video Monitoring................................................................................................................................ 46
Figure 23: eVPN Diagram............................................................................................................................................ 47
Figure 24: CGN Logical Topology................................................................................................................................ 50
Figure 25: CGNAT44+APP.......................................................................................................................................... 51
Figure 26: CGNAT44+APP+Different Mapping Behaviors........................................................................................... 52
Figure 27: CGNAT44+APP+Different Mapping/Filtering Behaviors.............................................................................53
Figure 28: Hair-pinning................................................................................................................................................ 54
Figure 29: IPv6 Private Address Allocated for Loopback Interfaces............................................................................60
Figure 30: Layer 3 VPN – VPN Prefix (RD + IPv4 Prefix)........................................................................................... 60
List of Tables
Table 1: Alphabetical Index of Terms............................................................................................................................... 9
Table 2: NPE Site Locations.......................................................................................................................................... 13
Table 3: ISIS Metrics-Scenario I.................................................................................................................................... 16
Table 4: ISIS Metrics-Scenario II................................................................................................................................... 17
Table 5: MX480 Scaling Table....................................................................................................................................... 19
Table 6: RSVP timers and multipliers............................................................................................................................ 26
Table 7: Queue design and METRO network links schedulers’ parameters..................................................................33
Table 8: Mapping of Scheduler priorities to Trio Hardware priorities..............................................................................34
Table 9: Inline Jflow Performance Scaling on Different MPC Cards..............................................................................36
Table 10: RPM Management Interface.......................................................................................................................... 40
Table 11: FTEL GPOL Address..................................................................................................................................... 48
Table 12: Administratively Scoped IPv4 Multicast address............................................................................................ 48
Table 13: IETF IMIX Ratio............................................................................................................................................. 49
Table 14: CGNAT Scaling Numbers.............................................................................................................................. 50
Table 15: JFLOW Scaling Numbers............................................................................................................................... 50
Table 16: System-Location variables............................................................................................................................. 57

Table 17: Naming convention for hosname................................................................................................................... 58

Table 18: IPv4 Private Address allocation per regions for IP/MPLS links......................................................................58
Table 19: IPv6 Private Address allocation per regions for PE-CE links.........................................................................59
Table 20: IPv4 Private Address allocation per regions for loopback interface...............................................................60
Table 21: AS Numbers Deployed Across Legacy Networks.......................................................................................... 60
Table 22: Route Distinguisher Format........................................................................................................................... 60
Table 23: Route Distinguisher Structure........................................................................................................................ 61
Table 24: VRF Route Target Format.............................................................................................................................. 61
Table 25: Service Vlan Allocation.................................................................................................................................. 61
Table 26: METRO Network platform Scaling Metrics..................................................................................................... 63

1. Introduction
1.1 Purpose
This document is a METRO NETWORK HLD leveraging best practices expertise and experience of Juniper
Professional Services based on a review of the FTEL business and technical requirements. It is an outcome of the
service engaged by SVTECH and will be used as input into subsequent Plan and Build Phase deliverables.
FTEL is planning of new METRO network to provide End-to-End MPL Transport to business services, L2VPN&VPLS, L3VPN,
Mobile Backhaul and multicast services (plain multicast and NG-MVPN) using SEAMLESS MPLS solution.
A tiered IP/MPLS architecture will be implemented providing scalability and resiliency. The seamless MPLS architecture is
utilized from access nodes to the core nodes.
This high-level design document is based on a review of FTEL’s business and technical requirements as a part of New
METRO Project. It provides a high level network design for deployment of Juniper’s products for METRO network. Juniper
Professional Services has reviewed the following list of requirements as part of reviewing this High Level Design (HLD)
Infrastructure service:
 Description of services to be supported by the METRO network.
 Requirements on resiliency and HA
 Requirements on Class of Service (CoS)
 Requirements on routing policies
 Scaling requirements and constraints and requirements on Capacity planning
 Requirements on Network Management and Monitoring.
 Requirements on Network planning ( Naming convention for hostname, interface description)
 Requirements on Ip addressing
 Requirements on new technologies ( eVPN, FW with MS-MIC etc…)
 Requirements on timing and synchronization ( SyncE, PTP)
The high level design and recommendations presented in this document are based on the expertise and experience of Juniper
Professional Services team in evaluating and recommending network architectures, products and technologies. The Juniper
PS team leverages our current knowledge of leading practices to provide an optimum design and minimize network disruption.
The proposed design and recommendations presented here are based on information provided by FTEL. Additional
information has been collected during meetings, conference calls or other forms of communication with FTEL’s teams. Based
on a review of network requirements, a description of physical and logical topologies was created. These topologies will serve
as the foundation for a low level design that provides technical detail, including device configurations.
1.2 Audience
The primary audience for this report is SVTECH network design and engineering team, network operations team and
any other service personnel directly or indirectly involved in this project. Juniper Networks personnel such as Service
Managers, JTAC engineers and other Professional Services consultants may also use this document as part of the
service delivery process.
1.3 Scope
The scope of this document is strictly confined on the high level design of the routing and connectivity principles
across the new METRO network. The document includes high level of discussion of any service, as all services
should be referenced in the respective SD document created for each service.
The solutions discussed are limited to the new Juniper components of FTEL new METRO project and do not cover
any third party products.
1.4 Comments
It is assumed that all comments and inputs from the stakeholders have been taken into account. For further comments
and inputs, the reader is required to contact the author of this document and/or SVTECH/FTEL design engineers.

1.5 Alphabetical Index of Terms

Following project specific terminology is used throughout this document to specify specific parts of the data network.
These terminologies may also reflect the physical place and function of the equipment with respect to the overall
project
Acronym Description
802.1Q IEEE specification for adding virtual local area network (VLAN) tags to an Ethernet frame.
802.3ad link Process that enables grouping of Ethernet interfaces at the physical layer to form a single link layer interface, also known
Distribution as a link Distribution group (LAG) or LAG bundle
802.3ah IEEE specification defining Ethernet between the subscriber and the immediate service provider. Also known as Ethernet in
the first or last mile.
AE Logical Aggregate Ethernet
ARP Address Resolution Protocol. Protocol for mapping IPv4 addresses to media access control (MAC) addresses; dynamically
binds the IP address (the logical address) to the correct MAC address.
BA classifier Behaviour Aggregate classifier. Method of classification that operates on a packet as it enters the router. The packet
header contents are examined, and this single field determines the class–of–service (CoS) settings applied to the packet.
BFD Bidirectional Forwarding Detection. Protocol that uses control packets and shorter detection time limits to more rapidly
detect failures in a network.
BGP Border Gateway Protocol (BGP) is a standardized exterior gateway protocol designed to exchange routing and reachability
information among autonomous systems (AS) on the Internet. The protocol is often classified as a path vector protocol but
is sometimes also classed as a distance-vector routing protocol. The Border Gateway Protocol makes routing decisions
based on paths, network policies, or rule-sets configured by a network administrator and is involved in making core routing
decisions.
BGP may be used for routing within an autonomous system. In this application it is referred to as Interior Border Gateway
Protocol, Internal BGP, or iBGP. In contrast, the Internet application of the protocol may be referred to as Exterior Border
Gateway Protocol, External BGP, or EBGP.
BGP-LU BGP Labeled Unicast (BGP-LU) is defined in RFC3107 to carry label mapping information in BGP. It is used to stitch
multiple areas in IP/MPLS network and provide end to end MPLS transport so called seamless MPLS
Bootp Bootstrap protocol. UDP/IP-based protocol that allows a booting host to configure itself dynamically and without user
supervision. BOOTP provides a means to notify a host of its assigned IP address, the IP address of a boot server host, and
the name of a file to be loaded into memory and executed. Other configuration information, such as the local subnet mask,
the local time offset, the addresses of default routers, and the addresses of various Internet servers, can also be
communicated to a host using BOOTP.
CoS Class of Service. Method of classifying traffic on a packet-by-packet basis using information in the type-of-service (ToS)
byte to provide different service levels to different traffic.
DC Data Centre
DHCP Dynamic Host Configuration Protocol. Mechanism through which hosts using TCP/IP can obtain protocol configuration
parameters automatically from a DHCP server on the network; allocates IP addresses dynamically so that they can be
reused when no longer needed.
FW Firewall
GRES Graceful Routing Engine switchover. In a router that contains a master and a backup Routing Engine, allows the backup
Routing Engine to assume mastership automatically, with no disruption of packet forwarding.
HA High Availability
HLD High Level Design
HTTP Hypertext Transfer Protocol. Method used to publish and receive information on the Web, such as text and graphic files.
HTTPS Hypertext Transfer Protocol over Secure Sockets Layer. Similar to HTTP with an added encryption layer that encrypts and
decrypts user page requests and pages that are returned by a Web server. Used for secure communication, such as
payment transactions.
IGMP Internet Group Management Protocol. Host-to-router signalling protocol for IPv4 to report their multicast group
memberships to neighbouring routers and determine whether group members are present during IP multicasting. Similarly,
multicast routers, use IGMP to discover which of their hosts belong to multicast groups and to determine whether group
members are present.
ISIS Intermediate System to Intermediate System (IS-IS) is a routing protocol designed to move information efficiently within a
Network, a group of physically connected computers or similar devices. It accomplishes this by determining the best route
for datagrams through a packet-switched network.
JSRP JUNOS redundancy protocol
LACP Link Distribution Control Protocol. Mechanism for exchanging port and system information to create and maintain LAG
bundles.
LAG Link Distribution Group. Two or more network links bundled together to appear as a single link. Distributes MAC clients
across the link layer interface and collects traffic from the links to present to the MAC clients of the LAG. Also known as a
bundle.
LFM Link Fault Management. Method used to detect problems on links and spans on an Ethernet network defined in IEEE

Acronym Description
802.3ah.
LLD Low Level Design
LLDP Link Layer Discovery Protocol. Is a standard Data Link Layer protocol used by network devices for advertising of their
identity, capabilities, and interconnections on a IEEE 802 LAN network.
M-BGP Multiprotocol Extensions for BGP (MBGP), sometimes referred to as Multiprotocol BGP or Multicast BGP and defined in
IETF 4760, is an extension to Border Gateway Protocol (BGP) that allows different types of addresses (known as address
families) to be distributed in parallel. Whereas standard BGP supports only IPv4 unicast addresses, Multiprotocol BGP
supports IPv4 and IPv6 addresses and it supports unicast and multicast variants of each.
MF classifier Method for classifying traffic flows. Unlike a behaviour aggregate (BA) classifier, a multi-field classifier examines multiple
fields in the packet to apply class-of-service (CoS) settings. Examples of fields that a multi-field classifier examines include
the source and destination address of the packet, as well as the source and destination port numbers of the packet.
MSDP Multicast Source Discovery Protocol. Used to connect multicast routing domains to allow the domains to discover multicast
sources from other domains. It typically runs on the same router as the PIM sparse mode rendezvous point (RP).
MSTP Multiple Spanning Tree Protocol. Spanning tree protocol used to prevent loops in bridge configurations. Unlike other types
of STPs, MSTP can block ports selectively by VLAN.
NMS Network Management System: network management station. System that enables a user to configure and monitor network
elements.
NTP Network Time Protocol. Used to synchronize the system clocks of hosts on the Internet to Universal Coordinated Time
(UTC). A router can update its clock automatically by configuring it as a Network Time Protocol (NTP) client. Using NTP
enables the system to record accurate times of events. You can view the log file of events to monitor the status of the
network.
OAM Operation, Administration, and Maintenance. ATM Forum specification for monitoring ATM virtual connections, verifying
that the connection is up and the router is operational. A set of Ethernet connectivity specifications and functions providing
connectivity monitoring, fault detection and notification, fault verification, fault isolation, loopback, and remote defect
identification. The primary specifications defining Ethernet OAM are IEEE 802.3ah link-fault management (LFM) and IEEE
802.1ag Ethernet connectivity-fault management (CFM).
OoB Out Of Band. OoB Management is when devices (switches, servers, fw) use a dedicated network for their management.
OSPF Open Shortest Path First. Dynamic routing protocol intended to operate within a single Autonomous System. It adverises
the states of local network links within the AS and makes routing decisions based on the shortest-path-first (SPF) algorithm
(also referred to as the Dijkstra algorithm). OSPF is a link-state routing protocol.
P2P Point-to-point.
PFE Packet Forwarding Engine. ASIC that is responsible of the forwarding functions.
PIM-SM Protocol Independent Multicast Sparse Mode. A sparse mode multicast protocol, which uses shared trees. In a shared tree,
sources forward multicast datagram’s to a directly connected router, the designated router. The designated router
encapsulates the datagram and unicasts it to an assigned rendezvous point router, which then forwards the datagram to
members of multicast groups.
POC Proof of Concept
PS Professional Services
QoS Quality of Service. Performance, such as transmission rates and error rates, of a communications channel or system. A
suite of features that configure queuing and scheduling on the forwarding path of networking equipment. QoS provides a
level of predictability and control beyond the best-effort delivery that the router provides by default.
RE Routing Engine
RG SRX Cluster Redundant Group
RP Rendezvous Point. For PIM sparse mode, a core router acting as the root of the distribution tree in a shared tree.
RETH SRX Cluster Redundant interface
RVI Routed Virtual Interface. L3 interface associated to a VLAN.
SNMP Simple Network Management Protocol. Protocol governs network management and the monitoring of network devices and
their functions.
SSH Secure Shell. Protocol that uses strong authentication and encryption for remote access across a non-secure network. SSH
provides remote login, remote program execution, file copy, and other functions.
STP Spanning Tree Protocol. Defined in the IEEE standard 802.1D, the Spanning Tree Protocol is an OSI Layer 2 protocol that
ensures a loop-free topology for any bridged LAN. This protocol creates a spanning tree within a mesh network of
connected Layer 2 bridges (typically Ethernet switches), and disables the links that are not part of that tree, leaving a single
active path between any two network nodes.
Syslog System log. Method for sending and storing messages to a log file for troubleshooting or record-keeping. It can also be
used as an action within a firewall filter to store information to the messages file.
VC Virtual Chassis. Interconnect two or more EX 4200 switches, enabling them to operate as a unified single high bandwidth
switch.
VDC Virtual Data Centre
VR A routing instance of type virtual router

Acronym Description
VRF A routing instance of type virtual routing and forwarding
Table 1: Alphabetical Index of Terms

2. Physical Topology
FTEL introduces a new METRO network to cater business services (L3VPN, L2VPN&VPLS, Mobile Backhaul and
Multicast service) to its enterprise customers. The new METRO network comprises the following parts:
 UPE (User provider Edge) network elements with MX104 platform to be used to provide 1G connectivity to
customer and 10G uplinks to NPE (Network Provider Edge) network elements and it is located at PoP sites.
 NPE (Network Provider Edge) network elements with MX480 platform to be used to provide 10G connectivity
to Core network and to UPE network elements. NPE is to have RR (Route Reflector) function as well as ABR (
Area Border Router) function to reflect ring VPN routes to Regional RRs and exchange IGP routes between
L1 and L2 networks
 DIS (Distribution) switch network elements with QFX5100 platform to be used to aggregate and switch
multiple rings to NPE network elements. It named as DIS switch in FTEL deployment and it is located at Metro
PoP sites.
 AGG (Aggregation) PoP Switch: 3rd party (Huawei) sitch will be used in PoP sites to connect local site
switches and routers. As it is 3rd party switches, it needs to be taken care of by FTEL for bandwidth/port
expansion. It is used for PoP+ sites
 In case of multiple 10G needed between uPEs and/or between uPE and NPE, NPE and MC, Multiple Layer 3
links are configured in between network platforms with ECMP enabled. Each layer 3 link has multiple single
hop LSPs to balance the traffic between network elements based on the traffic load balancing described in
section 4
2.1 UPE (USER PROVIDER EDGE) NETWORK ELEMENTS

MX104 router is deployed up to 6 locations per ring. It has four built-in 10-Gigabit Ethernet SFP+ ports and 20-port 1
Gigabit Ethernet Modular Interface Card (MIC). MX104 (UPE) platforms are connected via 10-Gigabit Ethernet ports
to adjacent UPE or NPE device in a ring. 1-Gigabit Ethernet ports are connected to site switches to routers based in
customer sites.
MX104 (UPE) platforms are powered by the Junos Trio chipset and runs Junos operating-system (Junos OS) for high-
performance routing and switching. The chassis has 4 slots accepting Modular Interface Cards (MICs). The remaining
3 slots will be used for capacity expansion and advance service enablement. One slot is reserved for 20-port 1-Gigabit
Ethernet Modular Interface Card. One slot is reserved for 2-port 10-Gigabit Ethernet Modular Interface card or 20x1G
MIC. The last slot is reserved for MIC to enable advanced services like Firewall or IDS.
2.1.1 UPE-Site Swicth Topolofy Options

There will be maximum 6 uPE platforms per a ring, however there would less number of uPE within a ring based on
port, link capacity requirements . However the following combinations considered for uPE-site switch connection
topologies.
 6 uPE per ring
 5 uPE per ring
 3 uPE per ring
 2 uPE per ring

Option 1
In this option, 6 uPE per is considered and is followed for all rings with odd number of uPES ( 6xuPE, 4xuPE and
2xuPE).
The uPEs are paired for each group of site switches. Each switch is to be connected to dedicated uPE pair. It allows
us to forecast capacity requirement for each uPE pair downlink and uplinks and also helps to forcast the number of
port required per uPE pair. It make troubleshooting and management easy and makes the network behavior
predictable. It is recommended to follow this model when Site switch-uPE connections are designed.
Odd numbered uPE is chosen as primary router for Site Swicth sub group and even numbered uPE chosen as a
primary router for other site switch sub group. All uPEs will share the load between each other in active-active model
Figure 1: Site Swicth-uPE Topology Option 1

Option 2
In this option, due to local link distance to uPEs from Site switch, some of uPEs may be used by more than one site
switch group. This option is also applicable to rings where there are odd number of uPEs.
As it is shown in figure 2, uPE5 is used by Site switch group A and group D and uPE04 is used by site switch group B
and D.it is important to keep this type of topology scenario controlled and uPE05 and upE04 performance is monitored
closely. The level predictability substantially is decreased in the scenario and it is not recommended unless it is
necessary due to number of uPEs within a ring ( odd numbers) or due to local link distance between respective uPE
and site switch.

2.2 Distributed Switch NETWORK ELEMENTS

A QFX5100 switch is deployed per location. Regional rings up to 24 are connected to the distributed switch per NPE
pair. The switch provides layer-2 connectivity from regional rings (Uplink UPE) to NPE.
2.3 NPE (NETWORK PROVIDER EDGE) NETWORK ELEMENTS

MX480 (NPE) router is deployed in two locations, Hanoi and Ho Chi Minh City. Four MX480 platforms form two pair
NPE elements in each location. MX480 platform has one MPC4E card with 32-port 10-Gigabit Ethernet ports. MX480
(NPE) platforms establish connectivity between regional rings and Core network. By doing so, MX480 provides end-
to-end connectivity between rings across core network. MX480 is ABR (Area Border Router) to change routes
between Level 1 (rings) and Level 2 (Core network) networks. MX480 platforms also reflect routes from UPEs to VPN
RR (Route Reflectors) with inline RR ( Route Reflector) function.
MX480 (NPE) platforms are powered by the Junos Trio chipset and runs Junos operating-system (Junos OS) for high-
performance routing and switching. The chassis has 6 slots accepting Modular Port Concentrator ( MPC). The initial
deployment is with one MPC4E card and the rest of 5 slots are reserved for capacity expansion in the future.
Table 1 summarizes the NPE sites location where two platforms are deployed in a pair in each location.
Site NPE sites

NPEHNI100001/ NPEHNI200002
Hanoi NPEHNI100003/ NPEHNI200004
Ho Chi Minh City NPEHCM100001/NPEHCM200002
NPEHCM100003/NPEHCM200004
Table 2: NPE Site Locations

Figure 2: New Metro Network Topology
All connections between UPE-PoP Aggregation Switch, PoP Aggregation Switch-Distribution Switch, Distribution
Switch-NPE,NPE –MC ( Core prouter) platforms are 10-Gigabit Ethernet or multiple 10-Gigabit Ethernet connections.
In Some provinces, there will be dedicated NPE routers due to site distance to Hanoi or Ho Chi Minh City NPE sites. It
is decided to leverage same transmission Broadband service (MP) routers use via L2 switch. The platform to be used
for NPE is MX104 in mentioned provinces. Because of MX104 platform capacity, number of uPE will be limited to
24xuPE per MX104 NPE pair . Provincial network topology is shown in the following figure
Figure 3: Provincial Metro Network Topology

3. PROTOCOL DESIGN
3.1 IGP Routing
FTEL is using ISIS for the IGP routing in its IP core network. The new Metro infrastructure will be integrated using the
same ISIS design deployed today. Access Network will be in Level1 ISIS network as Core network continues to be in
Level2 Network.
NPEs will be ABR (Area Border Router) to exchange router to/from Level1 and Level2 networks. Group of rings
connected to each NPE pair will be in different area in ISIS topology.
uPE routers use NPE routers as the next hop for the default route 0.0.0.0/0. The NPE routers do not advertise a
default route—instead, NPE routers set the attached bit, which is advertised to the uPE router and causes the uPE
router to install a default route with the NPE router as the next hop.
To prevent a default route from being installed, the attached bit must be ignored, and this can be accomplished by
configuring the “ignore-attached-bit” command.
It is recommended using unique IS-IS area-IDs to identify each region in conjunction with the ignore-attach-bit
command.
By default, IS-IS uPE internal routes are installed into the L2 database; which is an exchange of routing information
between levels that needs to be prevented in multi-region networks. To prevent uPE internal routes from being
installed into the L2 database, an IS-IS export policy must be configured on the NPE routers to reject L1 routes.
However, to provide end-to-end troubleshooting capabilities, aggregate routes for uPE loopback and interface address
and will be exported into L2 area on each NPE pair. L1 external routes are not installed into the L2 database by
default, and IS-IS L2 internal routes are not installed into the L1
3.1.1 Hierarchy and Area Architecture

All Core Network routers are part of the existing ISIS L2 area. However, all Metro Network routers are part of ISIS L1
area. This will add a few routers to the existing domain, which still remains within the scaling capability of the ISIS
database.
In the IS-IS routing domain, each router is identified by its NSAP (Network Service Access Point) address. Routers are
recognized as in the same area if the area parts of their NSAP addresses are the same. The NSAP addressing of the
new nodes will follow the encoding already in place in the existing IGP design.
Figure 4: NSAP Address Encoding
The same practice of mapping the loopback address of the node into the System-ID or Router-ID of the NSAP
address is followed in the new infrastructure.

3.1.2 Routes to be carried in ISIS

ISIS carries the loopback IPv4/IPv6 addresses of the router, the point-to-point link subnets, multicast sources and the
management subnets to support the following:
 Reachability for the setup of MPLS LSPs between PEs, NPE and UPEs;
 Reachability of resolve MP-BGP and BGP-U next hops;
 Reachability for network management and operations;
 Link capacities for traffic-engineering purposes.
All TLVs are level 2 TLVs in the ISIS database. L1 is explicitly disabled on all ISIS-enabled interfaces of core routers.
L2 is explicitly disabled on all ISIS-enabled interfaces of Access Network (UPEs). L1 is explicitly enabled on NPE
interface facing Metro network (facing UPE routers), L2 is explicitly enabled on NPE interfaced facing core network
(facing MC routers) No external subnets are injected into the domain. This protects the scale of the database in the
core network.
3.1.3 Link Metric

The metric assignment on the new Metro Network IGP links is a very critical part of the IGP design as metric design
influences the traffic routing to achieve the following requirements:
SCENARIO I
In this scenario, NPEs connecting core network will balance incoming and outgoing Metro network traffic.
 Outgoing traffic from UPE is to use the closest NPE to exit to core network.
 Incoming traffic from remote metro ring selects the closest NPE based on BGP best path selection.
 Both NPEs routes rings’ traffic to core network.
 Both NPEs are redundant to each other.
 Traffic destined to customer in same ring remain within the ring.
 Traffic destined to customer in different ring connected to same NPE ( same area) remains within the same
area.
 Traffic destined to customer in different ring connected to different NPE travels across core network to reach
to destination.
The metric assignment for Metro network links categories are shown in the tables below.
Metro
Link Level IGP Metric Comment
uPE-uPE L1 300
uPE-NPE01 L1 300
To define a determintistic
uPE-NPE02 L1 310 path for traffic coming
from middle uPE
NPE01-NPE02 L1&L2 100
NPE-MC L2 200
MC-RR L2 10000
MC-MC L2 10 Intra-region
MC-MC L2 100 Inter-region
MC-PE/MP L2 500

MC-GW L2 3000 International GW

Table 3: ISIS Metrics-Scenario I
Figure 5: Traffic Flow- Scenario I
SCENARIO II
In this scenario, NPE01 will route all ring incoming and outgoing traffic to core network (active-passive).
 Outgoing traffic from UPE is to use NPE01 to exit to core network.
 UPE link metric connecting to NPE02 will be 2000
 Incoming traffic from remote metro ring selects NPE01 as NPE01 will advertise routes with higher local
preference.
 Only NPE01 routes rings’ traffic to core network.
 NPE02 is redundant to NPE02.
 Traffic destined to customer in same ring remain within the ring.
 Traffic destined to customer in different ring connected to same NPE (same area) remains within the same
area.
 Traffic destined to customer in different ring connected to different NPE travels across core network to reach
to destination.
The metric assignment for Metro network links categories are shown in the tables below.
Link Level IGP Metric Comment

uPE-uPE L1 300
uPE-NPE01 L1 300
To force all ring traffic to be
uPE-NPE02 L1 2000
routed to NPE01
NPE01-NPE02 L1&L2 100
NPE-MC L2 200
MC-RR L2 10000
MC-MC L2 10 Intra-region
MC-MC L2 100 Inter-region
MC-PE/MP L2 500
MC-GW L2 3000 International GW

Table 4: ISIS Metrics-Scenario II
Figure 6: Traffic Flow-Scenario II
3.1.4 MTU size

Setting the MTU consistently across the network interfaces is crucial for transport performance and the formation of
ISIS adjacency. Adjacent routers will not form IGP adjacency unless they agree on the MTU size. To avoid
unnecessary fragmentation, core-facing interfaces should be set large enough to accommodate large packets taking
into consideration the label stack and any encapsulation on core links. Fragmentation is bandwidth-inefficient because
the complete IP header is duplicated on each fragment. It is also CPU intensive since fragments require additional
processing on both the end hosts and the IP routers. Reassembly also requires more memory on the receiver.
Moreover, if one fragment is lost, the complete packet has to be discarded due to the lack of selective-retransmission-
of IP-fragments support in IPv4/IPv6.
The MTU on all core physical interfaces (10GE) will be set to 9192 bytes. This is applicable to all NPE MX480 and
UPE MX104 routers.
On all customer-facing interfaces, the MTU size should be set to the same value consistently with the current design
for all the services.
3.1.5 ISIS Authentication

All IS-IS protocol exchanges can be authenticated to guarantee that only trusted routing devices participate in the
autonomous system (AS) routing. By default, IS-IS authentication is disabled on the routing device.
In FTEL METRO Network, MD5 authentication type is to be used.
3.2 BGP Routing

Multi-protocol BGP (MP-BGP) is required on all NPE and UPE routers to form multi-protocol internal peering session
with other NPEs and UPEs and to support the range of services needed by FTEL. Those services are:
 IPv4 unicast VPN service
 IPv6 unicast VPN service
 IPv4 multicast VPN service
 IPv6 multicast VPN service
 L2 VPN ( RFC6624. Previously known as Kompella draft) service
 VPLS (Virtual private LAN service)

 eVPN service for future use
Each of these services is supported through a specific BGP address family, which has to be negotiated upon BGP
session establishment between the peers. For session scalability, the peering is established with Route Reflector (RR)
routers, which reflect the routes between all the clients.
In New Metro network, UPE routers are peered with NPEs, which is in inline RR function. NPEs establish BGP-LU
peering with respective site VPN RRs (Hanoi and Ho Chi Minh City). This design is called Hierarchical BGP-LU
design.
NPE routers could suffer huge demand for RIB capacity, as they are inline RR routers for given group of uPE rings.
The following table provides a summary of the routes scaling on the different new platforms involved in the METRO
project setup. The scaling numbers are based on testing in isolated conditions as per the Junos 14.2 releases.
Platform RIB scaling FIB scaling Comment
RE-S-1800x4-32G TRIO MPC
MX480 12 millions IPv4 routes with 2.5 millions IPv4 unicast prefixes
next-hop resolution
2.5 millions VPNv4 prefixes (VRF-
table-label)
1 million 6PE IPv6 prefixes
Table 5: MX480 Scaling Table
3.3 BGP-LU
BGP-LU provides reachability between regions by advertising uPE loopbacks and label bindings to NPE, which in turn
advertises the loopbacks and label bindings to remote uPEs in other regions. The diagram in Figure 6 outlines the
operation of BGP-LU at a high level in each of the different types of routers in the regions.
uPE router advertises is loopback ip address and label with next-hop self. NPE receives loopback ip address and
assign label then advertise the loopback and label with next-hop self. Remote NPE receives loopback ip address and
assigns label and then advertise loopback and label with next-hop self to remote uPE. Remote uPE receives loopback
ip address and use preferred route and label for forwarding.
Note: Seamless MPLS solution requires 5 labels and current default maximum label is set to 3 on MX routers.
It is recommended to set the maximum labels to 5 on all uPE and NPE routers to support Seamless MPLS
solution

Figure 7: BGP-LU Diagram
3.3.1 BGP-LU Design Consideration on Existing PE routers

Existing PE routers do not have BGP-LU enabled today and to provide connectivity to the customers, who has been
connected to existing PE routers, from new METRO network, BGP-LU feature is to be enabled on existing routers. By
default, maximum number of labels is set to 3 on existing PE routers, to support label stack with BGP-LU which is 5
labels, maximum label needs to be set to 5.
The following strategy is recommended for enabling BGP-LU on existing PE routers.
 Increase maximum labels to 5 on all PE routers
 Configure end-to-end LSPs from one PE to new NPE routers with node-link protection enabled
 Enable BGP-LU on VPN RR (Router Reflectors) and on one PE router (it is recommended to issue the
change during maintenance call)
 Verify connectivity from new METRO network to the PE routers.
 Enable BGP-LU on the rest of PE routers as needed (the customers on existing PE router require connectivity
to new METRO network as they have a site connected to one of uPE router in METRO network)
3.4 Non-Stop-Routing (NSR)

Non-Stop Active Routing (NSR) is to be enabled on all new METRO network routers to allow for graceful routing-
engine switchover without breaking the routing connectivity. Enabling NSR results in the protocols states to be
synchronously available at the two routing engines such that sessions and adjacencies do not need to re-establish on
the backup RE upon mastership switchover.
The NSR support is highly protocol-dependent. For FTEL’s deployment, the following is the list of the main protocols
that are essential for the core operation and should therefore be supported by the replication process between the
master and backup routing engines. Though other protocols related to the access interfaces are also supported.

 ISIS
 BGP
 LDP
 RSVP
 BFD
All MP-iBGP sessions involved in this setup negotiate the following set of BGP address families.
 inet unicast
 inet flow
 inet6 label-unicast explicit null
 inet-vpn unicast
 inet-vpn flow
 inet-mvpn signaling
 inet6-mvpn signaling
 L2vpn signaling
 route target
As of Junos 14.2 OS releases, NSR is not supported for the flow and NG-MVPN address families. If the BGP peer in
the master Routing Engine has negotiated address-family capabilities that are not supported for nonstop active
routing, then the corresponding BGP neighbor state on the backup Routing Engine shows as idle. On switchover, the
BGP session is re-established from the new master Routing Engine, causing huge network churn and slow
convergence. To optimize the setup, it might be a good option to isolate those unsupported families in a separate BGP
session. This enables NSR for all other families, which actually carry the most of routes in the network.
There will be 2 BGP groups configured on uPE and NPE platforms. One BGP group for NSR supported family groups
mentioned above and the second BGP groups for unsupported families. The secondary loopback ip address is used
for the second BGP group.
Note: uPE platforms are to be deployed with single Routing Engine (RE) and NSR is not needed at this
moment. NSR is to be enabled only on NPE platforms. Once Redundant RE is installed on uPE platform, NSR
will be enabled accordingly. It is recommended to have second BGP group for unsupported BGP families
using secondary loopback ip address on uPE even though NSR is not enabled.
3.5 GRES ( Graceful Routing Engine SwitchOver)

The graceful Routing Engine switchover (GRES) feature in Junos OS enables a routing platform with redundant
Routing Engines to continue forwarding packets, even if one Routing Engine fails. GRES preserves interface and
kernel information. Traffic is not interrupted. However, GRES does not preserve the control plane.
To preserve routing during a switchover, GRES must be combined with either of the followings:
 Graceful restart protocol extensions
 Nonstop active routing
Any updates to the master Routing Engine are replicated to the backup Routing Engine as soon as they occur. In
METRO design GRES is used with nonstop active routing to preserve control plane during a switchover.
Mastership is switched over to the backup routing engine for any of the following cases
 The master Routing Engine kernel stops operating.
 The master Routing Engine experiences a hardware failure.
 The administrator initiates a manual switchover.
If the backup Routing Engine does not receive a keep-alive from the master Routing Engine within 2 seconds, it
concludes that the master Routing Engine has failed and takes mastership.

The Packet Forwarding Engine:

 Seamlessly disconnects from the old master Routing Engine
 Reconnects to the new master Routing Engine
 Does not reboot
 Does not interrupt traffic
The new master Routing Engine and the Packet Forwarding Engine then becomes synchronized. If the new master
Routing Engine detects that the Packet Forwarding Engine state is not up to date, it resends state update messages.
3.6 Fast Failure Detection

All connectivity layers across all METRO routers in the METRO network are entirely Ethernet-based. As shown in
section 2.3, 10GE are used for the interconnectivity within UPEs and NPE-UPE as well as NPEs-MCs. Hence,
mechanism for fast failure detection on Ethernet links, is required to ensure fast service restoration with minimum
impact on transported services.
Since all the new METRO network links are Ethernet based, the default Link Fault Signaling (LFS) mechanism is
considered for fast detection of failures in the lower hardware layers before the Ethernet framing level. The
mechanism works well across the DWDM system for 10GE signals. The BFD (Bidirectional Forwarding Detection) is
additionally used to run on top of the ISIS protocol to account for the corner cases where failures occur in the upper
connectivity layers and go undetected by the LFS mechanism. The choice to run the BFD as a client of the ISIS
protocol is rather a simple option to achieve fast convergence as it also triggers the RSVP adjacency failure.
However, The links between distributed switch (QFX5100) and NPEs (MX480) forms an aggregated Ethernet link with
multiple 10-Gigabit Ethernet links to provide more than 10-Gigabit Ethernet bandwidth as multiple 10-Gigabit Ethernet
rings merge on the distributed switch. Micro BFD is not supported on QFX as of today and BFD is considered on links
between Core Network platforms (MC) and Aggregate Network platforms (NPE). Micro BFD also is considered for
ISIS between NPE and UPE network platforms to provide fast failure detection on IP/dynamic protocol level with
single links. This is important to coverage for failure scenarios that are not supported by OAM/LFM and LFS.
3.6.1 LFS Advantages

The following advantages are gained with LFS mechanism.
 It is on by default. No configuration required
 It provides faster failure detection than using upper layer protocols ( BFD, Ethernet OAM, etc…)
 It provide immediate link down and trigger consequent actions ( Fast-Reroute, LFA, IGP convergence, etc…)
Figure 5 shows a pair of nodes interconnected through grey-optics Ethernet interfaces using a pair of optical fiber
cables. The following is an explanation of the LFS working logic:
 Fiber from Router A to Router B breaks yet the fiber from Router B to Router A is still functional.
 Router B detect this directly. Router B internally declares its interface as down, hence it stops sending traffic
on that link
 With LFS scheme, Router B sends Remote Faults statur to Router A. it is sent continuously while the fault is
present.
 As a result, Router A is aware of the break. Router A internally declares its interface as down and stops
sending traffic on the link.
Figure 8: LFS Illustration Diagram

LFS uses reserved code words (instruction-sets) from the 64B/66B line coding of Ethernet to convey the status
messages.

3.6.2 LFS Through the DWDM Transponders

Some of the Ethernet connections in the new METRO network may use DWDM systems to bridge the long distance
between the nodes. The LFS mechanism generally works through DWDM transponders but there are different
behaviors according to the Ethernet signal type.
The behavior depends on the way the transponder maps the 10GE signal into the OTN container. A summary follows:
 10GBASE-W (WAN PHY) via STM-64, so order-sets are transparently transported
 GFP-F mapping of 10GBASE-R (LAN-PHY) payload only into OPU2. So LFS order-sets are not supported.
This is because GFP (Generic Framing Procedure) mapping has no sufficient bandwidth.
 Bit transparent maping of 10GBASE-R signal into OPU2e;so LFS signaling is preserved.
 Bit transparent mapping of 10GBASE-R signal into OPU1e;so LFS signaling is preserved.
3.6.3 Distributed BFD Sessions
All BFD sessions are distributed to the PFEs to provide independency from the Routing Engines. The distributed
mode is the default for all single-hop BFD sessions. This allows for sessions of shorter BFD transmit and receive
intervals to survive even under NSR and Routing Engines graceful switchover, which contributes to faster detection.
Distributed BFD sessions also contribute to improved session scale and offloads the processing from the Routing
Engine CPU to the FPC CPU, resulting in improved scaling and performance for Routing Engine-based applications
3.6.4 OAM LFM (Link Fault Management)

To provide fast link failure detection for individual member links of aggregated Ethernet interface between NPE
distribution switch, distribution switch and PoP switch and also between UPE and Pop switch as well as between UPE
and UPE platforms, Ethernet OAM Link Fault Management (LFM) is recommended. The IEEE 802.3ah LFM works
across point-to-point Ethernet link either directly or through repeaters.
In case of aggregated Ethernet, LFM runs on each of the individual member link interface. LFM is link-layer protocol
hence does not need Layer 3 address to operate.
LFM provides the following functions:
 Failure detection on physical links in both directions, as well as unidirectional failures.
 Ability to put a port in link-loopback mode remotely for diagnostics.
 Report and receive link error events such as framing or symbol errors.
The following diagram shows LFM between network platforms;
Figure 9: LFM Illustration Diagram

As it is explained in previous section, BFD will be used to provide fast failure detection mechanism on IP/protocol
layer. The following diagram shows both LFM and BFD together.

Figure 10: BFD/LFM Illustration Diagram

3.7 MPLS
FTEL new METRO network is a single IGP and MPLS domain. All core interfaces of the new infrastructure including
uPE, NPE, MC, MP, and PE routers run MPLS to join this domain.
MPLS LDP tunneling allows the targeted LDP sessions to be established over a one-hop RVSP LSP. These sessions
are set up over each core link to provide RVSP FRR services.
3.7.1 MPLS Network Design Options

In IP/MPLS design, there are different options to design traffic engineering and each option has its advantages and
disadvantages. The following table shows them in details:
Figure 11: IP/MPLS Network Design Options
3.7.2 LDP (Label Distribution Protocol)

The Label Distribution Protocol (LDP) is a protocol that responsible for generating and distributing labels in a MPLS
network. LDP allows routers to establish Label-Switched-Path (LSP) through the network by mapping network-layer
routing information directly to data link layer-switched paths. These LSPs might have and endpoint at directly attached
neighbor (comparable to IP hob by hop forwarding), or at a network egress node, enabling switching through all
intermediary nodes. LSPs established by LDP can also traverse traffic-engineered LSPs created by Resource
Reservation Protocol (RSVP).
In METRO network, LDP protocol will be enabled on the loopback interface of NPE, uPE platforms to provide target
LDP as it is required for link protection features described in section 2.6.3.2..
3.7.2.1 FEC De-aggregation

When an LDP egress router advertises multiple prefixes, the prefixes are bound to a single label and aggregated into
a single forwarding equivalence class (FEC). By default, LDP maintains this aggregation as the advertisement
traverses the network.
Since an LSP is not split across multiple next hops and the prefixes are bound into a single LSP, load balancing
across equal-cost paths does not occur.
De-aggregating the FECs causes each prefix to be bound to a separate label and become a separate LSP. De-
aggregating a FEC allows the resulting multiple LSPs to be distributed across multiple equal-cost paths. LSPs are
distributed across the multiple next hops on the egress segments but the router installs only one next hop per LSP.
For all LDP sessions, FEC de-aggregation should be configured globally.
3.7.2.2 LDP-IGP Synchronization

LDP-IGP synchronization is useful only if the links are configured to run LDP as opposed to only enabling LDP on the
loopback interface. The loopback interface is a passive interface that does not support raising the IGP cost upon LDP

session failure. In FTEL Metro Network design, LDP is enabled only on loopback interface therefore this feature, ldp-
igp synchronization, is not needed.
3.7.2.3 Timers
The default values for the following timers are used:
 LDP Hello-interval: 5 seconds for link hello messages and 15 seconds for targeted hello messages
 LDP Hold-time: 3 times the Hello-interval
 LDP session keepalive-interval: 10 seconds
 LDP session keepalive-timeout: 3 times the keepalive-interval
3.7.3 RSVP (Resource Reservation Protocol

RSVP was designed to signal bandwidth requirements on a per-flow basis. RSVP later has been implemented with TE
(Traffic Engineering) extensions.
RSVP uses a messaging system to create LSPs. The message types are Path, Resv, PathTear, ResvTear, PathErr,
ResvErr.
3.7.3.1 RSVP Message Aggregation

Aggregate messages allow for packing multiple RSVP messages into a single transmission, thereby reducing network
overhead and enhancing efficiency. The number of supportable sessions and processing overhead are significantly
improved when aggregation is enabled.
Not all routers connected to a subnet need to support aggregation simultaneously. Each RSVP router negotiates its
intention to use aggregate messages on a per-neighbor basis. Only when both routers agree are aggregate messages
sent.
3.7.3.2 RSVP Authentication

Neighboring routers use an authentication key (password) to verify the authenticity of packets received from an
interface. RSVP uses HMAC-MD5 authentication, which is defined in RFC 2104, HMAC: Keyed-Hashing for Message
Authentication. All routers that are connected to the same IP subnet must use the same authentication scheme and
password.
3.7.3.3 RSVP timers and Multipliers

The following table lists RSVP timers and settings used in METRO network design
Feature Description Setting

RSVP PATH/RESV Refresh messages are sent periodically so that
messages refresh reservation states in neighbouring nodes do not 30 seconds
timeout time out.
Keep multiplier The keep multiplier determines the lifetime of a 3 times
reservation state of an LSP if no refresh
messages are returned
RSVP hello interval RSVP monitors the status of the interior gateway 9 seconds
protocol (IGP) neighbors and relies on the IGP
protocols to detect when a node fails. If an IGP
protocol declares a neighbor down (because
IGP hello packets are no longer being received),
RSVP also brings down that neighbor. However,
the IGP protocols and RSVP still act
independently when bringing a neighbor up.
Table 6: RSVP timers and multipliers
In Junos OS, RSVP typically relies on IGP hello packet detection to check for node failures. RSVP sessions are kept
up even if RSVP hello packets are no longer being received, so long as the router continues to receive IGP hello
packets. RSVP sessions are maintained until either the router stops receiving IGP hello packets or the RSVP PATH
and RESV messages time out. Configuring a short time for the IGP hello timers allows these protocols to detect node
failures quickly.
So to conserve resources, the RSVP hello interval could be set to a larger value between 1 to 60 seconds. The value
of 0 disables hello messages.

3.7.4 End-to-End RSVP Signaled LSP

End-to-End LSP is to be signaled from each NPE to each NPE router with explicit route, ERO. Node-link protection is
provided for alternative paths in case end-to-end LSP fails due to link or node failure in the path.
RSVP uses Constrained Shortest Path First (CSPF) to determine new LSP paths. CSPF algorithm is a shortest-path-
first algorithm that has been modified to take into account specific restrictions when calculating the shortest path
across the network. The CSPF algorithm takes into account the IGP metric, bandwidth and coloring. The CSPF
algorithm is implemented to follow the sequential selection criteria below:
 Pruning of links with insufficient bandwidth.
 Pruning of links that do not contain an included color.
 Pruning of links that contain an excluded color.
 Increase of the cost of the links that contain prohibited SRLG value.
 Calculation of shortest path from ingress to egress.
 If equal cost paths exist, selection of the one whose last hop address equals the LSP's destination.
 Selection of the best path among equal-cost paths (least hop, then least fill).
 Passing explicit route (ERO) to RSVP
In FTEL METRO network, there may be cases where equal cost multipath (ECMP) exist. With full-mesh RSVP
between NPE platforms, traffic balance is provided during link repair when there are failures in the network.
3.7.4.1 Link Protection

Link protection is introduced in access network and Core network ( uPE-uPE, uPE-NPE and NPE-MC) to provide link
and node resiliency. Link protection features provide fast traffic recovery with alternative path/LSP. In the case of
multipoint LSPs, when one of the links of the point-to-multipoint tree fails, the sub trees might get detached until the
IGP re-converges and the multipoint LSP is established using the best path from the downstream router to the new
upstream router.
In fast reroute using local repair for LDP traffic; a backup path (repair path) is pre-installed in the Packet Forwarding
Engine. When the primary path fails, traffic is rapidly moved to the backup path without having to wait for the routing
protocols to converge. Loop-free alternate (LFA) is one of the methods used to provide IP fast reroute capability in the
core and service provider networks.
However, LFA has some limitations to provide full coverage for some cases. To overcome LFA limitations, there are 2
options provided.
 Manual bypass LSP, it requires simulation of network and identify limitation of LFA coverage. Manual
bypass RSVP LSP is introduced to the network for links not covered by LFA .
 Dynamic bypass LSP, Dynamic RSVP LSPs are created automatically and pre-installed in the
system so that they can be used immediately when there is a link failure.
Without LFA, when a link or a router fails or is returned to service, the distributed routing algorithms compute the new
routes based on the changes in the network. The time during which the new routes are computed is referred to as
routing transition. Until the routing transition is completed, the network connectivity is interrupted because the routers
adjacent to a failure continue to forward the data packets through the failed component until an alternative path is
identified. It is recommended to enable automatically established backup RSVP tunnels feature on all uPE to uPE
amd uPE to NPE interfaces. The system establishes bypass RSVP tunnels only when required (no LFA backup
coverage exists), thus offloading the operations team from maintaining the database describing which links require,
and which links doesn’t require backup RSVP tunnels.

The following diagram illustrate dynamic bypass LSP in METRO network.
Figure 12: Dynamic bypass LSP Illustration Diagram
Note: LDP based LSP with LFA is used within rings in FTEL design. Dynamic RSVP is chosen to extend LFA
capabilities. Remote LFA will be considered in the future once the network is successfully deployed.
3.7.4.2 EGRESS PROTECTION

When uPE and/or NPE platform or link failures take place, it takes some time to restore service using traditional
routing table convergence. Local repair procedures can provide much faster restoration by establishing local
protection as close to a failure as possible. Node-link protection between NPEs and LFA with dynamic bypass LSP
between uPEs is provided for local repair. Fast protection for egress NPE is available to services in seamless MPLS
design where BGP labeled unicast interconnects IGP levels. If NPE and/or MC router detects that an egress router
(respective NPE) is down, it immediately forwards the traffic destined to that router to a protector router (in FTE case,
it is the second NPE platform) that forwards the traffic downstream to uPE then to the customer router/switch.
To provide egress protection for BGP labeled unicast, the protector node (The second NPE) must create a backup
state for downstream destinations before the failure happens. The basic idea of the solution is that the protector node
constructs a forwarding state associated with the protected node and relays the MPLS labels assigned by the
protected node further downstream to the final destination.
In Seamless MPLS Network, BGP labeled unicast provides end-to-end transport label-switched paths (LSPs) by
stitching the intra-level LSPs together. Area Border routers (NPE) run BGP labeled unicast to VPN RR in order to
exchange labels for /32 NPE/UPE loopback routes. BGP labeled unicast also runs between UPE routers and NPE
routers and NPE routers acts as RR for the UPE routers connected to.
In Figure 12, the traffic goes from CE1 to CE2. NPE3 is the protected router, NPE4 is the protector, and MC is the
point of local repair (PLR). The primary path is chosen from NPE1 to NPE3. When NPE3 fails, Router MC detects the
NPE3 failure and forwards the traffic to NPE4, which provides backup service and forwards the traffic downstream.

Figure 13: Egress Protection Diagram
3.7.4.3 LDP Tunneling

RSVP is used for traffic engineering in METRO network. Running LDP over RSVP eliminates the distribution of
external routes in Infrastructure network.
LSPs established by LDP are tunneled through RSVP signaled LSPs. LDPs treat traffic-engineered LSPs as single
hops. LDP tunneling is enabled on LSPs between NPE/uPE platforms.
It disables normal Time-To-Live (TTL) decrementing. The configuration on the MPLS hierarchy affects all RSVP-
signaled or LDP-signaled LSPs. When this router acts as an ingress router for an LSP, it pushes an MPLS header
with a TTL value of 255, regardless of the IP packet TTL. When the router acts as the penultimate router, it pops the
MPLS header without writing the MPLS TTL into the IP packet. The setting helps in hiding the core topology.
3.7.4.4 LSP Miscellaneous Features

LSP re-optimization is useful in the cases where the path is entirely determined by the CSPF algorithm. For the new
METRO, the paths are explicitly configured with strict hops for the one-hop LSPs and the associated secondary LSPs.
Therefore the “optimize-timer” is left at the default value of zero, meaning re-optimization is disabled. However, the
following knobs are globally set in all the new infrastructure nodes for any potential use by an LSP that has the re-
optimization enabled.
 optimize-switchover-delay:
Allows specifying the amount of time to delay the switching of the LSPs on the ingress nodes to the optimized paths.
The specified delay helps to ensure that the new optimized paths have been fully established before traffic is switched
over from the old paths.
 optimize-hold-dead-delay:
Allows specifying the amount of time to delay the tear down of old paths after the ingress router has switched traffic to
new optimized paths. The specified delay helps to ensure that old paths are not torn down before all routes have been
switched over to the new optimized paths. This delay timer starts when the timer specified by the “optimize-
switchover-delay” statement has elapsed.

 smart-optimize-timer:
When the smart optimization is enabled on a router, the Junos OS operates on the assumption that the original LSP
path is preferable to any alternate or secondary path. When you enable the smart optimization timer and an LSP fails
and its traffic is switched to an alternate path, the smart optimization timer starts and waits 3 minutes (this is the
default value). After 3 minutes have passed, the LSP is switched back to the original path. If the original path fails
again and the LSP is switched to an alternate path again, the router waits 1 hour before attempting to switch the LSP
back to its original path. Smart optimization is enabled by default. For TM’s deployment, since the one-hop RSVP
LSPs are strictly defined, the smart-optimize-timer is to be set to zero to disable the smart optimization.
3.7.4.5 Revert Timer

For all LSPs configured with secondary paths, a revert-timer applies to specify the amount of time (in seconds) that an
LSP must wait before traffic reverts back to the primary path. If during this time the primary path experiences any
connectivity problem or stability problem, the timer is restarted.
3.7.4.6 LSP Hop-limit

The setting specifies the maximum number of routers an LSP can traverse including the ingress and egress routers.
The limit could be applied to the primary LSP, secondary LSP or the fast-reroute detours.
3.7.4.7 LSP Least-fill

The least-fill option causes the CSPF algorithm to choose the least utilized path to establish the LSP on. The knob is
no relevant to the intended deployment since the EROs are strictly configured for all LSPs including the secondary
ones and no bandwidth reservation is used. However, it impacts the switchover decision when re-optimization is
configured for the LSP.
3.7.4.8 Adaptive LSP

All LSPs are set to be adaptive when attempting to reroute themselves. When it is adaptive, the LSP holds onto
existing resources until the new path is successfully established and traffic has been cut over to the new LSP. The
setting is particularly useful when bandwidth reservation is in use, because it avoids double counting of resources for
links that share the old and new paths.
3.7.4.9 IPv6 Tunneling

It Allows IPv6 routes to be resolved by converting LDP and RSVP routes stored in the inet.3 routing table to IPv4-
mapped IPv6 addresses and then copying them into the inet6.3 routing table. This routing table is then used to resolve
next hops for both 6PE and 6VPE routes.
3.7.4.10 ICMP Tunneling

An intermediate LSR (Label Switched Router) could generate an ICMP (Internet Control Message Protocol) message
in response to a received packet from a source router. The generated message is addressed to the source router but
the LSR copies the MPLS stack of the original received packet and pushes it to the generated ICMP packet. The
ICMP packet is then labelled switched until the egress router of which the original packet was to be delivered. The
egress router then route the ICMP packet either through an alternative layer 3 path or through a reverse LSP to the
source.
ICMP message tunnelling can be useful for debugging and tracing purposes if the message is an ICMP time
exceeded message. It also handles path MTU discovery.
3.8 LLDP (Link Layer Discovery Protocol)

The Link Layer Discovery Protocol (LLDP) is an industry-standard protocol to advertise capabilities, identity, and other
information onto a LAN. The Layer 2 protocol, detailed in IEEE 802.1AB-2005, replaces several proprietary protocols
implemented by individual vendors for their equipment.
LLDP allows network devices that operate at the lower layers of a protocol stack (such as Layer 2 bridges and
switches) to learn some of the capabilities and characteristics of LAN devices available to higher layer protocols, such
as IP addresses. The information gathered through LLDP operation is stored in a network device and is queried with
SNMP. Topology information can also be gathered from this database.
Some of the information that can be gathered by LLDP (only minimal information is mandatory) is:
 System name and description
 Port name and description

 VLAN name and identifier

 IP network management address
 Capabilities of the device (for example, switch, router, or server)
 MAC address and physical layer information
 Power information
 Link aggregation information

4. TRAFFIC LOAD BALANCING

Load balancing is used to evenly distribute traffic when the following conditions apply:
 There are multiple equal-cost next hops over different interfaces to the same destination.
 There is a single next hop over an aggregated interface.
By default, when load balancing is used to help distribute traffic, Junos OS employs a hash algorithm to select a next-
hop address to install into the forwarding table. Whenever the set of next hops for a destination changes in any way,
the next-hop address is reselected by means of the hash algorithm.
Current Juniper Networks routers support per-prefix (default) and per-flow load balancing. The latter involves hashing
against various Layer 3 and Layer 4 headers, including portions of the source address, destination address, transport
protocol, incoming interface, and application ports. The effect is that now individual flows are hashed to a specific next
hop, resulting in a more even distribution across available next hops, especially when routing between fewer source
and destination pairs. If per-flow is not configured, the default per-prefix mode is in effect. Per-prefix mode only
considers the destination prefixes in the hashing algorithm to deterministically map the prefix onto one of the available
next hops.
Juniper Networks routers can load-balance on a per-packet basis in MPLS. Load balancing can be performed on
information in both the IP header and the MPLS label stack, providing a more uniform distribution of MPLS traffic to
next hops. This feature is enabled on supported platforms by default and requires no configuration.
4.1 Load Balancing of the MPLS Packets

In FTELs MPLS core, a maximum of five labels is expected. MX routers that are Trio-based support a maximum of
five (5) labels by default in the hashing algorithm in Junos releases earlier than 14.1. Up to eight (8) labels are
supported starting from Junos release 14.1.
The MX router should also be configured to load-balance IPv4 traffic over Layer 2 Ethernet pseudowire whenever a
Layer 2 virtual circuit is deployed across the core network.
For Layer 2 VPNs, the router could encounter a packet-reordering problem. When a burst of traffic pushes the
customer traffic bandwidth to exceed its limits, the traffic might be affected in mid flow. Packets might be reordered as
a result. By excluding the EXP bit of the top label from the hash key, this reordering problem could be avoided. Thus
the EXP bit of the top label is excluded from the hash key.

5. CLASS OF SERVICE
The new METRO network is configured to support BA based Class of Service (CoS) model to provide the end-to-end
quality of service differentiation for all services offered over the METRO network..
The class of service model to be implemented in the core network is summarized in the following table:
Packet Loss
Delay Buffer
Drop Profile
Forwarding
MPLS EXP
Transmit
Priority
Priority
Queue
Class
Rate
Control 3 5% 5% Strict-high Default RED profile 111 Any

110
Diamond 6 5% 5% Medium-low Default RED profile 101 Any
Platinum 5 20% 10% High Default RED profile 100 Any
Gold 4 15% 15% High Default RED profile 011 Any
Silver 2 10% 10% Medium-high Default RED profile 010 Any
Bronze 1 30% 30% Medium-low Fill level 90-100 Drop 001 Any
probability:0-100
Best-Effort 0 15% 15% Medium-low Fill level 80-100 Drop 000 Any
probability:0-100
Table 7: Queue design and METRO network links schedulers’ parameters

All the new METRO network routers are Trio-based MX routers, MX480 and MX104. All the interfaces from uPE to
NPE/uPE and from NPE to Core routers are MPLS enabled and are egress port scheduled according to the seven
aggregate queues shown in Table 7.
5.1 Forwarding classes

Trio interfaces support 8 hardware queues per port. Each forwarding class (FC) is explicitly mapped to one of the
eight (8) queues. FTEL has seven (7) forwarding classes, which correspond to the seven aggregate bundles of
services transported over the core network. Queue 3 is the network-control class in default Junos implementation.
Control class carries both network-control and other signaling and voice traffic.
The default setting of switch fabric priority varies by interface mode. In port mode, any schedulers that use high priority
automatically map that FC to a high fabric priority. In H-CoS (Hierarchical COS) mode, all FCs default to normal fabric
priority, regardless of the associated scheduling priority. Therefore it is recommended to explicitly set the fabric priority
for each forwarding class.
5.2 Packet classification and pre-classification

In addition to the normal BA classification at the ingress interface, Trio interface also supports a pre-classification of
traffic received at the ingress. The goal of pre-classification is to prioritize network control traffic that is received over
WAN ports (i.e., network ports, as opposed to switch fabric ports) into one of two traffic classes: network control and
best effort. This classification is independent of any additional CoS processing that may be configured; here, the
classification is based on a combination of the traffic’s MAC address and deep packet inspection, which on Trio can
go more than 256 bytes deep into the packet’s payload.
Pre-classification is performed on all recognized control protocols, whether the packet is destined for the local RE or
for a remote host. The result is that control protocols are marked as high priority while non-control ones get best effort
or low-priority. Trio’s fabric CoS then kicks in to ensure vital control traffic is delivered to the egress port, even during
times of PFE oversubscription, hence the term intelligent oversubscription.
All host-generated traffic (traffic generated from the RE), is by default classified to either of network-control (queue 3)
or best effort (queue 0). TCP-related packets, such as BGP or LDP sessions, first use queue 0 (BE) and then fall back
to use queue 3 (network control) only when performing a retransmission. In general, control protocols packets are sent
over queue 3 and management protocols packets are sent over queue 0.
On all new METRO network routers, the defined custom BA classifiers of types inet-precedence and EXP are to be
applied on all the core logical interfaces. The EXP classifier is to be applied on all routing-instances to correctly
classify incoming VPN traffic. This is because all routing-instances are configured with the knob “vrf-table-label”.
5.3 Egress Scheduling

For all METRO network IP/MPLS interfaces, egress scheduling is implemented on the port level where the queues are
attached. No hierarchical or per-unit scheduling is used on IP/MPLS interfaces.
MX routers with Trio-based MPCs use a form of Priority Queue Deficit Weighted Round Robin (PQ-DWRR)
scheduling with five levels of strict priority. PQ-DWRR extends the basic deficit round robin (DWRR) mechanism by
adding support for one or more priority queues that exhibit minimal delay. The deficit part of the algorithm’s name
stems from the allowance of a small amount of negative credit in an attempt to keep queues empty. The resultant
negative balance from one servicing interval is carried over to the next quantum’s credit allocation, keeping the
average de-queuing rate near the configured transmit value.
A Scheduler is created and mapped to each queue. The scheduler is configured to set the following queue
parameters:
 Priority:
The priority determines the order in which queues are serviced. A strict priority scheduler services high-priority queues
in positive credit before moving to the next level of priority. In Trio, in-profile queues of the same priority are serviced
in a simple round-robin manner, where one packet is removed from each queue while they remain positive.
Trio supports five (5) hardware priority levels, mapped to the scheduler priority (software priority) as shown in the table
below.
Scheduler Priority Hardware Priority Comment

Strict-high Guaranteed High (GH=0) Can not be demoted
High Guaranteed High (GH=0) Demoted to EH by default
Medium-high Guaranteed Medium (GM=1) Demoted to EL by default
Medium-low Guaranteed Medium (GM=1) Demoted to EL by default
Low Guaranteed Low (GL=2) Demoted to EL by default
Excess priority high Excess High (EH=3)
Excess priority low Excess Low (EL=4)
Excess priority none NA Disable demotion
Table 8: Mapping of Scheduler priorities to Trio Hardware priorities
 Transmit rate:
The queue’s transmit rate specifies the amount of bandwidth allocated to the queue and can be set based on bits per
second or as a percentage of interface bandwidth. By default, a queue can be serviced when in negative credit, as
long as no other queues have traffic pending and it’s not blocked from excess usage. Transmit rate acts as the
guaranteed rate for the queue and hence the queue’s priority is demoted to the excess region if traffic exceeds the
specified queue transmit rate.
 Buffer size:
This is the delay buffer for the queue that allows it to accommodate traffic bursts. Buffer size could be configured as a
percentage of the output interface’s total buffer capacity, or as a temporal value from 1 to 200,000 microseconds,
which simply represents buffer size as a function of delay, rather than bytes. The value configured is mapped into the
closest matching hardware capability. Most Trio PFEs offer 500 milliseconds of delay bandwidth buffer, based on a 4
× 10GE MIC; each port is pre-assigned 100 milliseconds of that buffer with the balance left-for assignment via the CLI.
 WRED drop profile:
WRED (Weighted Random Early Detection) drop profile is applied only for Control class. WRED is effective in
combating congestion for TCP-based application. It proactively drops packets at certain fill level of the average

queue length. Trio WRED is implemented as a queue-tail process, meaning the packet marked for WRED
dropping is never en-queued in the first place.
5.4 Egress Re-write rule

Rewrite-rule at the egress interface works as a mirror image to the BA classifier applied at the ingress. The rewrite-
rule matches on the forwarding-class and packet loss priority (PLP) of the packet to set the marking bits before packet
is sent out the egress interface. Custom IP-precedence and MPLS rewrite rules are defined to be applied on the core
logical interfaces of METRO network routers to ensure the proper classification on the upstream core routers.

6. NETWORK SERVICES
Network services are often implemented on customer facing (uPE) or internet facing (GW) routers to gauge network
quality and collect sampling for flow analysis and reporting. All uPE routers in METRO network are MX104 chassis
with Trio-based line cards. Trio-based line cards support inline services (license –based) which provides huge
advantages:
 Total Cost of Ownership

As network services become available using the Trio chipset, no additional Services Modules would need to be
purchased. However, the real cost savings manifests itself in the additional FPC slot that is now free since the
Services Module is no longer required. This additional FPC can be used to provide additional WAN ports to realize
previously locked revenue.
 Configuration
The configuration of inline Trio services is largely the same as services on the MS- DPC. The largest difference is the
requirement to set aside a specific amount of bandwidth on the Trio chipset to be reserved for inline services.
 Performance
Inline Trio services are processed directly on the Lookup Block, which enables near line-rate performance of network
services. The Trio chipset is responsible for forwarding traffic; by being able to apply network services as part of the
same workflow, it is possible to provide near line-rate performance as opposed to having to send the packet to a
separate Services Module.
6.1 Inline Jflow

Inline J-Flow service should be implemented on uPE, GW routers to sample the traffic at a rate of 1/1000 and send
the sampled flows to a collector for analysis and further subsequent (D)DOS mitigation actions.
Inline sampling supports version 9 and IPFIX flow collection templates. Support for version “9” template was
introduced in Junos OS Release 13.2, and is limited to IPv4 flows. Inline Jflow V10 is used in METRO network routers
because it supports both IPv4 and IPv6 families in the standard IPFIX format. IP Flow Export Information (IPFIX) is the
latest version of flow statistics, which is based on RFCs 5101, 5102, and 5103. IPFIX is the official IETF protocol that
was created based on the need for a common and universal standard of exporting flow information. There’s little
change between IPFIX and J-Flow v9 aside from some cosmetic message headers and the introduction of variable-
length fields.
Because inline IPFIX is implemented within the Trio chipset directly on the lookup block “LU”, the performance is near
line rate. As traffic moves through the Trio chipset, it is inspected and sampled locally without having to take a longer
path to a Service Module and back. Being able to keep the packet within the Trio chipset speeds up the operations
and lowers the latency. Therefore, the performance of the inline Jflow is directly related to the number of LU chipsets
available within the line card. Table 9 summarizes the scaling of the inline Jflow performance on different Trio line
cards. Each trio PFE constitutes one LU chipset and hence the per-PFE performance is the same across all the
different MPC cards.
KPI Trio PFE MPC1 MPC2 MPC4E
Max flows 4 millions 4 millions 8 millions 8 millions
Flow setup rate (flow/sec) 150K 150K 300K 300K
Max flow export rate 100Kpps 100Kpps 200Kpps 200Kpps
Throughput at max flows 20Gbps 20Gbps 40Gbps -
Table 9: Inline Jflow Performance Scaling on Different MPC Cards
Table 9 shows MPC2 and MPC4E cards with the same inline Jflow performance as each has two PFEs. However, the
internal architecture of the MS-MPC4E is different. MPC4E has two LU blocks per PFE. The PFE main processor
sprays packets of the same flows across the two LU chips. Hence, each LU exports records for the same flows but
with different observation domain ID to the external flow collector. The collector is expected to aggregate the flow
records exported by the two LUs for the same flows.
As of Junos OS 13.1 release, a maximum of six records for IPv4 can be exported in one packet, and a maximum of 3
records for IPv6 can be exported in one packet. The flow records packets are transported to the collector over UDP

transport protocol. The collector should be reachable through the global routing instance over revenue ports (non-
management). Starting from Junos OS 13.3, the collector could be reachable through non-default routing instances as
well. Nevertheless, inline Jflow is limited to only one collector as opposed to the service-PIC-based Jflow, which can
send the flow records to multiple collectors.
The MTU size across the network should be set according to the guidelines in section 3.1.4 to minimize the chances
of fragmentation, which would lead to duplicate creation of flow records for fragmented packets.
Before configuring inline sampling, the hash tables should be adequately sized for IPv4 and IPv6 flow sampling.
These tables can use one to fifteen 256k areas, and each table is assigned a default value of one such area. The
values of 15x256K for IPv4 table and 1K for IPv6 table are generally recommended on all FPCs (MPCs). Changing
these values at any time causes the FPC to automatically reboot. Therefore, it is very important to ensure the
adequate sizing is met in each of the nodes to avoid subsequent changes.
The following shows an example of how to use flow information;
Figure 14: End-to-end Arbor DDOS design
The diagram above shows Arbor’s Distributed Denial of Service (DDoS) molution into IPCore.
The Arbor flagship product, Peakflow appliances is used and integrated into IPCore. The Peakflow appliances are
made of the following components, of which IP/MPLS Network has only subscribed to the first three:
 Peakflow SP Collector Platform (CP) appliances in the peering edge or backbone.
 Peakflow SP Flow Sensor (FS) appliances in the customer aggregation edge;
 Peakflow SP Threat Management System (TMS) appliances deployed in any part of the network to surgically
mitigate network threats.
 Peakflow SP Business Intelligence (BI) appliances to increase scalability and add redundancy for managing
critical business objects.
 Peakflow SP Portal Interface (PI) appliances to increase the scale, redundancy and profitability of Arbor-
based managed services.
Arbor Networks Peakflow SP product suite offers network-wide visibility and security to proactively fend off malicious
threats, thwart distributed denial of service (DDoS) attacks and strengthen the quality of service.

6.2 RPM (Real-Time Performance Monitoring)

Real-time performance monitoring enables customers to monitor network performance in real time and to assess and
analyse network efficiency. Typically, network performance is assessed in real time based on the jitter, delay, and
packet loss experienced on the network. RPM is a service available in Junos OS that enables a router to measure
metrics such as round-trip delays and unanswered echo requests. To achieve this, RPM exchanges a set of probes
with other IP hosts in the network for monitoring and network tracking purposes. These probes are sent from a source
node to other destination devices in the network that require tracking. Data such as transit delay and jitter can be
collected from these probes, and this data can be used to provide an approximation of the delay and jitter experienced
by live traffic in the network.
During a probe, the client device sends a packet across to a remote server, which in turn returns the packet with an
acknowledgement to the sender. Both the type and content of the queries sent from the client are user configurable.
Both the source and destination nodes running RPM services are aware that the packets in the probe are used to
compute information such as RTT and jitter delay. For each probe, RPM makes several measurements (for RTT as
well as for different positive and negative jitter). For each type of measurement, RPM calculates the minimum,
maximum, average, peak-to-peak, standard deviation, and overall sum over several different collections of
measurements.
The following diagram shows a probe packet sent from the client and a corresponding reply from the server. The
probe query and responses occur between specific user-defined source and destination addresses. The RPM service
is a JUNOS process (rmopd) that runs on the Routing Engine.
Figure 15: RPM Client-Server
The different packet types that can be used within the probe include:
 Internet Control Message Protocol (ICMP) echo
 ICMP timestamp
 HTTP GET
 UDP echo
 TCP connection
 UDP timestamp
ICMP ping is the default probe type in Junos devices.
6.2.1 Timestamping
The timestamping activity consists of the source (client) node applying a timestamp (T1) to the RPM packets with the
time at which they leave the node. The destination (server) node applies a timestamp (T2) when it receives the probe
and a timestamp (T3) when the probe leaves the destination back to the source. The source receives the response
and applies a timestamp (T4). Different metrics are calculated based on these timestamps collected from a series of
probes. For example, the RTT can be computed as the difference between the timestamps T1 and T4 less T3 minus
T2.
PFE-based timestamping is supported in Trio chipset. All edge routers in METRO network routers (uPE, NPEare
configured to do time stamping on the PFE before sending RPM probes to the local RE or to the remote probe

destination. This supports one-way hardware timestamp services to measure the one-way delay and jitter metrics.
Probes are still managed (request, response, and result analysis) by the RE. Figure 9 explains this PFE-based
timestamping.
Figure 16: PFE-based Time-Stamping on Trio Chipset
PFE-based timestamping eliminates the need for the service card, and the probes flow directly between the RE and
the PFEs. Timestamping is done on the distributed PFEs as opposed to sending all the probes through one service
card. Service-PIC based timestamping is explained in Figure 10 below.
Figure 17: Service-PIC based Time-Stamping

Figure 11 below shows the payload fields of a time-stamped probe packet. The “Version” and “Magic Number” fields
are used to verify the authenticity of the RPM probe packet at both the client and server ends.
Figure 18: Payload of a Time-stamped RPM Probe

RPM probes can be configured from the JUNOS CLI, or SNMP. The results can be reported via CLI or SNMP.
Alternatively, a third-party MIB browser or performance management application can be used to retrieve RPM
measurement data and plot the resulting graphs.
6.2.2 Reporting of Results

RPM probes can be configured from the JUNOS CLI, SNMP, and J-Web. The results can be reported via CLI or J-
Web or SNMP. Alternately, a third-party MIB browser or performance management application can be used to retrieve
RPM measurement data and plot the resulting graphs. The following summarizes management interfaces available for
RPM.
Management Information
Interface
CLI Command-line access to all configuration and monitoring of RPM
XML-JunosScript XML support for all CLI commands and capabilities
RFC 2925a Standards-based IETF DISMAN-PING-MIB
MIB-JNX-RPM • Supported as of JUNOS 9.6
• Juniper RPM MIB
•Leverages RFC 2925a
• Provides objects with the detailed RPMtest result
MIB-JNX-PING Juniper Ping MIB
• Leverages RFC 2925a
• Adds objects Relating to the different types of RPM probes
System Log messages Generated when configured thresholds are exceeded
Snmp Traps Generated when configured thresholds are exceeded
J-web GUI for configuration device to allow configuration and monitoring on individual device
Table 10: RPM Management Interface
For SNMP, MIB definitions allow SNMP operations on RPM variables. Two MIBs can be used for this purpose—the
Ping MIB that is based on the RFC2925 (DISMAN-PING-MIB) Definitions of Managed Objects for Remote Ping,
Traceroute, and Lookup Operations, and the RPM MIB or jnx-rpm.mib.
These two MIBs can be used either together or separately to collect RPM data (although the jnx-rpm.mib provides
more information because it is RPM-specific). The RPM MIB has been developed from the flat Ping MIB to provide a
hierarchical collection of data.
The jnx-ping.mib allows creation of hardware timestamp-based probes, and it provides thresholds for raising alarms
via SNMP to configure the number of samples in the moving average collection. The jnx-rpm.mib tables can be
classified into a Results and History group respectively. However, in case of the jnx-rpm.mib, the data is classified into
separate collection types and measurement sets.
Figure 18 shows the MIB object identifiers (OIDs) for each of the probes that are configured for different traffic
forwarding classes. The different metrics such as RTT, jitter, and so on are tracked for each of the probes belonging to
a different traffic class.

Figure 19: MIB OIDS for each probes
Figure 19 shows the metrics obtained from a third-party MIB browser
Figure 20: Prober Results from third-party MIB browser

6.3 Ethernet OAM (Operations, Administration and Maintenance) For Services

Ethernet OAM provides tools that a service can use to determine how a network of Ethernet links functions. Ethernet
OAM have the following characteristics
 Rely on the media access control (MAC) address or virtual local area network (VLAN) identifier for
troubleshooting.
 Work independently of the actual Ethernet transport and function over physical Ethernet ports or a virtual
service such a pseudowire and so on.
 Isolate faults over a network architecture.
OAM can provide link-level information, provide performance statistics, or track end-to-end connectivity across the
network.
Connectivity Fault Management ( CFM) is define in IEEE 802.1ag and it is used in a Metro Ethernet network.
The main CFM features are:
 Fault monitoring using the continuity check protocol. This is a neighbor discovery and health check protocol
that discovers and maintains adjacencies at the Vlan or link level.
 Path discovery and fault verification using the linktrace protocol. It maps the path to a destination MAC
address though one or more bridged networks between the source and destination like IP traceroute.
 Fault isolation using loopback protocol. It provides troubleshooting capabilities with the continuity check
protocol similar to Ip ping.
CFM partitions the service network into various administrative domains. Each administrative domain is mapped into
one maintenance domain providing enough information to perform its own management and making end-to-end
monitoring possible. Each maintenance domain is associated with a maintenance domain level from 0 through 7.
Level allocation is based on the network hierarchy. The highest domains are assigned a higher level than the lower
domains. Customer end points have the highest maintenance domain level. In a CFM maintenance domain, each
service instance is called a maintenance association. A maintenance association can be as a full mesh of
maintenance endpoints (MEPs) having similar characteristics. MEPs are active CFM entities generating and
responding to CFM protocol messages. There is also a maintenance intermediate point (MIP), which is a CFM entity
similar to the MEP, but more passive. MIPs only respond to CFM messages.
Figure 21: CFM diagram with multiple domains

To understand the figure 12, the followings are provided for manage objects;
 Maintenance Domain (MD) is part of a network that is controlled by an operator.

 Maintenance Association (MA) is used to monitor connectivity for a single service instance within
the Maintenance Domain.
 Maintenance End Point (MEP) resides at the edge of a maintenance domain.
o DOWN MEP: The CFM monitoring on physical interfaces and logical routed Ethernet interfaces is
done using DOWN MEPs.
o UP MEP: For bridge, VPLS and L2VPN services, the downstream service monitoring (AC side) is
done with DOWN MEPs and upstream service monitoring (core side) is done with UP MEPs.
 Maintenance Intermediate Point (MIP) is a passive functional entity located at intermediate points in a
Maintenance Domain
OAM frames from higher domain levels go transparent over the MEPs of lower level domains. With this approach, a
customer line monitored with Domain of level 7 will have its CFM packets all the way through the Service provider
domains untouched. If the Customer support wants to monitor the line between the border switches in this network,
the service provider domain will pass the OAM packets to their destination untouched. It is valid with the Network
operator Rings in the diagram. They will allow the level 5 domain CFM packets to go to their destination over their
rings. The End points (MEPs) are responsible to filter or process the CFM packets on the ports they are set. If an
equal to their Domain level CFM packets are received, the MEPs need to process them. If higher than their level CFM
frames are received, the MEPs need to pass them transparently. And if they receive lower level CFM packets –the
MEPs need to drop them so they don’t go to supposedly higher-level domains behind the Maintenance End point.
With this logic –customer equipment is not supposed to receive CFM packets from Provider or Operator equipment
and the provider equipment will not receive CFM packets from the Operators. Thus, each different organization
(customer, service provider) has the ability to isolate the fault within the organization's maintenance level, without the
service provider having to share its network information to the customer, or the operator having to share its network
information to the service provider.
6.3.1 Fault Detection (CCM)

Continuity Check protocol is used for fault detection by a MEP. MEP periodically sends Continuity Check (CC)
multicast messages at different ccm intervals (10ms, 100ms, 1s, 10s, 1minute, 10 minutes). CCMs are the heartbeat
of the OAM monitored network. If the Heartbeat stops –then there is a failure. This failure will generate an event or an
SNMP trap. Those events and traps can trigger actions or be logged for later analysis. Loss of connectivity is detected
if 3 consecutive CCM messages are not received from a remote MEP. The MEPs also distinguish the different
Maintenance associations (MAs) and do not process packets from other associations different from their own. So far,
any MA can have a set of no more than 8192 MEPs identified by their ID number (1...8192) and different MAs can
have MEPs with same ID numbers. No matter if the local MEP expects to receive a CCM message from MEP with ID
100, it will discard it if this CCM is sent from another MA. MIPs passively receive CFM frames and respond back to the
originating MEPs.MIP functionality:
 MIP is configurable at the service.
 MIP can be configured without configuring MEPs.
 When MIP is configured, the MIP levels are computed automatically for each interface in the bridge/vpls
instance based on its CFM configuration.
 MIP supports Linktrace and Loopback protocols.
 When a MIP is configured, the Linktrace/Loopback packets matching the MIP level are sent to CPU
for further processing.
 The packets with higher md-levels are transparently forwarded in the hardware while the packets with lower
md-levels are dropped in the hardware.
6.3.2 Fault Verification

Equivalent to IP “Ping” command loopback protocol is used for Fault Verification. MEP can transmit a unicast LBM to
a MEP or MIP in the same MA. Receiving MEP responds by transforming the LBM into a unicast LBR sent back to
the originating MEP.
6.3.3 Fault Isolation

A Linktrace Message (LTM), similar to the IP trace route function, can quickly determine the location of a fault. When
an LTM is sent to a service end (MEP), all intermediate nodes (MIPs) respond with an LTR. The returned LTRs
uniquely identify the segment or node where the fault originates. Under normal operating conditions, Linktrace is also
used by network elements to determine the path a service takes through the network –this route awareness is stored

in a local database to expedite fault isolation, and for link protection purposes. LTMs can also be used to identify
adjacency relationships with remote MEPs and MIPs at the same administrative level.
6.3.4 OAM on JUNOS

JUNOS supports the below
802.1ag & Y.1731 protocol features.
 Support all MD Levels (0-7).
 The CFM monitoring on physical interfaces and logical routed Ethernet interfaces is done using DOWN MEPs
 The bridge, vpls and l2vpn services support:
o Downstream service monitoring (AC side) with DOWN MEPs
o Upstream service monitoring (core side) with UP MEPs
 CFM has now support for filter based IPv6 L2TPv3 tunnel monitoring.
 10ms and higher hello intervals are supported.
 Linktrace & Loopback protocols are supported.
 Support for Y.1731 ETH-DM and ETH-LM/SLM( proactive/on-demand),
 The MIP functionality is supported on bridge, vpls and l2vpn services.
 Rate limiting of CFM packets is supported using policers.
The MX-series supports the 802.1ag & Y.1731 protocol suite for monitoring the following Ethernet services:
 Routed Services
 Physical Ethernet Interfaces (Port based CFM)
 Layer-2 Point-to-Point Ethernet Services
o Circuit-Cross-Connect / l2circuit local-switching/Interface-Switch
o BGP based L2VPN
o LDP based L2CIRCUIT
 Layer 2 Point-to-Multipoint bridged Services
o Bridge-Domains
o BGP/LDP based VPLS
The CFM service monitoring is also supported on psedowires/mpls-lsps for following multihoming applications:
o Point-to-Point Ethernet Service with PW-Redundancy
o In this case, the CFM monitoring is supported on Active and Backup PWs
o Point-to-Point Ethernet Service with LSP Redundancy
o In this case, the CFM monitoring is supported on Active and Backup LSPs
The transmission and reception of CCM packets are handled at the PFE level using distributed PPMD. Only exception
packets are handled in the RE. This allows load distribution across different PFEs and provides better scaling.
Multicast CFM packets are identified using DA-MAC and CFM-Ethertype (0x8902). Unicast CFM packets destined to
CPU (for MEP or MIP) are identified using CFM-Ethertype (0x8902) and destination-mac. CFM implementation uses
Implicit Firewall Filters to identify CFM packets.The Firewall Filters match on the above fields in the packet to match
the packet. Based on the service-type, the Firewall Filters can be installed at input or output of an interface.
For bridged/vpls services, the firewall filters are installed at ingress of the route-table.Linktrace Request (LTM) PDUs
use the multicast mac-address. The Linktrace Reply (LTR) PDUs, Loopback (LBM and LBR) PDUs and the
Y.1731 ETH-DM and ETH-LM PDUs use the unicast mac-address.
6.3.5 Junos Space OAM Insight Overview

The Junos Space OAM Insight application provides capabilities for automated end-to-end network management,
monitoring, and troubleshooting. OAM Insight provides a set of functions designed to monitor network operation to
detect network faults and measure network performance.
The OAM Insight application allows network operators to configure OAM (Operations, Administration and
Maintenance) functionality on all devices and to monitor, detect, isolate and troubleshoot networking faults in a quick
and efficient manner. Operators can also measure quality-of-service (QoS) attributes, such as availability, frame
delay, frame delay variation, and frame loss. Monitoring and fault isolation can be performed at the Link Layer,
Transport Layer, and Session Layer, thus providing a clear demarcation view to the operator, service provider, and
customer.
The OAM Insight application is compliant with ITU-T and IEEE standards that support cross-platform link fault
management, connectivity fault management (CFM), and performance management on all Juniper Networks devices
using the OAM Insight application in Junos Space.
The following standards are supported:
 802.3ah, Link Fault Management which defines OAM link-fault management mechanisms
 802.1ag, Connectivity Fault Management
 Y1731, Draft, Performance Management
The OAM Insight application supports service-based CFM provision, LFM provision, and performance measurements
of Y.1731 standard frames on the network services. The OAM Insight application also supports the following functions:
CFM configuration of the following features:
 CFM Action Profile—You can configure a CFM action profile and specify the action to be taken when any of
the configured events occur. Alternately, you can configure an action profile and specify default actions when
connectivity to a remote maintenance end point (MEP) fails.
 Service Level Agreement (SLA)-Iterator Profile—The OAM application uses the SLA-iterator profile to collect
Y.1731 statistics. These iterator profiles have to be configured before starting performance monitoring.
 Remote MEP—You can configure a remote MEP to wait for continuity check messages (CCMs). If
autodiscovery is not enabled, the remote MEP must be configured under a MEP. If the remote MEP is not
configured under a MEP, the CCMs from the remote MEP are treated as errors.
 CFM Modify Service—You can add or delete new endpoints, as well as modify attributes for remote MEP
configuration for MEP.
LFM configuration
 LFM Profile—You can define an LFM profile and specify the action to be taken when any of the configured
events occur.
 LFM Action Profile—You can configure an LFM action profile and define event fault flags and thresholds and
the action to be taken when any of the configured events occur.
CFM Integration with Network Activate Services
The CFM is enabled for Point-to-Point (PPP) (E-Line) and virtual private LAN service (VPLS) (E-LAN) services
through the Network Activate application. The Network Activate application now detects the availability of CFM service
profiles automatically. On availability, the Network Activate application imports the CFM service profiles (CFM service
definitions) from the OAM Insight application and attaches them to a Network Activate service order.
OAM Insight Performance Management
In Performance Management, the OAM Insight application provides an option to measure the frame delay, frame loss,
frame delay variation, and service availability. These measurements are achieved in either of the following ways:
 Triggering a one-way delay
 Triggering a two-way delay
Users can also get this information through the Loss Measurement/Delay Measurement (LM/DM) iterator.
The performance measurement is useful for generating periodic service level agreement conformance reports from
the deployed network and for studying traffic patterns in the network over a period of time. The iterator profiles are
configured on remote MEP for measurement of frame delay (ETH-DM), frame loss (ETH-LM) and statistical frame loss
(SFL).

6.4 Inline Video Monitoring

MX inline video monitoring feature runs inline of the normal packet forwarding path on trio line cards ( MPC2E, MPC-
3D-16XGE-SFPP)
The solution supported the followings:
 CBR traffic based on RFC 4445
 Monitoring of ingress redundant IPTV/RTP stream feeds
 Monitoring of all egress IPTV/RTP streams feeds
 No dependency on third party solution
 Bulk stats can be configured for accounting solution
The following figure illustrate inline video monitoring solution
Figure 22: Inline Video Monitoring

With Juniper solution, inline video monitoring captures delay factor (DF), Media Loss Rate (MLR) for UDP/IP, Media
Loss rate (MLR) for RTP/UDP/IP. Inline video monitoring with MPLS label is to be supported in future Junos release.
Junos Space is the proposed management platform for MX inline vides monitoring solution. It provides a single
management console for collecting and displaying video QoE alarms from the MX platforms. It allows for correlation of
customer fault reports to actual video QOE events occurring in real time or historically in the network.

7. SERVICES
7.1 eVPN (Ethernet VPN)
Ethernet VPN (EVPN) is a new standards-based technology that provides virtual multi-point bridged connectivity
between different Layer 2 domains over an IP or IP/MPLS backbone network. Similar to other VPN technologies such
as Layer3 VPN and VPLS, EVPN instances (EVIs) are configured on uPE (PE) routers to maintain logical service
separation between customers. The uPEs connect to customer devices, which can be a router, switch. The uPE
routers then exchange reachability information using Multi-Protocol BGP (MP-BGP) and encapsulated customer traffic
is forwarded between uPEs.
EVPN MAC address learning between uPEs occurs in the control plane. A new MAC address detected from a CE is
advertised by the local uPE to all remote PEs using an MP-BGP MAC route. This method differs from existing Layer 2
VPN solutions such as VPLS, which performs MAC address learning by flooding unknown unicast in the data plane.
This control plane-based MAC learning method provides better control over the virtual Layer 2 network.
Figure 23: eVPN Diagram

EVPN has the flexibility to be deployed using different topologies including E-LINE, E-LAN, and E-TREE. It supports
an all-active mode of multi-homing between the CE and uPE devices that overcomes the limitations of existing
solutions in the areas of resiliency, load balancing, and efficient bandwidth utilization. The control plane-based MAC
learning allows a network operator to apply policies to control Layer 2 MAC address learning between EVPN sites and
also provides many options for the type of encapsulation that can be used in the data plane.
EVPN’s integrated routing and bridging (IRB) functionality supports both Layer 2 and Layer 3 connectivity between CE
nodes along with built-in Layer 3 gateway functionality. By adding the MAC and IP address information of both hosts
and gateways in MAC routes, EVPN provides optimum intra-subnet and inter-subnet forwarding within and across
data centers.
It is recommended to deploy eVPN for new customer after METRO network is deployed. Wit successful customer
trials, FTEL can introduce eVPN services to existing customer with new site deployments and migrations.
7.2 Multicast
Multicast is required in FTEL IP/MPLS network to provide IPTV services .
During any development of new products, the following should be considered.
o PIM-SM is the preferred Multicast protocol due to its superior scaling properties in both the control and
forwarding planes.
o P2MP LSP’s are a valid alternative to PIM-SM
o IGMP can be supported in customer networks through the use of CE devices
7.2.1 GLOP
RFC 3180 [3] GLOP is an assignment of Global Multicast IPv4 Address space that is inherited due to the assignment
of an AS number. FTEL is the owner of AS18403. GLOP is driven from AS number and the following global multicast
address is unique.
IANA has allocated 233/8 for this purpose; by mapping the AS number to two middle octets.
For example, taking AS 18403 written in binary = 0100011111100011. Mapping these 16 bits into two middle octets
01000111&11100011 (71 & 227)

AS Multicast
address
AS1840 233.71.227.0/24
3
Table 11: FTEL GPOL Address
7.2.2 Administratively Scoped IPv4 Multicast Space

The administratively scoped IPv4 multicast address space is defined to be the range 239.0.0.0 to 239.255.255.255
according to RFC 2365.
The key properties of administratively scoped IP multicast are that
o Packets addressed to administratively scoped multicast addresses do not cross configured administrative
boundaries
o Administratively scoped multicast addresses are locally assigned, and are not required to be unique across
administrative boundaries.
In summary the administrative IPv4 Multicast address space is like a private IP address space (e.g. 10/8 etc).
Range Use
239.255.0.0/16 Local Scope
Organizational
239.192.0.0/14
Scope
224.0.1.0-238.255.255.255 Global Scope
Table 12: Administratively Scoped IPv4 Multicast address
7.2.3 PIM-SM
PIM-SM (Protocol Independent Multicast – Sparse Mode) is the multicast protocol of choice for the support of wide
scale “opt-in” style multicast solutions.
PIM-SM only forwards multicast traffic towards clients who have explicitly requested the multicast traffic, and it
requires the function of a RP (Rendezvous Point) to allow multicast clients to find sources of multicast traffic on the
network. Once a source is found a shortest path tree is built to provide optimal path forwarding of multicast traffic. In
other words the RP is required to start a multicast session, but is not required once that session has started between
that client and that specific source.
7.2.4 ANYCAST RP
The RP is an important function in the multicast network to allow clients to discover sources, and so redundancy
should be provided in the network for the RP. There are multiple methods for RP redundancy. These include Auto-RP,
Bootstrap and Anycast RP. Anycast RP is recommended due to its fast convergence and simpler troubleshooting.
Anycast RP requires that MSDP be configured between all RP’s in the backbone. MSDP is similar to BGP, must be
configured per neighbor. It is established via a TCP session and it advertises active sources between the RP’s in the
network.
7.2.5 Next Generation Multicast

Next Gen Multicast is to be deployed to support MVPN’s for general IPVPN . NG-MVPN in segmented inter-area
network is supported on Junos 15.1 and above
A point-to-multipoint MPLS LSP is an RSVP-signaled LSP with a single source and multiple destinations. Point-to-
multipoint LSPs avoid unnecessary packet replication at the ingress router by using MPLS packet replication capability
of the network. Packet replication takes place only when packets are forwarded to two or more different destinations
requiring different network paths.
The following are some of the properties of point-to-multipoint LSPs:
o A point-to-multipoint LSP allows you to use MPLS for point-to-multipoint data distribution. This functionality is
similar to the function provided by IP multicast.
o Branch LSPs can be added and be removed from a main point-to-multipoint LSP without disrupting traffic. The
unaffected parts of the point-to-multipoint LSP continue to function normally.

o uPE can be a transit and an egress router for different branch LSPs of the same point-to-multipoint LSP.
o Link protection provides bypass LSP for each of the branch LSPs that make up the point-to-multipoint
LSP.Traffic is switched to the bypass LSP in case of primary LSP failure
o sub-paths can be configured either statically or dynamically.
o Graceful restart can be used for on point-to-multipoint LSPs.
7.2.5.1 NG-MVPN On Segmented Inter-Area Network

P2MP LSPs in segmented inter-area network is supported and be segmented with Junos 15.1 and above. A
segmented P2MP LSP has ingress area segment, backbone area segment and egress area segment. Ingress area
segment could be at uPE or at NPE platforms; backbone area segment could be at transit NPE platforms. The egress
area segment is at uPE or NPE platforms.
Each of the intra-area segments can be carried over provider tunnels with P2MP RSVP-TE LSP. Segmentation of
inter-area P2MP LSP occurs when the S-PMSI autodiscovery (AD) routes are advertised which initiates the inclusion
of a new BGP extended community or inter-area P2MP segmented next-hop extended community in ingress uPE or
NPE, transit NPE, and egress uPE routes or NPE.
Within each NPE inter-area P2MP LSPs are carried over intra-area aggregated P2MP LSPs. Segments of inter-area
P2MP LSPs are stitched at NPEs using BGP as the stitching protocol.
7.3 SERVICES BY MULTISERVICE CARDS ( MS-MIC )

Multiservices Modular Interfaces Card (MS-MIC and the Multiservices Modular PIC concentrator (MS-MPC) are
introduced to enable application based services, CGNAT, DPI.
In Ftel design, it is considered to enable MS-MIC on uPE platforms for the following services;
 CGNAT
 Jflow
DPI and F functions are not supported on MS-MIC.
The following table shows features supported by MS-MIC;
 Static Source NAT
 DynamicSource NAT - Address Only
 Static Destination NAT
 NAPT - EIM/EIF/APP
 NAT64
 AMS Support
Using the following IMIX packet ratio, each card CGNAT and JFLOW scaling and performance numbers are shown in
below tables ;
Packet size (incl. IP Distribution (in
# Packets
header) packets)
64 7 58%
570 4 33%
1518 1 8%
Avg pkt size 354
Table 13: IETF IMIX Ratio
Traffic 16G MIC (800MHz)
SFW ( Firewall)+NAPT44+APP+EIM+EIF w/o syslog-UDP Max Flow 14 Million

SFW (Firewall)+NAPT44+APP+EIM+EIF w/o syslog-HTTP-Connection 1.45K

per second
SFW (Firewall)+NAPT44+APP+EIM+EIF w/o syslog-HTTP-Concurrent 6.65 Million
Connection
SFW (Firewall)+NAPT44+APP+EIM+EIF w/o syslog-HTTP-Throughput 12.6Gbps
Table 14: CGNAT Scaling Numbers
Module Template Max Flow Max Throughput ( Mbps) Flow Setup Rate
16G MIC (800MHz) IPv4 14000000 5928 95

IPv6 14000000 5945 105
Table 15: JFLOW Scaling Numbers
7.3.1 Logical Topology

The CGN-INSIDE routing instance in the uPE router is configured with one service interface in the CGN MS-MIC
service cards. Each MS-MIC has one procession unit so one service interface however, multiple MS-MIC could be
deployed to provide service redundancy or to provide capacity expansion. A static default route in the routing instance
points to those logical service interface as ECMP next-hops. Load balancing takes place between multiple MS-MIC
cards however, in this design, one MS-MIC per uPE is considered to use other slots for L3 and L2 VPN services for
capacity expansion.
The following figure shows logical topology for CGN services
Figure 24: CGN Logical Topology
7.3.2 CGNAT on MS-MIC

CGNAT solution uses the new generation of Trio service cards, known as MS-MIC. The solution is designed to serve
business customers that are assigned private IP addresses. Business customers who require Internet services will be
routed into CGN-inside VRF to be processed for CGNAT.
MS-MIC service card is equipped with one processing unit that perform several services such as NAT. In FTEL
design, the MS-MIC card is used solely for dynamic source NAT’ing. The role of the uPE nodes is to source-translate
traffic originating from private addresses to public addresses that belong to a pre-defined public pool. These public
pools are uniquely assigned to an MS-MIC NPU. This means that if some private traffic is served by a specific NPU
(or service interface), therefore translated to a public address belonging to the public pool assigned to that NPU, then
the reverse flow (public to private) must return to the same uPE node where the translation took place on the very
same NPU.

Because each of the new uPE nodes will be serving and translating many of business customers, the CGNAT service
must be as friendly as possible to the end user’s applications. This mandates the use of the following NAT features:
• Address Pooling Paired (APP)
• Endpoint Independent Mapping (EIM)
• Endpoint Independent Filtering (EIF)
• Application Layer Gateway (ALG)
• Hair pinning
The following subsections elaborate on all of the aspects summarized above that together carve out the CGNAT
solution in FTEL network.
7.3.2.1 Customer Friendly CGNAT Service

The friendliest NAT configuration implies the use of APP+EIM+EIF along with ALG’s. The same set of features is
preserved, as much as possible, across all uPE nodes to provide consistent level of application experience for all the
business customers.
7.3.2.1.1 Address Pooling Paired ( APP)

It is recommended that a NAT has an "IP address pooling" behavior of "Paired”: same public IP address for the same
private IP.
Figure 25: CGNAT44+APP
APP does not say anything on what public port is going to be used on the post-NAT’d IP address. Neither does it say
which connections are going to be accepted from the outside.
All it does is to guarantee that, if a private IP IPc is randomly translated to the public address IPn1, then all
subsequent other flows from the same IPc will be translated to the same public IP of IPn1. It solves the problems of an
application opening multiple connections. Some examples of use cases are:
 If a SIP client is sending RTP and RTCP, it should be expected that they come from the same IP address,
even after they go through NAT. Otherwise it should have been negotiated beforehand.
 Any Peer-2-Peer protocol that negotiated ports assuming address stability will benefit from address pooling
paired.
7.3.2.1.2 Endpoint Independent Mapping (EIM)

"Endpoint-Independent Mapping" behavior means a private source couple <IP:port> must be translated always to the
same public <IP:port> value regardless of the destination.

Figure 26: CGNAT44+APP+Different Mapping Behaviors

It means assigning the same external address and port for all connections from a given host if they use the same
internal port. This means if they come from a different source port (same host) different external port could be
assigned.
Enabling EIM means you have a stable external IP address + port (for a period of time) that external hosts can use to
connect. The association between the private IP address + port and the corresponding external IP address + port
associated by the NAT is called “mapping”.
The determination of who can initiate a connection from the public space to the private space is not done by EIM, but
EIF (explained in the next section). Note that EIM requires APP.
Moreover, a private subscriber that maintains the same private port and IP address could potentially send traffic to
many different IP destinations. Since in this case such user would maintain the same public IP and port (and hence
the user public port consumption would count as 1), this user could though generate a high number of different flows
and this could have an impact on the MS-MIC scalability. To limit the number of sessions that such user could do, a
new configuration knob was introduced starting from Junos 11.4R7 to limit the flows per internal IP address. This knob
however is supported only on the MS-DPC card and is yet to be supported in the MS-MIC card as per Junos 14.2R5.

7.3.2.1.3 Endpoint Independent Filtering (EIF)
Figure 27: CGNAT44+APP+Different Mapping/Filtering Behaviors

EIF works in conjunction with EIM, where EIF creates a mapping between the internal Host:port (H1:h1) and a
corresponding Address:port (N1:n1) chosen from the NAT pool.
Once the mapping is created, it can be used by other hosts:ports to reach the internal Host:port (H1:h1) by sending
unsolicited traffic to the associated NAT mapping (N1:n1).
This is what is described in RFC3489 as “Full-Cone NAT” behavior.
The recommendation to use Endpoint-Independent Filtering is aimed at maximizing application transparency; in
particular, for applications that receive media simultaneously from multiple locations (e.g. gaming).
EIF though allows connections to be initiated from the Internet to the pinholes specified by EIM. These new
connections are not going to consume other ports (since they use the same EIM port association) but each of them
will create a new flow through the MX.
Starting from Junos 11.4R7, a new configuration knob was introduced to limit the number of inbound flows to the EIF
mapping in order to avoid Denial of Service (DoS) attacks. Without this knob, the default behavior is to limit the
number of inbound connections on an EIF mapping based on the max-flows allowed in the system. However, the knob
is yet to be supported in the MS-MIC card as of Junos 14.2R5.
7.3.2.1.4 Hair-pinning
Hair-pinning allows two endpoints on the internal side of the same LSN to communicate even if they use each other’s
external IP addresses and ports. The following diagram illustrates a typical hair-pinning communication.
APP, EIM, EIF and Hair-pinning are very essential for many applications such as online gaming.

Figure 28: Hair-pinning
7.3.2.1.5 Application Layer Gateway (ALG)

Application layer gateways (ALGs) are advanced application-inspecting features available on the MS-MIC that serves
for dynamically opening pinholes traffic for applications allowing return inbound packets (e.g., for FTP there may be
multiple sessions for control and data for the same data connection between the source and destination).
The possibility to know how to open reverse pinholes is possible because an ALG understands the application
protocol and how it is supposed to function.
The following is the list of ALGs to be enabled in the CGNs:
 PPTP: Point-to-Point Tunnelling Protocol is a Layer 2 protocol used for tunnelling PPP over an IP network.
PPTP is often used as a way to implement virtual private networks (VPNs) and is tunnelled over TCP and a
Generic Route Encapsulation (GRE) tunnel encapsulating the PPP packets.
 FTP: The File Transfer Protocol ALG monitors the FTP connection for PORT, PASV, and 227 commands. The
ALG will handle all NAT functions and pin-holing of any additional ports necessary.
 RTSP: Real-Time Streaming Protocol is used to establish and control media connections between end hosts.
RTSP handles all client-to-media server requests such as play and pause, and is used to control real-time
playback of the media files from the server. RTSP does not, however, stream any media data. Commonly, that
is left to Real-time Transport Protocol (RTP), and the two are used in combination to deliver media to the
clients.
 SIP: Session Initiation Protocol is a signalling protocol used for initiating, modifying, and terminating
multimedia sessions such as voice and video calls over IP. Similar to the MS-DPC, The SIP ALG on the MS-
MIC only supports SIP over UDP and not over TCP. SIP sessions are limited to 12 hours (720 minutes) for
NAT processing on the MS-MIC cards. There is no time limit for SIP sessions on the MS-DPC.
 TFTP
 RPC-PORTMAP
 REXEC
 RLOGIN
 RSH
 SQLNET
 RPC-SERVICES
 POP3
 DNS

 DCE-RPC-PORTMAP
 Other basic TCP/UDP ALGs for user-defined applications
7.3.2.1.6 Best Practise for a combined use of APP+EIM+EIF+ALG

The features discussed in the previous sections are to be enabled in a controlled manner according to the following
guidelines to avoid impacting the scale and performance of the service card:
 Most Internet traffic uses HTTP or HTTPS, and there is no browser on any OS that reuses the same source
port for sending traffic to different destinations. EIM provides no benefit for HTTP traffic.
 Because none of the Junos pre-defined ALGs require EIM to work, EIM should not be enabled for applications
that have defined ALGs or are known not to use Session Traversal Utilities for NAT (STUN) servers to
discover the presence of a NAT router.
 EIM allocates memory for each mapping; this is in addition to the memory used for flow allocation. This
reduces the maximum number of flows that can be established through the services card, and causes
processing overhead for the creation and deletion of flows and mappings. Therefore, EIM should selectively
be enabled and only for applications that do reuse the source ports and rely on a CGNAT device to maintain
the same address:port mapping for all traffic sent to different destinations.
7.3.2.2 NAT Pool Sizing

Each uPE has one MS interface binding to a service-set which references a NAT rule and hence a NAT pool.
Therefore, each uPE node should be assigned one (1) public NAT pool, one MS interface.
The best practice for pool sizing is to avoid excessively large NAT pools and assign the pool size according to the
maximum supported flows. Considering one MS interface, the following applies:
• Round robin allocation of IP addresses is used
• Each public IP address provides 64512 ports (1024-65535) by default
• The maximum number of concurrent sessions (bidirectional flows) per service PIC (NPU) is 15 millions
• For (1) MS configuration, the required number of public IP addresses in the pool = 15,000,000/64512= 233.
This is equivalent to a NAT pool size of /24.
• The AMS interface allocates the whole pool (/24) to the single active service PIC (NPU)

8. SYSTEM MANAGEMENT
The following section provides high level design for METRO network routers system config. It includes user login
accounts (including user authentication), root level authentication, time-zone, NTP (Network Time Protocol)
addresses, and the routers auxiliary/console ports.
8.1 Time Zone

All METRO network routers are to be configured with Vietnam time-zone, i.e. GMT +7.
8.2 Root Authentication

Root authentication encrypted passwords should be generated and not shared outside the FTEL infrastructure group.
This password must be protected and it is critical to the security of the network, and should not be made available to
general operational staff.
8.3 RADIUS
RADIUS (Remote Authentication Dial In User Service) is to be used to authenticate users, in addition to the auditing of
CLI commands. Two (2) RADIUS servers will be used for authentication, ensuring a high level of redundancy. These
servers are hosted on the EMS segments (at different locations).
The METRO network routers will source all RADIUS messages from its primary loopback interface address. All
RADIUS messages will be sent and received from UDP ports 1812 and 1813 as per RFC 2865 with an encrypted
RADIUS key. A default timeout of 3 seconds will be used between RADIUS requests (three RADIUS retries) before
the redundant RADIUS server is used.
8.4 Authentication Order

The primary authentication method shall be RADIUS. If both the primary or secondary RADIUS servers are
unreachable, then the login shall be authenticated locally. If the RADIUS server is reachable but the login attempt is
unsuccessful (e.g. bad password), local authentication is not be attempted.
8.5 Multicast ICMP Response

By default, the Routing Engine responds to ICMP (Internet Control Message Protocol) echo requests sent to multicast
group addresses. This is to be disabled by including the knob “no-multicast-echo”.
8.6 Redirects
By default, METRO network router will sends protocol redirect messages. This is to be disabled by including the knob
“no-redirects”.
8.7 Routing Engine (RE) Synchronization

Routing Engines are to both synchronized automatically on a configuration commit. Including the “commit
synchronize” knob does this.
8.8 Default Address Selection

By default, the source address included in locally generated Transmission Control Protocol/IP (TCP/IP) packets, such
as FTP traffic, and in User Datagram Protocol (UDP) and IP packets, such as Network Time Protocol (NTP) requests,
is chosen as the local address for the interface on which the traffic is transmitted. This means that the local address
chosen for packets to a particular destination might change from connection to connection based on the interface that
the routing protocol has chosen to reach the destination when the connection is established. If multiple equal-cost next
hops are present for a destination, locally generated packets use the lo0 address as a source.
If you include the default-address-selection statement in the configuration, the software chooses the system default
address as the source for most locally generated IP packets. The default address is usually an address configured on
the lo0 loopback interface.
8.9 Internet Options

By default, the router accepts packets that have both the SYN and FIN bits set in the TCP flag. Accepting packets
with the SYN and FIN bits set can result in security vulnerabilities, such as denial-of-service attacks. Packets with
SYN and FIN bits set are to be dropped. Including the “tcp-drop-synfin-set” knob does this.
8.10 Location
The physical location information for a IP Core router will be configured with the parameters summarized in Table 9.
Variable Example Description
building-name Hanoi FTEL Well known name/abbreviation for site
country-code VN The ISO 3166 country code [link]
floor-number 2 Floor number
rack-identifier 02/3/4 Rack Identifier
Table 16: System-Location variables
8.11 NTP
NTP (Network Time Protocol) synchronize the METRO network router to a centralized clock source, enabling log
messages to be correlated to other events on the network. Each METRO network router will synchronize its clock to
the NTP servers hosted on the EMS segments (at different locations),
The METRO network routers will source all NTP messages from its primary loopback interface address, where all
messages are authenticated.
8.12 Ports
The console port on the craft interface is enabled by default @9600bps. The auxiliary port is disabled.
Console ports are to emulate a VT100 terminal and the “log-out-on-disconnect” knob is to be enabled to log out
sessions when the data carrier on the console port is lost.
8.13 Services
By default all remote access to the router is disabled. Only SSH v2 is proposed as a remote terminal application, and
ftp and telnet is implicitly not enabled. Note that SCP (sftp) is inherently part of the SSH protocol suite and is to be
used to provide remote file transfer as required. The default values for connection limit (75) and rate-limit (150/min)
shall be maintained.
Root login is only permitted via the console. It is important to emphasis how important securing the OOB (Out of Band)
network is for this reason.
The above requirement implies that clients must support SSHv2 to access the routers.
8.14 Syslog
SYSLOG provides the primary logging facility for the auditing of events, e.g. error events, login attempts, and CLI
commands. Events will be logged to redundant SYSLOG servers hosted on the EMS network segments. In addition
specific events are to be logged to METRO network routers file system.
8.15 SNMP
SNMPv3 is the preferred method to SNMP poll METRO network routers as it is more secure. SNMPv3 authenticates
and encrypts the SNMP messages. SNMP traps will be set to three (3) redundant SNMP trap servers hosted on the
EMS segments with the following IPv4 addresses. However, currently FTEL uses snmpv2 for its deployment with
authentication enabled

9. PLANNING
In this section, some of planning items are discussed in terms of naming convention, description format, Ip
addressing, RD format etc..
9.1 HOSTNAME
Each hostname of network platforms contains information about the followings;
 Function of the platform
 Location of the platform
 Number of the ring it is within
 Number of Pop it is within
Function and location of the platform is identified with 3 alpha-numeric character and number of ring and number of
Pop is identified with 3 numeric characters. The variables of hostname is shown in the following table:
Table 17: Naming convention for hosname
The following is an example for 6th uPE platform in Hanoi, PoP1, Ring5, connecting to NPE01
 UPEHNO11056
The following is an example for 3rd uPE platform in Ho Chi Minh City, PoP2, Ring2, connecting to NPE02
 UPEHCM22023
The following example shown is for 1st NPE in Ho Chi Minh City connecting to MC01 and 2 nd NPE in Hanoi connecting
to MC02
 NPEHCM10001
 NPEHNO20002
9.2 IP ADDRESSING
It is recommended to use /31 network address on point-to-point logical connectives between uPE-uPE and uPE-NPE
as well as NPE-MC in METWOR network. To facilitate efficient and ease of management, the 10.254.0.0/16 address
space will be sub-divided into regions. Ip subnet range 10.245.0.0/16-10.255.235.0.0/16 are reserved for P2P links.
The following table shown the ip address distribution on 10.245.0.0/16 subnet and it is recommended to follow same
structure for the rest of /16 spaces if it is needed in the future
IP Address Space Allocation
10.245.0.0/19 OOB (Out of Band)
10.245.32.0/22 Hanoi Region
10.245.36.0/22 HCM Region
10.245.40.0/24 Province Region A
10.245.41.0/24 Province Region B
10.245.42.0/24 Province Region C
10.245.43.0/24 Province Region D
10.245.44-10.245.159.0 Range For the rest of 58 Regions
10.245.160.-10.245.255 Unallocated
Table 18: IPv4 Private Address allocation per regions for IP/MPLS links

9.2.1 IPv6 IP Addressing

For lo0.0 ip address, /32 subnet is reserved for each network platforms in METRO network. There is no IPv6
addressing requirement at the time the document developed.
Note. The IPv6 private address range was derived based on RFC 4193 (Pseudo-Random Global ID Algorithm). All
IPv6 private addresses must fall within the range “fd00::/8”
The IPv6 address allocation within the FTEL IPv6 prefixes block is out of scope of this document. However, it will give
some suggestions on how to allocate IPv6 prefixes for the VPNv6 service.
It is suggested that FTEL follows the RFC6164 “Using 127-bit IPv6 Prefixes on Inter-Router Links” for allocating IPv6
prefixes for the PE-CE links. Juniper router supports the RFC6164 and it explains all issues found in the past when
using 127-bit IPv6 prefixes on Inter-Router links.
For CE LAN address allocation, usually the customer will provide the IPv6 address to be allocated for each VPNv6
site. However, there will be cases where FTEL may allocate IPv6 from its own IPv6 address block to be used in the
CE LAN. When allocating IPv6 prefixes from the TM address block, it is suggested to allocate the IPv6 addresses
from the FTEL’s Public IPv6 address range assigned to FTEL by APNIC .
It is suggested to avoid the allocation of private IPv6 address for the CE LAN because this will eliminate the
requirement for applying NAT66 in the gateway to the Internet in case the customer wants Internet access inside the
VPN in the future.
The following table could be used for PE-CE point-to-point links.

fd00::/48 OOB (Out of Band)
fd00:0:1::/48 Hanoi Region
fd00:0:2:/48 HCM Region
fd00:0:3::/48 Province Region A
fd00:0:4:/48 Province Region B
fd00:0:5::/48 Province Region C
fd00:0:6::/48 Province Region D
fd00:0:7::/48-fd00:0:3f::/48 For the rest of 58 Regions
Range
fd00:0:40::/48- fd00:0:3e8::/48 Unallocated
Table 19: IPv6 Private Address allocation per regions for PE-CE links
9.2.2 Loopback Ip Address

Each FTE METRO network router is provisioned with the following IP loopback addresses,
 Unique Private IPv4 /32 address taken from the 10.255.0.0/16 address pool.
 Unique Private IPv6 /128 address taken from the fd7d:2c4f:bc9c::/48 address pool.
The primary loopback interface address, which is also the router-id, is taken from the 10.255.0.0/16 address pool. To
facilitate efficient and ease of management, the 10.255.0.0/16 address space is sub-divided into regions, as
summarized in Table 20. 10.255.0.0/16-10.246.0.0/16 spaces are reserved for loopback interfaces and the following
table shows ip address allocation structure and it is recommended to follow same structure for the rest of spaces.
10.255.0.0/19 OOB (Out of Band)
10.255.32.0/22 Hanoi Region
10.255.36.0/22 HCM Region
10.255.40.0/24 Province Region A
10.255.41.0/24 Province Region B
10.255.42.0/24 Province Region C

10.255.43.0/24 Province Region D

10.255.44-10.255.223.0 Range For the rest of 58 Regions
10.255.224.0/19 Unallocated
10.255.255.0/24 Anycast Range (not used)
Table 20: IPv4 Private Address allocation per regions for loopback interface
The IPv6 private /128 loopback address to be provisioned on each METRO network router will be allocated from the
fd7d:2c4f:bc9c::/48 address range. The address is formed based on the routers assigned private IPv4 loopback
address, e.g. fd7d:2c4f:bc9c::10:233:32:32/128 (for 10.233.32.32). Refer to Figure 17 below.
1st byte of 2nd byte of 3rd byte of 4th byte of
private IPv4 private IPv4 private IPv4 private IPv4
fd7d:2c4f:bc9 loopback loopback loopback loopback
:: address : : :
c address address address
(decimal (decimal (decimal (decimal
format) format) format) format)
Figure 29: IPv6 Private Address Allocated for Loopback Interfaces
9.2.3 IP Address Mapping to Hostname

Mapping hostname to ip addressing will introduce some difficulties to assign continues subnets per region so it is not
recommended to map ip address to hostname vice versa
9.3 AS Numbers
The following table shows AS numbers used by FTEL today.
AS Number Purpose of AS Number
18403 FTEL Internet Service Provider
18403 FTEL IPVPN Layer 3 VPN Services
Table 21: AS Numbers Deployed Across Legacy Networks
9.4 VRF Route Distinguishers

IPv4 VPN prefixes exchanged between METRO network platforms and its regional route reflectors are to be
composed of a Route Distinguisher and an IPv4 prefix (up to 96 bits). The Route Distinguisher disambiguates IPv4
addresses, allowing overlapping IPv4 address space between Layer 3 VPN’s. This assumes the Route Distinguisher
is configured differently between the VRF’s provisioned on the METRO uPE’s.
Figure 30: Layer 3 VPN – VPN Prefix (RD + IPv4 Prefix)

Administrator Assigned Number
(Based on IP address format) (TM Policy/Procedure to Allocate)
METRO uPE Router-ID Unique number assigned to a VPN
Table 22: Route Distinguisher Format
A unique route distinguisher assigned number is to be allocated for each routing instances. A routing-instance route
distinguisher provisioned on UPE is made globally unique throughout the METRO network by assigning the uPE’s
router-ID to the administration field.
The route-distinguisher assigned number, as allocated by FTEL, for a routing instances, for example, is 1001, i.e.
route-distinguisher = <(uPE Router ID)>:1001.
The following structure is provided for FTEL to develop RD mapping for their customers
Service RD (Route Distinguisher)
OOB (Management) <Router ID>:0-499-
Internet Services <Router ID>:500-999

Services ( L2VPN or VPLS) <Router ID>:1000-1999

Mobile Backhaul <Router ID>:2000-2999
L3VPN <Router ID>:3000-3999
Table 23: Route Distinguisher Structure
9.5 VRF Route Targets

VRF route-targets control route-distribution between routing-instances. It is recommended to provide VRF route
targets for customer in hub-spoke topology. A summary of routing instance route-targets is provided in Table 3.
VRF Route Target Name VRF Route Target Description
IPv4 routes exported from Customer hub site are
to be tagged with this VRF route-target BGP
extended community. These routes may include,
target:18403:<unique #  Customer IGP routes.
<IPVPN_Customer>_HRT
assigned by FTEL>  Interface routes (loopback + data center).
All spoke site routing instances import MP-BGP
routes previously tagged with this VRF route
target BGP extended community.
IPv4 routes exported from a customer spoke
routing instance are to be tagged with this VRF
route-target BGP extended community. These
target:18403:<unique # routes may include,
<IPVPN_Customer>_SRT assigned by FTEL>  Interface routes (loopback)
 Customer IGP routes.
A Customer hub site routing instance imports MP-
BGP routes previously tagged with this VRF
route-target BGP extended community.
Table 24: VRF Route Target Format
9.6 Link Description

In each link connectivity between METRO network Platforms the following format is to be used to provide information
about local and remote router and interface information.
description "LINK:xe-<FPC>-<PIC>-0.<uPE/NPE Local Hostname>---xe-<FPC>-<PIC>-0.<uPE/NPE Remote
Hostname>";
9.7 VLAN STRUCTURE

The ID of vlans are divided into 2 classes, service Vlan IDs and customer Vlan IDs. Service Vlan ID represents the
vlans used in between NPE and uPE platforms. The service Vlan ID represents the vlans allocated in between uPE
and Customer Switch or router. The following allocation for service Vlan ID is applicable to each NPE
Service SERVICE VLAN RANGE
OOB (Management) 0-499-
Point to Point Links 500-999
Services ( L2VPN or VPLS) 100-1999
Mobile Backhaul 2000-2999
L3VPN 3000-3999
Table 25: Service Vlan Allocation

9.8 SCALING
METRO network consists of MX104 and MX480 platforms with the following features enabled
o OAM (LFM, CFM)
o BGP
o BFD
o NSR
o GRES
o ISIS
o Link Protection
o IPv4/IPv6
o Inline Jflow
With the following services
o L2VPN ( L2 circuits, VPLS)
o L3VPN
The following table also shows the scaling for each platform:
FIB MX480 MX104
FIB Capacity:IPv4 unicast 2.5M IPv4 1.8M
FIB Capacity:IPv6 Unicast 2.5M IPv6 1M
Max L3VPN prefixes with remote PE 5M 1M
using one CPNc4 Label per CE
RIB MX480 MX104
RIB Capacity:IPv4 Unicast (w NSR) 12M 4M
RIB Capacity:IPv6 Unicast (w NSR) 12M 3M
BGP Max IPv4 Prefixes MX480 MX104
Max BGP Routes without route 12M 3M (RR Capacity)
resolution (w NSR and single hop)
BGP Max Peers MX480 MX104
eBGP/iBGP with no prefixes 4K 1K
IGP MX480 MX104
ISIS Max Adjacency 4K 300
ISIS Max Routes 300K 100K
MPLS,VPLS,VPN MX480 MX104
Max RSVP-TE LSPs (head-end) 32K 32K
Max LDP targeted sessions 2400 2400
Maximum number of instances (VRF 8K 2K
L3VPN, L2VPN Kompella, Virtual
router)
BFD MX480 MX104
Distributed BFD ( with 100ms hello 300 300
interval)
Non-Distributed BFD (with 100ms 300 300

hello interval)
OAM MX480 MX104
CCM session with 100ms session 500 per line card and 2000 per 300
interval chassis
Table 26: METRO Network platform Scaling Metrics
With the scaling number shown in the table above, the following principles are decided for FTEL METRO network
o Each NPE Pair will have maximum of 128xuPE Platforms
o Each ring will have maximum of 6xuPEs
o In Provinces where MX104 is used as NPE platform will follow the followings
o Each NPE pair will have maximum of 24xuPE
o Each ring will have maximum of 6xuPE
JunOS 14.2R5 for MX104/MX480 routers and JunOS 14.1X53-D30 for QFX5100 are recommended for FTEL METRO
Project.

10. Sign Off

(Optional – remove this section in case the sign off is by email or on another document)
This final sign off is based on the understanding that formal review meetings have been conducted to present, explain
and discuss the Low Level Design document created. As per the observations made during these meetings, we have
incorporated <client name> input and feedback in this document.
Juniper Networks Inc Customer
Approved YES NO Approved YES NO
Name Name
Signature Signature
Position Position
Date Date

PS FTEL Metro HLD - 20162901

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PS FTEL Metro HLD - 20162901

Uploaded by

Copyright:

Available Formats

FTEL NEW METRO NETWORK HLD

Generated For : FTEL

Version Date Issued Status Author Reason for Change

Department Name Approval Date

Intellectual Property Rights

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 2 of 64

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 3 of 64

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 4 of 64

Figure 8: LFS Illustration Diagram................................................................................................................................. 22

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 5 of 64

Table 17: Naming convention for hosname................................................................................................................... 58

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 6 of 64

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 7 of 64

1.5 Alphabetical Index of Terms

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 8 of 64

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 9 of 64

Table 1: Alphabetical Index of Terms

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 10 of 64

2.1 UPE (USER PROVIDER EDGE) NETWORK ELEMENTS

2.1.1 UPE-Site Swicth Topolofy Options

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 11 of 64

Figure 1: Site Swicth-uPE Topology Option 1

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 12 of 64

2.2 Distributed Switch NETWORK ELEMENTS

2.3 NPE (NETWORK PROVIDER EDGE) NETWORK ELEMENTS

Site NPE sites

Table 2: NPE Site Locations

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 13 of 64

Figure 2: New Metro Network Topology

Figure 3: Provincial Metro Network Topology

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 14 of 64

3.1.1 Hierarchy and Area Architecture

Figure 4: NSAP Address Encoding

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 15 of 64

3.1.2 Routes to be carried in ISIS

3.1.3 Link Metric

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 16 of 64

MC-GW L2 3000 International GW

Figure 5: Traffic Flow- Scenario I

Link Level IGP Metric Comment

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 17 of 64

Table 4: ISIS Metrics-Scenario II

Figure 6: Traffic Flow-Scenario II

3.1.4 MTU size

3.1.5 ISIS Authentication

3.2 BGP Routing

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 18 of 64

 eVPN service for future use

Table 5: MX480 Scaling Table

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 19 of 64

Figure 7: BGP-LU Diagram

3.3.1 BGP-LU Design Consideration on Existing PE routers

3.4 Non-Stop-Routing (NSR)

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 20 of 64

3.5 GRES ( Graceful Routing Engine SwitchOver)

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 21 of 64

The Packet Forwarding Engine:

3.6 Fast Failure Detection

3.6.1 LFS Advantages

Figure 8: LFS Illustration Diagram

JUNIPER NETWORKS Copyright  2015 All rights reserved Page 22 of 64

3.6.2 LFS Through the DWDM Transponders

3.6.3 Distributed BFD Sessions