Data Center Interconnect Options

Advanced Data Center Switching
Chapter 6: Data Center Interconnect

We Will Discuss:
• The meaning of the term Data Center Interconnect;
• The control and data plane of an MPLS VPN; and
• The DCI options when using a VXLAN overlay with EVPN signaling.
Chapter 6–2 • Data Center Interconnect www.juniper.net

DCI Overview
The slide lists the topics we will discuss. We discuss the highlighted topic first.
www.juniper.net Data Center Interconnect • Chapter 6–3

What Is a Data Center Interconnect?

Data center interconnect (DCI) is basically a method to connect multiple data centers together. As the name implies, a
Layer 3 DCI uses IP routing between data centers while a Layer 2 DCI extends the Layer 2 network (VLANs) from one data
center to another.
Many of the DCI communication options rely on an MPLS network to transport frames between data centers. Although in
most cases an MPLS network can be substituted with an IP network (i.e., by encapsulating MPLS in GRE), there are several
advantages to using an MPLS network including availability, cost, fast failover, traffic engineering, and scalable VPN options.

Interconnect Network
Between two data centers that need to be interconnected is a network of some type. A typical interconnect network could be
a point-to-point line, an IP network, or an MPLS network. The slide shows that these networks can be owned by customer
(the owners of the data center) or by a service provider. All the DCI options that we discuss in this chapter will work in both a
customer-owned or service provider-owned interconnect network. The main difference is how much control a customer has
over the DCI. Sometimes it is just easier and cost effective to let the service provider manage the DCI.

Point-to-Point DCI
In general, if there is great distance between data centers, a point-to-point interconnect can be pretty expensive. However, if
the data centers are just down the street from one another, it might make sense to have a point-to-point interconnect. This
type of interconnect is usually provided as a dark fiber between the data center. The customer simply attaches equipment to
the fiber and has the choice of running any type of protocol they wish over the interconnect.

IP DCI
It is possible to provide a DCI over an IP network. If the DCI is meant to provide Layer 2 stretch (extending of VLANs) between
the data centers then the Ethernet frames will need to be encapsulated in IP as it traverses the DCI. VXLAN and GRE are
some of the typical IP encapsulations that provide the layer 2 stretch. If the DCI is to provide layer 3 reachability between
data center, then an IP network is well suited to meet those needs. However, sometime the DCI network may only support
globally routeable IP addressing while the data centers use RFC 1918 addressing. When that is the case, it might make
sense to create a layer 3 VPN between the two data centers, like GRE, IPSec, or RFC 4364 (MPLS Layer 3 VPN over GRE).

MPLS DCI
The slide shows the encapsulation boundary of an MPLS transport network. The boundaries are different depending on who
owns the MPLS network. If the customer own the MPLS network then MPLS can be used for encapsulation from end-to-end.
If the service provider owns the MPLS network then the encapsulations between DC and MPLS network completely depends
on what is allowed by the service provider. If the service provider is providing a layer 2 VPN service, then the customer should
expect that any Ethernet frames sent from one data center will appear unchanged as it arrives at the remote data center. If
the service provider is providing a layer 3 VPN service, then the customer should expect that any IP packets sent from one
data center will appear unchanged as it arrives at the remote data center. In some cases, the service provider will allow a
customer to established data center-to-data center MPLS label switched paths (LSPs).

MPLS Advantages
Many of the DCI technologies that we will discuss depend on an MPLS network to transport frames between data centers.
Although in most cases an MPLS network can be substituted with an IP network (i.e., by encapsulating MPLS in GRE), there
are several advantages to using an MPLS network:
1. Fast failover between MPLS nodes: Fast reroute and Node/Link protection are two features of an MPLS network
that allow for 50ms or better recovery time in the event of a link failure or node failure along the path of an
MPLS label switched path (LSP).
2. Scalable VPNs: VPLS, EVPN, L3 MPLS VPNs are DCI technologies that use MPLS to transport frames between
data centers. These same technologies allow for the interconnection of many sites (potentially hundreds)
without the need for the manual setup of a full mesh of tunnels between those sites. In most cases, adding a
new site only requires administrator to configure the devices at the new site. The remote sites do not need to be
touched.
3. Traffic engineering: MPLS allows for the administrator to decide the path takes over the MPLS network. You no
longer have to take the same path calculated by the IGP (i.e., all data takes the same path between sites). You
can literally direct different traffic types to take different paths over the MPLS network.
4. Any-to-any connectivity: When using an MPLS backbone to provide the DCI, it will allow you the flexibility to
provide any type of MPLS-based Layer 2 DCI, Layer 3 DCI, or both combinations that you choose. An MPLS
backbone is a network that can generally support most types of MPLS or IP-based connectivity at the same
time.

MPLS VPN Review

The slide highlights the topic we discuss next.

MPLS Shim Header: Part 1

MPLS is responsible for directing a flow of IP packets along a predetermined path across a network. This path is the LSP,
which is similar to an ATM VC in that it is unidirectional. That is, the traffic flows in one direction from the ingress router to an
egress router. Duplex traffic requires two LSPs—that is, one path to carry traffic in each direction. An LSP is created by the
concatenation of one or more label-switched hops that direct packets between label-switching routers (LSRs) to transit the
MPLS domain.
When an IP packet enters a label-switched path, the ingress router examines the packet and assigns it a label based on its
destination, placing a 32-bit (4-byte) label in front of the packet’s header immediately after the Layer 2 encapsulation. The
label transforms the packet from one that is forwarded based on IP addressing to one that is forwarded based on the
fixed-length label. The slide shows an example of a labeled IP packet. Note that MPLS can be used to label non-IP traffic,
such as in the case of a Layer 2 VPN.
MPLS labels can be assigned per interface or per router. The Junos operating system currently assigns MPLS label values on
a per-router basis. Thus, a label value of 10234 can only be assigned once by a given Juniper Networks router.
At egress the IP packet is restored when the MPLS label is removed as part of a pop operation. The now unlabeled packet is
routed based on a longest-match IP address lookup. In most cases, the penultimate (or second to last) router pops the label
stack in penultimate hop popping. In some cases, a labeled packet is delivered to the ultimate router—the egress LSR—when
the stack is popped, and the packet is forwarded using conventional IP routing.

MPLS Shim Header: Part 2

The MPLS shim header is composed of four fields:
• 20-bit label: Identifies the packet as belonging to a particular LSP. This value changes as the packet flows on
the LSP from LSR to LSR.
• Traffic Class (TC): Formerly called EXP (experimental), these three bits can be used to convey class-of-service
information, specifically the forwarding class a given packet belongs to. The 3-bit width of this field makes it
possible to give a frame a total of eight possible markings, each of them potentially linked to a different
forwarding behavior, for example a different queuing priority and a different buffer size.
• Bottom-of-stack bit: many MPLS applications require a packet to be tagged with several labels, one stacked on
top of the other. 
The bottom-of-stack bit of a MPLS header is set to 1 if this is the bottom of the label stack, and below is the
payload. The bottom-of-stack bit is set to zero instead if below the header lays another MPLS header (i.e.
another label).
Among the applications which require label stacking are for example VPNs. Here the outer label, or transport
label, indicates which label-switching router traffic should be delivered to. The inner label, called service label,
describes instead how the payload should be treated once it reaches its destination label-switching router.
• Time to live (TTL): As in the case for the equivalent IP field, TTL limits the number of hops a MPLS packet can
travel. It is decremented at each hop, and if its value drops to zero, the packet is discarded. When using MPLS
for IP traffic engineering, the default behavior is to copy the value of the IP TTL field into the MPLS TTL field. This
allows diagnostic tools like traceroute to continue working even when packets are encapsulated within MPLS
and sent down a label-switched path.

Labels Have Only Local Significance

A very important point to keep in mind is that labels have only local significance: they are assigned by each router according
to its own label availability. In other words, you can establish a label-switched path across a domain, between two endpoints,
and traffic following the path will typically be tagged with a different label at each hop.
A second important point is that generally labels are global to the router, and not tied to the incoming interface; a packet
tagged with a given label will be subject to the same forwarding treatment regardless from the interface it has been received
on. This apparently minor point will play a major role in MPLS traffic protection - a set of MPLS features that try and minimize
packet loss during a link or a node failure. 
There are only very few exceptions to this rule, mostly to do with specific (and very advanced) MPLS applications. One
example is carrier-of-carriers, where a MPLS-enabled service provider offers a MPLS transport service to other service
providers.

Reserved MPLS Label Values

Labels 0 through 15 are reserved (RFC 3032, MPLS Label Stack Encoding).
• A value of 0 represents the IP version 4 (IPv4) explicit null label. This label indicates that the label must be
popped, and the forwarding of the packet must then be based on what is below it, either another label or the
payload.
• A value of 1 represents the router alert label. This label value is legal anywhere in the label stack except at the
bottom. When a received packet contains this label value at the top of the label stack, it is delivered to a local
software module for processing. The label beneath it in the stack determines the actual forwarding of the
packet. However, if the packet is forwarded further, the router alert label should be pushed back onto the label
stack before forwarding. The use of this label is analogous to the use of the router alert option in IP packets.
• A value of 2 represents the IP version 6 (IPv6) explicit null label. This label value is legal only when it is the sole
label stack entry. It indicates that the label stack must be popped, and the forwarding of the packet then must
be based on the IPv6 header.
• A value of 3 represents the implicit null label. This is a label that a LSR can assign and distribute, but it never
actually appears in the encapsulation. When an LSR would otherwise replace the label at the top of the stack with
a new label, but the new label is implicit null, the LSR pops the stack instead of doing the replacement. Although
this value can never appear in the encapsulation, it can be specified by a label signaling protocol.
Continued on the next page.

Reserved MPLS Label Values (contd.)
The following list is a continuation of reserved Labels 0 through 15 (RFC 3032, MPLS Label Stack Encoding).
• A value of 7 is used for the Entropy Label Indicator (ELI). After determining a load balancing methodology, the ELI
allows the ingress LSR to notify the downstream LSRs of the chosen load balancing methodology.
• A value of 13 is used for Generic Associated Channel Label (GAL). This label informs an LSR that a received LSP
belongs to a Virtual Circuit Connectivity Verification (VCCV) control channel.
• A value of 14 is used as the OAM Alert Label. This label indicates that a packet is an MPLS OAM packet as
described in ITU-T Recommendation Y.1711.
• Values 4–6, 8-12, and 15 are reserved for future use.

Implicit and Explicit Null Labels

Two labels deserve special attention: Label 0 and Label 3. These two labels can only be used at the end of an LSP, between
the penultimate (that is, second-to-last) and the egress router.
• Label 0 (Explicit null): This label is always assigned an action of “decapsulate” (pop); the label-switching router
will just remove the MPLS header, and take a forwarding action based on what is below it (either another label,
or the actual LSP payload).
• Label 3 (Implicit Null): This is a special label value that is never actually found in MPLS frames, but only within
MPLS signaling protocols. It is used by the egress router, i.e. the last hop in a label-switched path, to request the
previous router to remove the MPLS header. This behavior, referred to as penultimate-hop popping, is the
Junos OS default.

Label-Switching Router
The original definition of label-switching router is “a router that takes forwarding decisions based only on the content of the
MPLS header”. In other words, a label-switching router always operates in label-switching mode. We will use a definition
which is slightly less restrictive, to include also ingress and egress routers, sometimes referred to as label edge routers.
Traffic at the ingress or at the egress of a label-switched path is typically not encapsulated into MPLS, so label-switching is
not possible, and a forwarding decision needs to be taken according to other rules.
We will use the term label-switching router (LSR) to mean any router which participates in MPLS forwarding, including both
the ingress and the egress nodes. For brevity, in the rest of the course we will also use the term router as synonym for
label-switching router.

MPLS Label Operations

The forwarding behavior of a label-switching router is defined according to three basic label operations:
• Push: add a MPLS header (a label) to a packet. 
This operation is typically done by the label-switching router at the beginning of a label-switched path, to
encapsulate a non-MPLS packet and allow it to be forwarded by label switching within the MPLS domain.
• Pop: remove a MPLS header from a MPLS-encapsulated packet. 
This is often done either at the end of an LSP or, as we will see shortly, by the second-to-last router (the
penultimate hop).
• Swap: replace the label value of a MPLS packet with another value.
This operation is typically performed by transit label-switching routers, as a packet traverses a label-switched
path.
After performing one of these MPLS basic operations, the packet is generally forwarded to the next-hop router.
In some cases the forwarding treatment can be more complex, involving different combinations of the three basic
operations. For some types of services, for example for VPNs, it is common to see a double-push forwarding action; while in
some traffic protection scenarios, when building a local detour to avoid a link failure, sometimes a transit router will have to
perform a swap-push operation.

Label-Switched Path
A label-switched path (LSP) is a unidirectional path through the network defined in terms of label switching operations (push,
pop, swap). You can think of a LSP as a tunnel: any packet that enters it is delivered to its endpoint, no matter what type of
payload it contains.
Establishing a label-switched path across a MPLS domain means determining the actual labels and label operations
performed by the label-switching routers on the path. This can be done with manual configuration, or by some type of
dynamic label distribution protocol.
Often a label-switched path will reside within a single MPLS domain, for example within a single service provider. However,
the development of advanced BGP-based MPLS signaling allows the creation of label-switched paths that span multiple
domains and multiple administrations.

Ingress Label Switching Router

The ingress router, sometimes called head end, is typically performing a label operation of push, inserting the MPLS header
between the layer-2 encapsulation and the payload packet. Its role is encapsulating non-MPLS traffic by adding one or more
labels to it, and forwarding it down a label-switched path.
The ingress router is not a pure label-switching router: the initial decision of which traffic to forward down which LSP is taken
not according to the content of labels (which are not present yet), but according to other criteria, e.g. a route lookup for IP
MPLS traffic engineering, or even the incoming interface, in case of point-to-point transport of layer-2 frames over MPLS
(layer-2 circuits, circuit-cross-connect).

Transit Label Switching Routers

The transit label-switching routers are LSRs that are neither at the beginning nor at the end of a label-switched path. They
typically operate in pure label switching mode, taking forwarding decisions only based on the label value of incoming MPLS
frames.
Very often transit LSRs will perform a swap operation, replacing the incoming label with the one expected by the next-hop of
the label-switched path. Transit LSRs are typically not aware of the content of the MPLS traffic they are forwarding, and do
not know if the payload is IP, IPv6, layer-2 frames or anything else.

The Label Information Base

The Label Information Base contains the actual MPLS label switching table which associates incoming MPLS labels to
forwarding actions, typically a label operation of either swap or pop and a forwarding next-hop.
Even if the label information base can be populated by static entries, generally this is done by a dynamic label distribution
protocol.

Penultimate-Hop Popping
Often the MPLS header is removed by the second-to-last (the penultimate) router in an LSP. This removal is an optimization
that helps in several cases, including using MPLS for IP traffic engineering. Removing the label at the penultimate hop
facilitates the work of the last-hop (egress) router, which, instead of having both to remove the MPLS header and then take
an IP routing decision, will only need to do the latter.
Penultimate-hop popping (PHP) is the default behavior on Juniper routers; however, it can be disabled in the configuration.
Some applications require PHP to be disabled, but that is often done automatically: the Junos OS is smart enough to detect
the need to signal the LSP so that PHP is disabled.

Egress Label Switching Router

The egress router (or tail end) of a LSP is the last router in the label-switched path. Exactly as in the case of the ingress LSR,
it is generally not a pure label-switching router, as it has to take a forwarding decision based on other factors than the
incoming label.
In case of MPLS IP traffic engineering, the egress router will be delivered ordinary IP packets due to penultimate-hop
popping, and will take a forwarding decision based on ordinary IP routing.

Label Information Base

On routers running the Junos OS, the label information base is stored into the mpls.0 table.
As soon as you enable MPLS processing, four default entries are automatically created: they are for label 0 (explicit null),
label 1 (router alert), label 2 (ipv6 explicit null) and label 13 (Generic Associated Label, used for Operation and Maintenance
and defined in RFC5586).

Label to Forwarding Action Mappings

On top of the pre-defined four labels, the mpls.0 table can be populated by static configuration or, much more frequently, by
dynamic label distribution protocols.
Each label is associated with a forwarding action, typically composed of a MPLS label operation (push, pop, swap or a
combination of these) and a next-hop. In this example, label 300576 has been installed by a dynamic protocol called LDP,
while the remaining label, 1004792, has been configured statically.
Note that there are two entries for this last label. This is because, in some cases, a label-switching router may have to take
different forwarding actions according to whether the label is or is not at the bottom of the label stack. In this case, the
forwarding actions turn out to be the same: pop the MPLS header and sent the content to 172.17.23.1 via interface ge-1/1/
5.0. The IP address of the next hop needs of course to be directly connected: it is only use to derive which MAC address to
use for layer-2 encapsulation.

Label Distribution Protocols

Label distribution protocols create and maintain the label-to-forwarding equivalence class (FEC) bindings along an LSP from
the MPLS ingress label-switching router (LSR) to the MPLS egress LSP. A label distribution protocol is a set of procedures by
which one LSR informs a peer LSR of the meaning of the labels used to forward traffic between them. MPLS uses this
information to create the forwarding tables in each LSR.
Label distribution protocols are often referred to as signaling protocols. However, label distribution is a more accurate
description of their function and is preferred in this course.
The label distribution protocols create and maintain an LSP dynamically with little or no user intervention. Once the label
distribution protocols are configured for the signaling of an LSP, the egress router of an LSP will send label (and other)
information in the upstream direction towards the ingress router based on the configured options.

RSVP
The Junos OS uses RSVP as the label distribution protocol for traffic engineered LSPs.
• RSVP was designed to be the resource reservation protocol of the Internet and “provide a general facility for
creating and maintaining distributed reservation state across a set of multicast or unicast delivery paths”
(RFC 2205). Reservations are an important part of traffic engineering, so it made sense to continue to use
RSVP for this purpose rather than reinventing the wheel.
• RSVP was explicitly designed to support extensibility mechanisms by allowing it to carry what are called opaque
objects. Opaque objects make no real sense to RSVP itself but are carried with the understanding that some
adjunct protocol (such as MPLS) might find the information in these objects useful. This encourages RSVP
extensions that create and maintain distributed state for information other than pure resource reservation. The
designers believed that extensions could be developed easily to add support for explicit routes and label
distribution.
Continued on the next page.

RSVP (contd.)
• Extensions do not make the enhanced version of RSVP incompatible with existing RSVP implementations. An
RSVP implementation can differentiate between LSP signaling and standard RSVP reservations by examining
the contents of each message.
• With the proper extensions, RSVP provides a tool that consolidates the procedures for a number of critical
signaling tasks into a single message exchange:
– Extended RSVP can establish an LSP along an explicit path that would not have been chosen by the
interior gateway protocol (IGP);
– Extended RSVP can distribute label-binding information to LSRs in the LSP;
– Extended RSVP can reserve network resources in routers comprising the LSP (the traditional role of
RSVP); and
– Extended RSVP permits an LSP to be established to carry best-effort traffic without making a specific
resource reservation.
Thus, RSVP provides MPLS-signaled LSPs with a method of support for explicit routes (“go here, then here, finally here…”),
path numbering through label assignment, and route recording (where the LSP actually goes from ingress to egress, which is
very handy information to have).
RSVP also gives MPLS LSPs a keepalive mechanism to use for visibility (“this LSP is still here and available”) and redundancy
(“this LSP appears dead…is there a secondary path configured?”).
LDP
LDP associates a set of destinations (prefixes) with each data link layer LSP. This set of destinations is called the FEC. These
destinations all share a common data LSP path egress and a common unicast routing path. LDP supports topology-driven
MPLS networks in best-effort, hop-by-hop implementations. The LDP signaling protocol always establishes LSPs that follow
the contours of the IGP’s shortest path. Traffic engineering is not possible with LDP.

Layer 2 Options
Three classifications exist for Layer 2 DCIs:
1. No MAC learning by the Provider Edge (PE) device: This type of layer 2 DCI does not require that the PE devices
learn MAC addresses.
2. Data plane MAC learning by the PE device: This type of DCI requires that the PE device learns the MAC
addresses of both the local data center as well as the remote data centers.
3. Control plane MAC learning - This type of DCI requires that a local PE learn the local MAC addresses using the
control plane and then distribute these learned MAC addressed to the remote PEs.
Layer 3 Options
A Layer 3 DCI uses routing to interconnect data centers. Each data center must maintain a unique IP address space. A
Layer 3 DCI can be established using just about any IP capable link. Another important consideration for DCIs is
incorporating some level of redundancy by using link aggregation groups (LAGs), IGPs using equal cost multipath, and BGP or
MP-BGP using the mutipath or multihop features.

Customer Edge Devices

CE devices are located in the DC and usually perform standard switching or routing functions. CE devices can interface to PE
routers using virtually any Layer 2 technology and routing protocol.

Provider Edge Devices

PE devices are located at the edge of the data center or at the edge of a Service Provider’s network. They interface to the CE
routers on one side and to the IP/MPLS core routers on the other. PE devices maintain site-specific VPN route and forwarding
(VRF) tables. In a Layer 3 VPN scenario, the PE and CE routers function as routing peers (RIP, OSPF, BGP, etc), with the PE
router terminating the routing exchange between customer sites and the IP/MPLS core. In a Layer 2 VPN scenario, the PE’s
CE-facing interface is configured with matching VLAN-tagging to the CE’s PE-facing interfaces and any frames received from
the CE device will be forwarding over the MPLS backbone to the remote site.
Information is exchanged between PE routers using either MP-BGP or LDP. This information exchange allows the PE routers to
map data to and from the appropriate MPLS LSPs traversing the IP/MPLS core.
PE routers, and Ingress and Egress LSRs, use MPLS LSPs when forwarding customer VPN traffic between sites. LSP tunnels
in the interconnect network separate VPN traffic in the same fashion as PVCs in a legacy ATM or Frame Relay network.

Provider Routers
Provider (P) routers are located in the IP/MPLS core. These routers do not carry VPN data center routes, nor do they interface
in the VPN control and signaling planes. This is a key aspect of the RFC 4364 scalability model; only PE devices are aware of
VPN routes, and no single PE router must hold all VPN state information.
P routers are involved in the VPN forwarding plane where they act as label-switching routers (LSRs) performing label
swapping (and popping) operations.

VPN Site
A VPN site is a collection of devices that can communicate with each other without the need to transit the IP/MPLS
backbone (i.e., a single data center). A site can range from a single location with one switch or router to a network consisting
of many geographically diverse devices.

MPLS VPN Packet

The slide shows how VPN data is encapsulated in an MPLS VPN scenario (MPLS L3 VPN as an example). PE1 receives IP
packets destined for CE2. PE1 performs a lookup in its Green VRF table (the table associated with the PE-CE interface). The
route to CE2’s address should list three things in terms nexthop. It will list the outgoing interface and the inner and outer
label that should be pushed onto the IP packet. The outer label is swapped by the P routers along the way to deliver the
MPLS packet to PE2. P3 performs a penultimate hop pop leaving only single labeled packets and forwards them to PE2. PE2
receives the labeled packets, pops the inner label, and uses the inner label to determine which VRF table to use (PE2 might
have many VRF table). PE2 performs a lookup on the Green VRF table (because label 1000=Green VRF) and forwards the
original IP packets to CE2.

MPLS VPN Stitching

Sometimes data from one CE may need to pass through multiple VPNs before reaching the remote CE. The top diagram
shows the situation where packets enter the green VPN at PE1, get decapsulated at PE2, and then forwarded in their original
format to PE3 where they enter the red VPN. From PE2’s perspective, PE3 is the CE for the green VPN. From PE3’s
perspective, PE2 is the CE for the red VPN. You might think that you need 2 physical devices, PE2 and PE3 to “stitch” the two
VPNs together. Well, as the bottom diagram shows, you can actually “stitch” two VPNs together using a single MX Series
router. You can use the logical tunnel interface feature which are internal interfaces that allow you to connect two virtual
routers together. The two virtual routers enabled on the MX Series device would simply perform the same functions as PE2
and PE3 in the top diagram.

The LSP
The next few slides are going to discuss the details of MPLS Layer 3 VPNs. One thing to remember with Juniper Networks
routers is that once an LSP is established (from PE1 to PE2 in the diagram) the ingress PE will install a host route (/32) to the
loopback interface of the egress router in the inet.3 with a next-hop of the LSP (i.e. outbound interface of LSP and push a
label). This default behavior means that not all traffic entering PE1 can get routed through the LSP. So what traffic gets
routed over the LSP then?
Looking at the example in the slide, remember PE1 and PE2 are MP-BGP peers. That means that PE2 will advertise VPN
routes to PE1 using MP-BGP which will have a BGP next-hop of 2.2.2.2 (PE2’s loopback). For these VPN routes to be usable
by PE1, PE1 must find a route to reach 2.2.2.2 in the inet.3 table. PE1 will not look in inet.0 to resolve the nexthop of MPLS
VPN MP-BGP routes.

VPN-IPv4 Route
The VPN-IPv4 route has a very simple purpose which is to advertise IP routes. PE2 installs locally learned routes in its VRF
table. That includes the directly connected PE-CE interface as well as any routes PE2 learns from CE2 (RIP, OSPF, BGP, etc).
Once PE2 has locally learned routes in its VRF table, it advertises it (based on configured policy) to remote PEs and attaches
a target community, target community “Orange” in the example. PE1, upon receiving the route must decide on whether it
should keep the route. It makes this decision based on resolving the BGP nexthop in inet.3 as well as looking at the received
route target community. PE1, in order to accept and use this advertisement, must be configured with an import policy that
accepts routes tagged with the “Orange” target community. Without a configured policy that matches on the “Orange” route
target, PE1 would just discard the advertisement. So, at a minimum, each VRF on each participating PE for a given VPN must
be configured with an export policy that attaches a unique target community to routes and also configured with an import
policy that matches and accepts advertisements based on that unique target community.

Route Distinguisher Formats Defined

The route distinguisher can be formatted two ways:
• Type 0: This format uses a 2-byte administration field that codes the provider’s autonomous system number,
followed by a 4-byte assigned number field. The assigned number field is administered by the provider and should
be unique across the autonomous system.
• Type 1: This format uses a 4-byte administration field that is normally coded with the router ID (RID) of the
advertising PE router, followed by a 2-byte assigned number field that caries a unique value for each VRF table
supported by the PE router.
The examples on the slide show both the Type 0 and Type 1 route distinguisher formats. The first example shows the 2-byte
administration field with the 4-byte assigned number field (Type 0).

Route Target Community

Each VPN-IPv4 route advertised by a PE router contains one or more route target communities. These communities are added
using VRF export policy or explicit configuration.
When a PE router receives route advertisements from remote PE routers, it determines whether the associated route target
matches one of its local VRF tables. Matching route targets cause the PE router to install the route into the VRF table whose
configuration matches the route target.
Because the application of policy determines a VPN’s connectivity, you must take extra care when writing and applying VPN
policy to ensure that the tenant’s connectivity requirements are faithfully met.

VRF Export Policy

For each VRF, you must apply a VRF export policy. A VRF export policy determine which routes in a PE’s VRF table will be
advertised to remote PEs. A VRF export policy gives you complete control over the connectivity from one site to another
simply by either advertising or not advertising particular routes to a remote site. Another important function of the VRF
export policy is that it will also cause the advertised routes to be tagged with a target community. In the slide, PE2 has a
locally learned route (10.1.2/24, the network between PE2 and CE2) in its VRF table. To ensure CE1 and PE1 can send data
to CE2, PE2 has an VRF export policy applied to its IBGP neighbor, PE1, which advertises locally learned routes tagged with
the target community, target:1:1. The next slide shows PE1’s process of installing the VPN-IPv4 route in its own VRF table.

VRF Import Policy

The slide shows the process that PE1 goes through when it receives a VPN-IPv4 route advertisement PE2. There is an
assumption that PE1 is configured with a VRF import policy that accepts the same route target, target:1:1, that PE2 is
attaching to its VPN routes.

DCI Options for a VXLAN Overlay

The slide highlights the topic we discuss next.

EVPN DCI Option

This slide shows the 4 options for DCI when the data centers are enabled for VXLAN using EVPN signaling. The next few
slides discuss each option in detail.

Over the Top (OTT) of an L3VPN

The slide shows an example of the signaling and data plane when using EVPN/VXLAN over a Layer 3 VPN. The two MX Series
devices are the PE routers for the Layer 3 VPN. The layer 3 VPN can be over a private MPLS network or could be a purchased
Service Provider service. From the two QFX perspectives, they are separated by an IP network. The QFXs simply forward
VXLAN packets between each other based on the MAC addresses learned through EVPN signaling. The MX devices have an
MPLS layer 3 VPN between each other (Bidirectional MPLS LSPs, IGP, L3 VPN MP-BGP routing, etc). The MXs advertise the
local QFX’s loopback address to the other MX.
When forwarding data from West to East, QFX1 takes a locally received Ethernet frame and encapsulates it in a VXLAN
packet destined to QFX2’s loopback address. MX1 performs a lookup for the received packet on the VRF table associated
with the VPN interface (the incoming interface) and encapsulates the VXLAN packet into two MPLS headers (outer for MPLS
LSP, inner for MX2 VRF mapping). Upon receiving the MPLS encapsulated packet, MX2 uses the inner MPLS header to
determine the VRF table so that it can route the remaining VXLAN packet to QFX2. QFX2 strips the VXLAN encapsulation and
forwards the original Ethernet frame to the destination host.

EVPN Stitching over MPLS Network

The slide shows an example of the signaling/data plane when using EVPN stitching between three EVPNs; EVPN/VXLAN
between QFX1 and MX1, EVPN/MPLS between MX1 and MX2, and an EVPN/VXLAN between MX2 and QFX2. Each EVPN is
signaled using EVPN MP-BGP signaling and are stitched together on the MX devices using logical tunnel interfaces.
packet destined to MX1’s loopback address. MX1 strips the VXLAN encapsulation and forwards the remaining Ethernet
frame out of a logical tunnel interface. MX1 receives the Ethernet frame over the associated (looped) logical tunnel
interface. MX1 takes the locally received Ethernet frame and encapsulates it in two MPLS headers (outer for MPLS LSP,
inner for MX2 VRF mapping). Upon receiving the MPLS encapsulated packet, MX2 uses the inner MPLS header to determine
the appropriate VRF and outgoing interface. MX2 forwards the remaining Ethernet frame out of a logical tunnel interface.
MX2 receives the Ethernet frame over the associated (looped) logical tunnel interface. MX2 takes the locally received
Ethernet frame and encapsulates it in a VXLAN packet destined to QFX2’s loopback address. QFX2 strips the VXLAN
encapsulation and forwards the remaining Ethernet frame to the destination host.

Over the Top (OTT) of a Public IP Network

The slide shows an example of the signaling/data plane when using EVPN stitching between three EVPNs; EVPN/VXLAN
between QFX1 and MX1, EVPN/VXLAN between MX1 and MX2, and EVPN/VXLAN between MX2 and QFX2. Each EVPN is
signaled using EVPN MP-BGP signaling and are stitched together on the MX devices using logical tunnel interfaces.
packet destined to MX1’s loopback address. MX1 strips the VXLAN encapsulation and forwards the remaining Ethernet
frame out of a logical tunnel interface. MX1 receives the Ethernet frame over the associated (looped) logical tunnel
interface. MX1 takes the locally received Ethernet frame and encapsulates it in a VXLAN packet destined to MX2’s loopback
address. MX2 strips the VXLAN encapsulation and forwards the remaining Ethernet frame out of a logical tunnel interface.
MX2 receives the Ethernet frame over the associated (looped) logical tunnel interface. MX2 takes the locally received
Ethernet frame and encapsulates it in a VXLAN packet destined to QFX2’s loopback address. QFX2 strips the VXLAN
encapsulation and forwards the remaining Ethernet frame to the destination host.

EVPN over IP
The slide shows an example of the signaling/data plane when using EVPN/VXLAN over an IP network. EVPN MP-BGP is used
to synchronize MAC tables.
packet destined to MX1’s loopback address. QFX2 strips the VXLAN encapsulation and forwards the remaining Ethernet
frame to the destination host.

Option 1 Example Topology: Part 1

The slide shows the topology that will serve as the underlay network. It is based on EBGP routing between the routers in the
same data center. In AS 64555, PE and P routers will run OSPF to advertise each router’s loopback address. They will run
LDP to automatically establish MPLS LSPs to each other’s loopback address. Finally, each PE will establish a VPN-IPv4
MP-IBGP session with each other. The PEs will exchange locally learned routes (the loopback addresses of the Leaf nodes)
so that they Leaf nodes can establish the overlay network (next slide).

Option 1 Example Topology: Part 2

Once the underlay network is established, each Leaf node will have learned a route from the local PE (using the EBGP
session) to reach the loopback address (VTEP source address) of the remote Leaf. Leaf1 and Leaf2 will act as VXLAN Layer
2 Gateways and also establish an EVPN MP-IBGP session with each other to exchange EVPN routes to advertise locally
learned MACs to the remote Leaf. Host A and Host B will be able to communicate as if they were on the same LAN segment.

PE1’s MPLS Configuration

The slide shows the MPLS configuration of PE1.

PE1’s MPLS Status

Any easy way to see that there are MPLS LSPs established when using LDP signaling is to view the inet.3 table. If there is a
route in the inet.3 table to the remote PE’s loopback address then there is a unidirectional MPLS LSP established to the
remote PE. Remember, there needs to be an MPLS LSP established in each direction so you must check the inet.3 table on
both PEs.

VRF Configuration
The slide shows the VRF configuration for PE1. Notice the use of the vrf-target statement. Originally, VRF import policies
could only be enabled by writing explicit policies under [edit policy-options] and applying them using the
vrf-import and vrf-export statements. However, more recent versions of the Junos Operating System allow you to
skip those steps and simple configure a single vrf-target statement. The vrf-target statement actually enables two
hidden policies. One policy is an VRF export policy that takes all locally learned routes in the VRF (direct interface routes as
well as routes learned from the local CE) and advertises them to the remote PE tagged with the specified target community.
The other policy is an VRF import policy that will accept all VPN-IPv4 routes learned from remote PEs that are tagged with the
specified target community.

MP-BGP Routing
The slide shows how to enable VPN-IPv4 signaling between PEs. Use the show bgp summary command to verify that the
MP-BGP neighbor relationship is established and that the PE is receiving routes from the remote neighbor.

VRF Table
Remember, the main purpose of establishing an underlay network and the DCI is to ensure that the routers in each site can
reach the loopback addresses (VTEP source addresses) of the remote Leaf nodes. The slide shows that PE1 has learnt the
loopback address of Leaf2.

Leaf1 Configuration
The slide shows the underlay and overlay network configuration of Leaf1. Leaf2 would be configured very similarly.

We Discussed:
• The meaning of the term Data Center Interconnect;
• The control and data plane of an MPLS VPN; and
• The DCI options when using a VXLAN overlay with EVPN signaling.

Review Questions:
1.
2.
3.

Lab: Data Center Interconnect

The slide provides the objective for this lab.

Answers to Review Questions
1.
A DCI can be provided by a point-to-point link, and IP network, or an MPLS network.
2.
The VPN-IPv4 NLRI includes an MPLS label, the route distinguisher, and an IP prefix. A target community is also tagged to the route
but it is not officially part of the NLRI.
3.
When the transport network of a DCI is a public IP network, the option available for a DCI is option 3.

Data Center Interconnect Options

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Center Interconnect Options

Uploaded by

Copyright:

Available Formats

Advanced Data Center Switching

Chapter 6: Data Center Interconnect

Chapter 6–2 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–3

What Is a Data Center Interconnect?

Chapter 6–4 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–5

Chapter 6–6 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–7

Chapter 6–8 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–9

MPLS VPN Review

Chapter 6–10 • Data Center Interconnect www.juniper.net

MPLS Shim Header: Part 1

www.juniper.net Data Center Interconnect • Chapter 6–11

MPLS Shim Header: Part 2

Chapter 6–12 • Data Center Interconnect www.juniper.net

Labels Have Only Local Significance

www.juniper.net Data Center Interconnect • Chapter 6–13

Reserved MPLS Label Values

Chapter 6–14 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–15

Implicit and Explicit Null Labels

Chapter 6–16 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–17

MPLS Label Operations

Chapter 6–18 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–19

Ingress Label Switching Router

Chapter 6–20 • Data Center Interconnect www.juniper.net

Transit Label Switching Routers

www.juniper.net Data Center Interconnect • Chapter 6–21

The Label Information Base

Chapter 6–22 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–23

Egress Label Switching Router

Chapter 6–24 • Data Center Interconnect www.juniper.net

Label Information Base

www.juniper.net Data Center Interconnect • Chapter 6–25

Label to Forwarding Action Mappings

Chapter 6–26 • Data Center Interconnect www.juniper.net

Label Distribution Protocols

www.juniper.net Data Center Interconnect • Chapter 6–27

Chapter 6–28 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–29

Chapter 6–30 • Data Center Interconnect www.juniper.net

Customer Edge Devices

www.juniper.net Data Center Interconnect • Chapter 6–31

Provider Edge Devices

Chapter 6–32 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–33

Chapter 6–34 • Data Center Interconnect www.juniper.net

MPLS VPN Packet

www.juniper.net Data Center Interconnect • Chapter 6–35

MPLS VPN Stitching

Chapter 6–36 • Data Center Interconnect www.juniper.net

www.juniper.net Data Center Interconnect • Chapter 6–37

Chapter 6–38 • Data Center Interconnect www.juniper.net

Route Distinguisher Formats Defined

www.juniper.net Data Center Interconnect • Chapter 6–39

Route Target Community

Chapter 6–40 • Data Center Interconnect www.juniper.net

VRF Export Policy