You are on page 1of 52

MPLS Management

Technology Overview and Applications

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

MPLS Operations Lifecycle


Build and plan the network
Capacity planning and resource monitoring
Internal-Focused Operations One-time Strategic Operations External-Focused Operations

Monitor the network


Node/link failure detection May impact multiple services

Network Configuration and Planning

Service Configuration and Planning

Provision new services and maintain existing services


Edge/service node configuration

Network Monitoring

Service Monitoring

Monitor service
End-to-end monitoring Linked to customer SLAs

Ongoing Tactical Operations

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

What s Needed for MPLS management?


What s needed beyond the basic MPLS CLI?
CLI used for basic configuration and trouble shooting (show commands) VRF-Aware commands for traditional troubleshooting tools

Traditional management tools: MIBs to provide management information for SNMP management applications
MIB counters and Trap notifications form MPLS

New management tools: MPLS OAM -> for reactive trouble shooting
Ping and trace capabilities of MPLS label switched paths

Monitoring and Performance Management via MPLS Aware Netflow and IP SLA for MPLS L3 VPN Automated MPLS OAM -> for proactive trouble shooting
Automated LSP ping/trace via Auto IP SLA

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Embedded Management for MPLS OAM Tools


LSP Ping and Trace for LDP, RSVP distribution mechanisms and VCCV
Deployment Scope Standard Compliance

OAM Feature

Cisco Value Add


LSP ping/trace for LDPsignaled LSPs OAM automation via IP SLA Discovery of available LDP ECMP LSPs between PEs Automation via IP SLA LSP ping/trace for RSVP signaled LSPs OAM automation via IP SLA Use of LSP Ping for liveliness detection Use of BFD over VCCV control channel for failure detection

MPLS LSP Ping/Trace for LDP IPv4 FECs

RFC4379

LDP MPLS Core


LSP Multipath (ECMP) Tree Trace RFC4379

Traffic Engineered MPLS Core

MPLS LSP Ping/Trace for RSVP IPv4 FECs VCCV LSP Ping (single and multi-segment PW) VCCV BFD (incl. Fault, AC Notification)

RFC4379 RFC 5085 IETF Draft IETF Draft

PW3E

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

MPLS LSP Ping/Traceroute


Detect MPLS traffic black holes or misrouting Isolate MPLS faults Verify data plane against the control plane Detect MTU of MPLS LSP paths MPLS LSP ping (ICMP) for connectivity checks

Requirement

Solution

MPLS LSP traceroute for hop-by-hop fault localization MPLS LSP traceroute for path tracing IPv4 LDP prefix, VPNv4 prefix: tunnel monitoring

Applications

TE tunnel L2 VPNs

RFC Standards
BRKRST-1101

RFC 4377, RFC 4378, RFC4379

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

MPLS OAM
Embedded management capabilities used for node-specific and end-toend MPLS failure detection A broken LSP will affect end to end connectivity and services, it is difficult to troubleshoot an MPLS failure:
Requires the operator to do manual/hop-by-hop work

MPLS
50 R3 LSP Broken 49 R2

Various reasons for an LSP to break:


Broken LDP adjacency MPLS not enabled (globally or per interface) Mismatched labels Software/hardware corruption

R1

MPLS OAM facilitates and speeds Up troubleshooting of MPLS failures Principlesimilar to traditional (ICMP based) tools:
LSP Ping: based on echo request and echo reply LSP Trace: packets with incremental TTL Virtual Circuit Connection Verification (VCCV): end-to-end fault detection and diagnostics for an emulated PW service
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

MPLS and VRF Aware Ping and Traceroute


Uses MPLS Encapsulation, if Available, to Verify Reachability of IP Prefix through; For the Traceroute : Labels Used along the LSP Are Displayed

The Cisco IOS CLI VRF ping and Cisco IOS CLI VRF traceroute : => ping and traceroute of IP prefixes stored in a VRF
VPN/VRF routing table used to route the ICMP packet IP datagram uses the MP BGP label and the LDP label ICMP echo reply/request or Traceroute mechanisms ICMP extensions for MPLS

Benefits of VRF aware ping and traceroute :


VPN prefixes test and troubleshooting capacity Rapid fault detection for MPLS VPNs Ease of use with a variety of servers and network equipment Functionality included in Cisco IOS feature set
7

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Traceroute from R1 to R4 in MPLS Environment


MPLS stack copied and top label switched, TTL set to 255

Label Used to Reach R4->67

Label Used to Reach R4->61

TTL=2

Label Used to Reach R4->Pop

R1

MPLS Packet Destination R4 and TTL=1

R2

Label Used to Reach R1->29

R3

Label Used to Reach R1->22

R4

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

VRF aware traceroute from PE1 to PE2


PE1
VRF NMtest Outgoing tag for 135.15.252.10->50

P1
Incoming tag ->50 Outgoing tag for 135.15.252.1->49

P2

P3

Incoming tag ->49 Incoming tag ->35 Outgoing tag for Outgoing tag for 135.15.252.1->35 135.15.252.1->pop tag

VRF IP add 10.0.0.1 Loopback


135.15.252.10

PE2

PE1 looks at the VRF routing table and finds 10.0.0.0 [200/0] via 135.15.252.10, 00:40:19
PE1#traceroute vrf NMtest ip 10.0.0.1 Type escape sequence to abort. MP BGP label->82 Tracing the route to 10.0.0.1 1 135.15.202.1 [MPLS: Labels 50/82 Exp 0] 0 msec 0 msec 0 msec 2 10.200.14.1 [MPLS: Labels 49/82 Exp 0] 0 msec 0 msec 0 msec 3 10.200.12.2 [MPLS: Labels 35/82 Exp 0] 0 msec 0 msec 0 msec 4 10.0.0.1 0 msec 0 msec * PE1#
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Why MPLS Ping


ICMP Ping works in MPLS/IP environments The ICMP Packets can be encapsulated in MPLS Labels However ICMP ping may not detect LSP failures ICMP packet will be forwarded as long as IP path is available So LSP forwarding may be broken, but ICMP will succeed MPLS OAM packets carry lot more information Helps in targeted debugging of LSP paths

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

MPLS AOM

50 R3 LSP Broken

49 R1 R2

Various reasons for LSP to break


Broken LDP adjacency MPLS not enabled Mismatch labels Software/hardware corruption

Regular IP ping will be successful


BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

11

MPLS OAM Packets Format


Principlesimilar to traditional (ICMP-based) tools
LSP Ping: Based on echo request and echo reply LSP Trace: Packets with incremental TTL

LSP Ping/Trace do not use ICMP packets


New packet format specifically designed for MPLS OAM IPv4 (IPv6) UDP packets with port 3503 UDP packets : MPLS echo-req. or MPLS echo-reply

Packets contain Control Information and Diagnostic Information from LSR at Failure Point for Fault Localization and Many Options to provide for Efficient Troubleshooting information
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

12

MPLS OAM Theory of Operation (1 of 2)


MPLS Echo-Req
50 SA DA=127/8 Echo SA=Source Addr DA=Destination Addr SA DA=127/8 Echo

50 R4

49 R2 R1 MPLS Echo-Reply

R3

LSP Broken

Label stack is same as used by the LSP and this makes the echo to be switched in-band of LSP Same label stack takes the same path as MPLS data Where the LSP is broken, the Packet Is consumed by the router trying to forward the packet using the IP header
IP-DA = Loopback

In this case R2 would not forward the echo-req to R1, but rather consumes the packet and reply to it accordingly
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

13

MPLS OAM Theory of Operation (2 of 2)


MPLS Echo-Req
50 SA DA=127/8 Echo 49 SA DA=127/8 Echo SA DA=127/8 Echo

50 R3
SA=Source Addr DA=Destination Addr

49 R4 R2 R1 MPLS Echo-Reply

LSP reply will be generated as an IP packet which may use an LSP path back if available Reply contains Return Code information An Echo reply, which may or not be labeled, Information is displayed on R3 which initiated the MPLS OAM test (probe) Diagnostic Capability at Failure Point for Fault Localization and More Options to provide for Efficient Troubleshooting information
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

14

Packet Format of an MPLS LSP Echo


IP/MPLS Header

Value
1

Meaning
MPLS Echo Request MPLS Echo Reply

Version Number
Message Type Reply Mode

Must Be Zero
Return Code Rtrn Subcode

Sender s Handle Sequence Number Timestamp Sent (NTP Seconds) Timestamp Sent (NTP Fraction of usecs) Timestamp Received (NTP Seconds) Timestamp Received (NTP Fraction of usecs) TLVs

Version Number: It s Set to One Message Type: Message Type Field Tells Whether the Packet Is an MPLS Echo Request or MPLS Echo Reply
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

15

Packet Format of an MPLS LSP Echo (Cont.)


IP/MPLS Header

Value
1

Meaning
Do Not Reply Reply via an IPv4 UDP Packet Reply via an IPv4 UDP packet with Router Alert

Version Number
Message Type Reply Mode

Must Be Zero
Return Code Rtrn Subcode

Sender s Handle Sequence Number Timestamp Sent (NTP Seconds) Timestamp Sent (NTP Fraction of usecs) Timestamp Received (NTP Seconds) Timestamp Received (NTP Fraction of usecs) TLVs

Reply Mode: The Reply Mode Is Used to Control How the Target Router Replies to MPLS Echo Request
Default Is to Send a Reply with a Value of 2
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

16

Return Code
IP/MPLS Header Version Number
Message Type Reply Mode
Value Meaning The Error Code Is Contained in the Error Code TLV Malformed Echo Request Received One Or More of the TLVs Was Not Understood Replying Router Is an Egress for the FEC Replying Router Has No Mapping for the FEC Replying Router Is Not One of the "Downstream Routers" Replying Router Is One of the "Downstream Routers", and Its Mapping for this FEC on the Received Interface Is the Given Label

Must Be Zero
Return Code Rtrn Subcode
0 1 2

Sender s Handle Sequence Number Timestamp Sent (NTP Seconds) Timestamp Sent (NTP Fraction of usecs) Timestamp Received (NTP Seconds) Timestamp Received (NTP Fraction of usecs) TLVs

The router initiating the LSP ping/trace would set the return code to zero The replying router would set it accordingly based on the following table
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

Target FEC Stack TLV


Sub Type Length
1 2 3 4 5 6 7 9
BRKRST-1101

ValueField
LDP IPv4 Prefix LDP IPv6 Prefix RSVP IPv4 Session Query RSVP IPv6 Session Query

This is a list of sub-TLVs Target stack TLV is included only in the echo-request and not in echoreply

5 17 20

56 13 25 10

Reserved VPN IPv4 Prefix VPN IPv6 prefix L2 Circuit ID


2009 Cisco Systems, Inc. All rights reserved. Cisco Public

18 18

LDP IP V4 Prefix (Sub-TLV)


The sender puts the IPv4 prefix for which we are selecting the LSP in the echo request The length field defines the mask for the prefix

7 8 0x0001
Prefix Length

15 16 Length = 5 Ipv4 Prefix 31

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

RSVP IPv4 (Sub-TLV)


Tunnel (Tun) endpoint address is the destination address of the TE Tunnel being used by the LSP ping/trace Tunnel ID is the TE tunnel number Extended Tunnel ID is usually the source address of the TE tunnel IPv4 tunnel sender address is again the source address of the TE tunnel

15 16 0x0003 Length = 20 IPv4 Tunnel Endpoint Address Must Be Zero Tunnel ID Extended Tunnel ID IPv4 Tunnel Sender Address LSP ID Must Be Zero

31

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

L2 Circuit Type (Sub-TLV)


Remote PE address is the address of the destination of AToM tunnel Source address is the LDP ID PWID Type is the VC Type PWID is the VC ID configure for the AtoM Tunnel

0x0009 Remote PE Address Source PE Address PWID Type PWID Length=4

Length = 16

PWID

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Downstream Mapping TLV


MTU Address Type Downstream IP Address ID Downstream Interface Address Multipath Type Depth Limit Multipath Length DS Flags

MTU: MTU of the outgoing interface Address type: IPv4, IPv6, Unnumbered IPv4, Unnumbered IPv6 DS Flags: Request for ILS TLV, etc. Downstream IP Address: IP Address of the Downstream Router Downstream Interface Address: IP address of the outgoing interface for that LSP Multipath Information: Encoded IP address information about the outgoing interface Downstream labels: The outgoing labels for the LSP on the downstream router

Multipath Information (Variable Length) Downstream Label Downstream Label Protocol Protocol

Both Request and Reply Packets Responding router uses incoming DSMAP for label verification Responding routers puts in one DSMAP for each outgoing interface New draft for Detailed DSMAP (DDMAP)

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

22

LSP Ping (Cont.)


R3#ping mpls ? ipv4 pseudowire traffic-eng Target specified as an IPv4 address Target VC specified as an IPv4 address and VC ID Target specified as TE tunnel interface

The Above Are Three of the Sub-TLVs That We Have Already Discussed Earlier

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

LSP Ping: IPv4 FEC


R3#ping mpls ipv4 10.200.0.1/32 ? destination Destination address or address range exp EXP bits in mpls header interval Send interval between requests in msec pad Pad TLV pattern repeat Repeat count reply Reply mode size Packet size source Source specified as an IP address sweep Sweep range of sizes timeout Timeout in seconds ttl Time to live verbose Verbose mode for ping output
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

24

Validation of PE-PE MPLS Connectivity


Connectivity of LSP path(s) between PE routers can be validated using LSP ping (ping mpls command via CLI)
pe1>ping mpls ipv4 10.1.2.249/32 Sending 5, 100-byte MPLS Echos to 10.1.2.249/32, timeout is 2 seconds, send interval is 0 msec: Codes: '!' - success, 'Q' - request not sent, '.' - timeout, 'L' - labeled output interface, 'B' - unlabeled output interface, 'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch, PE1 'M' - malformed request, 'm' - unsupported tlvs, 'N' PE2 label - no entry, P1 P2 'P' - no rx intf label prot, 'p' - premature termination of LSP, 'R' - transit router, 'I' - unknown upstream index, 'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort. !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 284/294/300 ms
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

25

Troubleshooting Using LSP Ping (IPv4)


(MPLS Disabled at the Egress Router)
P R3 LSP Broken R4 MPLS Disabled on R4 If a Regular Ping Is Done from R3 to R4, It Would Be Successful But an LSP Ping Would Fail
R3#ping mpls ip 10.200.0.4/32 Sending 5, 100-byte MPLS Echos to 10.200.0.1/32, timeout is 2 seconds, send interval is 0 msec: Codes: '!' - success, 'Q' - request not transmitted, '.' - timeout, 'U' - unreachable, 'R' - downstream router but not target Type escape sequence to abort. UUUUU Success rate is 0 percent (0/5) R3#
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

26

Loadbalancing (Cont.)
R3#sh mpls forwarding-table 10.200.0.1 Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 27 20 10.200.0.1/32 0 PO0/0 point2point 23 10.200.0.1/32 0 PO1/0 point2point R3#
R3#trace mpls ip 10.200.0.1/32 destination 127.0.0.3 Tracing MPLS Label Switched Path to 10.200.0.1/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not transmitted, '.' - timeout, 'U' - unreachable, 'R' - downstream router but not target Type escape sequence to abort. 0 10.200.134.3 MRU 4470 [Labels: 23 Exp: 0] R 1 10.200.14.4 MRU 1504 [implicit-null] 14 ms ! 2 10.200.14.1 5 ms
BRKRST-1101

R3#trace mpls ip 10.200.0.1/32 destination 127.0.0.1 Tracing MPLS Label Switched Path to 10.200.0.1/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not transmitted, '.' - timeout, 'U' - unreachable, 'R' - downstream router but not target Type escape sequence to abort. 0 10.200.123.3 MRU 4470 [Labels: 20 Exp: 0] R 1 10.200.12.2 MRU 1504 [implicit-null] 12 ms ! 2 10.200.12.1 3 ms
Cisco Public

2009 Cisco Systems, Inc. All rights reserved.

27

Troubleshooting Using LSP Ping (RSVP IPv4)


MPLS Echo-req R4
Pinging from R4 to R3 through TE Tunnel R4#ping mpls Traffic-eng Tunnel 1 R3# *Jan 21 13:43:56.200: LSPV: Echo Hdr decode: version 1, msg type 1, reply mode 2 , return_code 0, return_subcode 0, sender handle EA00000A, sequence number 1, ti mestamp sent 13:58:08 UTC Wed Jan 21 2004, timestamp rcvd 00:00:00 UTC Mon Jan 1 1900 *Jan 21 13:43:56.200: LSPV: tlvtype 1, tlvlength 24 *Jan 21 13:43:56.204: LSPV: RSVP IPV4 FEC decode: srcaddr 10.200.0.4, destaddr 1 0.200.0.3, tun id 1, ext tun id 180879364, lsp id 4142 *Jan 21 13:43:56.204: LSPV: Target FEC stack length = 24, retcode = 3 *Jan 21 13:43:56.204: LSPV: tlvtype 3, tlvlength 4 *Jan 21 13:43:56.204: LSPV: Pad TLV decode: type 1, size 4 *Jan 21 13:43:56.204: LSPV: Echo Hdr encode: version 1, msg type 2, reply mode 2 , return_code 3, return_subcode 0, sender handle EA00000A, sequence number 1, ti mestamp sent 13:58:08 UTC Wed Jan 21 2004, timestamp rcvd 13:43:56 UTC Wed Jan 2 1 2004
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

TE Tunnel R3 MPLS echo-reply

28

Generating an LSP Trace


For LSP Trace we generate an mpls-echo request and increment the TTL by 1 starting at 1 Within the echo-req we add the downstream TLV The TTL of the outermost label is set to 1 and then incremented by 1 on every other request that is being send out The downstream routers, receiving the echo-req, would decrement the TTL by 1 and if it expires and the router is one of the downstream router it would reply with a return code of 6 When the echo-req finally reaches the destination router it would reply with a return code of 3

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

LSP Trace: Path/Tree Trace (Cont.)


R3 R1 R2 R4 R5 R7 R6 R9

R8

Trace Can Be Divided into Two Types


Path trace would give us information of only one path out of all the possible ECMP paths In the above example if I do a path trace from R1 to R6; I might only be reported about R1-R2-R3-R4-R5-R6 Tree trace returns ALL of the possible paths between one source and destination So in the above case the LSP (tree) trace would give us information about both the paths R1-R2-R3-R4-R5-R6 and R1-R2-R7-R8-R5-R6 Path trace support available since 27S; tree trace would be supported as part of subsequent releases
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

30

LSP Trace (Cont.)


R3#traceroute mpls ? ipv4 traffic-eng Target specified as an IPv4 address Target specified as TE tunnel interface

The Above Are Two of the Sub-TLVs that We Have Already Discussed Earlier

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

LSP Trace (Cont.)


R1#traceroute mpls ipv4 10.200.0.3/32 ? destination Destination address or address range exp reply source timeout ttl EXP bits in mpls header Reply mode Source specified as an IP address Timeout in seconds Maximum time to live

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

32

Troubleshooting Using LSP Trace (IPv4)


R4 R3 R1 R2

There is an intermittent response for the data traffic using the LSP R3-R4-R1-R2 Sweeping LSP ping tells us that packets over 1500 are failing
Now if I do a regular trace, I ll get the following R3#tracer 10.200.0.2 Type escape sequence to abort. Tracing the route to 10.200.0.2 1 10.200.34.4 [MPLS: Label 44 Exp 0] 0 msec 0 msec 0 msec 2 10.200.14.1 [MPLS: Label 22 Exp 0] 0 msec 0 msec 0 msec 3 10.200.12.2 0 msec * 0 msec R3#
BRKRST-1101

But if I do an LSP trace I get the following R3#tracer mpls ip 10.200.0.2/32 Tracing MPLS Label Switched Path to 10.200.0.2/32, timeout is 2 seconds Codes: '!' - success, 'Q' - request not transmitted, '.' - timeout, 'U' - unreachable, 'R' - downstream router but not target Type escape sequence to abort. 0 10.200.34.3 MRU 4470 [Labels: 44 Exp: 0] R 1 10.200.14.4 MRU 1500 [Labels: 22 Exp: 0] 4 ms R 2 10.200.12.1 MRU 4474 [implicit-null] 15 ms ! 3 10.200.12.2 20 ms
Cisco Public

2009 Cisco Systems, Inc. All rights reserved.

33

Cisco IOS IP SLAs LSP Health Monitor


LSP Health Monitor
Automatic connectivity testing of label switch paths (LSP) between PE devices Combined end-to-end latency and connectivity testing utilizing LSP Ping and LSP Trace Ease of use with automatic configuration of IP SLAs operations based on BGP neighbors Equal cost multi-path discovery and measurement (future release)

MPLS Core Health Monitoring


Real-time automatic health monitoring for the L3 MPLS VPN network Reducing Operational expense and problem isolation times Locating and isolating MPLS core forwarding and path issues Measurement of all equal paths between PE edges measuring all customer traffic paths (future release)

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

34

Automated MPLS OAM


Automatic MPLS OAM probes between PE routers
Automatic discovery of PE targets via BGP next-hop discovery Automatic discovery of all available LSP paths for PE targets via LSP multi-path trace Scheduled LSP pings to verify LSP path connectivity 3 consecutive LSP ping failures result in SNMP Trap notification
PE1 - MPLS OAM Probe PE2 - MPLS OAM Probe PE3 - MPLS OAM Probe

PE3

P1 PE1

P2 PE2

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

35

Automated LSP Verification


IPSLA VPN
IP SLAs

CE

LSP-ping

IP SLAs LSP Health Monitor Proactive end-to-end LSP verification


Standards-based LSP-Ping Automatic Neighbor PE discovery (per VRF) LSP Path Discovery for each Egress PE (including multiple paths)

MPLS

100s of PEs Ingress + Egress


IP SLAs

Scalability
Fast retry on failure Ease of configuration- automated test setup Intelligent group-based notifications Group scheduling
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved.

CE

IP SLA VPN

Cisco Public

36

IP SLAs LSP Health Monitor Functionality - in Detailed Steps


0. User configures
Auto-Command per VRF or for the PE

3. IP SLA+ LSP-Ping
Send LSP ping to Neighbor at a time and rate controlled by IP SLA (random Start) Fast retry on failure; send trap on timeout/ connection loss

IPSLA VPN
IP SLAs

PE2

CE

PE1

LSP-ping

PE3 2. IP SLA Agent 1. Automated LSP Discovery


Find BGP Next hops For all VPNs, or for selected VPN(s) Use a single probe template

MPLS
PEx

Group-Schedule of IP SLA probes: Probes generated from source to all destination PEs using /32 MP-IBGP VPNv4 loopbacks

4. VPN Discovery interval updates


LSP Scan Rate (SR); add probes if new BGP neighbor LSP Scan Rate Factor N (SRxN) Delete probes (ex: VRF removed or no route in the VRF)
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved.

PE50
IP SLAs

IP SLA VPN
Cisco Public

37

Virtual Circuit Connection Verification (VCCV)


Ability to provide end-to-end fault detection and diagnostics for an emulated pseudowire service

Requirement

One tunnel can serve many pseudowires MPLS LSP ping is sufficient to monitor the PSN tunnel (PE-PE connectivity), but not VCs inside of tunnel

VCCV allows sending control packets in band of PseudoWires (PW) Two components

Solution

Signaling component: communicate VCCV capabilities as part of VC label Switching component: cause the PW payload to be treated as a control packet
Type 1: uses Protocol ID of PW Control word Type 2: use MPLS router alert label Type 3: manipulate TTL exhaust

Applications IETF Standards


BRKRST-1101

Layer 2 transport over MPLS


FRoMPLS, ATMoMPLS, EoMPLS

RFC 5085
2009 Cisco Systems, Inc. All rights reserved. Cisco Public

38

MPLS OAM: Virtual Circuit Connection Verification


VCCV checks connectivity between egress and ingress PEs VCCV capability is negotiated when the AToM tunnel is brought up (depends on the LDP peer and the VC type)
QinQ Customer VLAN
7600

MPLS Verify/Trace Path of LSP Tunnels between PEs.

7600

Customer VLAN

QinQ

Verify/Trace Emulated services (e.g. Ethernet) mapped to Customer VLANS (Attachment VCs) Trace/Verify packets take same path as data packets
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

39

Connectivity Trace Using VCCV EoMPLS


VCCV marks the payload as control packet for switching purpose; packet follows the PW data path Control packets sent over the AToM tunnels are intercepted by the egress PE
PE1#ping mpls pseudowire 172.16.255.4 333
Attachment Circuit PE1 TTL in VC label is set appropriately at the initiator to reach the node of interest to verify the connectivity to VCCV packets use the same path as the data packets (may use different path than signaling traffic) VCCV Packet is Lost Attachment Circuit PE2

Connectivity of single-segment PW is implemented using VCCV CC type1 (RFC 5085)


BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

40

Troubleshooting Using LSP Ping (L2 CKT)


Pinging from R3 to R1 through AToM Tunnel R3#ping mpls pseudowire 10.200.0.1 10
R3#ping mpls pseudowire <IPv4 peer IP addr > <VC ID>? destination Destination address or address range exp EXP bits in mpls header interval Send interval between requests in Routerc pad Pad TLV pattern repeat Repeat count reply Reply mode size Packet size source Source specified as an IP address sweep Sweep range of sizes timeout Timeout in seconds ttl Time to live verbose verbose mode for ping output

MPLS Echo-req

AToM Tunnel R1 MPLS Echo-reply with Return Code 4

R3

R1#

*Jan 19 19:32:17.726: LSPV: AToM echo request rx packet handler *Jan 19 19:32:17.726: LSPV: Echo packet received: src 10.200.0.3, dst 127.0.0.1, size 122 *Jan 19 19:32:17.734: LSPV: Echo Hdr decode: version 1, msg type 1, reply mode 2 , return_code 0, return_subcode 0, sender handle 850000D1, sequence number 1, ti mestamp sent 20:22:30 UTC Mon Jan 19 2004, timestamp rcvd 00:00:00 UTC Mon Jan 1 1900 *Jan 19 19:32:17.734: LSPV: tlvtype 1, tlvlength 20 *Jan 19 19:32:17.734: LSPV: AToM FEC decode: srcaddr 10.200.0.1, destaddr 10.200 . 0.3, vcid 10, vctype 5 *Jan 19 19:32:17.734: LSPV: Target FEC stack length = 20, retcode = 3 *Jan 19 19:32:17.734: LSPV: tlvtype 3, tlvlength 8 *Jan 19 19:32:17.734: LSPV: Pad TLV decode: type 1, size 8 *Jan 19 19:32:17.734: LSPV: Echo Hdr encode: version 1, msg type 2, reply mode 2 , return_code 4, return_subcode 0, sender handle 850000D1, sequence number 1, ti mestamp sent 20:22:30 UTC Mon Jan 19 2004, timestamp rcvd 19:32:17 UTC Mon Jan 1 9 2004 41 Cisco Public 2009 Cisco Systems, Inc. All rights reserved.

Return code 4 sent due to some error condition either of the following has occurred
Wrong VC ID Wrong VC Type Wrong Source Address
BRKRST-1101

VCCV Switching Types


Three different Switching Modes
Type 1 (in-band vccv) Type 2 (out-of-band VCCV) Type 3 (TTL expiry) Type 1 involves defining the upper nibble of the CW (control word) as a Protocol ID (PID) field to signal inband VCCV [RFC4385] Type 2 involves shimming a MPLS router alert label between the IGP label stack and VC label Manipulate and Signal TTL exhaust (TTL == 1) for multiple switching point PEs

Cisco Routers always use Type 1, if available, for LSP Ping over an AToM VC Control Channel. Type 2 Switching accommodates those VC types and implementations that do not support or interpret the AToM Control word. A new CC Type 3 new switching point TLV - is introduced to support VCCV in MS-PWs (RFC 5085)
BRKRST-1101 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

42

VCCV Switching Types


Two Types of Switching Modes
Type 1 involves defining the upper nibble of the control word as a Protocol Id (PID) field
Control Word Use Is Signalled in LDPStandard Form:
01234567890123456789012345678901
0 0 0 0 Flags FRG Length Sequence Number

OAM Uses a different 1st Nibble


01234567890123456789012345678901
0001 Reserved PPP DLL Protocol Number=IPvx IP OAM Packet: Ping/BFD/LSP Ping

CE
PE1#sh mpls l2transport binding 10 Destination Address: 10.200.0.1, VC ID: 10 Local Label: 16 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 1 Remote Label: 69 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 1
BRKRST-1101

PE1

PE2

CE

vccv Packet Sent from PE1 to PE2


IGP Label TTL=255 vc Label+CW IP Payload

vccv Packet Received from PE1 to PE2


vc Label+CW IP Payload

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

VCCV Switching Types (Cont.)


Type 2 involves shimming a MPLS router alert label between the IGP label stack and VC label

CE

PE1

PE2

CE

PE1#sh mpls l2transport binding 10 Destination Address: 10.200.0.1, VC ID: 10 Local Label: 16 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 2 Remote Label: 69 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 2

vccv Packet Sent from PE1 to PE2


IGP Label TTL=255 Rtr Alert Label 0x0001 vc Label+CW IP Payload

vccv Packet Received from PE1 to PE2


Rtr Alert Label 0x0001 vc Label+CW IP Payload

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

44

VCCV Switching Types (Type2)


Signal out-of-band VCCV using MPLS router alert label. Shim an MPLS Router Alert Label Between the IGP Label Stack and VC Label.
PE1#sh mpls l2transport binding 10 Destination Address: 10.200.0.1, VC ID: 10 Local Label: 16 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 2 Remote Label: 69 Cbit: 0, VC Type: Ethernet, GroupID: 0 MTU: 1500, Interface Desc: n/a VCCV Capabilities: Type 2

PE1

PE2

IGP Label TTL=255 Rtr Alert Label 0x0001 vc Label+CW L2 Payload

Rtr Alert Label 0x0001 vc Label+CW L2 Payload

VCCV Packet Sent to PE2

VCCV Packet Received from PE1

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

VCCV for Multi-Segmented Pseudowires Ping Operation using VCCV Type III
Ping from T-PE2 to S-PE1
CE1 AC
VCID: 100
ACCESS MPLS

S-PE1
VCID: 101 MPLS Core

S-PE2
VCID: 102
ACCESS MPLS

AC CE2

T-PE1
3.

T-PE2
1.

TTL 0 Punt to OAM


PSN Tunnel Pseudowire

2.

Label Switch Packet TTL 1

PWID 101 Sender IP: S-PE2 Remote IP: S-PE1 TTL 2 SRC IP: T-PE2 Dest IP: 127.0.0.1

4.

BRKRST-1101

Code 8 TTL 2 Src IP: 127.0.0.1 Dest IP: T-PE2

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

46

MPLS Aware NetFlow


Provides flow statistics per MPLS and IP packets
MPLS packets:
Labels information And NetFlow v5 fields for underlying IP packet

IP packets: Regular IP NetFlow records

Leverages the new NetFlow version 9 export format Configure on ingress interface Supported on sampled/non-sampled NetFlow VRF aware Netflow Export support
Router(config)# ip flow-export destination 10.10.10.10 9999 vrf terps <sctp|udp>

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

Example: MPLS VPN Aware Netflow


vrf = red 172.16.99.1 98.98.98.98 201

IP/MPLS

204

99.99.99.99

PE
VRF = red 10.100.1.201

PE
10.100.1.204

VRF = red Netflow Interface

VPN Traffic flow

201#sh ip bgp vpnv4 vrf red labels Network Next Hop Route Distinguisher: 1:1 (red) 24.24.24.24/32 10.100.1.204 98.98.98.98/32 172.16.98.2 99.99.99.99/32 10.100.1.204 172.16.98.0/24 0.0.0.0 172.16.99.0/24 10.100.1.204 201.201.201.201/32 0.0.0.0
BRKRST-1101

In label/Out label nolabel/21 19/nolabel nolabel/20 21/nolabel(red) nolabel/18 18/nolabel(red))


Cisco Public

2009 Cisco Systems, Inc. All rights reserved.

48

Example: MPLS VPN Aware Netflow


172.16.99.1 98.98.98.98 201

IP/MPLS

204

99.99.99.99

PE
VRF = red 10.100.1.201

PE
10.100.1.204

VRF = red Netflow Interface

VPN Traffic flow ip flow-cache mpls label-positions 1 ! 201#sh ip cache verbose flow SrcIf Port Msk AS Et1/0 0000 /0 SrcIPaddress 172.16.98.2 0 1:18-0-1 DstIf Port Msk AS Tu0* 0000 /0 DstIPaddress NextHop 172.16.99.1 0.0.0.0

Pr TOS Flgs Pkts B/Pk Active 00 05 10 60 276K 1550.9

Pos:Lbl-Exp-S

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Label = 18 EXP = 0 Stack = 1

Cisco Public

49

Embedded Management for MPLS: SNMP MIBs and Traps


Deployment Scope MIB Module/OAM Feature Standard Compliance
RFC3813 RFC3815 RFC3814

Cisco Value Add

MPLS-LSR-MIB MPLS-LDP-MIB

VRF-aware MIB capabilities VRF-aware MIB capabilities VRF-aware MIB capabilities

LDP MPLS Core

MPLS-FTN-MIB

MPLS-LDP-STD-MIB MPLS-L3VPN-STD-MIB
MPLS-TE-MIB RFC3812 IETF Draft

LDP session status Trap notifications VRF max-route Trap notifications


-

Traffic Engineered MPLS Core

MPLS-FRR-MIB

MPLS-TE-STD-MIB

TE Tunnel status Trap notifications

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

LDP Event Monitoring Using LDP Traps


Interface Shutdown (E1/0 on PE1)
Time = t: Received SNMPv2c Trap from pe1:
sysUpTimeInstance = 8159606 snmpTrapOID.0 = mplsLdpSessionDown mplsLdpSessionState.<index> = nonexistent(1) mplsLdpSessionDiscontinuityTime.<index> = 8159605 mplsLdpSessionStatsUnknownMesTypeErrors.<index> = 0 mplsLdpSessionStatsUnknownTlvErrors.<index> = 0 ifIndex.5 = 5

LDP Session Down (PE1 P01)


Time = t: Received SNMPv2c Trap from pe1:
sysUpTimeInstance = 8159606 snmpTrapOID.0 = mplsLdpSessionDown mplsLdpSessionState.<index> = nonexistent(1) mplsLdpSessionDiscontinuityTime.<index> = 8159605 mplsLdpSessionStatsUnknownMesTypeErrors.<index> = 0 mplsLdpSessionStatsUnknownTlvErrors.<index> = 0 ifIndex.5 = 5

Interface goes down Time = t+1: Received SNMPv2c Trap from pe1:
sysUpTimeInstance = 8159906 snmpTrapOID.0 = linkDown ifIndex.5 = 5 ifDescr.5 = Ethernet1/0 ifType.5 = ethernetCsmacd(6) locIfReason.5 = administratively down

LDP session goes down

Time = t+1: Received SNMPv2c Trap from p01:


sysUpTimeInstance = 8160579 snmpTrapOID.0 = mplsLdpSessionDown mplsLdpSessionState.<index> = nonexistent(1) mplsLdpSessionDiscontinuityTime.<index> = 8160579

PE1

Time = t+2: Received SNMPv2c Trap from p01: LDP session


sysUpTimeInstance = 8160579 snmpTrapOID.0 = mplsLdpSessionDown mplsLdpSessionState.<index> = nonexistent(1) mplsLdpSessionDiscontinuityTime.<index> = 8160579 mplsLdpSessionStatsUnknownMesTypeErrors.<index> = 0 mplsLdpSessionStatsUnknownTlvErrors.<index> = 0 ifIndex.5 = 5

P1

mplsLdpSessionStatsUnknownMesTypeErrors.<index> = 0 mplsLdpSessionStatsUnknownTlvErrors.<index> = 0 ifIndex.5 = 5

PE1

LDP session

P1

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

MPLS Management Summary


MPLS management operations include MPLS node and service configuration, and monitoring In addition to CLI, SNMP MIBs and OAM capabilities are available for MPLS management MPLS MIBs provide LDP, VPN, and TE management information, which can be collected by SNMP tools
MIB counters, Trap notifications

Advanced MPLS management capabilities can be implemented via MPLS OAM


LSP path discovery and connectivity validation Proactive monitoring via automated MPLS OAM

BRKRST-1101

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

52