Multicast Troubleshooting 1 PDF

Multicast Troubleshooting
Denise “Fish” Fishburne

Customer Proof of Concept Team Lead
CCIE #2639, CCDE 2009:0014

BRKIPM-2264
Agenda
• Quick Review
• Multicast Troubleshooting Methodology
• Troubleshooting ASM
• Troubleshooting SSM
3
Housekeeping Items
• Cell Phones
• “Intermediate” Class
• Assumes Multicast Operational Experience
• Questions
4
Quick Review
5
Quick Review Agenda
• Frequently Used Terms

• Tree Growth
• Sending a (*,G) PIM Join
• Registering a Source
• Designated Routers
6
Frequently Used Terms
ASM - Any Source Multicast
SSM - Source Specific Multicast
MDT - Multicast Distribution Tree
RP - Rendezvous Point
FHR - First Hop Router
LHR - Last Hop Router
IR - Intermediate Router
DR - Designated Router
IGMP- Internet Group Management Protocol
7
• Any Source Multicast (ASM)
• Original (Classic) PIM-SM
• Supports both Shared and Source Trees
• Source Specific Multicast (SSM)

• a.k.a. Single Source Multicast
• Supports only Source Trees
8
(*,G) = *,G
= Shared Tree
= RP tree
*,G MDT
RP
LHR
9
(S,G) = S,G
= Shortest Path Tree
= Source tree
S,G MDT
FHR
LHR
10
(*,G) = *,G
= Shared Tree
= RP tree
(S,G) = S,G
= Source Tree
= Shortest Path Tree
Q: What is the Root of the (*,G) Tree?

A: RP
Q: What is the Root of the (S,G) Tree?
A: Source
11
IIF = Incoming Interface
Interface towards the root of the tree
OIL = Outgoing interface List
Interfaces that have received PIM Joins or IGMP membership requests
Q: What does the IIF of the (*,G) Tree point towards?

A: RP
Q: What does the IIF of the (S,G) Tree point towards
A: Source
12
Tree Growth
Nature’s Way
In Nature, trees grow
• UP and
• OUT
from the ROOT
13
Tree Growth
Multicast Distribution Tree (MDT)
With Multicast, trees grow
• TOWARD
the ROOT
14
Sending a (*,G) PIM Join
10.4.5.0
.5 .4
PIM Join
Router A Router B RP
Question:
When Router A sends a PIM (*,G) Join out, what will be the Destination IP?
IP address of the RP? 224.0.0.13? IP of Router B
(All PIM Routers) (10.4.5.4)
**Assume Router A knows the RP, has the RP in its RIB via an IGP with B and has a PIM
neighbor with Router B
15
The PIM Header
PIM header common to all PIM messages:
Some PIM Types are:

Message Type Destination
0 = Hello Multicast to ALL-PIM-ROUTERS
1 = Register Unicast to RP
2 = Register-Stop Unicast to source of Register packet
3 = Join/Prune Multicast to ALL-PIM-ROUTERS
4 = Bootstrap Multicast to ALL-PIM-ROUTERS
5 = Assert Multicast to ALL-PIM-ROUTERS
http://www.ietf.org/rfc/rfc4601.txt
16
10.4.5.0
.5 .4
Router A Router B RP
DIP 224.0.0.13 PIM Join
Q: When Router A sends a PIM (*,G) Join out, what will be the Destination IP?
IP of the RP?
224.0.0.13? IP of Router B
(All PIM Routers) (10.4.5.4)
17
A B
.5 10.4.5.0 .4
RP
.1
C
Question:
If the destination IP address of the Join is 224.0.0.13 (all PIM Routers), then
how do we keep Router B and Router C from BOTH acting on that join?
18
Sending a (*,G) Join
A B
.5 10.4.5.0 .4

RP
.1
C
Answer:
The address of the upstream neighbor (B or C) is the target of the message
http://www.ietf.org/rfc/rfc4601.txt
19
A B
.5 10.4.5.0 .4
PIM Join
.1
Destination IP C
224.0.0.13
IP address of B
10.4.5.4
RP
2.2.2.2
20
***When a shared MDT is already
Registering a Source*** active for that group on the RP
FHR RP = 2.2.2.2
gig0
Mcast packet is encapsulated into unicast PIM
Mcast Data packet:
Dest IP=RP PIM Mcast data Destination IP is the RP.
PIM header type is Register
gig0 added to the RP sends PIM (S,G) Join Back Towards Source
OIL for (S,G) if there is an active shared tree for that G
Mcast packets now get sent out twice
Mcast data 1.With Register
2.Out (S,G)
Mcast data Once the RP sees the packets come in
on the (S,G),
…. it sends a unicast Register Stop
Register Stop to the FHR
Mcast data Mcast data Mcast packets now only out the (S,G) tree
21
Designated Routers (DRs)
Router A
.3
Vlan
FHR Q: Which router should register the source
with the RP?
.2
A: The FHR that is the DR
Router B
22
Designated Routers (DRs)
.3
Vlan
Q: Which router should send the PIM (*,G)
Join?
.2
A: The LHR that is the DR
LHR
23
Multicast Troubleshooting
Methodology
24
Multicast Troubleshooting Methodology Agenda
• The Plumbing
• Troubleshooting Toolbox
25
Building the Plumbing
Unicast
EIGRP EIGRP EIGRP

R1 R1 Subnet
R1 R1
10.1.1.0/24
advertise advertise advertise
Ping
10.1.1.0/24 10.1.1.0/24 10.1.1.0/24
10.1.1.1 10.1.1.1
Point A Point B
26
Building the Plumbing ***In this example, LHR is configured
to not cut over to the S,G tree.
Multicast***
PIM PIM PIM PIM

R1 R1 R1 R1
FHR RP PIM (*,G) PIM (*,G) LHR
Source PIM Unicast Register Join Join Join
(Tx)
PIM (S,G) PIM (S,G)

Join Join
27
Sherlock Holmes
28
Troubleshooting the ASM Plumbing LHR
FHR RP
MDTs Built Backwards Towards Root
Traffic
Troubleshooting
29
Troubleshooting the SSM Plumbing LHR
FHR
MDTs Built Backwards Towards Root
Traffic
Troubleshooting
30
Troubleshooting Toolbox
“Go To” Commands
show ip mroute
show ip mroute count
show ip pim neighbor

show ip pim interface
show ip igmp group
show ip pim rp [group]

show ip pim rp mapping [group]
show ip rpf [address]
31
Facts to Remember
• Multicast Trees get built backwards towards the root
• Multicast traffic in ASM and SSM are triggered via a “pull”
• High on the “food chain” dependent on routing working and PIM working
32
Questions to Remember
• Who is the root?
• Where is the root?
• What is the PIM RPF neighbor towards the root?
33
Checklist
 IGMP Membership Report Received by LHR

 “WHO:” is the root?
 “WHERE” is the root?
 “WHAT” is the PIM RPF neighbor towards the root?
34
ASM Troubleshooting
35
ASM Troubleshooting
R3
R3
R1
R1 R2
R2 R5
R5
FHR RP LHR
R4
(*,G) from LHR towards the RP 1
FHR registers source with RP

2
RP sends PIM (S,G) Join Towards Source
3
LHR Cutting over to SPT (S,G)
36
ASM Troubleshooting
Shared Tree
R3
R3
R1
R1 R2
R2 R5
R5
FHR RP LHR
R4

2
3
37
ASM Troubleshooting:
Shared Tree
R3
RP LHR
R2 R5
1a
LHR receives
1c R4
IGMP Membership Report
R4 (an IR) sends a PIM (*,G) 1b
To R2 (RP)
LHR (R5) sends a PIM (*,G)
To R4

38
Shared Tree
R3
RP LHR
R2 R5
1a
R4 LHR receives
 “WHO:” LHR knows who the RP is for this group
 “WHERE:” LHR knows where (RIB) the RP is
 “WHAT:” LHR has a PIM Neighbor to send the PIM (*,G) Join to
39
Shared Tree
1a LHR Receives IGMP Membership Report LHR
R5
LHR#show ip mroute Is completely empty

LHR#show ip igmp group Is completely empty
LHR#show ip igmp interface Is completely empty
40
Shared Tree
R5
• PIM not enable on interface of Receiver
• Layer 1 issues
• Layer 2 issues
• Access-List
• IGMP membership report not being sent
LHR#show run interface gig1/1 interface GigabitEthernet1/1

no switchport
ip address 100.1.1.1 255.255.255.0
ip pim sparse-mode
41
Shared Tree
R5
LHR#show run interface gig1/1 interface GigabitEthernet1/1

no switchport
ip address 100.1.1.1 255.255.255.0
LHR#show ip igmp groups
ip pim sparse-mode
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
239.1.1.1 GigabitEthernet1/1 00:18:35 00:02:31 100.1.1.100
42
Shared Tree
1b LHR Sends PIM (*,G) Join R3
LHR
R2 R5
RP
R4 1b
To R4
 “WHO:” LHR knows who the RP is for this group
 “WHERE:” LHR knows where (RIB) the RP is
 “WHAT:” LHR has a PIM Neighbor to send the PIM (*,G) Join to
43
Shared Tree
R3
1b LHR Sends PIM (*,G) Join
R2 R5
RP
Who? RP LHR
R4
Where?
RPF Nbr?
LHR#show ip mroute (*, 239.1.1.1), 00:01:23/00:02:48, RP 0.0.0.0, flags: SJC

Incoming interface: Null, RPF nbr 0.0.0.0 ?
Outgoing interface list:
GigabitEthernet1/1, Forward/Sparse, 00:01:23/00:02:48, H
LHR#show ip pim rp 239.1.1.1 Group: 239.1.1.1, RP: 0.0.0.0
?
44
Shared Tree
R3
Note:
R2 R5
RP information can be shared 3 ways
• Static RP LHR
R4
• Auto-rp
• BSR
LHR(config)#ip pim rp-address 2.2.2.2 ?

<1-99> Access-list reference for group
<1300-1999> Access-list reference for group (expanded range)
WORD IP Named Standard Access list
override Overrides dynamically learnt RP mappings
45
Shared Tree
R3
R2 R5
RP
✓
Who?
RP
R4 LHR
Where?
RPF Nbr?
LHR#sh ip pim rp 239.1.1.1

Group: 239.1.1.1, RP: 2.2.2.2, uptime 00:00:39, expires never
LHR#sh ip pim rp mapping 239.1.1.1

PIM Group-to-RP Mappings
Group(s): 224.0.0.0/4, Static-Override
RP: 2.2.2.2 (?) 46
Shared Tree
R3
R2 R5
RP
✓
Who?
RP LHR
Where? R4
RPF Nbr?
LHR#show ip mroute ✓
(*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists
LHR#show ip route 2.2.2.2 % Network not in table
47
Shared Tree
R3
R2 R5
RP
✓
Who?
✓
Where? RP
R4 LHR
RPF Nbr?
LHR#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets

O 2.2.2.2 [110/42] via 10.4.5.4, Gig 1/5
[110/42] via 10.3.5.3, Gig 1/4
48
Shared Tree
R3
RP R2 R5
✓
Who?
✓
Where? RP
R4 LHR
RPF Nbr?
LHR#show ip mroute
(*, 239.1.1.1), 00:08:18/00:02:46, RP 2.2.2.2, flags: SJC
49
Shared Tree
R3
R2 R5
RP
✓
Who?
✓
Where? RP LHR
RPF Nbr? R4
LHR#show ip rpf 2.2.2.2

RPF information for ? (2.2.2.2) failed, no route exists
LHR#show ip pim neighbor No PIM neighbors listed
50
Shared Tree
R3
R2 R5
RP LHR
R4
LHR#show ip pim neighbor

PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable
Neighbor Interface Uptime/Expires
10.3.5.3 GigabitEthernet1/4 00:17:57/00:01:21
10.4.5.4 GigabitEthernet1/5 00:17:56/00:01:31
51
Shared Tree
R3
Question:
Why was 10.4.5.4 chosen as the RPF neighbor? R2 R5
Answer:
If ECMP are available, the RPF for the multicast traffic RP LHR
R4
will be based on highest IP address for the RPF
neighbor
LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2)

RPF interface: GigabitEthernet1/5
RPF neighbor: ? (10.4.5.4)
RPF route/mask: 2.2.2.2/32
RPF type: unicast (ospf 1)
52
Shared Tree
1b LHR Sends PIM (*,G) Join R3
Note:
LHR now has all the information it needs to
R2 R5
send a PIM (*,G) join to its RPF neighbor.
RP RP
✓
Who? R4 LHR
✓
Where? 1b
✓ Nbr?
RPF LHR (R5) sends a PIM
(*,G) To R4
LHR#show ip mroute RP flags

*,G (*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC
IIF Incoming interface: Gig 1/5, RPF nbr 10.4.5.4
OIL Outgoing interface list: RPF nbr
Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H
53
Shared Tree
1c IR Sends PIM (*,G) Join
R3
R2 R5
RP LHR
R4
R4 (an IR) sends a PIM (*,G) R4
To R2 (RP) 1c
 “WHO:” IR knows who the RP is for this group

 “WHERE:” IR knows where (RIB) the RP is
 “WHAT:” IR has a PIM Neighbor to send the PIM (*,G) Join to
54
Shared Tree
R3
Note:
Telnetting backwards towards the root is
a good way to troubleshoot. R2 R5
RP LHR
LHR#show ip mroute 239.1.1.1 R4
R4
(*, 239.1.1.1), 00:36:39/00:02:56, RP 2.2.2.2, flags: SJC
Incoming interface: GigabitEthernet1/5, RPF nbr 10.4.5.4
Gig 1/1, Forward/Sparse-Dense, 00:08:08/00:02:56, H
LHR#telnet 10.4.5.4 RP
Trying 10.4.5.4 ... Open
R4#show ip mroute 239.1.1.1 Who?
✓
(*, 239.1.1.1), 00:48:49/00:02:33, RP 2.2.2.2, flags: S Where?
✓
Incoming interface: Gig 3/3, RPF nbr 10.2.4.2, RPF-MFD
Outgoing interface list: RPF
✓ Nbr?
Gig 3/5, Forward/Sparse, 00:47:16/00:02:33, H 55
Shared Tree
R3
Note:
R4 had all the information it needed to
send a PIM (*,G) join to its RPF
R2 R5
neighbor.
RP LHR
R4
R4#telnet 10.2.4.2 R4 (an IR) sent a PIM (*,G)R4
Trying 10.2.4.2... Open To R2 (RP) 1c
R2#show mrib route 239.1.1.1
(*,239.1.1.1) RPF nbr: 2.2.2.2 Flags: C
Up: 00:52:12
Incoming Interface List
Decapstunnel0 Flags: A, Up: 00:52:12
Outgoing Interface List
GigabitEthernet0/0/0/3 Flags: F NS, Up: 00:52:12
56
Shared Tree
R3
RP LHR
R2 R5
1a
LHR receives
1c R4
R4 (an IR) sends a PIM (*,G) 1b
To R2 (RP)
(*,G) Between RP and LHR To R4
✓ (*,G) from LHR towards the RP 1

57
ASM Troubleshooting
R3
R3
R1
R1 R2
R2 R5
R5
FHR RP LHR
R4

2
3
58
Source Tree
R1
R1 R2
R2
FHR RP
2a
FHR receives 2b
Multicast Data FHR registers source with RP
2c
RP builds S,G Back Towards Source

2
59
***When a shared MDT is already active for that group on the RP

Source Tree
R1
R1 R2
R2
FHR RP
2a
FHR receives 2b
2c

2
60

Source Tree
2a FHR Receives Multicast Data
R1
R1 R2
R2
FHR RP
2a
FHR receives
Multicast Data
2a
FHR receives Multicast Data
61
Source Tree
2a FHR Receives Multicast Data R1
R1 R2
R2
 FHR Receives Multicast Data S: 200.1.1.100 RP

FHR
R1 R2
G: 239.1.1.1
• PIM not enable on interface of Receiver
• Layer 1 issues
2a
• Layer 2 issues
• Access-List FHR receives
• IGMP membership report not being sent Multicast Data
R1#show ip mroute Empty
R1#show run interface gig3/2 interface GigabitEthernet3/2

no switchport
ip address 200.1.1.1 255.255.255.0
ip pim sparse-mode
62
Source Tree
2a FHR Receives Multicast Data R1
R1 R2
R2
Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been G: 239.1.1.1 FHR
built.
R1#show ip mroute (*, 239.1.1.1), 00:02:17/00:00:42, RP 0.0.0.0, flags: SP

Outgoing interface list: Null
63
Source Tree
R1
R1 R2
R2
FHR RP
2a
FHR receives 2b
2c

2
64

Source Tree
2b FHR Registers Source with RP
R1
R1 R2
R2
FHR RP
2b
2b
65
Source Tree
2b FHR Registers Source with RP R2
R2
R1
R1
Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been built. G: 239.1.1.1 FHR
Question:
Why is there no (S,G)?

66
Source Tree
R2
R1
R1
Note: S: 200.1.1.100 RP
We see that a (*,G) for that group has been built. G: 239.1.1.1 FHR
Question:
Why is there no (S,G)?
R1#show ip mroute 239.1.1.1 count

IP Multicast Statistics
2 routes using 1230 bytes of memory
2 groups, 0.00 average sources per group
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)
Group: 239.1.1.1, Source count: 0, Packets forwarded: 0, Packets received: 6027

RP-tree: Forwarding: 0/0/0/0, Other: 6027/6026/1
RPF failed
67
Source Tree
R2
R1
R1
S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
FHR does not know who the RP is for
this group.
RP
Who?
Where?
RPF Nbr?

R1#sh ip rpf 2.2.2.2
RPF information for ? (2.2.2.2) failed, no route exists
68
Source Tree
2b FHR Registers Source with RP
R1
R1 R2
R2
Note:
RP information can be shared 3 ways S: 200.1.1.100 RP
• Static G: 239.1.1.1 FHR
• Auto-rp
• BSR
RP
✓
Who?
Where?
RPF Nbr?
R1#sh ip pim rp mapping 239.1.1.1
“ip pim rp-address 2.2.2.2 override” PIM Group-to-RP Mappings
Group(s): 224.0.0.0/4, Static-Override
RP: 2.2.2.2 (?)
✓
69
Source Tree
R1 R2
R2
S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
FHR doesn’t know who the RPF
neighbor is for the shared tree for this
group. RP
✓
Who?
✓
Where?
RPF Nbr?
R1#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets
O 2.2.2.2 [110/2] via 10.1.2.2, 01:11:17, Gig3/1
R1#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2) failed, no route exists
R1#show ip pim neighbor No PIM neighbors
70
Source Tree
R1 R2
R2
S: 200.1.1.100 RP
Note:
G: 239.1.1.1 FHR
“ip pim sparse-mode” added to Gig3/1 on
FHR (R1)
R1#show ip mroute
(*, 239.1.1.1), 00:02:45/stopped, RP 2.2.2.2, flags: SPF
Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD
(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

✓
Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC
71
Source Tree
R1
R1 R2
R2
FHR RP
2a
FHR receives 2b
2c

2
72

Source Tree
R1
R1 R2
R2
FHR RP
2c
2c
73
Source Tree
2c RP builds S,G Back Towards Source R1
R1 R2
R2
S: 200.1.1.100 RP
Problem:
G: 239.1.1.1 FHR
(S,G) stuck in “registering” on FHR (R1)
R1#show ip mroute
Incoming interface: Gig3/1, RPF nbr 10.1.2.2, RPF-MFD
(200.1.1.100, 239.1.1.1), 00:01:34/00:02:59, flags: PFT

Incoming interface: Gig3/21, RPF nbr 0.0.0.0, Registering (data-header), Partial-SC
74
Source Tree
R1 R2
R2
Note:
Not only is the (S,G) stuck in registering, but S: 200.1.1.100 RP
G: 239.1.1.1 FHR
the CPU on FHR is also higher than normal.
Question:
Is the high CPU related to our (S,G) that is
stuck in registering?
R1#show proc cpu

CPU utilization for five seconds: 74%/31%; one minute: 73%; five
minutes: 41%
PID 5Sec 1Min 5Min TTY Process
156 42.31% 41.93% 23.57% 0 IP Input
(note: 1 mcast stream is being sent at Tx rate of 20Mbps)
75
Source Tree
R1 R2
R2
Question:
Is the high CPU related to our (S,G) that is S: 200.1.1.100 RP
stuck in registering? G: 239.1.1.1 FHR
Answer:
We actually have only 2 interfaces that are
up/up on R1. So yes, the high CPU is
related to the multicast traffic stream
There is a space after ^

R1#sh int summary | exclude ^
* GigabitEthernet3/1 0 0 0 0 0 0 20079000 2385 0
* GigabitEthernet3/21 24 96 0 0 19614000 2393 0 0 0
RXBS: rx rate (bits/sec) TXBS: tx rate (bits/sec)

RXPS: rx rate (pkts/sec) TXPS: tx rate (pkts/sec)
76
Source Tree
R1 R2
R2
Question:
S: 200.1.1.100 RP
Why is the FHR having higher CPU?
G: 239.1.1.1 FHR
Clue:
OIL on FHR for (S,G) is null
Note:
S,G tree will get built backwards from the RP.
So troubleshoot backwards from RP.
R2#show pim rpf 200.1.1.100
* 200.1.1.100/32 [-1/-1]
via Null with rpf neighbor 0.0.0.0
R2#show route 200.1.1.100
% Network not in table
77
Source Tree
Question:
Why is the FHR having higher CPU?
R1
R1 R2
R2
***When a shared MDT is already
RP active for that group on the RP
FHR
Mcast packet is encapsulated into
Mcast data unicast PIM packet:
Dest IP=RP PIM Mcast data Destination IP is the RP.
PIM header type is Register
RP sends PIM (S,G) Join Back Towards
Source if there is an active shared tree
for that G
Mcast data Once the RP sees the packets
Register Stop to the FHR come in on the (S,G), it sends a
unicast Register Stop 78
Source Tree
R1 R2
R2
Problem:
RP doesn’t know where the Source is for this S: 200.1.1.100 RP
G: 239.1.1.1 FHR
group.
Issue:
Source address not in RIB of RP –
routing problem Source
✓
Who?
Where?
RPF Nbr?
R2#show pim rpf 200.1.1.100
* 200.1.1.100/32 [-1/-1]
via Null with rpf neighbor 0.0.0.0
R2#show route 200.1.1.100
% Network not in table
79
Source Tree
Note: R1
R1 R2
R2
Fix routing problem. RP now has
Source in its RIB. S: 200.1.1.100 RP
G: 239.1.1.1 FHR
Source
✓
Who?
✓
Where?
✓ Nbr?
RPF
R2#show pim rpf 200.1.1.100 * 200.1.1.100/32 [110/2]

via Gig /0/1 with rpf neighbor 10.1.2.1
R2#show mrib route (200.1.1.100,239.1.1.1) RPF nbr: 10.1.2.1 Flags: L

Up: 00:50:36
GigabitEthernet0/1 Flags: A, Up: 00:11:56
GigabitEthernet0/3 Flags: F NS, Up: 00:49:00
80
ASM Troubleshooting
Source Tree
Source Tree
R1 R2
R2
Note: S: 200.1.1.100
Going back to FHR (R1) with this issue fixed FHR RP
G: 239.1.1.1
we now see that the (S,G) is no longer stuck
in registering and the CPU is no longer high.
R1#show ip mroute
Incoming interface: GigabitEthernet3/1, RPF nbr 10.1.2.2, RPF-MFD
(200.1.1.100, 239.1.1.1), 00:10:47/00:03:20, flags: FT

81
ASM Troubleshooting
Source Tree
Source Tree
R1 R2
R2
Note: S: 200.1.1.100 RP
G: 239.1.1.1 FHR
Going back to FHR (R1) with this issue
fixed we now see that the (S,G) is no
longer stuck in registering and the CPU
is no longer high.
R1#show proc cpu

CPU utilization for five seconds: 0%/0%
82
Source Tree
S: 200.1.1.100
G: 239.1.1.1
FHR Registering of Source to the RP RP
Mcast data
Potential Reasons for “Black Hole” RP Wants to Join (S,G)

1. Source not known in IGP at RP or some IR on the way back to the source
(this is the situation we just covered)
2. Usual suspects for any kind of packet loss (physical, firewall, ACL, etc)
3. Missing PIM Neighbor Relationship
4. Multicast Boundary Configured for this Multicast Group
83
ASM Troubleshooting
Source Tree
R1
R1 R2
R2
RP
S: 200.1.1.100 FHR
G: 239.1.1.1
Registering of Source to the RP

2
RP building S,G Back Towards Source
(S,G) Between RP and FHR
✓ 84
PIM-SM Frequently Forgotten Fact
85
ASM Troubleshooting
R3
R3
R1
R1 R2
R2 R5
R5
FHR RP LHR
R4

2
3
86
ASM Troubleshooting
Joining the SPT
R3
R3
R1
R1 R2
R2 R5
R5
R4
3
--If SPT Threshold is not infinity (default 0) when the mcast stream comes down the shared
tree and into the LHR,
--then the LHR will try to cutover to the shortest path tree
87
ASM Troubleshooting
Joining the SPT
3 LHR Cuts over to the SPT
R3
R3
R1
R1 R2
R2 R5
R5
R4
LHR Cutting over to SPT (S,G) 3

(PIM (S,G) Joins towards FHR for source
LHR Now must answer 3 questions before Source

sending PIM (S,G) Join Who?
Where? 88
RPF Nbr?
ASM Troubleshooting
Joining the SPT
3 LHR Cuts over to the SPT
R3
R3
R1
R1 R2
R2 R5
R5
R4
(S,G) Between LHR and FHR
✓ 3
89
ASM Designated Routers
Receivers
Last Hop Designated Router Responsibilities
The LH DR is responsible for sending the PIM (*,G) Join towards the RP
The LH DR will also be responsible for cutting over to the (S,G) tree
.3
Vlan
.2
LHR
90
ASM Designated Routers
Sources
First Hop Designated Router Responsibilities
The FH DR is responsible for registering the Source/Group with the RP
Router A
.3
Vlan
FHR
.2
Router B
91
ASM Troubleshooting
R3
R3
R1
R1 R2
R2 R5
R5
R4

Registering of Source to the RP
2
3
92
Troubleshooting the ASM Plumbing LHR
FHR RP
93
SSM Specific
Troubleshooting
94
SSM Specific Troubleshooting
R3
R3
R1
R1 R2
R2 R5
R5
R4
(S,G) from LHR towards Source
95
R3
R3 LHR receives
IGMPv3 Membership Report
ssm1
R1
R1 R2
R2 R5
R5
ssm4
R4
R2 (IR) send (S,G) Join to R1(FHR) ssm2
ssm3
LHR (R5) sends a PIM (S,G)
R4 (IR) send (S,G) Join to R2 To R4

96
ssm1 LHR Receives IGMPv3 Membership Report
LHR receives
R3
R3
ssm1
R1
R1 R2
R2 R5
R5
R4
ssm1
LHR receives
97
R5
R5
R1 R2
ssm1
Problem: LHR receives
LHR not seeing IGMPv3 Membership Report IGMPv3 Membership Report
from Receiver
LHR#show ip mroute Shows nothing but the autorp group 224.0.1.40

LHR#show ip igmp group Is completely empty except for 224.0.1.40
LHR#show ip pim interface gig1/1
Address Interface Ver/ Nbr Query DR DR
Mode Count Intvl Prior
100.1.1.1 Gig1/1 v2/S 0 30 1 100.1.1.1
98
Problem:
LHR not seeing R5
R5
IGMPv3 Membership
Report from Receiver
ssm1
Note:
Sniffer trace of IGMPv3 Membership report LHR receives
from receiver IGMPv3 Membership Report
Destination IP
224.0.0.22
IGMP v3
http://www.rfc-editor.org/rfc/rfc3376.txt 99
Problem:
LHR not seeing IGMPv3 R5
Membership Report
R5
from Receiver
Issue:
IGMPv3 not enabled on interface ssm1
LHR receives
Other Things to Look For:
Layer 1 issues, Layer 2 issues, Access-lists. IGMPv3 Membership Report
LHR#show ip igmp interface gig3/11 GigabitEthernet3/11 is up, line protocol is up
Internet address is 100.1.1.1/24
IGMP is enabled on interface
Current IGMP host version is 2
Current IGMP router version is 2
interface GigabitEthernet3/11
LHR#show run interface gig3/11 no switchport
ip address 100.1.1.1 255.255.255.0
ip pim sparse-mode 100
R5
R5
Note:
“ip igmp version 3” interface command ssm1
added under gig3/11 on the LHR
LHR receives
LHR#show ip igmp group g3/11

IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter
232.1.1.1 Gig1/1 17:53:07 stopped 100.1.1.100
101
ssm2 LHR Sends (S,G) Join
R3
R3
R1
R1 R2
R2 R5
R5
R4 ssm2
To R4
ssm2
To R4 102
R3
Problem:
LHR is configured for this group R2 R5
to be an ASM group, not an SSM
group.
Mode of Group LHR
ASM R4
SSM
LHR#show ip mroute ssm Is completely empty Flags:
D - Dense,
S - Sparse,
LHR#show ip mroute B - Bidir Group
??
(*, 232.1.1.1), 17:51:47/00:02:58, RP 2.2.2.2, flags: SJC s - SSM Group
103
ssm2 LHR Sends (S,G) Join R3
Note:
Some platforms (XR) have the
232/8 group as SSM by default. R2 R5
In IOS it must be configured.
Mode of Group If SSM Source LHR
ASM Who? R4
SSM Where?
RPF Nbr?
LHR(config)#ip pim ssm ?
default Use 232/8 group range for SSM
range ACL for group range to be used for SSM
LHR(config)#ip pim ssm default
“232.0.0.0-232.255.255.255 (232/8) Source-Specific Multicast Block”

-- http://www.iana.org/assignments/multicast-addresses/ 104
Question:
Assuming equal cost paths to the source from the
LHR thru R3 and R4, which router will the LHR use
as its RPF neighbor?
R3
R2 R5
R4 LHR
105
R3
Note:
With this group now an SSM group, the
LHR asks itself where is the Source and
R2 R5
who is the RPF nbr for this source.
Then it sends the (S,G) Join. LHR
Mode of Group If SSM Source R4
ASM ✓
Where? PIM (S,G)
✓
SSM ✓ Nbr?
RPF Join Sent
LHR#show ip mroute
(200.1.1.100, 232.1.1.1), 00:01:24/00:01:59, flags: sTI
s - SSM Group
T - SPT-bit set 106
I - Received Source Specific Host Report

ssm3 IR (R4) Sends (S,G) Join
R3
R3
R1
R1 R2
R2 R5
R5
ssm3 R4
R4 (IR) send (S,G) Join to R2
ssm3
R4 (IR) send (S,G) Join to R2 107
ssm3 IR (R4) Sends (S,G) Join R3
Problem:
IR is configured for this group to be R2
an ASM group, not an SSM group.
R5
Mode of Group
ASM LHR
R4
SSM R4
R4#show ip mroute
??
(*, 232.1.1.1), 18:13:34/stopped, RP 2.2.2.2, flags: SP
Incoming interface: GigabitEthernet3/3, RPF nbr 10.2.4.2, Partial-SC
(200.1.1.100, 232.1.1.1), 00:00:12/00:03:17, flags:
GigabitEthernet3/5, Forward/Sparse, 00:00:12/00:03:17 108
R3
Note:
With this group now an SSM group, the
IR (R4) asks itself where is the Source
and who is the RPF nbr for this source. R2 R5
Then it sends the (S,G) Join.
Mode of Group If SSM Source LHR
ASM ✓
Where? R4
✓
SSM ✓ Nbr?
RPF
R4
R4#show ip mroute
(200.1.1.100, 232.1.1.1), 00:05:41/00:02:44, flags: sT

109
R3
R3
R1
R1 R2
R2 R5
R5
ssm4
R4
R2 (IR) send (S,G) Join to R1(FHR)
ssm4
R2 (IR) send (S,G) Join to R1(FHR) 110
R3
ssm4 IR (R2) Sends (S,G) Join LHR
R1 R2 R5
Note:
XR has a very useful command
“show pim group-map”
R4
R2#show pim group-map

IP PIM Group Mapping Table
Group Range Proto Client Groups RP address Info
224.0.1.39/32* DM perm 0 0.0.0.0

224.0.1.40/32* DM perm 1 0.0.0.0
224.0.0.0/24* NO perm 0 0.0.0.0
232.0.0.0/8* SSM config 1 0.0.0.0
224.0.0.0/4* SM config 0 2.2.2.2 RPF: De0,2.2.2.2
224.0.0.0/4 SM static 0 0.0.0.0 RPF: Null,0.0.0.0
111
SSM Specific Troubleshooting R3
Note: R1 R2 R5
IR (R2) asks itself where is the Source
and who is the RPF nbr for this source.
Then it sends the (S,G) Join. R4
Mode of Group If SSM Source
✓ R2 sends
ASM Where?
✓
SSM ✓ Nbr?
RPF PIM (S,G) Join
R2#show mrib ipv4 route 232.1.1.1

(200.1.1.100,232.1.1.1) RPF nbr: 10.1.2.1 Flags:
Up: 01:22:22
GigabitEthernet0/1/0/1 Flags: A, Up: 01:22:22
GigabitEthernet0/1/0/3 Flags: F NS, Up: 01:22:22
112
SSM Specific Troubleshooting R3
Note: R1 R2 R5
(S,G) MDT built all the way from the
LHR to the FHR. But note how no
R4
traffic has yet been sent from the
source. R2 sends
PIM (S,G) Join
R1#show ip mroute 232.1.1.1
(200.1.1.100, 232.1.1.1), 00:00:28/00:03:01, flags: sT
R1#show ip mroute 232.1.1.1 count
Group: 232.1.1.1, Source count: 1, Pkts forwarded: 0, Pkts received: 0
Source: 200.1.1.100/32, Forwarding: 0/0/0/0, Other: 0/0/0
113
R3
R3
ssm1
LHR receives
R1
R1 R2
R2 IGMPv3 Membership
R5R5 Report
ssm4 R4
R2 (IR) send (S,G) Join to R1(FHR) ssm2
ssm3
R4 (IR) send (S,G) Join to R2 To R4

114
Troubleshooting with Fish
115
NO Receivers Getting Mcast
FHR RP
LHR
Start at RP for this Group
 *,G w/ OIL?
• NO? Pick 1 LHR and start there
 S,G
• NO? Go to FHR
116
Some Receivers Getting Mcast
FHR RP
LHR
Start at LHR of Broken Receiver

 Pull?
 Who, Where, What
117
SSM Troubleshooting with Fish LHR
FHR
Start at FHR
 S,G w/ OIL?
• NO? Pick 1 LHR and start there
• YES? Check to see if traffic coming in from Source
118
Facts to Remember
• Multicast Trees get built backwards towards the root
• Multicast traffic in ASM and SSM are triggered via a “pull”
• High on the “food chain” dependent on routing working and PIM working
119
Questions to Remember
• Who is the root?
• Where is the root?
• What is the PIM RPF neighbor towards the root?
120
Checklist

 “WHO:” is the root?
 “WHERE” is the root?
 “WHAT” is the PIM RPF neighbor towards the root?
121
Questions?
122
Participate in the “My Favorite Speaker” Contest
Promote Your Favorite Speaker and You Could be a Winner
• Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle <@DeniseFishburne>
• Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
123
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
124
Continue Your Education
• Demos in the Cisco Campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
125
Thank you
126

Multicast Troubleshooting 1 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multicast Troubleshooting 1 PDF

Uploaded by

Copyright:

Available Formats

Multicast Troubleshooting

Denise “Fish” Fishburne

CCIE #2639, CCDE 2009:0014

• Multicast Troubleshooting Methodology

• Frequently Used Terms

SSM - Source Specific Multicast

MDT - Multicast Distribution Tree

FHR - First Hop Router

LHR - Last Hop Router

IGMP- Internet Group Management Protocol

• Source Specific Multicast (SSM)

Q: What is the Root of the (*,G) Tree?

Interface towards the root of the tree

OIL = Outgoing interface List

Interfaces that have received PIM Joins or IGMP membership requests

Q: What does the IIF of the (*,G) Tree point towards?

In Nature, trees grow

from the ROOT

With Multicast, trees grow

Some PIM Types are:

DIP 224.0.0.13 PIM Join

EIGRP EIGRP EIGRP

PIM PIM PIM PIM

PIM (S,G) PIM (S,G)

MDTs Built Backwards Towards Root

MDTs Built Backwards Towards Root

show ip pim neighbor

show ip igmp group

show ip pim rp [group]

• Multicast Trees get built backwards towards the root

• Multicast traffic in ASM and SSM are triggered via a “pull”

• Who is the root?

• Where is the root?

• What is the PIM RPF neighbor towards the root?

 IGMP Membership Report Received by LHR

(*,G) from LHR towards the RP 1

FHR registers source with RP

(*,G) from LHR towards the RP 1

(*,G) from LHR towards the RP 1

 IGMP Membership Report Received by LHR

LHR#show ip mroute Is completely empty

LHR#show run interface gig1/1 interface GigabitEthernet1/1

 IGMP Membership Report Received by LHR

LHR#show run interface gig1/1 interface GigabitEthernet1/1

LHR#show ip mroute (*, 239.1.1.1), 00:01:23/00:02:48, RP 0.0.0.0, flags: SJC

LHR(config)#ip pim rp-address 2.2.2.2 ?

LHR#sh ip pim rp 239.1.1.1

LHR#sh ip pim rp mapping 239.1.1.1

LHR#show ip route 2.2.2.2 % Network not in table

LHR#show ip route 2.2.2.2 2.0.0.0/32 is subnetted, 1 subnets

LHR#show ip rpf 2.2.2.2

LHR#show ip pim neighbor

LHR#show ip rpf 2.2.2.2 RPF information for ? (2.2.2.2)

LHR#show ip mroute RP flags

 “WHO:” IR knows who the RP is for this group

✓ (*,G) from LHR towards the RP 1

(*,G) from LHR towards the RP 1

FHR registers source with RP

***When a shared MDT is already active for that group on the RP

FHR registers source with RP

***When a shared MDT is already active for that group on the RP

 FHR Receives Multicast Data S: 200.1.1.100 RP

R1#show run interface gig3/2 interface GigabitEthernet3/2