ACI L3 Multicast Troubleshoot – Part-3: Source and Receiver inside ACI – Topology 2
Topology:
Contents
I- Receiver sends out IGMP Report
In this first, we focus on Receiver side and how shared tree is formed (no traffic from source yet):
1- Receiver sends IGMP Join:
Upon receive of the IGMP join from receiver for the group 239.251.1.1, IGMP entry added to leaf-3-4:
leaf3# show ip igmp groups vrf bameur_MC:VRF-MC3
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
IGMP Connected Group Membership for VRF "bameur_MC:VRF-MC3"
Group Address Type Interface Uptime Expires Last Reporter
239.251.1.1 D vlan1 00:06:08 00:03:18 172.16.30.100 172.16.30.100
Code language: CSS (css)
(*,G) tree installed to leaf-3-4 (where receiver resides):
leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 00:06:35, igmp ip pim
Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
Outgoing interface list: (count: 1)
Vlan1, uptime: 00:06:35, igmp
Incoming interface: pointing to the fabric via tunnel interface with destination being VRF GiPo:
leaf3# show interface tunnel 29
Tunnel29 is up
MTU 9000 bytes, BW 0 Kbit
Transport protocol is in VRF "bameur_MC:VRF-MC3"
Tunnel protocol/transport is ivxlan
Tunnel source 127.0.0.100/32
Tunnel destination 225.1.192.48/32
Code language: PHP (php)
RPF pointing to root of the shared: the RP 192.168.30.10 (RPF neighbor to reach RP is BL PTEP 10.0.48.64)
Outgoing Interface: pointing to the BD VLAN of the Receiver EP
2- IGMP Join interest Published to COOP:
After receiving interest from the receiver EP via IGMP Join, Leaf-3-4 will publish it to COOP:
COOP log:
Spine-1# log_trace_bl_print_tool coop_trace.bl | grep 239.251 -A 5 -B 5 | egrep "10.0.168.65|10.0.168.70"
[2024-04-08T09:46:17.361525000+02:00:D:coop_oracle_process_publish:1439] TID 36:[DBG_COOP_TRACE_DETAIL]:(1712562380:959405422) Received PUBLISH message from 10.0.168.70 Publisher_id: 10.0.168.70 with trans_id 1552
[2024-04-08T09:46:17.361557000+02:00:D:coop_objs_add_mgroup_link_generic:9460] TID 36:[DBG_COOP_TRACE_DETAIL]:Add mgv4 <1, 0.0.0.0, 239.251.1.1, 240002> to leaf 10.0.168.70 linkage
[2024-04-08T09:46:17.364210000+02:00:D:coop_oracle_process_publish:1439] TID 17:[DBG_COOP_TRACE_DETAIL]:(1712562380:958776264) Received PUBLISH message from 10.0.168.65 Publisher_id: 10.0.168.65 with trans_id 7954
[2024-04-08T09:46:17.364220000+02:00:D:coop_objs_add_mgroup_link_generic:9460] TID 17:[DBG_COOP_TRACE_DETAIL]:Add mgv4 <1, 0.0.0.0, 239.251.1.1, 240002> to leaf 10.0.168.65 linkage
Code language: HTML, XML (xml)
verification in COOP DB:
spine1# show coop internal info repo mgroup
Repo Hdr Checksum : 31363
Repo Hdr record timestamp : 04 08 2024 07:53:10 284993671
Repo Hdr last pub timestamp : 04 08 2024 07:53:10 285217368
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ EXPORT
VRF Vnid : 2359298
mgroup src ip : 0.0.0.0
mgroup group ip : 239.251.1.1
Flags : 0x0x1 afi 0
Local leafs 2 (active: 2 deleted: 0)
Leaf 0 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 04 08 2024 07:51:21 457853613
Leaf Repo Hdr last pub timestamp : 04 08 2024 07:51:21 458024081
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.168.65
Leaf Flags : 0x0
Leaf 1 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 04 08 2024 07:53:10 284993671
Leaf Repo Hdr last pub timestamp : 04 08 2024 07:53:10 285217368
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.168.70
Leaf Flags : 0x0
oldest local publish timestamp: 04 08 2024 07:46:20 959376706
MPOD Pub id : 0.0.0.0
MPOD leafs 0 (active: 0 deleted: 0)
MSITE Pub id : 0.0.0.0
MSITE leafs 0 (active: 0 deleted: 0)
Hash: 990247176 owner: 10.0.168.71
Code language: CSS (css)
Group interest illustrated in the previous output was published to spine COOP from leaf-103-104, we can verify the mentioned PTEPs:
spine1# acidiag fnvread | grep 10.0.168.65
103 1 leaf3 10.0.168.65/32 leaf active 0
spine1# acidiag fnvread | grep 10.0.168.70
104 1 leaf4 10.0.168.70/32 leaf active 0
Code language: PHP (php)
3- COOP Notify the border leafs about the IGMP Interest for the group:
Below is COOP log about notifying the border leaf about multicast group 239.251.1.1 interest:
COOP log:
spine-1# log_trace_bl_print_tool coop_trace.bl | grep 239.251 -A 5 -B 5 | egrep "10.0.48.65|10.0.48.64"
[2024-04-08T09:59:32.798331000+02:00:M:coop_oracle_publish_mgroup_send_buf_xmit:551] TID 28:[DBG_COOP_TRACE_DETAIL_MC]:Sending mgroup notification to borde
r leaf 10.0.48.65 num_recs=1
[2024-04-08T09:59:32.798340000+02:00:M:coop_oracle_publish_mgroup_send_buf_xmit:551] TID 28:[DBG_COOP_TRACE_DETAIL_MC]:Sending mgroup notification to borde
r leaf 10.0.48.64 num_recs=1
Code language: PHP (php)
Notifications sent to BL PTEPs:
Apic-1# acidiag fnvread
ID Pod ID Name IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
101 1 leaf1 10.0.48.65/32 leaf active 0
102 1 leaf2 10.0.48.64/32 leaf active 0
Code language: PHP (php)
4- Stripe Winner Border leaf install (*,G):
The stripe winner is responsible for sending PIM joins back to the source/RP
Checking the stripe winner via command below: we can see that we have 2 BL and winner for the group 239.251.1.1 (based on hash) is BL with Router-id 1.1.1.1, which is leaf-1 in our example:
leaf-1# show ip pim internal stripe-winner 239.251.1.1 vrf bameur_MC:VRF-MC3
PIM Stripe Winner info for VRF "bameur_MC:VRF-MC3" (BL count: 2)
(*, 239.251.1.1)
BLs:
Group hash 1827639267 VNID 2359298
1.1.1.1 hash: 1281198598 (local)
2.2.2.2 hash: 961945459
Winner: 1.1.1.1 best_hash: 1281198598
Code language: CSS (css)
As a result , leaf-1 (Stripe winner) will install (*,G) in MRIB and send (*,G) toward the RP:
leaf-1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 00:20:09, ngmvpn ip pim
Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1) (Fabric OIF)
Tunnel17, uptime: 00:20:09, ngmvpn
Incoming interface: pointing to the L3OUT interface from where we learn the RP route:
leaf-1# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.30.10/32, ubest/mbest: 1/0
*via 192.168.50.2, [20/0], 08:38:00, bgp-65001, external, tag 65033
Code language: PHP (php)
leaf-1# show ip arp vrf bameur_MC:VRF-MC3
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context bameur_MC:VRF-MC3
Total number of entries: 1
Address Age MAC Address Interface
192.168.50.2 00:13:02 a093.51ac.d741 Ethernet1/19.187
RPF pointing to the NH to reach RP (which is PIM neighbor).
leaf-1# show ip pim neighbor vrf bameur_MC:VRF-MC3
PIM Neighbor Status for VRF "bameur_MC:VRF-MC3"
Neighbor Interface Uptime Expires DR Bidir- BFD
Priority Capable State
2.2.2.2 Tunnel17 1w0d 00:01:38 1 no n/a
192.168.50.2 Ethernet1/19.187 08:39:44 00:01:30 1 yes n/a 1 yes n/a
Outgoing interface: pointing to the fabric via VRF GiPO:
leaf-1# show interface tunnel 17
Tunnel17 is up
MTU 9000 bytes, BW 9 Kbit
Transport protocol is in VRF "bameur_MC:VRF-MC3"
Tunnel protocol/transport ivxlan
Tunnel source 1.1.1.1, destination 225.1.192.48
Code language: PHP (php)
As expected, nothing installed in mRIB for the non stripe winner BL:
leaf-2# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
Group not found
Code language: CSS (css)
5-Stripe winner leaf-1 sends (*,G) PIM join to RP:
The RP receive the (*,G) from the Stripe winner and build the shared tree to the receiver:
RP# show ip mroute
(*, 239.251.1.1/32), uptime: 00:23:41, pim ip
Incoming interface: loopback155, RPF nbr: 192.168.30.10
Outgoing interface list: (count: 1)
Ethernet1/3.1058, uptime: 00:23:41, pim
Outgoing interface pointing to leaf-1, from where the (*,G) join was received.
II- Source starts sending Multicast traffic
6-Source starts sending Multicast traffic for the group
Source 10.29.1.10 located on leaf-5-6 starts sending multicast traffic for the group 239.251.1.1:
leaf6# show system internal epm endpoint ip 10.29.1.10
MAC : a093.51ac.d741 ::: Num IPs : 1
IP# 0 : 10.29.1.10 ::: IP# 0 flags : host-tracked| ::: l3-sw-hit: No
Vlan id : 50 ::: Vlan vnid : 15092 ::: VRF name : bameur_MC:VRF-MC3
BD vnid : 16318393 ::: VRF vnid : 2359298
Phy If : 0x16000001 ::: Tunnel If : 0
Interface : port-channel2
Flags : 0x80004c05 ::: sclass : 32771 ::: Ref count : 5
EP Create Timestamp : 04/08/2024 08:53:44.751902
EP Update Timestamp : 04/08/2024 10:39:48.626531
EP Flags : local|vPC|IP|MAC|sclass|timer|
7- FHMR install (S,G):
Source tree state installed in mRIB:
leaf6# show ip mroute vrf bameur_MC:VRF-MC3
(10.29.1.10/32, 239.251.1.1/32), uptime: 00:28:59, ip pim
Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
Outgoing interface list: (count: 0)
8- FHMR Send PIM Register to RP:
On the FHMR, the RP subnet must be reachable, in our example, we have a route for it pointing to the PTEP of the border leafs: Leaf-1 and Leaf-2:
leaf6# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.30.10/32, ubest/mbest: 2/0
*via 10.0.48.65%overlay-1, [200/0], 09:22:36, bgp-65001, internal, tag 65033
recursive next hop: 10.0.48.65/32%overlay-1
*via 10.0.48.64%overlay-1, [200/0], 09:22:36, bgp-65001, internal, tag 65033
recursive next hop: 10.0.48.64/32%overlay-1
Code language: PHP (php)
Like any unicast traffic, we have ECMP for the destination and based on hash, a next-hop via a BL is picked:
The FHMR leaf-6, sends PIM Register as unicast toward the RP:
leaf6# show ip pim internal event-history null-register | grep 239.251.1.1 | grep "pim_send_null_register" | more
2024-04-08T09:47:02.014681000+02:00 pim [24758]: TID 25073:pim_send_null_register:1295:(bameur_MC:VRF-MC3-base) Send Null Register to RP 192.168.30.10 for
(10.29.1.10/32, 239.251.1.1/32)
Code language: PHP (php)
9- RP install (S,G) in mRIB and send (S,G) to the source
The RP install (S,G) and starts joining the source tree:
RP# show ip mroute
(10.29.1.10/32, 239.251.1.1/32), uptime: 00:43:07, pim mrib ip
Incoming interface: Ethernet1/3.1058, RPF nbr: 192.168.50.1, internal
Outgoing interface list: (count: 0)
The RP sends (S,G) Join to the source.
From the unicast routing table of RP, I see that the route to the source subnet is via leaf-1:
RP# show ip route vrf bameur_mc
IP Route Table for VRF "bameur_mc"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.29.1.0/24, ubest/mbest: 1/0
*via 192.168.50.1, [20/0], 09:29:39, bgp-65033, external, tag 65001
Code language: PHP (php)
10- Leaf-1 receive the (S,G) join from the RP
leaf1# show ip pim internal event-history pim-internal | grep 10.29.1.10 | grep receive
2024-04-08T10:12:40.443791000+02:00 pim [4016]: TID 4463:pim_receive_join:1722:(bameur_MC:VRF-MC3-base) Route (10.29.1.10/32, 239.251.1.1/32) in PIM but not in MRIB, so force_add it to MRIB
2024-04-08T10:12:40.443782000+02:00 pim [4016]: TID 4463:pim_receive_join:1566:(bameur_MC:VRF-MC3-base) Create route for (10.29.1.10/32, 239.251.1.1/32)
Code language: PHP (php)
Leaf-1 install (S,G):
Leaf-1 act the LHMR and install the (S,G) tree:
Incoming and outgoing interfaces pointing to the fabric since source in ACI:
leaf1# show ip mroute vrf bameur_MC:VRF-MC3
(*, 239.251.1.1/32), uptime: 01:47:23, ngmvpn ip pim
Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1) (Fabric OIF)
Tunnel17, uptime: 01:47:23, ngmvpn
(10.29.1.10/32, 239.251.1.1/32), uptime: 01:21:00, pim mrib ip
Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
Outgoing interface list: (count: 1)
Tunnel17, uptime: 01:21:00, mrib, (RPF)
Once the BL Leaf-1 receives multicast traffic (via VRF GiPO, through FTAG tree) from the source, it send a Prune to the RP:
RP# show ip mroute
(*, 239.251.1.1/32), uptime: 01:58:46, pim ip
Incoming interface: loopback155, RPF nbr: 192.168.30.10
Outgoing interface list: (count: 1)
Ethernet1/3.1058, uptime: 01:58:46, pim
(10.29.1.10/32, 239.251.1.1/32), uptime: 01:32:22, pim mrib ip
Incoming interface: Ethernet1/3.1058, RPF nbr: 192.168.50.1, internal
Outgoing interface list: (count: 0): 0)
Final mRoute state:
Source Leafs (FHMR):
leaf6# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(10.29.1.10/32, 239.251.1.1/32), uptime: 01:56:44, ip pim
Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
Outgoing interface list: (count: 0)
Receiver leafs:
leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 02:23:58, igmp ip pim
Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
Outgoing interface list: (count: 1)
Vlan1, uptime: 02:23:58, igmp
Border leafs:
Stripe winner:
leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 02:24:52, ngmvpn ip pim
Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1) (Fabric OIF)
Tunnel17, uptime: 02:24:52, ngmvpn
(10.29.1.10/32, 239.251.1.1/32), uptime: 01:58:29, pim mrib ip
Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
Outgoing interface list: (count: 1)
Tunnel17, uptime: 01:58:29, mrib, (RPF)
III- Packet flow and Data-Plane Verification
1- On the source Leafs:
Source starts sending Multicast traffic:
ELAM triggered on leaf-6:
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
IP Version : 4
DSCP : 0
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : not set
TTL : 255
IP Protocol Number : ICMP
IP CheckSum : 54322( 0xD432 )
Destination IP : 239.251.1.1
Source IP : 10.29.1.10
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM FILTERS
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM Filter Key : mcast_sg_hit, fib_star_g_hit
PIM Filter Match : no
------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block: : IFABRIC_IG MC TENANT MYTEP ROUTE HIT
------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------
Lookup Drop
------------------------------------------------------------------------------------------------------------------------------------------------------
LU drop reason : no drop
Leaf-6 hit the (S,G) entry:
(10.29.1.10/32, 239.251.1.1/32), uptime: 00:21:20, ip pim
Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
Outgoing interface list: (count: 0)
In fact, to see to the complete status, we check the FIB:
leaf6# show forwarding distribution multicast route vrf bameur_MC:VRF-MC3 group 239.251.1.1
(10.29.1.10/32, 239.251.1.1/32), RPF Interface: Tunnel17, flags: O
Received Packets: 24 Bytes: 1560
Number of Outgoing Interfaces: 1
Outgoing Interface List Index: 8206
Tunnel17
As a result, the traffic forwarded to the fabric via tunnel 17 with destination as VRF GiPo:
leaf6# show interface tunnel 17
Tunnel17 is up
MTU 9000 bytes, BW 9 Kbit
Transport protocol is in VRF "bameur_MC:VRF-MC3"
Tunnel protocol/transport ivxlan
Tunnel source 127.0.0.100, destination 225.1.192.48
Code language: PHP (php)
The packet is distributed through the FTAG tree ID 10:
module-1(DBG-elam-insel6)# ereport | grep FTAG
FTAG : 10( 0xA )
Code language: PHP (php)
2- On the Receiver Leafs:
Leaf-3 (where receiver resides) receive the multicast traffic:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc udp ivxlan
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer iEth Header
------------------------------------------------------------------------------------------------------------------------------------------------------
iEth SUP code : NONE
Packet from CPU : no
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 56
Don't Fragment Bit : 0x0
TTL : 31
IP Protocol Number : UDP
Destination IP : 225.1.192.58
Source IP : 10.0.160.64
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 254
IP Protocol Number : ICMP
Destination IP : 239.251.1.1
Source IP : 10.29.1.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 0
Dst Policy Applied Bit : 0
sclass (src pcTag) : 0x8003
VRF or BD VNID : 2359298( 0x240002 )
FINAL FORWARDING LOOKUP
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block: : IFABRIC_EG MC INFRA ENCAP MYTEP ROUTE HIT
------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------
Lookup Drop
------------------------------------------------------------------------------------------------------------------------------------------------------
LU drop reason : no drop
Code language: PHP (php)
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG : 10( 0xA )
Code language: PHP (php)