ACI L3 Multicast Troubleshoot – Part-3: Source and Receiver inside ACI – Topology 2

ACI L3 Multicast Troubleshoot – Part-3: Source and Receiver inside ACI – Topology 2

Topology:


I- Receiver sends out IGMP Report

In this first, we focus on Receiver side and how shared tree is formed (no traffic from source yet):


1- Receiver sends IGMP Join:

Upon receive of the IGMP join from receiver for the group 239.251.1.1, IGMP entry added to leaf-3-4:

leaf3# show ip igmp groups vrf bameur_MC:VRF-MC3
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
IGMP Connected Group Membership for VRF "bameur_MC:VRF-MC3"
Group Address        Type   Interface    Uptime               Expires              Last Reporter       
239.251.1.1          D      vlan1        00:06:08             00:03:18             172.16.30.100            172.16.30.100Code language: CSS (css)


(*,G) tree installed to leaf-3-4 (where receiver resides):

leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 00:06:35, igmp ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
  Outgoing interface list: (count: 1)
    Vlan1, uptime: 00:06:35, igmp

Incoming interface: pointing to the fabric via tunnel interface with destination being VRF GiPo:

leaf3# show interface tunnel 29
Tunnel29 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport is ivxlan
    Tunnel source 127.0.0.100/32
    Tunnel destination 225.1.192.48/32Code language: PHP (php)

RPF pointing to root of the shared: the RP 192.168.30.10 (RPF neighbor to reach RP is BL PTEP 10.0.48.64)

Outgoing Interface: pointing to the BD VLAN of the Receiver EP


2- IGMP Join interest Published to COOP:

After receiving interest from the receiver EP via IGMP Join, Leaf-3-4 will publish it to COOP:

COOP log:

Spine-1# log_trace_bl_print_tool coop_trace.bl | grep 239.251 -A 5 -B 5 | egrep "10.0.168.65|10.0.168.70"

[2024-04-08T09:46:17.361525000+02:00:D:coop_oracle_process_publish:1439] TID 36:[DBG_COOP_TRACE_DETAIL]:(1712562380:959405422) Received PUBLISH message from 10.0.168.70 Publisher_id: 10.0.168.70 with trans_id 1552
[2024-04-08T09:46:17.361557000+02:00:D:coop_objs_add_mgroup_link_generic:9460] TID 36:[DBG_COOP_TRACE_DETAIL]:Add mgv4 <1, 0.0.0.0, 239.251.1.1, 240002> to leaf 10.0.168.70 linkage 

[2024-04-08T09:46:17.364210000+02:00:D:coop_oracle_process_publish:1439] TID 17:[DBG_COOP_TRACE_DETAIL]:(1712562380:958776264) Received PUBLISH message from 10.0.168.65 Publisher_id: 10.0.168.65 with trans_id 7954
[2024-04-08T09:46:17.364220000+02:00:D:coop_objs_add_mgroup_link_generic:9460] TID 17:[DBG_COOP_TRACE_DETAIL]:Add mgv4 <1, 0.0.0.0, 239.251.1.1, 240002> to leaf 10.0.168.65 linkage Code language: HTML, XML (xml)


verification in COOP DB:

spine1# show coop internal info repo mgroup

Repo Hdr Checksum : 31363
Repo Hdr record timestamp : 04 08 2024 07:53:10 284993671
Repo Hdr last pub timestamp : 04 08 2024 07:53:10 285217368
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ EXPORT 
VRF Vnid : 2359298 
mgroup src ip : 0.0.0.0 
mgroup group ip : 239.251.1.1 
Flags : 0x0x1 afi 0
Local leafs 2 (active: 2 deleted: 0)

Leaf 0 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 04 08 2024 07:51:21 457853613
Leaf Repo Hdr last pub timestamp : 04 08 2024 07:51:21 458024081
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.168.65

Leaf Flags : 0x0 
Leaf 1 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 04 08 2024 07:53:10 284993671
Leaf Repo Hdr last pub timestamp : 04 08 2024 07:53:10 285217368
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.168.70
Leaf Flags : 0x0 

oldest local publish timestamp: 04 08 2024 07:46:20 959376706
MPOD Pub id : 0.0.0.0 
MPOD leafs 0 (active: 0 deleted: 0)
MSITE Pub id : 0.0.0.0 
MSITE leafs 0 (active: 0 deleted: 0)
Hash: 990247176 owner: 10.0.168.71Code language: CSS (css)


Group interest illustrated in the previous output was published to spine COOP from leaf-103-104, we can verify the mentioned PTEPs:

spine1# acidiag fnvread | grep 10.0.168.65
     103        1    leaf3          10.0.168.65/32    leaf         active   0
spine1# acidiag fnvread | grep 10.0.168.70
     104        1    leaf4          10.0.168.70/32    leaf         active   0Code language: PHP (php)


3- COOP Notify the border leafs about the IGMP Interest for the group:

Below is COOP log about notifying the border leaf about multicast group 239.251.1.1 interest:

COOP log:

spine-1# log_trace_bl_print_tool coop_trace.bl | grep 239.251 -A 5 -B 5 | egrep "10.0.48.65|10.0.48.64"

[2024-04-08T09:59:32.798331000+02:00:M:coop_oracle_publish_mgroup_send_buf_xmit:551] TID 28:[DBG_COOP_TRACE_DETAIL_MC]:Sending mgroup notification to borde
r leaf 10.0.48.65 num_recs=1
[2024-04-08T09:59:32.798340000+02:00:M:coop_oracle_publish_mgroup_send_buf_xmit:551] TID 28:[DBG_COOP_TRACE_DETAIL_MC]:Sending mgroup notification to borde
r leaf 10.0.48.64 num_recs=1Code language: PHP (php)


Notifications sent to BL PTEPs:

Apic-1# acidiag fnvread
     ID   Pod ID    Name           IP Address    Role        State   LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
     101        1    leaf1       10.0.48.65/32    leaf         active   0
     102        1    leaf2       10.0.48.64/32    leaf         active   0Code language: PHP (php)


4- Stripe Winner Border leaf install (*,G):

The stripe winner is responsible for sending PIM joins back to the source/RP

Checking the stripe winner via command below: we can see that we have 2 BL and winner for the group 239.251.1.1 (based on hash) is BL with Router-id 1.1.1.1, which is leaf-1 in our example:

leaf-1# show ip pim internal stripe-winner 239.251.1.1 vrf bameur_MC:VRF-MC3
PIM Stripe Winner info for VRF "bameur_MC:VRF-MC3" (BL count: 2)
(*, 239.251.1.1)
BLs:  
Group hash 1827639267 VNID 2359298
1.1.1.1  hash: 1281198598 (local)
      2.2.2.2  hash: 961945459
Winner: 1.1.1.1 best_hash: 1281198598Code language: CSS (css)


As a result , leaf-1 (Stripe winner) will install (*,G) in MRIB and send (*,G) toward the RP:

leaf-1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 00:20:09, ngmvpn ip pim 
  Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel17, uptime: 00:20:09, ngmvpn

Incoming interface: pointing to the L3OUT interface from where we learn the RP route:

leaf-1# show ip route 192.168.30.10 vrf  bameur_MC:VRF-MC3

IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

192.168.30.10/32, ubest/mbest: 1/0
    *via 192.168.50.2, [20/0], 08:38:00, bgp-65001, external, tag 65033Code language: PHP (php)
leaf-1# show ip arp vrf bameur_MC:VRF-MC3

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       D - Static Adjacencies attached to down interface

IP ARP Table for context bameur_MC:VRF-MC3
Total number of entries: 1
Address         Age       MAC Address     Interface
192.168.50.2    00:13:02  a093.51ac.d741  Ethernet1/19.187


RPF pointing to the NH to reach RP (which is PIM neighbor).

leaf-1# show ip pim neighbor vrf bameur_MC:VRF-MC3

PIM Neighbor Status for VRF "bameur_MC:VRF-MC3"
Neighbor        Interface            Uptime    Expires   DR       Bidir-  BFD
                                                         Priority Capable State
2.2.2.2         Tunnel17             1w0d      00:01:38  1        no      n/a  
192.168.50.2    Ethernet1/19.187     08:39:44  00:01:30  1        yes     n/a            1          yes        n/a 


Outgoing interface: pointing to the fabric via VRF GiPO:

leaf-1# show interface tunnel 17

Tunnel17 is up
    MTU 9000 bytes, BW 9 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport ivxlan
    Tunnel source 1.1.1.1, destination 225.1.192.48Code language: PHP (php)


As expected, nothing installed in mRIB for the non stripe winner BL:

leaf-2# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

Group not foundCode language: CSS (css)


5-Stripe winner leaf-1 sends (*,G) PIM join to RP:

The RP receive the (*,G) from the Stripe winner and build the shared tree to the receiver:

RP# show ip mroute

(*, 239.251.1.1/32), uptime: 00:23:41, pim ip 
  Incoming interface: loopback155, RPF nbr: 192.168.30.10
  Outgoing interface list: (count: 1)
    Ethernet1/3.1058, uptime: 00:23:41, pim

Outgoing interface pointing to leaf-1, from where the (*,G) join was received.


II- Source starts sending Multicast traffic

6-Source starts sending Multicast traffic for the group

Source 10.29.1.10 located on leaf-5-6 starts sending multicast traffic for the group 239.251.1.1:

leaf6# show system internal epm endpoint ip 10.29.1.10

MAC : a093.51ac.d741 ::: Num IPs : 1
IP# 0 : 10.29.1.10 ::: IP# 0 flags : host-tracked| ::: l3-sw-hit: No
Vlan id : 50 ::: Vlan vnid : 15092 ::: VRF name : bameur_MC:VRF-MC3
BD vnid : 16318393 ::: VRF vnid : 2359298
Phy If : 0x16000001 ::: Tunnel If : 0
Interface : port-channel2
Flags : 0x80004c05 ::: sclass : 32771 ::: Ref count : 5
EP Create Timestamp : 04/08/2024 08:53:44.751902
EP Update Timestamp : 04/08/2024 10:39:48.626531
EP Flags : local|vPC|IP|MAC|sclass|timer|


7- FHMR install (S,G):

Source tree state installed in mRIB:

leaf6# show ip mroute vrf bameur_MC:VRF-MC3

(10.29.1.10/32, 239.251.1.1/32), uptime: 00:28:59, ip pim 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


8- FHMR Send PIM Register to RP:

On the FHMR, the RP subnet must be reachable, in our example, we have a route for it pointing to the PTEP of the border leafs: Leaf-1 and Leaf-2:

leaf6# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

192.168.30.10/32, ubest/mbest: 2/0
    *via 10.0.48.65%overlay-1, [200/0], 09:22:36, bgp-65001, internal, tag 65033
         recursive next hop: 10.0.48.65/32%overlay-1
    *via 10.0.48.64%overlay-1, [200/0], 09:22:36, bgp-65001, internal, tag 65033
         recursive next hop: 10.0.48.64/32%overlay-1Code language: PHP (php)

Like any unicast traffic, we have ECMP for the destination and based on hash, a next-hop via a BL is picked:

The FHMR leaf-6, sends PIM Register as unicast toward the RP:

leaf6# show ip pim internal event-history null-register | grep 239.251.1.1 | grep "pim_send_null_register" | more
2024-04-08T09:47:02.014681000+02:00 pim [24758]: TID 25073:pim_send_null_register:1295:(bameur_MC:VRF-MC3-base) Send Null Register to RP 192.168.30.10 for 
(10.29.1.10/32, 239.251.1.1/32)Code language: PHP (php)


9- RP install (S,G) in mRIB and send (S,G) to the source

The RP install (S,G) and starts joining the source tree:

RP# show ip mroute 

(10.29.1.10/32, 239.251.1.1/32), uptime: 00:43:07, pim mrib ip 
  Incoming interface: Ethernet1/3.1058, RPF nbr: 192.168.50.1, internal
  Outgoing interface list: (count: 0)


The RP sends (S,G) Join to the source.

From the unicast routing table of RP, I see that the route to the source subnet is via leaf-1:

RP# show ip route vrf bameur_mc
IP Route Table for VRF "bameur_mc"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.29.1.0/24, ubest/mbest: 1/0
    *via 192.168.50.1, [20/0], 09:29:39, bgp-65033, external, tag 65001Code language: PHP (php)


10- Leaf-1 receive the (S,G) join from the RP

leaf1# show ip pim internal event-history pim-internal | grep 10.29.1.10 | grep receive 
2024-04-08T10:12:40.443791000+02:00 pim [4016]: TID 4463:pim_receive_join:1722:(bameur_MC:VRF-MC3-base) Route (10.29.1.10/32, 239.251.1.1/32) in PIM but not in MRIB, so force_add it to MRIB
2024-04-08T10:12:40.443782000+02:00 pim [4016]: TID 4463:pim_receive_join:1566:(bameur_MC:VRF-MC3-base) Create route for (10.29.1.10/32, 239.251.1.1/32)Code language: PHP (php)


Leaf-1 install (S,G):

Leaf-1 act the LHMR and install the (S,G) tree:

Incoming and outgoing interfaces pointing to the fabric since source in ACI:

leaf1# show ip mroute vrf bameur_MC:VRF-MC3

(*, 239.251.1.1/32), uptime: 01:47:23, ngmvpn ip pim 
  Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel17, uptime: 01:47:23, ngmvpn


(10.29.1.10/32, 239.251.1.1/32), uptime: 01:21:00, pim mrib ip 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1)
    Tunnel17, uptime: 01:21:00, mrib, (RPF)


Once the BL Leaf-1 receives multicast traffic (via VRF GiPO, through FTAG tree) from the source, it send a Prune to the RP:

RP# show ip mroute 

(*, 239.251.1.1/32), uptime: 01:58:46, pim ip 
  Incoming interface: loopback155, RPF nbr: 192.168.30.10
  Outgoing interface list: (count: 1)
    Ethernet1/3.1058, uptime: 01:58:46, pim


(10.29.1.10/32, 239.251.1.1/32), uptime: 01:32:22, pim mrib ip 
  Incoming interface: Ethernet1/3.1058, RPF nbr: 192.168.50.1, internal
  Outgoing interface list: (count: 0): 0)


Final mRoute state:

Source Leafs (FHMR):

leaf6# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(10.29.1.10/32, 239.251.1.1/32), uptime: 01:56:44, ip pim 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


Receiver leafs:

leaf3# show ip mroute vrf bameur_MC:VRF-MC3 
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 02:23:58, igmp ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
  Outgoing interface list: (count: 1)
    Vlan1, uptime: 02:23:58, igmp


Border leafs:

Stripe winner:

leaf1# show ip mroute vrf bameur_MC:VRF-MC3 
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 02:24:52, ngmvpn ip pim 
  Incoming interface: Ethernet1/19.187, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel17, uptime: 02:24:52, ngmvpn


(10.29.1.10/32, 239.251.1.1/32), uptime: 01:58:29, pim mrib ip 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1)
    Tunnel17, uptime: 01:58:29, mrib, (RPF)


III- Packet flow and Data-Plane Verification

1- On the source Leafs:

Source starts sending Multicast traffic:

ELAM triggered on leaf-6:

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
IP Version                    : 4                                       
DSCP                          : 0                                       
IP Packet Length              : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit            : not set                                 
TTL                           : 255                                     
IP Protocol Number            : ICMP                                    
IP CheckSum                   : 54322( 0xD432 )                         
Destination IP                : 239.251.1.1                             
Source IP                     : 10.29.1.10   



------------------------------------------------------------------------------------------------------------------------------------------------------
PIM FILTERS
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM Filter Key                          : mcast_sg_hit,  fib_star_g_hit 
PIM Filter Match                        : no                            

------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP 
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block:     : IFABRIC_IG MC TENANT MYTEP ROUTE HIT 

------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------------------------------
Lookup Drop
------------------------------------------------------------------------------------------------------------------------------------------------------
LU drop reason                          : no drop      


Leaf-6 hit the (S,G) entry:

(10.29.1.10/32, 239.251.1.1/32), uptime: 00:21:20, ip pim 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


In fact, to see to the complete status, we check the FIB:

leaf6# show forwarding distribution multicast route vrf bameur_MC:VRF-MC3 group 239.251.1.1

  (10.29.1.10/32, 239.251.1.1/32), RPF Interface: Tunnel17, flags: O
    Received Packets: 24 Bytes: 1560 
    Number of Outgoing Interfaces: 1
    Outgoing Interface List Index: 8206
      Tunnel17


As a result, the traffic forwarded to the fabric via tunnel 17 with destination as VRF GiPo:

leaf6# show interface tunnel 17
Tunnel17 is up
    MTU 9000 bytes, BW 9 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport ivxlan
    Tunnel source 127.0.0.100, destination 225.1.192.48Code language: PHP (php)


The packet is distributed through the FTAG tree ID 10:

module-1(DBG-elam-insel6)# ereport  | grep FTAG
FTAG                                    : 10( 0xA )Code language: PHP (php)


2- On the Receiver Leafs:

Leaf-3 (where receiver resides) receive the multicast traffic:

======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2mc ipv4 ip ipmc ipv4mc udp ivxlan     
Opcode                        : OPCODE_L3MC                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer iEth Header
------------------------------------------------------------------------------------------------------------------------------------------------------
iEth SUP code                 : NONE                                    
Packet from CPU               : no                                                                          

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 56                                      
Don't Fragment Bit            : 0x0                                     
TTL                           : 31                                      
IP Protocol Number            : UDP                                     
Destination IP                : 225.1.192.58                            
Source IP                     : 10.0.160.64                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x0                                     
TTL                           : 254                                     
IP Protocol Number            : ICMP                                    
Destination IP                : 239.251.1.1                             
Source IP                     : 10.29.1.10                              

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : iVxLAN                                  
Don't Learn Bit               : 0                                       
Src Policy Applied Bit        : 0                                       
Dst Policy Applied Bit        : 0                                       
sclass (src pcTag)            : 0x8003                                  
VRF or BD VNID                : 2359298( 0x240002 )     


FINAL FORWARDING LOOKUP 
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block:     : IFABRIC_EG MC INFRA ENCAP MYTEP ROUTE HIT 

------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------------------------------
Lookup Drop
------------------------------------------------------------------------------------------------------------------------------------------------------
LU drop reason                          : no drop                   
Code language: PHP (php)
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG                                    : 10( 0xA )                     Code language: PHP (php)



You can review the previous parts from the following:

Bilel-A

Enthusiastic Network Engineer specializing in Cisco ACI, passionate about solving challenges. A lifelong learner who loves gaining and sharing knowledge. Profile: https://www.linkedin.com/in/bilel-ameur-71116b2b5
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x