Troubleshoot ACI L3 Multicast – Part-2: Source in ACI and Receiver Outside

Topology:


I- IGMP Join from Receiver Outside of ACI

1- Receiver send IGMP Report

IGMP Report received from the Receiver for the multicast group 239.252.1.1 (in our topology, it happen that receiver is connected to the RP, but not relevant for ACI part):

RP# show ip igmp groups 

Group Address      Type Interface              Uptime    Expires   Last Reporter
239.252.1.1        D    Ethernet1/48           00:00:15  00:04:09  172.16.20.3


RP build the (*,G):

(*, 239.252.1.1/32) installed on the RP and outgoing interface pointing to the receiver:

RP# show ip mroute

(*, 239.252.1.1/32), uptime: 00:07:23, igmp ip pim 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 1)
    Ethernet1/48, uptime: 00:07:23, igmp


II- Source starts sending multicast traffic


3- Source start transmitting multicast traffic

  • Source 172.16.30.100 start sending multicast traffic for the group 239.252.1.1

On the connected leaf-3-4, we have local learn for the source EP 172.16.30.100:

leaf3# show system internal epm endpoint ip 172.16.30.100

MAC : 0050.56a0.4aaa ::: Num IPs : 1
IP# 0 : 172.16.30.100 ::: IP# 0 flags : peer-aged|host-tracked| ::: l3-sw-hit: Yes ::: flags2 : 
Vlan id : 14 ::: Vlan vnid : 14897 ::: VRF name : bameur_MC:VRF-MC3
BD vnid : 14909454 ::: VRF vnid : 2359298
Phy If : 0x16000004 ::: Tunnel If : 0
Interface : port-channel5
Flags : 0x80005c05 ::: sclass : 16386 ::: Ref count : 5
EP Create Timestamp : 03/26/2024 15:39:33.859945
EP Update Timestamp : 04/01/2024 02:43:08.896538
EP Flags : local|vPC|IP|MAC|host-tracked|sclass|timer|


4- FHMR Leaf-3-4 install (S,G)

  • (S,G) entry is installed pm leaf-3-4 (FHMR):
leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 00:54:45, ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


Incoming interface is pointing to the fabric (via Tunnel with destination VRF GiPo)

leaf3# show interface tunnel 29
Tunnel29 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport is ivxlan
    Tunnel source 127.0.0.100/32
    Tunnel destination 225.1.192.48/32Code language: PHP (php)


The RPF (Root of the source tree) is the Pervasive since source is an EP learned in ACI and pointing to the spine PROXY-ACAST-V4 (10.0.128.65).

leaf3# show isis dteps vrf overlay-1

IS-IS Dynamic Tunnel End Point (DTEP) database:
DTEP-Address       Role    Encapsulation   Type                          
10.0.168.66        SPINE   N/A             PHYSICAL                      
10.0.128.64        SPINE   N/A             PHYSICAL,PROXY-ACAST-MAC      
10.0.128.65        SPINE   N/A             PHYSICAL,PROXY-ACAST-V4       
10.0.128.66        SPINE   N/A             PHYSICAL,PROXY-ACAST-V6Code language: PHP (php)


5- PIM Register sent to RP by the FHMR

  • Source leaf-3-4 (FHMR) send PIM Register (unicast) to RP:

Mainly, the first multicast packet to the multicast group hitting the leaf-3-4 is redirected to the leaf CPU and PIM Register is crafted (IP encapsulating the multicast packet) and sent as unicast to the RP for registering purpose:

Basically, since we learn the route to the RP from both border leafs (leaf-1 and leaf-2), for the PIM Register to RP IP (192.168.30.10), it’s based on ECMP hash similar to any unicast traffic :

leaf3# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3

192.168.30.10/32, ubest/mbest: 2/0
    *via 10.0.48.65%overlay-1, [200/0], 01:58:52, bgp-65001, internal, tag 65033
         recursive next hop: 10.0.48.65/32%overlay-1
    *via 10.0.48.64%overlay-1, [200/0], 02:21:50, bgp-65001, internal, tag 65033
         recursive next hop: 10.0.48.64/32%overlay-1Code language: PHP (php)
leaf3# acidiag fnvread | grep 101
     101        1    leaf1        10.0.48.65/32    leaf         active   0

leaf3# acidiag fnvread | grep 102
     102        1    leaf2        10.0.48.64/32    leaf         active   0Code language: PHP (php)


Note

in ACI, for the PIM Register message sent by FHMR, by default, uses the source BD SVI (subnet IP) as source IP. For the RP to be able to respond with Register stop, BD subnet should be advertised externally, otherwise, we could end up stuck in registering state.


6- RP receive the PIM Register and send (S,G) Join to the source

  • RP receives the PIM Register message and will start joining the source tree by sending an (S,G) Join to the source router:

From the RPF of the (S,G) on the RP router, I see that it’s pointing to leaf-2 (it depends on uRIB (routing table of RP) to the source subnet):

RP# show ip mroute vrf bameur_mc 
IP Multicast Routing Table for VRF "bameur_mc"


(*, 239.252.1.1/32), uptime: 01:45:50, igmp ip pim 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 1)
    Ethernet1/48, uptime: 01:45:50, igmp


(172.16.30.100/32, 239.252.1.1/32), uptime: 01:11:50, pim mrib ip 
  Incoming interface: Ethernet1/7.1058, RPF nbr: 192.168.60.1, internal
  Outgoing interface list: (count: 1)
    Ethernet1/48, uptime: 01:11:50, mrib


Border leaf-2 Receive the (S,G) Join

  • The border leaf Leaf-2 will receive the (S,G) join from RP and will install (S,G) entry in its mRIB:
leaf2# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:21:39, pim ip 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1)
    Ethernet1/19.205, uptime: 01:21:39, pim

The incoming interface: pointing to the fabric (VRF GiPo)

leaf2# show interface Tunnel17
Tunnel17 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport is ivxlan
    Tunnel source 2.2.2.2
    Tunnel destination 225.1.192.48/32Code language: PHP (php)

RPF is pointing to pervasive (Spine proxy) toward the source.

The Outgoing interface: Ethernet1/19.205 L3OUT to the RP (Interface from which the (S,G) join was received)


On the other Border leaf (leaf-1):

we see Incoming interface and outgoing interface for the (S,G) are pointing to the fabric (Fabric OIF)

(since (S,G) join from RP was received on leaf-2, the multicast traffic from source is expected to exist from leaf-2)

leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:15:20, ngmvpn ip pim 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel17, uptime: 01:15:20, ngmvpn, (RPF)



leaf1# show ip pim internal stripe-winner 239.252.1.1 vrf  bameur_MC:VRF-MC3
PIM Stripe Winner info for VRF "bameur_MC:VRF-MC3" (BL count: 2)
(*, 239.252.1.1)
BLs:  
Group hash 995921891 VNID 2359298
1.1.1.1  hash: 636258822 (local)
      2.2.2.2  hash: 139927411
Winner: 1.1.1.1 best_hash: 636258822


7- BL Leaf-2 Publish mgroup to COOP (including group and source information)

  • Leaf-2, which received the (S,G) PIM join from the RP, will publish this mcast group and source information to the spine COOP:
spine1# show coop internal info repo mgroup 

Repo Hdr Checksum : 19683
Repo Hdr record timestamp : 04 01 2024 12:19:44 746614155
Repo Hdr last pub timestamp : 04 01 2024 12:19:44 746745093
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ EXPORT 
VRF Vnid : 2359298 
mgroup src ip : 172.16.30.100 
mgroup group ip : 239.252.1.1 
Flags : 0x0x1 afi 0
Local leafs 1 (active: 1 deleted: 0)

Leaf 0 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 04 01 2024 12:19:44 746614155
Leaf Repo Hdr last pub timestamp : 04 01 2024 12:19:44 746745093
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.48.64
Leaf Flags : 0x0 

spine1# acidiag fnvread | grep 10.0.48.64
     102        1   leaf2       10.0.48.64/32    leaf         active   0Code language: PHP (php)

We see from the previous output that this (S,G) Join information represented in mgroup is published by the border leaf leaf-2.

(will the COOP notify the source leafs (LEAF-3-4) about this?)

8- RP send a register-stop message to the BL leaf-2

Once the RP start receiving multicast traffic natively on (S,G) tree (not via the Register message), the RP will send a Register-stop message to the source. indicating that (S,G) was built from the RP to the source.

On the RP:

we see that RP send out Register-Stop to 172.16.30.254 (SVI or subnet for the BD where source reside, because the PIM Register was source from 172.16.30.254 in the first place):

2024 Apr  1 12:47:15.844578 pim [32647]: (bameur_mc) : Send Register-Stop to 172.16.30.254 for (172.16.30.100/32, 239.252.1.1/32)
2024 Apr  1 12:47:15.844553 pim [32647]: (bameur_mc) : Received NULL Register from 172.16.30.254 for (172.16.30.100/32, 239.252.1.1/32) (pktlen 20) Code language: JavaScript (javascript)


The Border leaf “leaf-2” receive this PIM Register-stop:

2024-04-01T14:46:29.018166000+02:00 pim [63423]: TID 63833:pim_receive_register_stop:1830:(bameur_MC:VRF-MC3-base) Received Register-Stop from 192.168.30.1
0 for (172.16.30.100/32, 239.252.1.1/32)Code language: JavaScript (javascript)


Final Mroute:

RP (also LHMR for our topology):

RP# show ip mroute vrf bameur_mc 
IP Multicast Routing Table for VRF "bameur_mc"

(*, 239.252.1.1/32), uptime: 14:12:56, igmp ip pim 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 1)
    Ethernet1/48, uptime: 14:12:56, igmp


(172.16.30.100/32, 239.252.1.1/32), uptime: 01:41:54, pim mrib ip 
  Incoming interface: Ethernet1/7.1058, RPF nbr: 192.168.60.1, internal
  Outgoing interface list: (count: 1)
    Ethernet1/48, uptime: 01:41:54, mrib


Leaf-2 (BL):

leaf2# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:44:30, pim ip 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1)
    Ethernet1/19.205, uptime: 01:44:30, pim


Leaf-1 (BL)

leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:45:15, ngmvpn ip pim 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel17, uptime: 01:45:15, ngmvpn, (RPF)


Leaf-3 (FHMR):

bdsol-aci02-leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"


(172.16.30.100/32, 239.252.1.1/32), uptime: 01:46:22, ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


Packet flow and Data-Plane Verification

ELAM on the FHMR (leaf-3-4):

Multicast traffic from source 172.16.30.100 to group 239.252.1.1 hit on leaf-3:

======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2mc ipv4 ip ipmc ipv4mc udp            
Opcode                        : OPCODE_L3MC                             


------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
IP Version                    : 4                                       
DSCP                          : 0                                       
IP Packet Length              : 1498 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit            : set                                     
TTL                           : 32                                      
IP Protocol Number            : UDP                                     
IP CheckSum                   : 37338( 0x91DA )                         
Destination IP                : 239.252.1.1                             
Source IP                     : 172.16.30.100                           

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : UDP                                     
Source Port                   : 46463( 0xB57F )                         
Destination Port              : 5001( 0x1389 )                          
TCP/UDP CheckSum              : 0x478A( 0x478A )  
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM FILTERS
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM Filter Key                          : mcast_sg_hit,  fib_star_g_hit 
PIM Filter Match                        : no                            

------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP 
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block:     : IFABRIC_IG MC TENANT MYTEP ROUTE HIT 

------------------------------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------------------------------
Lookup Drop
------------------------------------------------------------------------------------------------------------------------------------------------------
LU drop reason                          : no drop                       Code language: PHP (php)


we see we hit (S,G):

  • The Incoming interface for the (S,G) is the fabric and RPF is Spine proxy indicating that the source for this (S,G) is in ACI fabric
  • Despite the (S,G) is showing no OIL (no outgoing interface)
leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:56:34, ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 0)


In fact, if we check MFDM (Sup), we see that Tunnel29 is also the outgoing interface for this (S,G) on the FHMR (leaf-3):

leaf3# show forwarding distribution multicast route vrf bameur_MC:VRF-MC3 group 239.252.1.1

  (172.16.30.100/32, 239.252.1.1/32), RPF Interface: Tunnel29, flags: O
    Received Packets: 128 Bytes: 8320 
    Number of Outgoing Interfaces: 1
    Outgoing Interface List Index: 8212
    Platform Index: 0x0
      Tunnel29


Meaning that source traffic received on leaf-3 will be sent out via Tunnel29, which is the VRF GiPO:

Tunnel29 is up
    MTU 9000 bytes, BW 9 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport ivxlan
    Tunnel source 127.0.0.100, destination 225.1.192.48Code language: JavaScript (javascript)


This traffic should then be received by the Border leafs:

ELAM on Border leaf ‘Leaf-2’:

Leaf-2 is will get traffic from VRF GiPo and send it out via L3OUT interface toward receiver (or RP if SPT not formed by LHMR):

leaf2# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(172.16.30.100/32, 239.252.1.1/32), uptime: 01:44:30, pim ip 
  Incoming interface: Tunnel17, RPF nbr: 10.0.128.65 (pervasive)
  Outgoing interface list: (count: 1)
    Ethernet1/19.205, uptime: 01:44:30, pim
======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2mc ipv4 ip ipmc ipv4mc udp ivxlan     
Opcode                        : OPCODE_L3MC                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer iEth Header
------------------------------------------------------------------------------------------------------------------------------------------------------
iEth SUP code                 : NONE                                    
Packet from CPU               : no                                      



------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x0                                     
TTL                           : 31                                      
IP Protocol Number            : UDP                                     
Destination IP                : 225.0.218.140                           
Source IP                     : 10.0.128.67                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x1                                     
TTL                           : 32                                      
IP Protocol Number            : UDP                                     
Destination IP                : 239.252.1.1                             
Source IP                     : 172.16.30.100                           

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : iVxLAN                                  
Don't Learn Bit               : 1                                       
Src Policy Applied Bit        : 0                                       
Dst Policy Applied Bit        : 0                                       
sclass (src pcTag)            : 0x4002                                  
VRF or BD VNID                : 14897( 0x3A31 )                         

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : UDP                                     
Source Port                   : 46463                                   
Destination Port              : 5001   Code language: PHP (php)


It seems that destination using source EP BD FTAG!

module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG                                    : 12( 0xC )Code language: PHP (php)
# fv.BDDef
bdDn                     : uni/tn-bameur_MC/BD-Receiver_BD
isSvc                    : no
OptimizeWanBandwidth     : no
accEncap                 : unknown
arpFlood                 : yes
bcastP                   : 225.0.218.128
bdStretch                : no
childAction              : 
configIssues             : l3-mcast-is-enabled-on-vrf-where-bd-has-ip-learn-disable
ctxDefDn                 : uni/ctx-[uni/tn-bameur_MC/ctx-VRF-MC3]
ctxDefStQual             : none
descr                    : 
dn                       : uni/bd-[uni/tn-bameur_MC/BD-Receiver_BD]-isSvc-noCode language: PHP (php)

Bilel-A

Enthusiastic Network Engineer specializing in Cisco ACI, passionate about solving challenges. A lifelong learner who loves gaining and sharing knowledge. Profile: https://www.linkedin.com/in/bilel-ameur-71116b2b5

Related articles

ACI L3 Multicast Troubleshoot – Part-3: Source and Receiver inside ACI – Topology 2

ACI L3 Multicast Troubleshoot – Part-3: Source and Receiver inside ACI – Topology 1

ACI L3 Multicast Troubleshoot – Part-1: Receiver in ACI and Source Outside

Quality of Service (QoS) in Cisco ACI – Overview and Configuration Examples

MACsec configuration in ACI Fabric

Cisco ACI Floating L3OUT [Explained & Configuration]

ACI Multiste PBR Configuration and Troubleshooting

ACI Route Redistribution

ACI Q in Q Dot1Q Tunnel Data Forwarding

ACI Multisite Connectivity Design and Deployment use cases

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x