ACI L3 Multicast Troubleshoot – Part-1: Receiver in ACI and Source Outside

Topology:


Pre-checks

  • Verify PIM neighbor is up between Border leaf and External L3OUT node:
leaf1# show ip pim neighbor vrf bameur_MC:VRF-MC3

PIM Neighbor information for Dom:bameur_MC:VRF-MC3
Neighbor             Interface            Uptime               Expires              DRPriority Bidir      BFDState  
192.168.50.2/32      eth1/19.213          5d3h                 00:01:31             1          yes        n/a       
2.2.2.2/32           tunnel21             5d3h                 00:01:44             1          no         n/a       


  • verify that multicast Border were registered in COOP mrouter on the spine:
spine1# show coop internal info repo mrouter key 2359298 

Repo Hdr Checksum : 32349
Repo Hdr record timestamp : 03 26 2024 09:37:24 500587101
Repo Hdr last pub timestamp : 03 31 2024 13:03:32 868964254
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ 
BD Vnid : 2359298 
flags : 0x4 

num of leafs in record : 2 
num of valid leafs in record : 2 

Leaf 0 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 26 2024 09:37:24 500587101
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:49:08 590196850
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.48.64
Leaf Flags : 0x4 MCAST 

Leaf 1 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 26 2024 09:37:24 498508281
Leaf Repo Hdr last pub timestamp : 03 31 2024 13:03:32 868964254
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.48.65
Leaf Flags : 0x4 MCAST 
Hash: 3519002267 owner: 10.0.168.66Code language: CSS (css)


I- No traffic from the source


1- Receiver sends IGMP report for the group

The Receiver Host show its intent to receive Multicast traffic for a group by sending IGMP Report (called IGMP join as well), we can verify on leaf-3 via command “show ip igmp groups vrf VRF_name”:

leaf3# show ip igmp groups vrf bameur_MC:VRF-MC3
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
IGMP Connected Group Membership for VRF "bameur_MC:VRF-MC3"
Group Address        Type   Interface    Uptime               Expires              Last Reporter       
239.251.1.1          D      vlan1        00:00:39             00:03:50             172.16.30.100 Code language: CSS (css)


we can see that EP 172.16.30.100 reported its interest to receive Multicast traffic for group 239.251.1.1 and we can see in our example that it was on vlan1.

Checking further, we can see that vlan 1 is the PIVLAN Id for the BD where the Receiver endpoints resides:

leaf3# show system internal epm endpoint ip 172.16.30.100

MAC : 0050.56b0.aaa8 ::: Num IPs : 1
IP# 0 : 172.16.30.100 ::: IP# 0 flags : peer-aged|host-tracked| ::: l3-sw-hit: No
Vlan id : 14 ::: Vlan vnid : 14897 ::: VRF name : bameur_MC:VRF-MC3
BD vnid : 14909454 ::: VRF vnid : 2359298
Phy If : 0x16000004 ::: Tunnel If : 0
Interface : port-channel5
Flags : 0x80005c25 ::: sclass : 16386 ::: Ref count : 5

EP Flags : local|vPC|peer-aged|IP|MAC|host-tracked|sclass|timer|



leaf3# show system internal epm vlan 1
+----------+---------+-----------------+----------+------+----------+-----------
   VLAN ID    Type      Access Encap     Fabric    H/W id  BD VLAN    Endpoint  
                        (Type Value)     Encap                          Count   
+----------+---------+-----------------+----------+------+----------+-----------
 1          Tenant BD NONE            0 14909454   16     1          0         


2- Receiver Leaf install (*,G) and publish group interest to the Spine COOP

We can see on the leaf-3 where receiver is located that an (*,G) mroute was installed:

leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 00:11:37, igmp ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
  Outgoing interface list: (count: 1)
    Vlan1, uptime: 00:11:37, igmp

  • The incoming interface pointing to tunnel 29 (toward the fabric, tunnel destination is VRF GiPo)
leaf3# show interface tunnel 29
Tunnel29 is up
    MTU 9000 bytes, BW 0 Kbit
    Tunnel protocol/transport is ivxlan
    Tunnel source 127.0.0.100/32
    Tunnel destination 225.1.192.48/32Code language: PHP (php)
  • RPF is toward the root of the shared tree (which is the rendez-vous Point)
    10.0.48.64 is PTEP of BL leaf-2 from where fabric received the RP subnet 192.168.30.10/32.
leaf3# acidiag fnvread | grep 10.0.48.64
     102        1    leaf2   10.0.48.64/32    leaf         active   0Code language: PHP (php)
  • Outgoing interface is populated via IGMP from the receiver.


3- IGMP mgroup entry installed in spine COOP DB

In the following command output from the spine, we can see that COOP received interest for the group 239.251.1.1 from 2 Leafs which are leaf-103 and leaf-104:

spine1# show coop internal info repo mgroup            

Repo Hdr Checksum : 31363
Repo Hdr record timestamp : 03 31 2024 12:51:27 1132741
Repo Hdr last pub timestamp : 03 31 2024 12:51:27 1369883
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ EXPORT 
VRF Vnid : 2359298 
mgroup src ip : 0.0.0.0 
mgroup group ip : 239.251.1.1 
Flags : 0x0x1 afi 0

Local leafs 2 (active: 2 deleted: 0)

Leaf 0 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 31 2024 12:51:27 1132741
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:51:27 1369883
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.168.65
Leaf Flags : 0x0 

Leaf 1 Info : 
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 31 2024 12:47:19 428948239
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:47:19 429150390
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ 
Leaf tep ip : 10.0.168.70
Leaf Flags : 0x0 

oldest local publish timestamp: 03 31 2024 12:47:19 429150390
MPOD Pub id : 0.0.0.0 
MPOD leafs 0 (active: 0 deleted: 0)
MSITE Pub id : 0.0.0.0 
MSITE leafs 0 (active: 0 deleted: 0)
Hash: 990247176 owner: 10.0.168.71Code language: CSS (css)
spine1# acidiag fnvread | grep 10.0.168.65
     103        1    leaf3     10.0.168.65/32    leaf         active   0
spine1# acidiag fnvread | grep 10.0.168.70
     104        1    leaf4     10.0.168.70/32    leaf         active   0Code language: PHP (php)


4-COOP notify Stripe Winner about the IGMP interest for the group

What is the stripe winner in ACI:

  • As per definition, the border leaf that is the stripe winner for a group is responsible for sending PIM joins on behalf of the fabric and forwarding multicast traffic into the fabric.
  • As per current latest, the stripe winner is decided based on a hash and not manually configured, The hash is computed using the S,G and Loopback IP of the Leaf.

You can verify the stripe winner Border leaf via command below:

leaf1# vsh -c "show ip pim internal stripe-winner 239.251.1.1 vrf bameur_MC:VRF-MC3"
PIM Stripe Winner info for VRF "bameur_MC:VRF-MC3" (BL count: 2)
(*, 239.251.1.1)
BLs:  
Group hash 1827639267 VNID 2359298
1.1.1.1  hash: 1281198598 (local)
      2.2.2.2  hash: 961945459
Winner: 1.1.1.1 best_hash: 1281198598Code language: CSS (css)


In our example, the border leaf (leaf-1) is the stripe winner.


5- Stripe winner installs (*,G) and sends out (*,G) PIM join to RP

leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 00:59:36, ngmvpn ip pim 
  Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel21, uptime: 00:59:36, ngmvpn
  • Incoming interface for (*,G) pointing to the L3OUT interface from where we received the RP subnet:
leaf1# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

192.168.30.10/32, ubest/mbest: 1/0
    *via 192.168.50.2%bameur_MC:VRF-MC3, [20/0], 5d04h, bgp-65001, external, tag 65033
         recursive next hop: 192.168.50.2/32%bameur_MC:VRF-MC3
leaf1# show ip arp vrf bameur_MC:VRF-MC3

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       D - Static Adjacencies attached to down interface

IP ARP Table for context bameur_MC:VRF-MC3
Total number of entries: 1
Address         Age       MAC Address     Interface
192.168.50.2    00:13:45  a093.51ac.d741  eth1/19.213Code language: HTML, XML (xml)
  • Outgoing interface is pointing to the Fabric (since receiver is In ACI fabric), it points to VRF GiPO with Loopback as source Interface:
Tunnel21 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport is ivxlan
    Tunnel source 1.1.1.1
    Tunnel destination 225.1.192.48/32Code language: JavaScript (javascript)


Note: the non stripe winner (in our example, leaf-2) doesn’t install the (*,G) in its mRIB and not responsible of sending (*,G) join to the RP:

leaf2# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

Group not found

Code language: CSS (css)


The RP receives the (*,G) PIM Join and build the shared tree (*,G) to the receiver:

(Root) Rendez-vous Point > Leaf-1 > leaf-3-4

RP# show ip mroute 

(*, 239.251.1.1/32), uptime: 01:16:43, pim ip 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 1)
    Ethernet1/3.1058, uptime: 01:16:43, pim


II- Source starts sending multicast traffic

  • Once source 172.16.20.3 starts sending the Multicast traffic for the group 239.251.1.1, the PIM Register is sent unicast to the RP from the FHMR and RP will join the (S,G) tree once receives native multicast traffic. This is outside of ACI in this use case.
  • Once multicast traffic is sent through the (*,G) tree toward the receiver, it will be received by the Stripe-winner (Leaf-1), Leaf-1 will join the Source tree (S,G)

Note: By default, the LHMR router (the one connected to the receiver) will join the (source tree) SPT tree and send out (S,G) to the source. and we also know that this behavior can be modified using SPT threshold.

For more details about SPT threshold and PIM sparse mode control plane and packet flow:


In ACI, by design, the LHMR function (the one which is expected to the join to (S,G) after receiving multicast traffic from the source is performed by the Border leaf (stripe winner).

We verify on leaf-1, it joined the (S,G) tree: (172.16.20.3/32, 239.251.1.1/32)

leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 01:35:05, ngmvpn ip pim 
  Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1) (Fabric OIF)
    Tunnel21, uptime: 01:35:05, ngmvpn


(172.16.20.3/32, 239.251.1.1/32), uptime: 00:00:19, ip mrib pim 
  Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
  Outgoing interface list: (count: 1)
    Tunnel21, uptime: 00:00:19, mrib


For the (172.16.20.3/32, 239.251.1.1/32) tree, we have:

  • Incoming interface: pointing to interface for the next-hop to reach the source subnet 172.16.20.3.
    RPF pointing to the source.

Note: In this example, the incoming interface for the Shared tree is the same as the source tree, because according to leaf-1 unicast RIB, it can reach to RP and Source subnet from the same interface (next-hop). But, for a different topology, it is possible that they can diverge, if BL have a better path to the source for example.

  • The outgoing interface: toward the fabric via VRF GiPo (this way the receivers on the fabric for this group will get multicast traffic from the source)
leaf1# show interface tunnel 21
Tunnel21 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "bameur_MC:VRF-MC3"
    Tunnel protocol/transport is ivxlan
    Tunnel source 1.1.1.1
    Tunnel destination 225.1.192.48/32Code language: PHP (php)


On the non-stripe winner, it still doesn’t install the mroute in the mRIB.


Now, if we verify on the “real LHMR” which is leaf-103-104, you can see that it didn’t join the SPT as we expect and it will rely on the Border leaf (Stripe winner) to receive the multicast traffic for this group from the source.

But, in same time, we still have the shared tree installed:

leaf3# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"

(*, 239.251.1.1/32), uptime: 02:06:42, igmp ip pim 
  Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
  Outgoing interface list: (count: 1)
    Vlan1, uptime: 02:06:42, igmp


Packet flow and Data-Plane verification

1- On the Striper winner:

We can perform a Packet capture via ELAM Tool on the Border leaf-1 where we expect to receive traffic from source for the group 239.251.1.1:

ELAM Leaf-1:

======================================================================================================================================================
                                                            Captured Packet                                                                           
======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2mc ipv4 ip ipmc ipv4mc                
Opcode                        : OPCODE_L3MC                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
IP Version                    : 4                                       
DSCP                          : 0                                       
IP Packet Length              : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit            : not set                                 
TTL                           : 254                                     
IP Protocol Number            : ICMP                                    
IP CheckSum                   : 50215( 0xC427 )                         
Destination IP                : 239.251.1.1                             
Source IP                     : 172.16.20.3                             
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM FILTERS
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM Filter Key                          : mcast_sg_rpf_pass, mcast_rpf_pass, mcast_sg_hit,  fib_star_g_hit
PIM Filter Match                        : no                            

------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP 
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block:     : IFABRIC_IG MC TENANT MYTEP ROUTE HIT 

------------------------------------------------------------------------------------------------------------------------------------------------------

Leaf-1 (stripe winner) hit the (S,G) entry in the mRIB and forward the multicast traffic to the fabric via the VRF GiPo (which is tunnel destination for the outgoing interface of the (S,G)).


2- On the leafs where Receiver resides:

the Leaf-103-104 receive the multicast traffic from leaf-1 through the fabric encapsulated in VXLAN (VRF VNID, Outer destination is VRF GiPo:

ELAM Leaf-3 (potentially same on other leafs):

======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2mc ipv4 ip ipmc ipv4mc udp ivxlan     
Opcode                        : OPCODE_L3MC                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 57                                      
Don't Fragment Bit            : 0x0                                     
TTL                           : 31                                      
IP Protocol Number            : UDP                                     
Destination IP                : 225.1.192.52                            
Source IP                     : 10.0.48.65                              

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x0                                     
TTL                           : 253                                     
IP Protocol Number            : ICMP                                    
Destination IP                : 239.251.1.1                             
Source IP                     : 172.16.20.3                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : iVxLAN                                  
Don't Learn Bit               : 1                                       
Src Policy Applied Bit        : 0                                       
Dst Policy Applied Bit        : 0                                       
sclass (src pcTag)            : 0xc007                                  
VRF or BD VNID                : 2359298( 0x240002 )                     
Code language: PHP (php)


Notes:

  • We can notice that outer IP source is the PTEP of leaf-1 (stripe winner) and outer destination is VRF GiPO.
  • VNID 2359298 (VRF VNID)
  • If you remember from previous verification, the VRF GiPo is 225.1.192.48, but we see Outer IP destination IP as: 225.1.192.52.

    That’s because the FTAG ID 4 was used, which means 225.1.192.48 + 4 –> 225.1.192.52
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG                                    : 4( 0x4 )  Code language: PHP (php)


Part 2:

Bilel-A

Enthusiastic Network Engineer specializing in Cisco ACI, passionate about solving challenges. A lifelong learner who loves gaining and sharing knowledge. Profile: https://www.linkedin.com/in/bilel-ameur-71116b2b5
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x