ACI L3 Multicast Troubleshoot – Part-1: Receiver in ACI and Source Outside
Topology:
Contents
Pre-checks
- Verify PIM neighbor is up between Border leaf and External L3OUT node:
leaf1# show ip pim neighbor vrf bameur_MC:VRF-MC3
PIM Neighbor information for Dom:bameur_MC:VRF-MC3
Neighbor Interface Uptime Expires DRPriority Bidir BFDState
192.168.50.2/32 eth1/19.213 5d3h 00:01:31 1 yes n/a
2.2.2.2/32 tunnel21 5d3h 00:01:44 1 no n/a - verify that multicast Border were registered in COOP mrouter on the spine:
spine1# show coop internal info repo mrouter key 2359298
Repo Hdr Checksum : 32349
Repo Hdr record timestamp : 03 26 2024 09:37:24 500587101
Repo Hdr last pub timestamp : 03 31 2024 13:03:32 868964254
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ
BD Vnid : 2359298
flags : 0x4
num of leafs in record : 2
num of valid leafs in record : 2
Leaf 0 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 26 2024 09:37:24 500587101
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:49:08 590196850
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.48.64
Leaf Flags : 0x4 MCAST
Leaf 1 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 26 2024 09:37:24 498508281
Leaf Repo Hdr last pub timestamp : 03 31 2024 13:03:32 868964254
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.48.65
Leaf Flags : 0x4 MCAST
Hash: 3519002267 owner: 10.0.168.66Code language: CSS (css)I- No traffic from the source
1- Receiver sends IGMP report for the group
The Receiver Host show its intent to receive Multicast traffic for a group by sending IGMP Report (called IGMP join as well), we can verify on leaf-3 via command “show ip igmp groups vrf VRF_name”:
leaf3# show ip igmp groups vrf bameur_MC:VRF-MC3
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
IGMP Connected Group Membership for VRF "bameur_MC:VRF-MC3"
Group Address Type Interface Uptime Expires Last Reporter
239.251.1.1 D vlan1 00:00:39 00:03:50 172.16.30.100 Code language: CSS (css)
we can see that EP 172.16.30.100 reported its interest to receive Multicast traffic for group 239.251.1.1 and we can see in our example that it was on vlan1.
Checking further, we can see that vlan 1 is the PIVLAN Id for the BD where the Receiver endpoints resides:
leaf3# show system internal epm endpoint ip 172.16.30.100
MAC : 0050.56b0.aaa8 ::: Num IPs : 1
IP# 0 : 172.16.30.100 ::: IP# 0 flags : peer-aged|host-tracked| ::: l3-sw-hit: No
Vlan id : 14 ::: Vlan vnid : 14897 ::: VRF name : bameur_MC:VRF-MC3
BD vnid : 14909454 ::: VRF vnid : 2359298
Phy If : 0x16000004 ::: Tunnel If : 0
Interface : port-channel5
Flags : 0x80005c25 ::: sclass : 16386 ::: Ref count : 5
EP Flags : local|vPC|peer-aged|IP|MAC|host-tracked|sclass|timer|
leaf3# show system internal epm vlan 1
+----------+---------+-----------------+----------+------+----------+-----------
VLAN ID Type Access Encap Fabric H/W id BD VLAN Endpoint
(Type Value) Encap Count
+----------+---------+-----------------+----------+------+----------+-----------
1 Tenant BD NONE 0 14909454 16 1 0
2- Receiver Leaf install (*,G) and publish group interest to the Spine COOP
We can see on the leaf-3 where receiver is located that an (*,G) mroute was installed:
leaf3# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 00:11:37, igmp ip pim
Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
Outgoing interface list: (count: 1)
Vlan1, uptime: 00:11:37, igmp
- The incoming interface pointing to tunnel 29 (toward the fabric, tunnel destination is VRF GiPo)
leaf3# show interface tunnel 29
Tunnel29 is up
MTU 9000 bytes, BW 0 Kbit
Tunnel protocol/transport is ivxlan
Tunnel source 127.0.0.100/32
Tunnel destination 225.1.192.48/32Code language: PHP (php)- RPF is toward the root of the shared tree (which is the rendez-vous Point)
10.0.48.64 is PTEP of BL leaf-2 from where fabric received the RP subnet 192.168.30.10/32.
leaf3# acidiag fnvread | grep 10.0.48.64
102 1 leaf2 10.0.48.64/32 leaf active 0Code language: PHP (php)- Outgoing interface is populated via IGMP from the receiver.
3- IGMP mgroup entry installed in spine COOP DB
In the following command output from the spine, we can see that COOP received interest for the group 239.251.1.1 from 2 Leafs which are leaf-103 and leaf-104:
spine1# show coop internal info repo mgroup
Repo Hdr Checksum : 31363
Repo Hdr record timestamp : 03 31 2024 12:51:27 1132741
Repo Hdr last pub timestamp : 03 31 2024 12:51:27 1369883
Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Repo Hdr dampen penalty : 0
Repo Hdr flags : IN_OBJ EXPORT
VRF Vnid : 2359298
mgroup src ip : 0.0.0.0
mgroup group ip : 239.251.1.1
Flags : 0x0x1 afi 0
Local leafs 2 (active: 2 deleted: 0)
Leaf 0 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 31 2024 12:51:27 1132741
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:51:27 1369883
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.168.65
Leaf Flags : 0x0
Leaf 1 Info :
Leaf Repo Hdr Checksum : 0
Leaf Repo Hdr record timestamp : 03 31 2024 12:47:19 428948239
Leaf Repo Hdr last pub timestamp : 03 31 2024 12:47:19 429150390
Leaf Repo Hdr last dampen timestamp : 01 01 1970 00:00:00 0
Leaf Repo Hdr dampen penalty : 0
Leaf Repo Hdr flags : IN_OBJ
Leaf tep ip : 10.0.168.70
Leaf Flags : 0x0
oldest local publish timestamp: 03 31 2024 12:47:19 429150390
MPOD Pub id : 0.0.0.0
MPOD leafs 0 (active: 0 deleted: 0)
MSITE Pub id : 0.0.0.0
MSITE leafs 0 (active: 0 deleted: 0)
Hash: 990247176 owner: 10.0.168.71Code language: CSS (css)spine1# acidiag fnvread | grep 10.0.168.65
103 1 leaf3 10.0.168.65/32 leaf active 0
spine1# acidiag fnvread | grep 10.0.168.70
104 1 leaf4 10.0.168.70/32 leaf active 0Code language: PHP (php)4-COOP notify Stripe Winner about the IGMP interest for the group
What is the stripe winner in ACI:
- As per definition, the border leaf that is the stripe winner for a group is responsible for sending PIM joins on behalf of the fabric and forwarding multicast traffic into the fabric.
- As per current latest, the stripe winner is decided based on a hash and not manually configured, The hash is computed using the S,G and Loopback IP of the Leaf.
You can verify the stripe winner Border leaf via command below:
leaf1# vsh -c "show ip pim internal stripe-winner 239.251.1.1 vrf bameur_MC:VRF-MC3"
PIM Stripe Winner info for VRF "bameur_MC:VRF-MC3" (BL count: 2)
(*, 239.251.1.1)
BLs:
Group hash 1827639267 VNID 2359298
1.1.1.1 hash: 1281198598 (local)
2.2.2.2 hash: 961945459
Winner: 1.1.1.1 best_hash: 1281198598Code language: CSS (css)
In our example, the border leaf (leaf-1) is the stripe winner.
5- Stripe winner installs (*,G) and sends out (*,G) PIM join to RP
leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 00:59:36, ngmvpn ip pim
Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1) (Fabric OIF)
Tunnel21, uptime: 00:59:36, ngmvpn
- Incoming interface for (*,G) pointing to the L3OUT interface from where we received the RP subnet:
leaf1# show ip route 192.168.30.10 vrf bameur_MC:VRF-MC3
IP Route Table for VRF "bameur_MC:VRF-MC3"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.30.10/32, ubest/mbest: 1/0
*via 192.168.50.2%bameur_MC:VRF-MC3, [20/0], 5d04h, bgp-65001, external, tag 65033
recursive next hop: 192.168.50.2/32%bameur_MC:VRF-MC3
leaf1# show ip arp vrf bameur_MC:VRF-MC3
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
D - Static Adjacencies attached to down interface
IP ARP Table for context bameur_MC:VRF-MC3
Total number of entries: 1
Address Age MAC Address Interface
192.168.50.2 00:13:45 a093.51ac.d741 eth1/19.213Code language: HTML, XML (xml)- Outgoing interface is pointing to the Fabric (since receiver is In ACI fabric), it points to VRF GiPO with Loopback as source Interface:
Tunnel21 is up
MTU 9000 bytes, BW 0 Kbit
Transport protocol is in VRF "bameur_MC:VRF-MC3"
Tunnel protocol/transport is ivxlan
Tunnel source 1.1.1.1
Tunnel destination 225.1.192.48/32Code language: JavaScript (javascript)
Note: the non stripe winner (in our example, leaf-2) doesn’t install the (*,G) in its mRIB and not responsible of sending (*,G) join to the RP:
leaf2# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
Group not found
Code language: CSS (css)The RP receives the (*,G) PIM Join and build the shared tree (*,G) to the receiver:
(Root) Rendez-vous Point > Leaf-1 > leaf-3-4
RP# show ip mroute
(*, 239.251.1.1/32), uptime: 01:16:43, pim ip
Incoming interface: Null, RPF nbr: 0.0.0.0
Outgoing interface list: (count: 1)
Ethernet1/3.1058, uptime: 01:16:43, pimII- Source starts sending multicast traffic
- Once source 172.16.20.3 starts sending the Multicast traffic for the group 239.251.1.1, the PIM Register is sent unicast to the RP from the FHMR and RP will join the (S,G) tree once receives native multicast traffic. This is outside of ACI in this use case.
- Once multicast traffic is sent through the (*,G) tree toward the receiver, it will be received by the Stripe-winner (Leaf-1), Leaf-1 will join the Source tree (S,G)
Note: By default, the LHMR router (the one connected to the receiver) will join the (source tree) SPT tree and send out (S,G) to the source. and we also know that this behavior can be modified using SPT threshold.
For more details about SPT threshold and PIM sparse mode control plane and packet flow:
In ACI, by design, the LHMR function (the one which is expected to the join to (S,G) after receiving multicast traffic from the source is performed by the Border leaf (stripe winner).
We verify on leaf-1, it joined the (S,G) tree: (172.16.20.3/32, 239.251.1.1/32)
leaf1# show ip mroute vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 01:35:05, ngmvpn ip pim
Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1) (Fabric OIF)
Tunnel21, uptime: 01:35:05, ngmvpn
(172.16.20.3/32, 239.251.1.1/32), uptime: 00:00:19, ip mrib pim
Incoming interface: Ethernet1/19.213, RPF nbr: 192.168.50.2
Outgoing interface list: (count: 1)
Tunnel21, uptime: 00:00:19, mrib
For the (172.16.20.3/32, 239.251.1.1/32) tree, we have:
- Incoming interface: pointing to interface for the next-hop to reach the source subnet 172.16.20.3.
RPF pointing to the source.
Note: In this example, the incoming interface for the Shared tree is the same as the source tree, because according to leaf-1 unicast RIB, it can reach to RP and Source subnet from the same interface (next-hop). But, for a different topology, it is possible that they can diverge, if BL have a better path to the source for example.
- The outgoing interface: toward the fabric via VRF GiPo (this way the receivers on the fabric for this group will get multicast traffic from the source)
leaf1# show interface tunnel 21
Tunnel21 is up
MTU 9000 bytes, BW 0 Kbit
Transport protocol is in VRF "bameur_MC:VRF-MC3"
Tunnel protocol/transport is ivxlan
Tunnel source 1.1.1.1
Tunnel destination 225.1.192.48/32Code language: PHP (php)
On the non-stripe winner, it still doesn’t install the mroute in the mRIB.
Now, if we verify on the “real LHMR” which is leaf-103-104, you can see that it didn’t join the SPT as we expect and it will rely on the Border leaf (Stripe winner) to receive the multicast traffic for this group from the source.
But, in same time, we still have the shared tree installed:
leaf3# show ip mroute 239.251.1.1 vrf bameur_MC:VRF-MC3
IP Multicast Routing Table for VRF "bameur_MC:VRF-MC3"
(*, 239.251.1.1/32), uptime: 02:06:42, igmp ip pim
Incoming interface: Tunnel29, RPF nbr: 10.0.48.64
Outgoing interface list: (count: 1)
Vlan1, uptime: 02:06:42, igmpPacket flow and Data-Plane verification
1- On the Striper winner:
We can perform a Packet capture via ELAM Tool on the Border leaf-1 where we expect to receive traffic from source for the group 239.251.1.1:
ELAM Leaf-1:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
IP Version : 4
DSCP : 0
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : not set
TTL : 254
IP Protocol Number : ICMP
IP CheckSum : 50215( 0xC427 )
Destination IP : 239.251.1.1
Source IP : 172.16.20.3 ------------------------------------------------------------------------------------------------------------------------------------------------------
PIM FILTERS
------------------------------------------------------------------------------------------------------------------------------------------------------
PIM Filter Key : mcast_sg_rpf_pass, mcast_rpf_pass, mcast_sg_hit, fib_star_g_hit
PIM Filter Match : no
------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits set in Final Forwarding Block: : IFABRIC_IG MC TENANT MYTEP ROUTE HIT
------------------------------------------------------------------------------------------------------------------------------------------------------Leaf-1 (stripe winner) hit the (S,G) entry in the mRIB and forward the multicast traffic to the fabric via the VRF GiPo (which is tunnel destination for the outgoing interface of the (S,G)).
2- On the leafs where Receiver resides:
the Leaf-103-104 receive the multicast traffic from leaf-1 through the fabric encapsulated in VXLAN (VRF VNID, Outer destination is VRF GiPo:
ELAM Leaf-3 (potentially same on other leafs):
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc udp ivxlan
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 57
Don't Fragment Bit : 0x0
TTL : 31
IP Protocol Number : UDP
Destination IP : 225.1.192.52
Source IP : 10.0.48.65
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 253
IP Protocol Number : ICMP
Destination IP : 239.251.1.1
Source IP : 172.16.20.3
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 1
Src Policy Applied Bit : 0
Dst Policy Applied Bit : 0
sclass (src pcTag) : 0xc007
VRF or BD VNID : 2359298( 0x240002 )
Code language: PHP (php)
Notes:
- We can notice that outer IP source is the PTEP of leaf-1 (stripe winner) and outer destination is VRF GiPO.
- VNID 2359298 (VRF VNID)
- If you remember from previous verification, the VRF GiPo is 225.1.192.48, but we see Outer IP destination IP as: 225.1.192.52.
That’s because the FTAG ID 4 was used, which means 225.1.192.48 + 4 –> 225.1.192.52
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG : 4( 0x4 ) Code language: PHP (php)