ACI Packet Forwarding on EX Hardware [Notes]
This document consists of some notes about ACI Packet forwarding from the Cisco document by Joseph Ristaino.
I- Two Endpoints in the same EPG – Switched traffic
- check endpoints via:
show mac address-table | grep <mac_address>
This command will show the learned endpoint and their PI VLAN.
- To verify the encapsulation VLAN:
show endpoint mac 0050.56a5.fccc or show vlan extended
- To check directly on Hardware:
leaf4# vsh_lc module-1# show system internal eltmc info vlan 30
30 is the PIVLAN in this example.
- Validate that hardware programmed the L2 information of the Endpoints via HAL:
HAL’s ((Hardware Abstraction Layer) job is to take software programming requests and push them to hardware.
leaf4# vsh_lc module-1# show platform internal hal ep l2 mac 0050.56a5.fccc
- Verify traffic forwarding via ELAM:
With ELAM we can check an index called: ovector_idx. This index is the physical port index that the frame/packet should be forwarded out of.
Once you have the ovector_idx, we can use this command to find what port it maps to:
module-1(DBG-TAH-elam-insel6)# show platform internal hal l2 port gpd
LEAF_4# vsh_lc
module-1# debug platform internal tah elam asic 0
module-1(DBG-elam)# trigger reset
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer l2 src_mac 0050.56a5.fccc dst_mac 0050.56a5.6794
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Triggered
Asic 0 Slice 1 Status Armed
module-1(DBG-elam-insel6)# report | grep ovec
sug_elam_out_sidebnd_no_spare_vec.ovector_idx: 0xB8
module-1(DBG-elam-insel6)# show platform internal hal l2 port gpd | grep b8
------ Eth1/49 ---II- Two Endpoints in different EPG/Same Leaf – Routed Packet
- Check the Endpoints in the Endpoint table:
leaf4# show endpoint ip 192.168.20.2 leaf4# show endpoint ip 192.168.21.2
- Look at the EP learning info in hardware (via HAL):
leaf4# vsh_lc
module-1# show platform internal hal ep l3 all
===========================================================================================================================================================================================
B E I S D S D D V EP-NH N |
Vrf EP S Age S S L N N B D P P P P S I U S L3 H | BD EP L3 L2 FD
Name T IP Class Intvl T E D D D E L I I A A S L B O IfName T | Name Mac IfName Ifname Name IP
===========================================================================================================================================================================================
Joey-T*ternal Pl 192.168.20.2 800a 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 - L2 BD-28 00:50:56:a5:fc:cc - Po3 FD-30 -
Joey-T*ternal Pl 192.168.21.2 800c 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 - L2 BD-7 00:50:56:a5:0c:11 - Po4 FD-8 - - Traffic Capture via ELAM:
leaf4# vsh_lc
module-1# debug platform internal tah elam asic 0
module-1(DBG-TAH-elam)# trigger init in-select 6 out-select 0
module-1(DBG-TAH-elam-insel6)# set outer ipv4 src_ip 192.168.20.2 dst_ip 192.168.21.2
module-1(DBG-TAH-elam-insel6)# start
module-1(DBG-TAH-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Armed
module-1(DBG-TAH-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status TriggeredIII- Two Endpoints in different EPG/Different Leaf – Routed Packet
- Verify Endpoints learning:
we can see in this example that on Leaf-4, 192.168.20.2 was locally learned on Po4 and 192.168.1.100 remotely learned on Tunnel2.
leaf4# vsh_lc
module-1# show platform internal hal ep l3 all
===========================================================================================================================================================================================
B E I S D S D D V EP-NH N |
Vrf EP S Age S S L N N B D P P P P S I U S L3 H | BD EP L3 L2 FD
Name T IP Class Intvl T E D D D E L I I A A S L B O IfName T | Name Mac IfName Ifname Name IP
===========================================================================================================================================================================================
Joey-T*ternal Xr 192.168.1.100 8013 128 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 - L3 - 00:0c:0c:0c:0c:0c Tunnel2 Tunnel2 - 0.0.0.0
Joey-T*ternal Pl 192.168.20.2 800a 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 - L2 BD-28 00:50:56:a5:fc:cc - Po3 FD-30 - - Verify Tunnel2
module-1# show system internal eltmc info interface tunnel2
IfInfo:
interface: Tunnel2 ::: ifindex: 402718722
iod: 66 ::: state: up
Mod: 0 ::: Port: 0
Tunnel Index: 0 ::: Tunnel Dst ip: 0xc0a87843
Tunnel Encap: ivxlan ::: Tunnel VPC Peer: 0
Tunnel Dst ip str: 192.168.120.67 ::: Tunnel ept: 0x1In the example, The destination exists off of a vPC, and that Destination IP should be the vPC Virtual IP of the remote leafs. Let’s check on a remote leaf and see:
leaf1# show system internal epm vpc
Local TEP IP : 192.168.160.95
Peer TEP IP : 192.168.160.93
vPC configured : Yes
vPC VIP : 192.168.120.67
MCT link status : Up
Local vPC version bitmap : 0x7
Peer vPC version bitmap : 0x7
Negotiated vPC version : 3
Peer advertisement received : Yes
Tunnel to vPC peer : UpWhen we check the ovector_idx in the ELAM report, we will see that the outgoing port is eth1/49 (interface to Spine), since, this traffic is from EPGs on different Leaf switches.
we will verify that Eth1/49 is used for Tunnel2 Encap:
module-1(DBG-TAH-elam-insel6)# show platform internal hal tunnel rtep pi
================================================================================================================================================================================================
=======
I N N |
E Vrf Hw V I P P P I I C U B B NH | Vrf L3 L3 L2 L2
IfId Ifname T Lid VrfId Name IP Enc P L 4 6 M I C OBd D T Id Cnt | VrfId Name IP Mac IfId IfName IfId IfName
================================================================================================================================================================================================
=======
18010002 Tunnel2 I 3005 2 overlay-1 192.168.120.670 0 0 0 0 0 0 0 1 0 E 2 2 2 overlay-1 0.0.0.0 0d:0d:0d:0d:0d:00 1a030001 Eth1/49.1 1a030000 Eth1/49
* This output gives us a few values we care about:
- IfId: The interface ID allocated to the tunnel
- IP: The IP of the destination. This should match ELTMC.
- L3 IfId: The layer 3 interface(s) the switch can use to forward to the appropriate destination.
