Cisco ACI ELAM Explained and Configuration Example [Step By Step]

What is Cisco ACI ELAM

Embedded Logic Analyzer Module (ELAM) is a tool that allows a user to set conditions in hardware and capture the first packet or frame that matches the set conditions. A successful capture will cause the ELAM status to show as ‘triggered’. Once triggered, the ELAM is disabled and a dump can be collected to analyze the vast number of forwarding decisions that the switch ASIC is making with that packet/frame.

ELAM is a tool that gives you the ability to look inside Cisco ASICs and understand how a packet is forwarded. It is embedded within the forwarding pipeline, and it can capture a packet in real-time without disruptions to performance of CPU or the control-plane resources. It helps to answer questions such as:

• Did the packet reach the Forwarding Engine (FE)?
  
• On what port and VLAN is the packet received?
  
• How does the packet appear (Layer 2 (L2) – Layer 4 (L4) data)?
  
• How is the packet altered, and where is it sent?

ELAM is extremely powerful, granular, and non-intrusive. It is a valuable troubleshooting tool for Cisco Technical Assistance Center (TAC) engineers who work on hardware-switching platforms.

Each platform follows a similar workflow, as shown in this diagram:

1. Identify the expected ingress FE. When platforms have more than one FE, it is critical to identify the FE that makes the forwarding decision for the packet you want to capture. Configure the ELAM on the correct FE.
2. Configure the ELAM trigger. You must configure a trigger with details specific to the packet that you want to capture. Common triggers include a source and destination IP address or L4 port numbers. ELAM allows multiple fields to be specified and performs a logical AND on all fields configured.
3. Start the ELAM.
4. Wait for the ELAM to trigger and display the result.
   

– ELAM on the First Generation Switches:

There are two ASIC types on Nexus 9300: Broadcom Trident T2 and the Cisco Northstar ASIC

• Northstar: ns: – Gen1
• Alpine: alp – Spines
• Rocky: roc – F/FX/FXP
• Tahoe: tah – GC/E/EX

Some considerations before using ELAM (@arielik):

• ELAM operates only in the Northstar ASIC for leafs (Gen1).
• ELAM operates only in the Alpine ASIC for Spines.
• Any packets that are locally switched in the BCM ASIC will not trigger the ELAM, since in this scenario the packet will not reach Northstar, and will not trigger an ELAM event.
• We must be sure that the packet is sent from the BCM ASIC to the Northstar ASIC.

– ELAM on the Second Generation Switches:

The second generation switches leverages the Cisco Cloud Scale ASICs:

• We will first look at the syntax of the ELAM on ACI switches and what’s the purpose of each command:

vsh_lc                                       [This command enters the line card shell where ELAMs are run]
debug platform internal <asic> elam asic 0   [refer to the ASICs table]

trigger reset                                           [ensures no existing triggers are running]
trigger init in-select <number> out-select <number>     [determines what information about a packet is displayed and which conditions can be set]
set outer/inner                                         [sets conditions]
start                                                   [starts the trigger]
status                                                  [checks if a packet is captured]

ereport             [display detailed forwarding decision for the packet]

ELAM Configuration Example Configuration:

• Let’s take an example of ELAM configuration and then go through the configuration step by step:

module-1# debug platform internal tah elam asic 0 
module-1(DBG-elam)# trigger reset
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 10.0.0.1 dst_ip 10.0.0.2
module-1(DBG-elam-insel6)# start 
module-1(DBG-elam-insel6)# status 
module-1(DBG-elam-insel6)# ereport

• Input Select command: “in-select”:

The ‘in-select’ defines which headers the packet/frame is expected to have, and which to match on.

☑Examples:

A packet coming from a downlink port that is not VXLAN encapsulated would only have outer Layer 2, Layer 3, and Layer 4 headers.

A packet coming from a front-panel (downlink) port that is VXLAN encapsulated (such as Cisco ACI Virtual Edge in VXLAN mode) or coming from an upstream spine would have VXLAN encapsulation. This means it would have potentially both outer and inner Layer 2, Layer 3, and Layer 4 headers.

☑ Input Select Lines:

module-1(DBG-elam)# trigger init in-select ?
 10  Outerl4-innerl4-ieth
 13  Outer(l2|l3|l4)-inner(l2|l3|l4)-noieth 
 14  Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth
 15  Outer(l2|l3|l4)-inner(l2|l3|l4)-ieth
 6   Outerl2-outerl3-outerl4
 7   Innerl2-innerl3-innerl4
 8   Outerl2-innerl2-ieth
 9   Outerl3-innerl3

☑ Coomon Usages and Best Practices:

• If ‘in-select 6’ is selected the only option is to set conditions and display headers from the outer Layer 2, 3, or 4 headers.
  –> To capture a packet coming with VLAN encapsulation on a downlink port, use ‘in-select 6’

• If ‘in-select 14’ is selected the only option is to set conditions for and see the details of the outer and inner Layer 2, 3, and 4 headers.
  –> To capture a packet with VXLAN encapsulation (either from a spine or from a vleaf with VXLAN encapsulation) use ‘in-select 14’

• Output Select command: “out-select”:

The ‘out-select’ allows some ability to control which lookup results are displayed in the ELAM report.

For most practical purposes ‘out-select 0′ can be used as it contains most information including the ‘drop vector‘ which will tell if the result of the lookup is to drop the packet/frame.

☑ Output Select Lines

0 – Pktrw 

1 – Sideband

• ELAM Set conditions:

ELAM supports a large amount of Layer 2, 3, and 4 conditions to look for in a packet:

module-1(DBG-elam-insel6)# set outer ?
  arp   ARP Fields
  fc    FC Fields
  ipv4  IPv4 Fields
  ipv6  IPv6 Fields
  l2    All Layer 2 Fields
  l3    Layer 3 Fields
  l4    L4 Fields
  mpls  MPLS Fields

module-1(DBG-elam-insel6)# set outer l3 ?
  ether-type  ARP Ether Type - 0x0806 -> ARP 0x8035 -> RARP
  packet-len  Packet Total Length

Specifying ‘inner’ vs. ‘outer’ determines if the condition should be checked in the inner
header (VXLAN encapsulated packet) or outer header.

☑ Examples:

1- Source and destination Inner IPv4 addresses, to capture packet VXLAN packets:

set inner ipv4 src_ip 10.0.0.1 dst_ip 10.0.0.2

2- Source and destination Outer IPv4 addresses, to capture packet from front ports (Not encapsulated packets):

set outer ipv4 src_ip 10.0.0.1 dst_ip 10.0.0.2

3- Source and destination Inner MAC addresses, to capture packet VXLAN packets:

set inner ipv4 src_ip 10.0.0.1 dst_ip 10.0.0.2

4- ARP , to capture ARP packet from front ports matching the source and target ip address:

set outer arp source-ip-address 10.0.0.1 target-ip-address 10.0.0.2

References:

This posts consists of some notes from Cisco document and Underlay blog.