Cisco ACI APIC Teaming [Bond0 Explained]

Cisco ACI APIC Teaming

Cisco APICs are equipped with two Network Interface Cards (NICs) for fabric connectivity. These NICs should be connected to different leaf switches for redundancy.

Cisco APIC connectivity is automatically configured for active-backup teaming, which means that only one interface is active at any given time:

APIC’s bond0 is active/standby.

You can verify (but not modify) this configuration from the Bash shell under /proc/net/bonding.

apic1# cat /proc/net/bonding/bond0
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2-2
MII Status: up
MII Polling Interval (ms): 60
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2-1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 58:f3:9c:5a:b8:b8
Slave queue ID: 0

Slave Interface: eth2-2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 58:f3:9c:5a:b8:b9
Slave queue ID: 0

Third-Generation APIC Connections Consideration:

Port-1 and port-2 is one pair, corresponding to eth2-1 on APIC. Port-3 and port-4 is another pair, corresponding to eth2-2 on APIC. Only one connection is allowed for each pair.

APIC-M3/L3, the VIC card has 4 ports, and usually labeled like this:

Example:

you can connect one cable to either port-1 or port-2, and you can connect another cable to either port-3 or port-4. Do not connect two cables on any pair:

• Option 1:

• Option 2:

Notes:

Previous Figures are for illustration purpose, if you want to use (Port 1 and Port 4) or (Port 2 and Port 4) , they also valid choices, as long, they are not mapped to the same Internal APIC port.

Second-Generation APIC Connections:

The second-generation APICs are M2 or L2 APIC clusters, to learn more about APIC generations and check the full list of APICs models, visit this article.

The network interfaces are as follows:

●   bond0: This is the NIC bonding interface for in-band connection to the leaf switch. No IP address is assigned for this interface.

●   bond0.<infra VLAN>: This subinterface connects to the leaf switch. The infra VLAN ID is specified during the initial Cisco APIC software configuration. This interface obtains a dynamic IP address from the pool of TEP addresses specified in the setup configuration.

Common VIC Misconfiguration Issues

1- Miscabling issue:

For L3/M3 APIC, mis-cabling of Fabric port will lead to situation where both leaf connected to the APIC are connected to the same Internal APIC port:

when you run the command “show lldp neighbors” on both leaf switches connected to the APIC, the same APIC interface is shown.

2- VIC Enabled LLDP issue:

While LLDP is a critical functionality for the ACI Initial setup process, it should only enabled on the APIC Internal Interfaces only, not the VIC Ports.

When configuring the APIC CIMC, you should never Enable LLDP (maybe only for troubleshoot purposes only).

When you enable LLDP on the VIC, the leaf will see the wrong LLDP neighbor, usually without name, just MAC showing as LLDP neighbor.

In simple word, the LLDP received by the Leafs is by the APIC VIC (External ports) instead of the APIC Internal ports.

How to resolve the issue:

To resolve the issue, you just need to the Disable LLDP on the VIC from the APIC CIMC Interface:

Log into the CIMC GUI interface > System settings > VIC configuration. (it may differ accrding to CIMC version).

or you can Login via ssh to the CIMC:

scope chassis
scope adapter 1
set lldp disabled
commit

then Power cycle the APIC and verify.

After disabling LLDP on VIC and rebooting the APIC, the Leaf should be able to detect the APIC and received LLDP TLV from the APIC.