Port Channel configuration on Nexus
Topology:
Contents
L2 Port-Channel configuration on NXOS:
NXOS-01(config)# feature lacp
NXOS-01(config)# interface ethernet 1/1-3
NXOS-01(config-if-range)# channel-group 10 mode activeNXOS-02(config)# feature lacp
NXOS-02(config)# interface ethernet 1/1-3
NXOS-02(config-if-range)# channel-group 10 mode activeNXOS-01# show interface ethernet 1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Belongs to Po10
Hardware: 100/1000/10000 Ethernet, address: 5020.0000.0101 (bia 5020.0000.0101
)
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/sNXOS-02# show interface eth1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Belongs to Po10
Hardware: 100/1000/10000 Ethernet, address: 5021.0000.0101 (bia 5021.0000.0101
)
MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 1000 Mb/sVerification:
- verify port-channel is up:
NXOS-01# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
b - BFD Session Wait
S - Switched R - Routed
U - Up (port-channel)
p - Up in delay-lacp mode (member)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
10 Po10(SU) Eth LACP Eth1/1(P) Eth1/2(P) Eth1/3(P)
- Check local and partner system information:
NXOS-01# show lacp port-channel interface port-channel 10
port-channel10
Port Channel Mac=50-20-0-0-1b-8
Local System Identifier=0x8000,50-20-0-0-1b-8
Admin key=0x9
Operational key=0x9
Partner System Identifier=0x8000,50-21-0-0-1b-8
Operational key=0x9
Max delay=0
Aggregate or individual=1
Member Port List=Eth1/1 Eth1/2 Eth1/3- Verify LACP neighbor information per interface:
NXOS-01# show lacp neighbor
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
port-channel10 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/1 32768,50-21-0-0-1b-8 0x101 8874 SA
LACP Partner Partner Partner
Port Priority Oper Key Port State
32768 0x9 0x3d
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/2 32768,50-21-0-0-1b-8 0x102 8874 SA
LACP Partner Partner Partner
Port Priority Oper Key Port State
32768 0x9 0x3d
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/3 32768,50-21-0-0-1b-8 0x103 8873 SA
LACP Partner Partner Partner
Port Priority Oper Key Port State
32768 0x9 0x3d- Check the counter for the sent and received LACP PDU:
NXOS-01# show lacp counters detail
port-channel10
Ethernet1/1
PDU sent: 475
PDU rcvd: 459
Marker rcvd: 0
Marker resp sent: 0
Marker sent: 0
Marker resp rcvd: 0
Pkts error: 0
PDU timeout count: 1
Flap count: 1
Ethernet1/2
PDU sent: 475
PDU rcvd: 459
Marker rcvd: 0
Marker resp sent: 0
Marker sent: 0
Marker resp rcvd: 0
Pkts error: 0
PDU timeout count: 1
Flap count: 1
Ethernet1/3
PDU sent: 475
PDU rcvd: 460
Marker rcvd: 0
Marker resp sent: 0
Marker sent: 0
Marker resp rcvd: 0
Pkts error: 0
PDU timeout count: 1
Flap count: 1- Check the LACP max bundle links:
NXOS-01# show port-channel internal sdb
PCM SDB
=======
Number of channels: 1
Channel Ifindex Size Mode Max-active Ungra Susp-Dis FOP LacpV
pcCon GirConv
Po10 0x16000009 3 active 32 0 0 Eth1/1 0
0 (0x1a000000)
lacp min-links 1
lacp max-bundle 32
port-channel max active member 32
port-channel load defer timeout 0
lacp fast-select-hot-standby disable 0
lacp delayed-enable 0
lacp delayed-enable primary-port 0x0
-------------------------------------------------------------
Member Ifindex Status Channel-status Bundle-num
Eth1/1 0x1a000000 0x1 2 0
Eth1/2 0x1a000200 0x1 2 1
Eth1/3 0x1a000400 0x1 2 2
------------------------------------------------------------
We can change the lacp max-bundle value:
NXOS-01(config)# int port-channel 10
NXOS-01(config-if)# lacp max-bundle ?
<1-32> Enter the max-bundle number
NXOS-01(config-if)# lacp max-bundle 6NXOS-01# show port-channel load-balance
System config:
Non-IP: src-dst mac
IP: src-dst ip-l4port rotate 0
Port Channel Load-Balancing Configuration for all modules:
Module 1:
Non-IP: src-dst mac
IP: src-dst ip-l4port rotate 0
LACP control plane packet capture
Let’s take an example the interface Eth1/1:
Step-1: Initially, After enabling Port-channel LACP on interface ethernet1/1, NXOS-01, the LACP packet sent by this interface contain the Actor information, but no Partner information yet, since it didn’t receive any LACP PDU packet from it’s peer yet:
- Actor system Priority: 32768
- Actor System id: 50:20:00:00:1b:08
This can be verified on the switch via command:
NXOS-01# show lacp system-identifier
32768,50-20-0-0-1b-8- Actor key: it represent the Port-channel ID configured.
The LACP key defines the ability of a port to aggregate with other ports. You must configure a key on each port running LACP. When 2 or more ports with the same key are configured, a LACP Etherchannel is established. - Actor port and priority
- Actor state: Out of sync, Collecting: Disabled, Distributing: Disabled
Step-2: Once Ethernet 1/1 on Switch NX-OS-01 receives an LACP PDU from it’s peer, it will sent an LACP PDU including the received partner information.
Step-3: interface Ethernet 1/1 on Switch NX-OS-02 receives an LACP PDU from it’s peer including both Actor and Partner information.
At this point, NX-OS-02 have see its local system-ID in the sent LACP PDU by the switch NXOS-01 (eth1/1) as partner. NXOS-01 will verify the consistency of the remote system-ID and key within the other links in the LAG.
– Once NXOS-02 verified both conditions:
- Sees its local system ID being advertised by the peer as partner
- verified consistency within the LAG (same key and remote system-ID across other links in the LAG).
It will sent an LACP PDU with Actor state flag, synchronization: 1, (in sync):
Step-4: NXOS-01 send an LACP PDU with flag, in-sync:
Now, both are in sync:
Step-5: NXOS-02 received the Partner with sync flag set and sends Collecting set:
Once the Port on the Remote Peer is in Sync, (Flag set), the local device (NXOS-02) will send an LACP with Collecting Flag set indicating that the Device is ready to receive traffic on its Port:
Same for NXOS-01, it will send an LACP PDU with flag collecting set.
Step-6: NXOS-02 will received the LACP frame with distributing flag being set on actor and partner, and it will send LACPDU with the Distributing flag set to indicate they are transmitting Data traffic on the Port:
same for NXOS-01.
Troubleshooting LACP port-channel:
Please refer to the following Cisco document
Also, you can check the LACP deep dive post:
Port-channel Compatibility Parameters
In order for a links to be members of a port-channel, they have to match some compatibility parameters, (in order to ensure consistency across LAG members). If you configure a member port with an incompatible attribute, the software suspends that port in the port channel.
The following parameters are extracted from Cisco Nexus 9k switch for reference:
- port mode:
Members must have the same port mode configured, either E,F or AUTO. If they are configured in AUTO port mode, they have to negotiate E or F mode when they come up. If a member negotiates a different mode, it will be suspended.
- speed
Members must have the same speed configured. If they are configured in AUTO speed, they have to negotiate the same speed when they come up. If a member negotiates a different speed, it will be suspended.
- MTU
Members have to have the same MTU configured. This only applies to ethernet
port-channel.
- MEDIUM
Members have to have the same medium type configured. This only applies to
ethernet port-channel.
- Span mode
Members must have the same span mode.
- load interval
Member must have same load interval configured.
- sub interfaces
Members must not have sub-interfaces.
- Duplex Mode
Members must have same Duplex Mode configured.
- Ethernet Layer
Members must have same Ethernet Layer (switchport/no-switchport) configured.
- Span Port
Members cannot be SPAN ports.
- Storm Control
Members must have same storm-control configured.
- Flow Control
Members must have same flowctrl configured.
- Capabilities
Members must have common capabilities.
- Capabilities speed
Members must have common speed capabilities.
- Capabilities duplex
Members must have common speed duplex capabilities.
- rate mode
Members must have the same rate mode configured.
- Capabilities FabricPath
Members must have common fabricpath capability.
- Port is PVLAN host
Port Channel cannot be created for PVLAN host
- 1G port is not capable of acting as peer-link
Members must be 10G to become part of a vPC peer-link.
- EthType
Members must have same EthType configured.
- shared interface
Members can not be shared-interfaces.
- Capabilities SpanDest
Members must be capable of span destination configuration
- Module Type Incompatible
Module type for interfaces is not compatible.
- Port Mode Fabricpath Incompatible
Members are Fabricpath Enforce locked, not compatible.
- Port auto negotiation Incompatible
Members must have same auto negotiation configured.
- port VLAN
Members port VLAN info.
- port
Members port does not exist.
- switching port
Members must be switching port, Layer 2.
- port access VLAN
Members must have the same port access VLAN.
- port native VLAN
Members must have the same port native VLAN.
- port allowed VLAN list
Members must have the same port allowed VLAN list.
- Members should have same fex config
Members must have same FEX configuration.
- FEX pinning max-links not one
FEX pinning max-links config is not one.
- Multiple port-channels with same Fex-id
Multiple port-channels to same FEX not allowed.
- Pinning Params
Members must have the same pinning parameters.
- All HIF member ports not in same pinning group
All HIF member ports not in same pinning group
- Slot in host vpc mode
Cannot add cfged slot member to fabric po vpc.
- Members in multiple FEX
Members must belong to same FEX.
- Members are of different type
Members must of same interface type.
- port egress queuing policy
10G port-channel members must have the same egress queuing policy as the
port-channel.
- Port Security policy
Members must have the same port-security enable status as port-channel
- Port priority-flow-control
PFC config should be the same for all the members
- Dot1x policy
Members must have host mode as multi-host with no mab configuration. Dot1X cannot be enabled on members when Port Security is configured on port channel
- PC Queuing policy
Queuing policy for Non-DCE PC should be non-dce
- PC Queuing policy
Queuing policy for the PC should be same as system queuing policy
- Emulated switch port type policy
vPC ports in emulated switch complex should be L2MP capable.
- VFC bound to port
Members cannot have VFCs bound to them.
- VFC bound to port channel
Port Channels that have VFCs bound to them cannot have more than one member
- VFC bound to FCoE capable port channel
Port Channels that have VFCs bound to them cannot have non fcoe capable member
- VFC bound to FCoE capable port channel
Port Channels that have VFCs bound to them cannot have non fcoe licensed member
- Fex ports for span
Port-Channel is already a SPAN source. Cannot add FEX ports connected through Ricard to this PC
- CTS mode
Members must have the same CTS mode configured (either “cts manual” or “cts dot1x” or no cts)
- CTS SGT propagation
SGT propagation must either be enabled or disabled on all members
- CTS SGT policy
Members must all have either “policy static” or “policy dynamic” or no
policy configured
- CTS peer identity
Members must all have the same peer identity configured
- CTS SGT configuration
Members must all have the same SGT configured
- CTS replay protection
Replay protection must either be enabled or disabled on all members
show interface status error policy [detail]
Displays the interfaces and VLANs that produce an error during policy programming to ensure that policies are consistent with hardware policies.
You can force ports with incompatible parameters to join the port channel if the following parameters are the same:
- (Link) Speed capability
- Speed configuration
- Duplex capability
- Duplex configuration
- Flow-control capability
- Flow-control configuration
