Port Channel configuration on Nexus

Topology:


L2 Port-Channel configuration on NXOS:

NXOS-01(config)# feature lacp
NXOS-01(config)# interface ethernet 1/1-3
NXOS-01(config-if-range)# channel-group 10 mode active
NXOS-02(config)# feature lacp
NXOS-02(config)# interface ethernet 1/1-3
NXOS-02(config-if-range)# channel-group 10 mode active
NXOS-01# show interface ethernet 1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
  Belongs to Po10
  Hardware: 100/1000/10000 Ethernet, address: 5020.0000.0101 (bia 5020.0000.0101
)
  MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 1000 Mb/s
NXOS-02# show interface eth1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
  Belongs to Po10
  Hardware: 100/1000/10000 Ethernet, address: 5021.0000.0101 (bia 5021.0000.0101
)
  MTU 1500 bytes, BW 1000000 Kbit , DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is access
  full-duplex, 1000 Mb/s


Verification:

  • verify port-channel is up:
NXOS-01# show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
10    Po10(SU)    Eth      LACP      Eth1/1(P)    Eth1/2(P)    Eth1/3(P)


  • Check local and partner system information:
NXOS-01# show lacp port-channel interface port-channel 10
port-channel10
  Port Channel Mac=50-20-0-0-1b-8
  Local System Identifier=0x8000,50-20-0-0-1b-8
  Admin key=0x9
  Operational key=0x9
 
 Partner System Identifier=0x8000,50-21-0-0-1b-8
  Operational key=0x9
  Max delay=0
  Aggregate or individual=1
  Member Port List=Eth1/1 Eth1/2 Eth1/3


  • Verify LACP neighbor information per interface:
NXOS-01# show lacp neighbor
Flags:  S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode
port-channel10 neighbors
Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/1      32768,50-21-0-0-1b-8   0x101           8874        SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x9                         0x3d

Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/2      32768,50-21-0-0-1b-8   0x102           8874        SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x9                         0x3d

Partner's information
            Partner                Partner                     Partner
Port        System ID              Port Number     Age         Flags
Eth1/3      32768,50-21-0-0-1b-8   0x103           8873        SA

            LACP Partner           Partner                     Partner
            Port Priority          Oper Key                    Port State
            32768                  0x9                         0x3d


  • Check the counter for the sent and received LACP PDU:
NXOS-01# show lacp counters detail
port-channel10
Ethernet1/1
  PDU sent: 475
  PDU rcvd: 459
  Marker rcvd: 0
  Marker resp sent: 0
  Marker sent: 0
  Marker resp rcvd: 0
  Pkts error: 0
  PDU timeout count: 1
  Flap count: 1

Ethernet1/2
  PDU sent: 475
  PDU rcvd: 459
  Marker rcvd: 0
  Marker resp sent: 0
  Marker sent: 0
  Marker resp rcvd: 0
  Pkts error: 0
  PDU timeout count: 1
  Flap count: 1

Ethernet1/3
  PDU sent: 475
  PDU rcvd: 460
  Marker rcvd: 0
  Marker resp sent: 0
  Marker sent: 0
  Marker resp rcvd: 0
  Pkts error: 0
  PDU timeout count: 1
  Flap count: 1


  • Check the LACP max bundle links:
NXOS-01# show port-channel internal sdb
PCM SDB
=======
Number of channels: 1
Channel   Ifindex       Size    Mode     Max-active  Ungra  Susp-Dis  FOP  LacpV
pcCon  GirConv
Po10      0x16000009    3       active   32          0      0         Eth1/1   0
      0     (0x1a000000)
lacp min-links 1
lacp max-bundle 32
port-channel max active member 32
port-channel load defer timeout 0
lacp fast-select-hot-standby disable 0
lacp delayed-enable 0
lacp delayed-enable primary-port 0x0
          -------------------------------------------------------------
          Member       Ifindex     Status  Channel-status   Bundle-num
          Eth1/1       0x1a000000  0x1     2                0
          Eth1/2       0x1a000200  0x1     2                1
          Eth1/3       0x1a000400  0x1     2                2
          ------------------------------------------------------------

We can change the lacp max-bundle value:

NXOS-01(config)# int port-channel 10
NXOS-01(config-if)# lacp max-bundle ?
  <1-32>  Enter the max-bundle number

NXOS-01(config-if)# lacp max-bundle 6


NXOS-01# show port-channel load-balance
 System config:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port rotate 0
Port Channel Load-Balancing Configuration for all modules:
Module 1:
  Non-IP: src-dst mac
  IP: src-dst ip-l4port rotate 0


LACP control plane packet capture

Let’s take an example the interface Eth1/1:

Step-1: Initially, After enabling Port-channel LACP on interface ethernet1/1, NXOS-01, the LACP packet sent by this interface contain the Actor information, but no Partner information yet, since it didn’t receive any LACP PDU packet from it’s peer yet:

  • Actor system Priority: 32768
  • Actor System id: 50:20:00:00:1b:08
    This can be verified on the switch via command:
NXOS-01# show lacp system-identifier
32768,50-20-0-0-1b-8
  • Actor key: it represent the Port-channel ID configured.
    The LACP key defines the ability of a port to aggregate with other ports. You must configure a key on each port running LACP. When 2 or more ports with the same key are configured, a LACP Etherchannel is established.
  • Actor port and priority
  • Actor state: Out of sync, Collecting: Disabled, Distributing: Disabled


Step-2: Once Ethernet 1/1 on Switch NX-OS-01 receives an LACP PDU from it’s peer, it will sent an LACP PDU including the received partner information.

Step-3: interface Ethernet 1/1 on Switch NX-OS-02 receives an LACP PDU from it’s peer including both Actor and Partner information.

At this point, NX-OS-02 have see its local system-ID in the sent LACP PDU by the switch NXOS-01 (eth1/1) as partner. NXOS-01 will verify the consistency of the remote system-ID and key within the other links in the LAG.

– Once NXOS-02 verified both conditions:

  • Sees its local system ID being advertised by the peer as partner
  • verified consistency within the LAG (same key and remote system-ID across other links in the LAG).

It will sent an LACP PDU with Actor state flag, synchronization: 1, (in sync):


Step-4: NXOS-01 send an LACP PDU with flag, in-sync:

Now, both are in sync:

Step-5: NXOS-02 received the Partner with sync flag set and sends Collecting set:

Once the Port on the Remote Peer is in Sync, (Flag set), the local device (NXOS-02) will send an LACP with Collecting Flag set indicating that the Device is ready to receive traffic on its Port:

Same for NXOS-01, it will send an LACP PDU with flag collecting set.


Step-6: NXOS-02 will received the LACP frame with distributing flag being set on actor and partner, and it will send LACPDU with the Distributing flag set to indicate they are transmitting Data traffic on the Port:

same for NXOS-01.

Troubleshooting LACP port-channel:

Please refer to the following Cisco document

Also, you can check the LACP deep dive post:


Port-channel Compatibility Parameters

In order for a links to be members of a port-channel, they have to match some compatibility parameters, (in order to ensure consistency across LAG members). If you configure a member port with an incompatible attribute, the software suspends that port in the port channel.


The following parameters are extracted from Cisco Nexus 9k switch for reference:

  • port mode:

Members must have the same port mode configured, either E,F or AUTO. If they are configured in AUTO port mode, they have to negotiate E or F mode when they come up. If a member negotiates a different mode, it will be suspended.

  • speed

Members must have the same speed configured. If they are configured in AUTO speed, they have to negotiate the same speed when they come up. If a member negotiates a different speed, it will be suspended.

  • MTU

Members have to have the same MTU configured. This only applies to ethernet
port-channel.

  • MEDIUM

Members have to have the same medium type configured. This only applies to
ethernet port-channel.

  • Span mode

Members must have the same span mode.

  • load interval

Member must have same load interval configured.

  • sub interfaces

Members must not have sub-interfaces.

  • Duplex Mode

Members must have same Duplex Mode configured.

  • Ethernet Layer

Members must have same Ethernet Layer (switchport/no-switchport) configured.

  • Span Port

Members cannot be SPAN ports.

  • Storm Control

Members must have same storm-control configured.

  • Flow Control

Members must have same flowctrl configured.

  • Capabilities

Members must have common capabilities.

  • Capabilities speed

Members must have common speed capabilities.

  • Capabilities duplex

Members must have common speed duplex capabilities.

  • rate mode

Members must have the same rate mode configured.

  • Capabilities FabricPath

Members must have common fabricpath capability.

  • Port is PVLAN host

Port Channel cannot be created for PVLAN host

  • 1G port is not capable of acting as peer-link

Members must be 10G to become part of a vPC peer-link.

  • EthType

Members must have same EthType configured.

  • shared interface

Members can not be shared-interfaces.

  • Capabilities SpanDest

Members must be capable of span destination configuration

  • Module Type Incompatible

Module type for interfaces is not compatible.

  • Port Mode Fabricpath Incompatible

Members are Fabricpath Enforce locked, not compatible.

  • Port auto negotiation Incompatible

Members must have same auto negotiation configured.

  • port VLAN

Members port VLAN info.

  • port

Members port does not exist.

  • switching port

Members must be switching port, Layer 2.

  • port access VLAN

Members must have the same port access VLAN.

  • port native VLAN

Members must have the same port native VLAN.

  • port allowed VLAN list

Members must have the same port allowed VLAN list.

  • Members should have same fex config

Members must have same FEX configuration.

  • FEX pinning max-links not one

FEX pinning max-links config is not one.

  • Multiple port-channels with same Fex-id

Multiple port-channels to same FEX not allowed.

  • Pinning Params

Members must have the same pinning parameters.

  • All HIF member ports not in same pinning group

All HIF member ports not in same pinning group

  • Slot in host vpc mode

Cannot add cfged slot member to fabric po vpc.

  • Members in multiple FEX

Members must belong to same FEX.

  • Members are of different type

Members must of same interface type.

  • port egress queuing policy

10G port-channel members must have the same egress queuing policy as the
port-channel.

  • Port Security policy

Members must have the same port-security enable status as port-channel

  • Port priority-flow-control

PFC config should be the same for all the members

  • Dot1x policy

Members must have host mode as multi-host with no mab configuration. Dot1X cannot be enabled on members when Port Security is configured on port channel

  • PC Queuing policy

Queuing policy for Non-DCE PC should be non-dce

  • PC Queuing policy

Queuing policy for the PC should be same as system queuing policy

  • Emulated switch port type policy

vPC ports in emulated switch complex should be L2MP capable.

  • VFC bound to port

Members cannot have VFCs bound to them.

  • VFC bound to port channel

Port Channels that have VFCs bound to them cannot have more than one member

  • VFC bound to FCoE capable port channel

Port Channels that have VFCs bound to them cannot have non fcoe capable member

  • VFC bound to FCoE capable port channel

Port Channels that have VFCs bound to them cannot have non fcoe licensed member

  • Fex ports for span

Port-Channel is already a SPAN source. Cannot add FEX ports connected through Ricard to this PC

  • CTS mode

Members must have the same CTS mode configured (either “cts manual” or “cts dot1x” or no cts)

  • CTS SGT propagation

SGT propagation must either be enabled or disabled on all members

  • CTS SGT policy

Members must all have either “policy static” or “policy dynamic” or no
policy configured

  • CTS peer identity

Members must all have the same peer identity configured

  • CTS SGT configuration

Members must all have the same SGT configured

  • CTS replay protection

Replay protection must either be enabled or disabled on all members


show interface status error policy [detail]

Displays the interfaces and VLANs that produce an error during policy programming to ensure that policies are consistent with hardware policies.

You can force ports with incompatible parameters to join the port channel if the following parameters are the same:

  • (Link) Speed capability
  • Speed configuration
  • Duplex capability
  • Duplex configuration
  • Flow-control capability
  • Flow-control configuration
Bilel

Bilel

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Inline Feedbacks
View all comments
Learn Duty
0
Would love your thoughts, please comment.x
()
x