Cisco ACI Microsoft NLB Configuration and Troubleshooting
Contents
What is Microsoft NLB
The Microsoft Network Load Balancing (NLB) feature distributes the client traffic across many servers, with each server running its individual copy of the application. Network Load Balancing uses Layer 2 unknown unicast or multicast to simultaneously distribute the incoming network traffic to all cluster hosts.
A group of Microsoft NLB nodes is collectively known as an NLB cluster. Every node within the cluster receives every packet of traffic, but only one node services a request. NLB cluster use a load-balancing algorithm to decide which individual node will service the particular traffic flow that is destined for the NLB VIP.
Microsoft NLB Modes
Microsoft NLB functions in three different operational modes:
- Unicast Mode: In this mode, each NLB cluster VIP is assigned a unicast MAC address. This mode relies on unknown unicast flooding to deliver traffic to the cluster.
- Multicast Mode: In this mode, each NLB cluster VIP is assigned a non-Internet Assigned Numbers Authority (IANA) multicast MAC address (03xx.xxxx.xxxx).
- IGMP Mode: In this mode, an NLB cluster VIP is assigned a unique IPv4 multicast group address. The multicast MAC address for this is derived from the standard MAC derivation for IPv4 multicast addresses
I- NLB Unicast Mode Configuration in Cisco ACI
In unicast mode, NLB uses the NLB MAC address to replace the original hardware MAC address of each adapter in each node of the cluster. Because multiple adapters now have the same address.
Traffic Flow:
- Traffic from Client to NLB nodes (VIP):
In the unicast mode of operation, Network Load Balancing reassigns the MAC address of the network adapter on which it is enabled (called the cluster adapter), and all cluster hosts are assigned the same MAC address. This MAC address is derived from the cluster’s primary IP address. For example, for a primary IP address of 1.2.3.4, the unicast MAC address is set to 02-BF-1-2-3-4.
- Return Traffic from NLB nodes to Client:
The use of a common MAC address would normally create a conflict, since Layer 2 switches expect to see unique source MAC addresses on all switch ports. To avoid this problem, Network Load Balancing uniquely modifies the source MAC address for outgoing packets. If the cluster MAC address is 02-BF-1-2-3-4, then each host’s source MAC address is set to 02-x-1-2-3-4, where x is the host’s priority within the cluster.
NLB Unicast Mode configuration in ACI
1- Make sure BD unknown Unicast is set to Flood:
2- Configure VIP under EPG subnet as MSNLB:
- Specify “NLB in unicast mode”
- Configure the unicast MAC address of the NLB cluster
NLB Unicast mode Verification and troubleshooting:
leaf1# moquery -c ipv4.NlbVip
Total Objects shown: 1
# ipv4.NlbVip
mode : mode-uc
mac : 00:50:56:A0:CE:AB
childAction :
dn : sys/ipv4/inst/dom-MC:VRF-MC3/rtipep-[172.16.31.2/32]/nlb-mode-uc-mac-00:50:56:A0:CE:AB
encapPgmOperSt : dst-info-pgmed
encapPgmOperStQual : unspecified
epmVipOperStQual :
fabEncap : vxlan-15335397
fabEncapGrpIp : 225.0.138.176
group : 0.0.0.0
lcOwn : local
mfdmCfgFailedBmp :
mfdmCfgFailedTs : 00:00:00:00.000
mfdmCfgState : 0
modTs : 2024-04-28T14:09:50.905+02:00
monPolDn :
name :
operSt : up
pcTag : 49161
rn : nlb-mode-uc-mac-00:50:56:A0:CE:AB
status : Code language: PHP (php)leaf1# show nlb vip
NLB Vip MO Database:
=============================================================================================
FabEncap |FabGipo |Vip |Dst |sclass | v4 |Vrf/Vip/Mode-Mac,
| |Oper |Pgm | | or |Igmp-Group,
| |State |State | | v6 | Vip Down Resaon
=========|=================|======|======|=======|===========================================
15335397 | 225.0.138.176 | up | up | 49161 | v4 | dom-MC:VRF-MC3/172.16.31.2/uc-00:50:56:A0:CE:AB
Total v4 Vip: 1
Total v6 Vip: 0Code language: PHP (php)ELAM on Egress leaf:
This MAC address of the NLB isn’t learned for ACI leaf, so, forwarding toward the NLB VIP is based on unknown unicast in the BD.
We started a ping from a client to the VIP 172.16.31.2 and at the egress leaf where NLB nodes resides, we see traffic being flooded in the BD:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc udp ivxlan l2uc ipuc ipv4uc
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer iEth Header
------------------------------------------------------------------------------------------------------------------------------------------------------
iEth SUP code : NONE
Packet from CPU : no
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 0050.56A0.CEAB
Source MAC : 0022.BDF8.19FF
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 31
IP Protocol Number : UDP
Destination IP : 225.0.138.178
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 64
IP Protocol Number : ICMP
Destination IP : 172.16.31.2
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4004
VRF or BD VNID : 15335397( 0xE9FFE5 )
Code language: PHP (php)
Packet flooded in BD using FTAG tree id 2:
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG : 2( 0x2 ) Code language: PHP (php)
On the ingress leaf, the unicast packet destination MAC address was rewrited to the Unicast MAC of the NLB cluster and then flooded in BD (where NLB is configured).
Basically traffic to the VIP is sent as unicast with destination MAC 0050.56A0.CEAB (based on NLB configuration) to flooded all Interfaces in the Bridge domain:
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC (Lookup Key)
------------------------------------------------------------------------------------------------------------------------------------------------------
Dst MAC Lookup was performed : yes
Dst MAC Lookup BD : 4656( 0x1230 )
( Hw BDID in "show plat int hal l2 bd pi" )
Dst MAC Address : 0050.56A0.CEAB Code language: CSS (css)
We can verify that BD floodlist include the interface to the NLB nodes, specifically for the Unkown unicast traffic UUC:
module-1(DBG-elam-insel14)# show platform internal hal l2 bd pi | egrep 1230
9f BD-159 BD-Reg 1230 0 Vx 0 IV e9ffe5 2 1 1 1 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1 4009 2 0 0 0 0 0 0 0 0 1 0 0 0
module-1(DBG-elam-insel14)# show platform internal hal l2 mcast bd_flood_list bd 0x9f
===========================================================================================================================================================
================================================
| ------------------------------------------------------------------ Mcast Replication List ------------------------------------
------------------------------------------------
| | --------------------------------------------------- Mcast Replication Entry ------------------------------------
------------------------------------------------
F | Mcast No. | Mc Rpl Sti Sti O O | --------------------------
--------- IFs ----------------------------------
BD L U I U | Rpl Rpl | Repl Ent Repl Ent Epg E E Vrf E Mc Alt Num | L2
BdId Name T M S B | Lst-Id Ent | Id Typ Id Typ Val V T VrfId Name GIPo/GIPo' M Idx T Dvif Dvif Ifs | Ifs
===========================================================================================================================================================
================================================
9f BD-159 UUC 1 0 0 86 2 16c EPG - - a0 0 - - - - - 6 - 0 800 1 Po2(16000001)
20000110STI 4000010f OvF - - IV 4 overlay-1 225.0.138.176 F 1ffc - 0 0 2 Eth1/49(1a030000) E
th1/50(1a031000)
Code language: PHP (php)II- NLB Static Multicast Mode Configuration in Cisco ACI
Instead of changing the MAC addresses on the network adapters, NLB converts the NLB virtual IP (VIP) address to an NLB multicast MAC address.
This MAC has the format of 03-BF-XX-XX-XX-XX. NLB also makes sure that the cluster’s primary IP address resolves to this multicast address as part of the Address Resolution Protocol (ARP). Although the individual network adapters retain their original MAC addresses, the NLB traffic is addressed to the NLB multicast MAC address.
Network switches cannot learn the NLB multicast MAC address in the course of their usual operations. So, we need to have a way to statically map VIP to the Mulicast MAC address (kind of static ARP and mapping of unicast VIP to multicast MAC address)
Assuming our NLB hosts are configured with VIP 172.16.31.2 and listening to MAC 03:BF:AC:10:1F:02:
NLB static Multicast mode Configuration:
1- Configure EPG subnet:
In ACI, we configure the VIP under the NLB, we specify NLB mode static Multicast
and we enter the Multicast MAC address of the cluster:
This way, we statically mapped the VIP IP to this MAC (similar to static ARP).
2- Configure static path for static Multicast NLB:
The remaining part is to provide static path to inform ACI where this MAC address resides, because in the endpoint, we are not learning this MAC neither from Control plane or Data plane, it’s just static:
NLB static Multicast mode Verification and Troubleshooting:
NLB VIP Managed Object is properly pushed with multicast static mode:
leaf1# moquery -c ipv4.NlbVip
Total Objects shown: 1
# ipv4.NlbVip
mode : mode-mcast--static
mac : 03:BF:AC:10:1F:02
childAction :
dn : sys/ipv4/inst/dom-MC:VRF-MC3/rtipep-[172.16.31.2/32]/nlb-mode-mcast--static-mac-03:BF:AC:10:1F:02
encapPgmOperSt : dst-info-pgmed
encapPgmOperStQual : unspecified
epmVipOperStQual :
fabEncap : vxlan-15335397
fabEncapGrpIp : 225.0.138.176
group : 0.0.0.0
lcOwn : local
mfdmCfgFailedBmp :
mfdmCfgFailedTs : 00:00:00:00.000
mfdmCfgState : 0
modTs : 2024-04-28T15:41:40.782+02:00
monPolDn :
name :
operSt : up
pcTag : 49161
rn : nlb-mode-mcast--static-mac-03:BF:AC:10:1F:02
status : Code language: PHP (php)
NLB VIP is programmed on the leaf:
leaf1# show nlb vip
NLB Vip MO Database:
=============================================================================================
FabEncap |FabGipo |Vip |Dst |sclass | v4 |Vrf/Vip/Mode-Mac,
| |Oper |Pgm | | or |Igmp-Group,
| |State |State | | v6 | Vip Down Resaon
=========|=================|======|======|=======|===========================================
15335397 | 225.0.138.176 | up | up | 49161 | v4 | dom-MC:VRF-MC3/172.16.31.2/mcast-03:BF:AC:10:1F:02
Total v4 Vip: 1
Total v6 Vip: 0Code language: PHP (php)
And static path are seen in macoif:
leaf1# show nlb macoif
LB Mac Oif MO Database:
==============================================================
Vxlan Encap Mac Interface
==============================================================
15335397 2008 03:BF:AC:10:1F:02 po2
Now, let’s static a communication from client to the VIP, for testing, I’m using Ping:
on the ingress, Traffic from client with destination VIP 172.16.31.2 will have its destination MAC rewritten to Multicast MAC specified in the NLB configuration (which is mapped to the VIP) and then will be flooded in the Bridge Domain:
ELAM on Egress Leaf:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc udp ivxlan ipuc ipv4uc
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 03BF.AC10.1F02
Source MAC : 0022.BDF8.19FF
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 31
IP Protocol Number : UDP
Destination IP : 225.0.138.178
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 64
IP Protocol Number : ICMP
Destination IP : 172.16.31.2
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4004
VRF or BD VNID : 15335397( 0xE9FFE5 ) Code language: PHP (php)
The Egress leaf receive traffic with destination MAC 03:BF:AC:10:1F:02, it will use the information from the static path configuration and will send the inner payload to Po2 (with encap 2008):
leaf1# show nlb macoif
LB Mac Oif MO Database:
==============================================================
Vxlan Encap Mac Interface
==============================================================
15335397 2008 03:BF:AC:10:1F:02 po2 III- NLB IGMP Mode Configuration in Cisco ACI
Microsoft NLB servers can also be configured to use IGMP to join the multicast group. This, combined with a querier on a switch and IGMP snooping, can optimize the scope of the flooding of multicast messages.
Microsoft NLB servers send IGMP joins to a multicast group address, where the last two octets of the multicast address correspond to the last two octets of the cluster IP (VIP).
This mode is basically the same as multicast mode, except that the leaf can automatically build their MAC address tables in this mode.
NLB IGMP mode configuration in ACI:
Since IGMP Snooping is required for this mode, we will need to make some pre-configuration were done:
1- Make sure an IGMP Querier is enabled under the BD subnet:
2- IGMP Snooping policy configured under the BD:
Please refer to following article for more details about ACI IGMP Snooping configuration:
3- Configure NLB with IGMP mode under the EPG subnet:
NLB IGMP mode Verification and Troubleshooting:
Verify NLB MO properly pushed to the leaf:
- Verify the mode and group.
- An interesting point to mention is MAC 01:00:5E:7F:1F:02 associated to the configured multicast group 239.255.31.2:
leaf1# moquery -c ipv4.NlbVip
Total Objects shown: 1
# ipv4.NlbVip
mode : mode-mcast-igmp
mac : 01:00:5E:7F:1F:02
childAction :
dn : sys/ipv4/inst/dom-bameur_MC:VRF-MC3/rtipep-[172.16.31.2/32]/nlb-mode-mcast-igmp-mac-01:00:5E:7F:1F:02
encapPgmOperSt : dst-info-pgmed
encapPgmOperStQual : unspecified
epmVipOperStQual :
fabEncap : vxlan-15335397
fabEncapGrpIp : 225.0.138.176
group : 239.255.31.2
lcOwn : local
mfdmCfgFailedBmp :
mfdmCfgFailedTs : 00:00:00:00.000
mfdmCfgState : 0
modTs : 2024-04-28T16:54:51.230+02:00
monPolDn :
name :
operSt : up
pcTag : 49161
rn : nlb-mode-mcast-igmp-mac-01:00:5E:7F:1F:02
status : Code language: PHP (php)- Verify NLB VIP configuration via “show nlb vip”:
leaf1# show nlb vip
NLB Vip MO Database:
=============================================================================================
FabEncap |FabGipo |Vip |Dst |sclass | v4 |Vrf/Vip/Mode-Mac,
| |Oper |Pgm | | or |Igmp-Group,
| |State |State | | v6 | Vip Down Resaon
=========|=================|======|======|=======|===========================================
15335397 | 225.0.138.176 | up | up | 49161 | v4 | dom-bameur_MC:VRF-MC3/172.16.31.2/igmp-01:00:5E:7F:1F:02
| | | | | | IGMP-Group: 239.255.31.2
Total v4 Vip: 1
Total v6 Vip: 0Code language: PHP (php)leaf1# show forwarding distribution l2 multicast nlb gs
###########Global NLB GS entries###########
=============================================================================
BDVNID Group Source => Ref BDVNID Vmac
=============================================================================
15335397 239.255.31.2 0.0.0.0 1 15335397 0100.5e7f.1f02Code language: PHP (php)- Verify IGMP querier is active on the BD subnet:
leaf1# show ip igmp snooping querier vlan 159 detail
Vlan IP Address Version Expires Port
159 172.16.31.254 v2 00:00:20 Switch querier
IGMP snooping querier configuration for VLAN 159
IGMP snooping last member query count: 0, configured value: 0
IGMP snooping version: 2, configured value: 2
IGMP snooping robustness variable: 2, configured value: 2
IGMP snooping startup query count: 2, configured value: 2
IGMP snooping startup query interval: 31 secs, configured value: 31 secs
IGMP snooping last member query interval: 1 secs, configured value: 1 secs
IGMP snooping query interval: 125 secs, configured value: 125 secs
IGMP snooping query max response time: 10 secs, configured value: 10 secs
IGMP snooping configured querier timeout: 255 secs, defaultCode language: CSS (css)- Most importantly, verify that NLB is sending IGMP report and that the leaf has IGMP snooping table populated for the configured group:
leaf1# show ip igmp snooping groups vlan 159
Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core port
Vlan Group Address Ver Type Port list
159 239.251.1.1 v2 D Po2
leaf1# show ip igmp snooping groups vlan 159 detail
IGMP Snooping group membership for vlan 159
Group addr: 239.255.31.2
Group ver: v2 [old-host-timer: not running]
Last reporter: 172.16.31.2
Group Report Timer: 00:00:00
Group v3 Expiry Report Timer: 00:00:00
IGMPv1/v2 memb ports:
Po2 [0 GQ missed]
vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: includeCode language: PHP (php)For testing, we start a communication from client with IP 192.168.65.10 to the VIP:
A ping from 192.168.65.10 to VIP 172.16.31.2:
- On Ingress leaf, the destination MAC is rewritten to 0100.5E7F.1F02 based on the NLB configuration we did previously.
- Then, it will be flooded in BD GiPo using FTAG tree:
ELAM on Egress leaf:
We can see the destination Inner MAC 0100.5E7F.1F02 and inner destination IP is the VIP: 172.16.31.2:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2mc ipv4 ip ipmc ipv4mc udp ivxlan ipuc ipv4uc
Opcode : OPCODE_L3MC
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 0100.5E7F.1F02
Source MAC : 0022.BDF8.19FF
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 31
IP Protocol Number : UDP
Destination IP : 225.0.138.178
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 64
IP Protocol Number : ICMP
Destination IP : 172.16.31.2
Source IP : 192.168.65.10 Code language: PHP (php)This packet was flooded by ingress leaf in BD GiPo via FTAG id 2:
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4004
VRF or BD VNID : 15335397( 0xE9FFE5 )
module-1(DBG-elam-insel14)# ereport | grep FTAG
FTAG : 2( 0x2 )
The Egress leaf (where VIP resides) received the packet with destination IP as VIP 172.16.31.2 and destination MAC as 0100.5E7F.1F02,
The Egress leaf will verify its IGMP snooping entries for group mapped to this MAC which is 239.255.31.2:
and the packet is send out via Po2.
leaf1# show ip igmp snooping groups 239.255.31.2 detail
IGMP Snooping group membership for vlan 159
Group addr: 239.255.31.2
Group ver: v2 [old-host-timer: not running]
Last reporter: 172.16.31.2
Group Report Timer: 00:00:00
Group v3 Expiry Report Timer: 00:00:00
IGMPv1/v2 memb ports:
Po2 [0 GQ missed]
vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: includeCode language: PHP (php)I verified that the VIP host was listening for this multicast MAC 0100.5E7F.1F02 (because it had sent an IGMP join for the group 239.255.31.2):
and traffic was flowing properly:
The return traffic will be like any unicast traffic, the Node will use it’s original MAC address as source and leaf will forward it to the destination IP:
Return traffic:
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 0022.BDF8.19FF
Source MAC : 0050.56A0.CEAB
802.1Q tag is valid : yes( 0x1 )
CoS : 0( 0x0 )
Access Encap VLAN : 2008( 0x7D8 )
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
IP Version : 4
DSCP : 0
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : not set
TTL : 64
IP Protocol Number : ICMP
IP CheckSum : 25430( 0x6356 )
Destination IP : 192.168.65.10
Source IP : 172.16.31.2
