Fiber Channel over Ethernet (FCoE) [Explained & Basic configuration]

What is Fibre Channel over Ethernet FCoE:

Fibre Channel over Ethernet (FCoE) encapsulates Fibre Channel frames over Ethernet networks. This allows Fibre Channel to use 10 Gigabit Ethernet networks (or higher speeds) while preserving the Fibre Channel protocol.

Hosts connect to FCoE with converged network adapters (CNAs), which contain both Fibre Channel host bus adapter (HBA) and Ethernet network interface controller (NIC) functionality on the same physical card. CNAs have one or more physical Ethernet ports. FCoE encapsulation can be done in software with a conventional Ethernet network interface card, however, FCoE CNAs offload (from the CPU) the low-level frame processing and SCSI protocol function traditionally performed by Fibre Channel host bus adapters.

This converged adapter not only provides the required physical connectivity, but it also enables lossless Ethernet. This is essential because Fibre Channel is a lossless protocol, and storage area networks (SANs) expect lossless communications.

FCoE Frame format:

FCoE is encapsulated over Ethernet with the use of a dedicated Ethertype, 0x8906

FCoE Initiation Protocol

Fibre Channel over Ethernet (FCoE) provides a method of transporting Fibre Channel traffic over a physical Ethernet connection. FCoE requires the underlying Ethernet to be full-duplex and to provide lossless behaviour for Fibre Channel traffic.

The FCoE Initialization Protocol (FIP) allows the switch to discover and initialize FCoE-capable entities that are connected to an Ethernet LAN. Two versions of FIP are supported by the Cisco Nexus 5000 Series switch:

• FIP: The Converged Enhanced Ethernet Data Center Bridging Exchange (CEE-DCBX) protocol supports T11-compliant Gen-2 CNAs.
• Pre-FIP: The Cisco, Intel, Nuova Data Center Bridging Exchange (CIN-DCBX) protocol supports Gen-1 converged network adapters (CNAs).

The Cisco Nexus 5000 Series switch detects the capabilities of the attached CNA and switches to the correct FIP mode.

FIP is used to perform device discovery, initialization, and link maintenance. FIP performs the following protocols:

• FIP Discovery: When a FCoE device is connected to the fabric, it sends out a Discovery Solicitation message. A Fibre Channel Forwarder (FCF) or a switch responds to the message with a Solicited Advertisement that provides an FCF MAC address to use for subsequent logins.
• FCoE Virtual Link instantiation: FIP defines the encapsulation of fabric login (FLOGI) , fabric discovery (FDISC), logout (LOGO), and exchange link parameters (ELP) frames along with the corresponding reply frames. The FCoE devices use these messages to perform a fabric login.
• FCoE Virtual Link maintenance: FIP periodically sends maintenance messages between the switch and the CNA to ensure the connection is still valid.

Establishing the Virtual Link between VN_Ports and VF_Ports

In addition to performing the VLAN and FCF discoveries (operations that are specific to FCoE), FIP is also responsible for performing the fabric login of the VN_Port. Although this can be performed by FC transparently over FCoE, it is done by FIP because at this stage, the virtual link has not been established yet. As previously mentioned, the VN_Port MAC address (required to set up the virtual link) is built from the FCID, and the FCID is available only after login. FIP is thus solving this chicken-and-egg problem by handling the whole fabric login process. The VN_Port is assigned a fabric-provided Mac address (FPMA) that is built by concatenating a 24-bit FCoE MAC address prefix (FC-MAP), ranging from 0x0E-FC-00 to 0x0E-FC-FF, to the 24-bit FCID, as shown in Figure 7. Being able to build a unique MAC address for the VN_Port directly from its FCID saves the switch from having to maintain a table that associates FCID and MAC addresses.

Figure 7 Fabric-Provided Mac Address

The FC-MAP range was introduced so that different values can be assigned to different SANs. For example, SAN A would be associated to 0x0EFC00 and SAN B to 0x0EFC01. This additional configuration ensures the uniqueness of the produced FPMA in the whole network. FC-MAPs are different for different SANs, FCIDs are uniquely assigned within a SAN, and the resulting FC-MAP and FCID are unique across the different SANs in the entire network.

To minimize the risk of crosstalk between SANs, Cisco recommends that each SAN uses a different VLAN for its FCoE traffic. This makes the configuration of the FC-MAP less critical, because the FPMAs have to be unique only on a per-VLAN basis.

The Data Center Bridging Exchange (DCBX) protocol is an extension of the Link Layer Discovery Protocol (LLDP). DCBX endpoints exchange request and acknowledgement messages. For flexibility, parameters are coded in a type-length-value (TLV) format.

DCBX:

DCBX runs on the physical Ethernet link between the Cisco Nexus 5000 Series switch and the CNA. By default, DCBX is enabled on Ethernet interfaces. When an Ethernet interface is brought up, the switch automatically starts to communicate with the CNA.

During the normal operation of FCoE between the switch and the CNA, DCBX provides link-error detection.

DCBX is also used to negotiate capabilities between the switch and the CNA and to send configuration values to the CNA.

The CNAs that are connected to a Cisco Nexus 5000 Series switch is programmed to accept the configuration values sent by the switch, allowing the switch to distribute configuration values to all attached CNAs, which reduces the possibility of configuration errors and simplifies CNA administration.

* DCBX Feature Negotiation

The switch and CNA exchange capability information and configuration values. The Cisco Nexus 5000 Series switches support the following capabilities:

• FCoE: If the CNA supports FCoE capability, the switch sends the IEEE 802.1p CoS value to be used with FCoE packets.
• Priority Flow Control (PFC): If the adapter supports PFC, the switch sends the IEEE 802.1p CoS values to be enabled with PFC.
• Priority group type-length-value (TLV)
• Ethernet logical link up and down signal
• FCoE logical link up and down signal for pre-FIP CNAs

FCoE configuration step by step :

First, we need to activate LLDP and FCoE features, but why do we need LLDP?

As we mentioned previously, to ensure a lossless behaviour on the Ethernet link, the DCBX protocol (Data Center Bridging Exchange) is used to exchange capability information and configuration values between the switch and the host’s CNA port.

And DCBX is an extension of the LLDP protocol, so we need to enable LLDP before using FCoE.

Step-1- Enable FCoE and LLDP features:

switch(config)# feature fcoe
switch(config)# feature lldp

Step-2- Create the VLAN and map the VLAN to VSAN:

switch(config)# vlan 100
switch(config-vlan)# fcoe vsan 100

Step-3- Configure the physical Ethernet interface:

switch(config)# interface Ethernet 1/1
switch(config-if)# shutdown
switch(config-if)# switchport mode trunk
switch(config-if)# switchport trunk native vlan 50
switch(config-if)# switchport trunk allowed vlan 50, 100
switch(config-if)# spanning-tree port type edge trunk
switch(config-if)# no shutdown

Step-4- Bind the virtual interface to the physical Ethernet interface:

vFC is a logical representation of the Fibre Channel interface associated with the Ethernet Interface.

switch(config)# interface vfc 100
switch(config-if)# bind interface Ethernet 1/8

Step-5- Associate vFC interface to its VSAN:

switch(config)# vsan database
switch(config-vsan)# vsan 100 interface vfc 100

References:

• en.wikipedia.org/wiki/Fibre_Channel_over_Ethernet
• searchstorage.techtarget.com/definition/FCoE-Fibre-Channel-over-Ethernet
• https://docplayer.net/136208339-Brocade-fabric-congestion-troubleshooting-guide.html
• www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_1/Cisco_Nexus_5000_Series_Switch_CLI_Software_Configuration_Guide_chapter30.html