VXLAN EVPN L2 VNI, L3VNI Configuration and Troublshooting

Part-1: Configure L2VNI 40100 For Intra-subnet Communication

I- Underlay Configuration and feature initialization:

nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature nv overlay

interface Ethernet1/1
  no switchport
  ip address 10.0.0.1/30
  ip router ospf 10 area 0.0.0.0
  no shutdown

interface loopback1
  ip address 1.1.1.1/32
  ip router ospf 10 area 0.0.0.0

router ospf 10
  router-id 1.1.1.1Code language: PHP (php)

II- Configure the EVPN tenant VRF

vrf context Tenant-1
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

III- Configure the EVPN L2VNI for Intra-subnet communication

vlan 100
  vn-segment 40100

interface Ethernet1/5
  switchport access vlan 100

evpn
  vni 40100 l2
    rd auto
    route-target import auto
    route-target export autoCode language: JavaScript (javascript)

IV- Configure SVI for L2VNI and enable Anycast Gateway

This is needed for seamless VM mobility across VTEPS, same configuration under all Leafs (VTEPs):

fabric forwarding anycast-gateway-mac 0000.1111.2222

interface Vlan100
  no shutdown
  vrf member Tenant-1
  ip address 172.16.10.254/24
  fabric forwarding mode anycast-gatewayCode language: PHP (php)

V- Configure VXLAN tunnel interface nve1 and associate Layer-2 VNIs with it.

Nve1 is a logical interface where VXLAN packets are encapsulated and decapsulated:

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 40100
    ingress-replication protocol bgpCode language: PHP (php)

VI- Configure MP-BGP with L2VPN EVPN address-family

router bgp 65001
  router-id 1.1.1.1
  neighbor 2.2.2.2
    remote-as 65001
    update-source loopback1
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 3.3.3.3
    remote-as 65001
    update-source loopback1
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Tenant-1
    address-family ipv4 unicast
      advertise l2vpn evpnCode language: CSS (css)

Verification:

The “route-target export auto” generates the RT values by merging BGP AS number and L2VNI, which will gives: 65001:40100

This imply that when using different BGP ASN, defining route target manually will be your best option.

Since, all Leafs are part of same ASN, auto is suitable for us. Below a capture done on leaf-1 illustrating the BGP update for L2VPN EVPN route advertisement:

From above, we see that BGP update carrying the L2 EVPN route from Leaf-1 (1.1.1.1) have a Extended community Route Target 65001:40100.

The L2VPN EVPN route information is stored in the BGP UPDATE MP_REACH_NLRI path attribute:

Network Layer Reachability Information

Example: Below BGP update sent by Leaf-1 toward peers Leaf-2 and Leaf-3:

VxLAN EVPN building blocks on N9K:

We will start with lower level (L2FM), aka MAC learning:

Verify that MAC of Endpoint connected to leaf-3 is learned on leaf-1 on VTEP NVE1 interface:

and MAC connected to Eth1/5 is locally learnt on leaf-1:

We will focus on verifying Control plane Update for the Host (connected on Eth1/5) information on leaf-1 and then on remote VTEP side on leaf-3:

• Verify that the route is installed into L2RIB:

We see the local MAC 5254.000f.b36e installed in L2RIB:

• Check that l2rib exports the update to BGP l2vpn evpn

From previous command, we know that Local MAC learnt from host on eth1/5 (leaf-1) is exported from L2RIB and installed as a path in BGP L2VPN EVPN address family.

Focusing the the host MAC update that we are focusing on:

From following, we can see that L2VPN EVPN for the MAC (also for the MAC/IP) exist in BGP table for address family L2VPN EVPN. We notice that:

• Path is locally originated (because it was learnt from L2RIB).
• Path is advertised to peers 2.2.2.2 and 3.3.3.3 (that’s where VNI was configured).
• We can see the Route Distinguisher and Route target which where auto generate in our example.

Leaf-1# show bgp l2vpn evpn 5254.000f.b36e
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 1.1.1.1:32867    (L2VNI 40100)
BGP routing table entry for [2]:[0]:[0]:[48]:[5254.000f.b36e]:[0]:[0.0.0.0]/216, version 42
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    1.1.1.1 (metric 0) from 0.0.0.0 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 40100
      Extcommunity: RT:65001:40100 ENCAP:8

  Path-id 1 advertised to peers:
    2.2.2.2            3.3.3.3        

BGP routing table entry for [2]:[0]:[0]:[48]:[5254.000f.b36e]:[32]:[172.16.10.10]/248, version 41
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    1.1.1.1 (metric 0) from 0.0.0.0 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 40100
      Extcommunity: RT:65001:40100 ENCAP:8

  Path-id 1 advertised to peers:
    2.2.2.2            3.3.3.3 Code language: PHP (php)

Now, let’s check on the remote VTEP (Leaf-3), and verify the EVPN route (for Endpoint connected on leaf-1 eth1/5) is properly shared and installed on the switch:

Since, we verified on leaf-1: L2FM > L2RIB > BGP EVPN (for MAC endpoint update).

Now we check in opposite direction on leaf-3:

• We see the route in BGP learnt from leaf-1 (1.1.1.1):

Leaf-3# show bgp l2vpn evpn 5254.000f.b36e
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 1.1.1.1:32867
BGP routing table entry for [2]:[0]:[0]:[48]:[5254.000f.b36e]:[0]:[0.0.0.0]/216, version 26
Paths: (1 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-40100
  AS-Path: NONE, path sourced internal to AS
    1.1.1.1 (metric 81) from 1.1.1.1 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 40100
      Extcommunity: RT:65001:40100 ENCAP:8

  Path-id 1 not advertised to any peer

BGP routing table entry for [2]:[0]:[0]:[48]:[5254.000f.b36e]:[32]:[172.16.10.10]/248, version 25
Paths: (1 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-40100
  AS-Path: NONE, path sourced internal to AS
    1.1.1.1 (metric 81) from 1.1.1.1 (1.1.1.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 40100
      Extcommunity: RT:65001:40100 ENCAP:8

  Path-id 1 not advertised to any peerCode language: PHP (php)

Verify L2RIB, we can see that the MAC is now leant from BGP (not local):

Leaf-3# show l2route evpn mac evi 100

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (V):vPC link 
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending
(S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override
(Pf):Permanently-Frozen, (Orp): Orphan

Topology    Mac Address    Prod   Flags         Seq No     Next-Hops                              
----------- -------------- ------ ------------- ---------- ---------------------------------------
100         5254.000f.b36e BGP    SplRcv        0          1.1.1.1 (Label: 40100)                 Code language: CSS (css)

Part-2: Configure L2VNI 40200 For Intra-subnet Communication (vPC use case)

vPC Configuration:

feature vpc

vlan 200
  vn-segment 40200

vpc domain 1
  peer-keepalive destination 192.168.10.2 source 192.168.10.1 vrf default
  peer-gateway

interface Vlan200
  no shutdown
  vrf member Tenant-1
  no ip redirects
  ip address 172.16.20.254/24
  no ipv6 redirects
  fabric forwarding mode anycast-gateway

interface port-channel10
  description ## Peer-link ##
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel20
  switchport access vlan 200
  vpc 20

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 40100
    ingress-replication protocol bgp
  member vni 40200
    ingress-replication protocol bgp

interface Ethernet1/2
  description ## Peer-link ##
  switchport mode trunk
  channel-group 10 mode active

interface Ethernet1/3
  description ## KeepAlive ##
  no switchport
  ip address 192.168.10.1/30
  no shutdown

interface Ethernet1/4
  description ## Connected to Downstream Device ##
  switchport access vlan 200
  channel-group 20

interface loopback1
  ip address 1.1.1.1/32
  ip address 12.12.12.12/32 secondary
  ip router ospf 10 area 0.0.0.0

  vni 40200 l2
    rd auto
    route-target import auto
    route-target export autoCode language: PHP (php)

Verification:

Leaf-1# show vpc brief 
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1   
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : failed  
Type-2 inconsistency reason       : SVI type-2 configuration incompatible
vPC role                          : primary                       
Number of vPCs configured         : 1   
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled
Virtual-peerlink mode             : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans    
--    ----   ------ -------------------------------------------------
1     Po10   up     1,100,200                                                            

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
20    Po20          up     success     success               200                    Code language: PHP (php)

The key configuration is to configure secondary IP for the NVE1 interface:

same secondary IP across both vPC peers:

Leaf-1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [notified]
 Local Router MAC: 5206.3500.1b08
 Host Learning Mode: Control-Plane
 Source-Interface: loopback1 (primary: 1.1.1.1, secondary: 12.12.12.12)Code language: CSS (css)

Leaf-1# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [notified]
 Local Router MAC: 5206.3500.1b08
 Host Learning Mode: Control-Plane
 Source-Interface: loopback1 (primary: 1.1.1.1, secondary: 12.12.12.12)Code language: CSS (css)

Part-3: Configure L3VNI For Inter-subnet Communication

feature vn-segment-vlan-based

vlan 500
  vn-segment 110110

vrf context Tenant-1
  vni 110110
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan500
  no shutdown
  vrf member Tenant-1
  no ip redirects
  ip forward
  no ipv6 redirects

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 110110 associate-vrf

interface loopback1
  ip address 1.1.1.1/32
  ip address 12.12.12.12/32 secondary
  ip router ospf 10 area 0.0.0.0Code language: PHP (php)

Part-4: Configure L3VNI For External Communication

Reference: Ciscolive