Cisco ACI Multi-Pod Deployment – Basic Configuration

Topology


I- IPN Nodes Configuration

With Link Layer Discovery Protocol (LLDP) you can verify the connections between the spine switch and the IPN ports.

Configuration include:

  • sub-interface with VLAN-4
  • OSPF adjacency with the Spines
  • DHCP Relay pointing to APIC Infra IP
  • Multicast PIM

IPN-1A configuration:

IPN-1A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150


vrf context mpod
 ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8 bidir
 ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidir

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 201 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 201 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.2/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 202 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 202 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.6/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/30
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.10.2



IPN-2A configuration:

IPN-2A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
  ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8 bidir
  ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidir

ip pim mtu 9150

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 201 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 201 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.10/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 202 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 202 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.14/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/29
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.10.10



IPN-1B configuration:

IPN-1B# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
 ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8 bidir
 ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidir

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 203 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 203 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.2/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip dhcp relay address 10.1.0.1
  ip dhcp relay address 10.1.0.2
  ip dhcp relay address 10.1.0.3 
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 204 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 204 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.6/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/28
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.20.2



IPN-2B configuration:

IPN-2B# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
  ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8 bidir
  ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidir

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 203 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 203 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.10/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip dhcp relay address 10.1.0.1
  ip dhcp relay address 10.1.0.2
  ip dhcp relay address 10.1.0.3 
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 204 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 204 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.14/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/27
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.20.10

II- Multipod Configuration from Wizard Setup

We will use the Multi-pod Setup wizard twice, the first for local Pod Spines configuration and the second for remote pod Spines configuration:

1- Configure Local Pod Parameters

Navigate to: Fabric > Inventory and Click on Quick Start > Add Pod


We will pass through the information needed for the local pod for IPN connectivity and External Pool Configuration, click on Get started:


  • IP Connectivity (Local Spines – IPNs)

Make sure MTU is set properly as 9150 (as per recommendation). if you are certain all EP (workload) using 1500 (you can set it as 1650). you just need to consider VXLAN overhead in addition to the expected EP MTU.


  • Routing protocol configuration (OSPF with IPN):


We change the OSPF policy to match the IPN devices side (like OSPF point to point)

  • Configure External TEP for POD-1:

This IP range is used to dynamically assign a unique Router-ID address to each spine node and a common Data Plane TEP IP address to all the spines in the pod, also called ETEP (It is an anycast TEP per Pod).

The Router-ID will be used to establish MP-BGP EVPN peerings with the spine nodes in remote pods, whereas the anycast TEP (ETEP) address represents the next-hop for all the EVPN prefixes (MAC and IP addresses for locally discovered endpoints) advertised between pods (serve as Next Hop for BGP EVPN updates across pods).


  • Validate configuration and submit:


2- Configure Remote Pod Parameters

Under Fabric > Inventory, re-Click on Quick Start > Add Pod

  • IP Connectivity (Remote pod Spines – IPNs):


  • Routing protocol configuration (OSPF with IPN):

  • Remote Pod External TEP Pool:

This TEP pool is used for Remote Spines Router-id and Data Plane TEP which serve as the next hop in the EVPN routes advertised from the remote Pod (placeholder in the BGP next hop field in the EVPN L2VPN route updates, but not really used in data plane).


and that’s basically it:


We can verify the configuration pushed properly:

L3OUT in tenant Infra:


Fabric External connection policy:

Navigate to: Infra tenant > Policies > Protocol > Fabric External connection Policies

Verify that in the infra tenant the ‘Fabric Ext Policy default’ object is defined:

The following screenshot were taken after completing first Pod configuration, but the configuration in the Ext connection policy is taken from Mpod setup wizard (basically Data plane TEPs and Spine <-> IPNs subnets):


The Fabric External Routing Profile enables the user to verify whether all routed subnets of the IPN defined are on it. Make sure it includes “Spines <-> IPNs” subnets. this is used for leaking (redistribution) into fabric ISIS (needed for mPOD no touch bring up and DHCP relay connectivity.)

Bilel

Bilel

Related articles

IGMP Snooping in Cisco ACI [Explained and configuration]

Cisco ACI SAN FC (Fiber Channel) and FCoE Configuration

ACI IPv6 Configuration in L3OUT

IPv6 Neighbor Discovery in ACI Explained with Examples

ACI Maintenance Mode

ACI L3 Multicast Configuration [Step By Step]

ACI L2 Multicast Configuration and Troubleshooting

Cisco ACI Service Graph PBR with L3OUT as Destination

ACI Micro EPGs forwarding

L2 PBR Configuration – Cisco ACI

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x