Cisco ACI Multi-Pod Deployment – Basic Configuration

Topology

I- IPN Nodes Configuration

With Link Layer Discovery Protocol (LLDP) you can verify the connections between the spine switch and the IPN ports.

IPN-1A configuration:

IPN-1A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150


vrf context mpod
 ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8
 ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 201 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 201 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.2/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 202 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 202 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.6/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/30
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.10.2

IPN-2A configuration:

IPN-1A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
  ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8
  ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28

ip pim mtu 9150

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 201 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 201 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.10/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 202 Pod 1 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 202 Pod 1 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.10.14/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/29
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.10.10

IPN-1B configuration:

IPN-1A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
 ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8
 ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 203 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 203 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.2/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip dhcp relay address 10.1.0.1
  ip dhcp relay address 10.1.0.2
  ip dhcp relay address 10.1.0.3 
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 204 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 204 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.6/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/28
  ip router ospf mpod area 0.0.0.0
  ip ospf network point-to-point
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.20.2

IPN-2B configuration:

IPN-1A# show run

policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216

feature telnet
feature ospf
feature pim
feature interface-vlan
feature dhcp
feature lldp

ip domain-lookup
system default switchport
system jumbomtu 9150

vrf context mpod
  ip pim rp-address 192.168.100.2 group-list 225.0.0.0/8
  ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28

ip pim mtu 9000

Vlan 4

service dhcp
ip dhcp relay
ipv6 dhcp relay

interface Ethernet1/1
  description To Spine 203 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/1.4
  description To Spine 203 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.10/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip dhcp relay address 10.1.0.1
  ip dhcp relay address 10.1.0.2
  ip dhcp relay address 10.1.0.3 
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  description To Spine 204 Pod 2 Site 1
  no switchport
  speed 40000
  no negotiate auto
  mtu 9150
  vrf member mpod
  no shutdown

interface Ethernet1/2.4
  description To Spine 204 Pod 2 Site 1
  mtu 9150
  encapsulation dot1q 4
  vrf member mpod
  ip address 172.16.20.14/30
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface loopback99
  vrf member mpod
  ip address 192.168.100.1/27
  ip ospf network point-to-point
  ip router ospf mpod area 0.0.0.0
  ip pim sparse-mode

router ospf mpod
  vrf mpod
    router-id 172.16.20.10

II- Multipod Configuration from Wizard Setup

We will use the Multi-pod Setup wizard twice, the first for local Pod Spines configuration and the second for remote pod Spines configuration:

1- Configure Local Pod Parameters

Click on Quick Start > Add Pod

We will pass through the information needed for the local pod for IPN connectivity and External Pool Configuration, click on Get started:

IP Connectivity (Local Spines – IPNs)

Routing protocol configuration (OSPF with IPN):

We change the OSPF policy to match the IPN devices side (like OSPF point to point)

Configure External TEP for POD-1:

This IP range is used to dynamically assign a unique Router-ID address to each spine node and a common Data Plane TEP IP address to all the spines in the pod, also called ETEP (It is an anycast TEP per Pod).

The Router-ID will be used to establish MP-BGP EVPN peerings with the spine nodes in remote pods, whereas the anycast TEP (ETEP) address represents the next-hop for all the EVPN prefixes (MAC and IP addresses for locally discovered endpoints) advertised between pods (serve as Next Hop for BGP EVPN updates across pods).

Validate configuration and submit:

2- Configure Remote Pod Parameters

Under Fabric > Inventory, re-Click on Quick Start > Add Pod

IP Connectivity (Remote pod Spines – IPNs):

Routing protocol configuration (OSPF with IPN):

Remote Pod External TEP Pool:

This TEP pool is used for Remote Spines Router-id and Data Plane TEP which serve as the next hop in the EVPN routes advertised from the remote Pod (placeholder in the BGP next hop field in the EVPN L2VPN route updates, but not really used in data plane).

and that’s basically it:

We can verify the configuration pushed properly:

L3OUT in tenant Infra:

Ext connection policy:

The following screenshot were taken after completing first Pod configuration, but the configuration in the Ext connection policy is taken from Mpod setup wizard (basically Data plane TEPs and Spine <-> IPNs subnets):

The Fabric External Routing Profile enables the user to verify whether all routed subnets of the IPN defined are on it. it includes “Spine > IPN subnets” and it’s used for leaking into ISIS (needed for mpod no touch bring up and DHCP relay connectivity.)