ACI Multisite Service Graph PBR Configuration and Troubleshooting
Contents
Configuration steps
Step-1: Create Service Device template
Under the Device template, create service Device cluster:
Specify the Service node properties:
After the template-level configuration is completed, the next step is to configure site-level configurations. Select one of the sites and click the service device that you just created.
Specify how service node is connected for each site:
Site-1:
Site-2:
One this properly deployed, we should seeL4-L7 Device and PBR Redirect Policy deployed on APIC for each site:
APIC Site-1:
APIC Site-2:
Note
In the case of EPG-to-EPG contracts with PBR, one or more IP prefixes must be configured under each consumer EPG. Because this IP prefix is not intended to be used for network connectivity (but only to derive the class-ID information for the consumer EPG), “No Default SVI Gateway” needs to be checked.
Starting from Cisco ACI Release 6.0(3), it is also supported to specify /32 IPv4 prefixes (or /128 IPv6 prefixes) under the consumer EPGs.
Step-2: Configure service chaining
From template property level, select/create contract to be used for service graph:
Under Service chaining:
click on (+) and L4-L7 device and check redirect:
After deploying the template, the service graph, device-selection policy, and deployed graph Instance are created for the tenant on each APIC.
Verification and Packet flow
Verification sclass translation table from Site-1:
Spine-S1# show dcimgr repo sclass-maps
---------------------------------------------------------------------------------------
Remote | Local | PcTag-DnName
site Vrf PcTag | Vrf PcTag Rel-state |
----------------------------------------------------------------------------------------
2 2424832 16388 | 2260992 32771 formed | uni/tn-test1/mscGraphXlateCont/epgDefXlate-[uni/tn-test1/GraphInst_C-[uni/tn-test1/brc-ctr-pbr1]-G-[uni/tn-test1/AbsGraph-Graph_ctr-pbr1]-S-[uni/tn-test1/ctx-VRF1]/NodeInst-node-1/LegVNode-0/EPgDef-consumer]
2 2424832 49154 | 2260992 16387 formed | uni/tn-test1/ap-APP-S1/epg-EPG-1
2 2424832 32771 | 2260992 49154 formed | uni/tn-test1/ap-APP-S2/epg-EPG-2Code language: PHP (php)S1-Leaf101# show zoning-rule scope 2260992
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+|
| 4285 | 16387 | 49154 | default | uni-dir-ignore | enabled | 2260992 | | redir(destgrp-6) | src_dst_any(9) |
| 4290 | 32771 | 16387 | default | uni-dir | enabled | 2260992 | | permit | src_dst_any(9) |
| 4286 | 49154 | 16387 | default | bi-dir | enabled | 2260992 | | redir(destgrp-6) | src_dst_any(9) |
| 4278 | 32771 | 49154 | default | uni-dir | enabled | 2260992 | | permit | src_dst_any(9) |
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+Code language: PHP (php)Consumer to Provider Direction
Consumer to Provider Direction (from site-2 perspective):
S2-Leaf102# show zoning-rule scope 2424832
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+
| 4193 | 32771 | 49154 | default | uni-dir | enabled | 2424832 | | redir(destgrp-3),redir_override | src_dst_any(9) |
| 4192 | 16388 | 32771 | default | uni-dir | enabled | 2424832 | | permit | src_dst_any(9) |
| 4198 | 49154 | 32771 | default | uni-dir | enabled | 2424832 | | redir(destgrp-3) | src_dst_any(9) |
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+Code language: PHP (php)Note: The zoning-rule for EPG-to-EPG contract with PBR is created with action “redir_override”: this is required in the specific PBR deployment with Cisco ACI Multi-Site. With this action, the hardware creates two entries to take different actions depending on whether the destination (provider) is in the local site or not:
- If the destination is in the local site, the PBR policy is applied.
- If the destination is NOT in the local site, the traffic is just permitted so that the redirection can instead happen on the leaf in the site where the provider endpoint resides. That’s how to get a provider leaf to always apply PBR policy.
When the consumer EPG-2 endpoint sends traffic toward the EPG-1 endpoints, the consumer leaf just forwards the traffic toward the provider leaf.
The consumer leaf must not apply the PBR policy even if the consumer leaf can resolve the destination class ID of the provider EPG.
In the case of an EPG-to-EPG contract with PBR, the zoning-rule for consumer-to-provider traffic has a special flag called “redirect override” based on which the leaf avoids applying the policy unless the destination endpoint is locally learned. Thus, the use of the “redirect override” zoning-rule ensures that the provider leaf always applies the PBR policy even for the consumer-to-provider traffic.
Troubleshooting and packet capture
I- Consumer to Provider flow:
we will start ping from VM-2:
1- ELAM on leaf-102 (Site-2), which is the consumer leaf:
module-1(DBG-elam-insel6)# debug platform internal roc elam asic 0
module-1(DBG-elam)# trigger reset
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 172.16.31.20 dst_ip 172.16.29.10
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# status
ELAM STATUS
===========
Asic 0 Slice 0 Status Triggered
module-1(DBG-elam-insel6)# ereport
Python available. Continue ELAM decode with LC Pkg
ELAM REPORT
======================================================================================================================================================
Trigger/Basic Information
======================================================================================================================================================
ELAM Report File : /tmp/logs/elam_2024-09-16-42m-08h-58s.txt
In-Select Trigger : Outerl2-outerl3-outerl4( 6 )
Out-Select Trigger : Pktrw-sideband-sb_info( 0 )
ELAM Captured Device : LEAF
Packet Direction : ingress
Triggered ASIC type : Homewood
Triggered ASIC instance : 0
Triggered Slice : 0
Incoming Interface : 0x14( 0x14 )
( Slice Source ID(Ss) in "show plat int hal l2 port gpd" )
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2uc ipv4 ip ipuc ipv4uc
Opcode : OPCODE_UC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
IP Version : 4
DSCP : 0
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : set
TTL : 64
IP Protocol Number : ICMP
IP CheckSum : 10156( 0x27AC )
Destination IP : 172.16.29.10
Source IP : 172.16.31.20
We can see that on Consumer leaf, the contract wasn’t applied, meaning that redir_override took effect since destination isn’t local to site-2:
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 29368( 0x72B8 )
sclass (src pcTag) : 32771( 0x8003 )
dclass (dst pcTag) : 49154( 0xC002 )
src pcTag is from local table : yes
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : no
Contract Hit : yes
Contract Aclqos Stats Index : 81787
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81787" )Code language: JavaScript (javascript)We can confirm by checking the aclqos rule index:
module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81787"
Extra TCAM resource:
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 151 | hw_index = 133 | stats_idx = 81787
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81787Code language: PHP (php)S2P1-Leaf102# show zoning-rule rule-id 4193
Config State
============
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | Intent |
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+
| 4193 | 32771 | 49154 | default | uni-dir | enabled | 2424832 | ctr-pbr1 | redir(destgrp-3),redir_override | src_dst_any(9) | install |
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+Code language: PHP (php)we can see that this rule has two entries:
- A redirect rule (grp-3) applied when destination is local to site.
- Extra entry used for override_redir, which will “permit” traffic (policy applied bit NOT set).
=============================
module-1# show sys int aclqos zoning-rules | grep -A 25 "4193"
Rule ID: 4193 Scope 13 Src EPG: 32771 Dst EPG: 49154 Filter 65535
Redir group: 3
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 148 | hw_index = 116 | stats_idx = 81804
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81804
76
Tcam Total Entries: 1
HW Stats: 3243
Extra TCAM resource:
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 151 | hw_index = 133 | stats_idx = 81787
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81787
78
Tcam Total Entries: 1
HW Stats: 1011Code language: PHP (php)2- ELAM on leaf-101 (Site-1), which is the provider leaf:
We can see that Src Policy Applied Bit is “0” –> contract not applied:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 28
IP Protocol Number : UDP
Destination IP : 10.0.168.64
Source IP : 172.16.200.101
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.29.10
Source IP : 172.16.31.20
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 0
Dst Policy Applied Bit : 0
sclass (src pcTag) : 0xc002
VRF or BD VNID : 2260992( 0x228000 ) Code language: PHP (php)On the provider leaf, as per design, the contract will be applied and traffic will be redirect to FW in site-1:
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 46616( 0xB618 )
sclass (src pcTag) : 49154( 0xC002 )
dclass (dst pcTag) : 16387( 0x4003 )
src pcTag is from local table : no
derived from group-id in iVxLAN header of incoming packet
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81728
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81728" )Code language: JavaScript (javascript)
Checking the rule entry, we see it’s pointing to the redirect to group-6:
module-1(DBG-elam-insel14)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81728"
Rule ID: 4286 Scope 23 Src EPG: 49154 Dst EPG: 16387 Filter 65535
Redir group: 6
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 232 | hw_index = 192 | stats_idx = 81728
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81728Code language: PHP (php)Redirect group detail:
S1-Leaf101# show service redir info group 6
===============================================================================================================================================================
LEGEND
TL: Threshold(Low) | TH: Threshold(High) | HP: HashProfile | HG: HealthGrp | BAC: Backup-Dest | TRA: Tracking | RES: Resiliency | W: Weight
===============================================================================================================================================================
GrpID Name destination HG-name BAC W operSt operStQual TL TH HP TRA RES
===== ==== =========== ============== === === ======= ============ === === === === ===
6 destgrp-6 dest-[192.168.1.10]-[vxlan-2260992] test-1::DeviceCluster1--ndo--imp N 1 enabled no-oper-grp 0 0 sym yes no Code language: PHP (php)
Next the leaf-101 in site-1 (provider) will apply the redirect rule and send traffic to spine with Service BD VNID to make L2 lookup in spine COOP for the PBR destination MAC:
List of destinations
Name bdVnid vMac vrf operSt operStQual HG-name
==== ====== ==== ==== ===== ========= =======
enabled no-oper-dest Not attached
dest-[192.168.1.10]-[vxlan-2260992] vxlan-15859680 00:EA:BD:07:**:7C test1:VRF1 enabled no-oper-dest test1::DeviceCluster1--ndo--implctCode language: PHP (php)ELAM on spine Site-1:
From the spine, we see the packet coming from leaf-101 (site-1) with Service BD VNID 15859680 to perform spine proxy lookup for the destination service node MAC:
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x0
TTL : 27
IP Protocol Number : UDP
Destination IP : 10.0.128.64
Source IP : 10.0.168.64
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 62
IP Protocol Number : ICMP
Destination IP : 172.16.29.10
Source IP : 172.16.31.20
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 1
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0xc002
VRF or BD VNID : 15859680( 0xF1FFE0 )
------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits Set in LUC Forwarding Mode are: : ISPINE_SC SPINE_PROXY UC INFRA ENCAP BRIDGE HIT
------------------------------------------------------------------------------------------------------------------------------------------------------Code language: PHP (php)Then packet is forwarded to the FW and packet will be permitted from FW to VM-1.
II- Provider to Consumer flow:
For the return traffic, it’s not different, but we will illustrate how the Provider leaf (Leaf-101 site-1) will also apply the policy and redirect to FW of site-1, which ensure that no asymmetry is encountered:
1- ELAM on leaf-101 (site-1), Provider:
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
IP Version : 4
DSCP : 0
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : not set
TTL : 64
IP Protocol Number : ICMP
IP CheckSum : 13546( 0x34EA )
Destination IP : 172.16.31.20
Source IP : 172.16.29.10
An interesting part to check is the destination IP lookup for Provider to consumer traffic, since we configured the subnet under the consumer EPG, the pctag is tied to consumer EPG subnet:
======================================================================================================================================================
Forwarding Lookup ( FPB )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination IP (Lookup Key)
------------------------------------------------------------------------------------------------------------------------------------------------------
Dst IP Lookup was performed : yes
Dst IP Lookup VRF : 4656( 0x1230 )
( Hw VrfId in "show plat int hal l3 vrf pi" )
Dst IP Address : 172.16.31.20
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination IP (Lookup Result)
------------------------------------------------------------------------------------------------------------------------------------------------------
Dst IP is Hit : yes
Dst IP Hit Index : 24781( 0x60CD )
( HIT IDX in "show plat int hal l3 routes")Code language: JavaScript (javascript)Which the provider leaf will rely on it to apply policy (since policy need to be applied on provider leaf) in case the sclass not learn in EPM (from data plane):
module-1(DBG-elam-insel6)# show plat int hal l3 routes | grep 172.16.31
| 4656| 172.16.31.0/ 24| UC| ade| 131| TCAM| 812| 0| 812|A| 7591| 801a| NA| NA| NA| NA| 0|c002|
3| 0| 0| 0| sc, dr, le
| 4656| 172.16.31.20/ 32| EP| ae6| c304| TRIE| cd| 5/ 0| 60cd|A| 80000e81| 88d2| 1e8e| 1/ 2| cd| 0| 0|c002|
1| 128| 0| 0| sc, le,sneCode language: PHP (php)Also, we can verify that contract is applied:
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 0( 0x0 )
L4 Dst Port : 50583( 0xC597 )
sclass (src pcTag) : 16387( 0x4003 )
dclass (dst pcTag) : 49154( 0xC002 )
src pcTag is from local table : yes
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81739
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81739" )Code language: JavaScript (javascript)and redirect rule was hit:
module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81739"
Rule ID: 4285 Scope 23 Src EPG: 16387 Dst EPG: 49154 Filter 65535
Redir group: 6
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 229 | hw_index = 181 | stats_idx = 81739
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81739Code language: PHP (php)
