ACI Multiste PBR Configuration and Troubleshooting

ACI Multiste PBR Configuration and Troubleshooting

Configuration steps

Step-1: Create Service Device template

Under the Device template, create service Device cluster:


Specify the Service node properties:

After the template-level configuration is completed, the next step is to configure site-level configurations. Select one of the sites and click the service device that you just created.


Specify how service node is connected for each site:

Site-1:

Site-2:


One this properly deployed, we should seeL4-L7 Device and PBR Redirect Policy deployed on APIC for each site:

APIC Site-1:

APIC Site-2:


Note

In the case of EPG-to-EPG contracts with PBR, one or more IP prefixes must be configured under each consumer EPG. Because this IP prefix is not intended to be used for network connectivity (but only to derive the class-ID information for the consumer EPG), “No Default SVI Gateway” needs to be checked.

Starting from Cisco ACI Release 6.0(3), it is also supported to specify /32 IPv4 prefixes (or /128 IPv6 prefixes) under the consumer EPGs.


Step-2: Configure service chaining

From template property level, select/create contract to be used for service graph:

Under Service chaining:

click on (+) and L4-L7 device and check redirect:

After deploying the template, the service graph, device-selection policy, and deployed graph Instance are created for the tenant on each APIC.


Verification and Packet flow:


Verification sclass translation table from Site-1:

Spine-S1# show dcimgr repo sclass-maps
---------------------------------------------------------------------------------------
     Remote            |        Local                  |   PcTag-DnName
site  Vrf       PcTag  |  Vrf      PcTag     Rel-state |
----------------------------------------------------------------------------------------
  2   2424832   16388  |  2260992   32771       formed | uni/tn-test1/mscGraphXlateCont/epgDefXlate-[uni/tn-test1/GraphInst_C-[uni/tn-test1/brc-ctr-pbr1]-G-[uni/tn-test1/AbsGraph-Graph_ctr-pbr1]-S-[uni/tn-test1/ctx-VRF1]/NodeInst-node-1/LegVNode-0/EPgDef-consumer]
  2   2424832   49154  |  2260992   16387       formed | uni/tn-test1/ap-APP-S1/epg-EPG-1
  2   2424832   32771  |  2260992   49154       formed | uni/tn-test1/ap-APP-S2/epg-EPG-2Code language: PHP (php)

S1-Leaf101# show zoning-rule scope 2260992
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID |      Dir       |  operSt |  Scope  | Name |      Action      |       Priority       |
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+|  
|   4285  | 16387  | 49154  | default  | uni-dir-ignore | enabled | 2260992 |      | redir(destgrp-6) |    src_dst_any(9)    |
|   4290  | 32771  | 16387  | default  |    uni-dir     | enabled | 2260992 |      |      permit      |    src_dst_any(9)    |
|   4286  | 49154  | 16387  | default  |     bi-dir     | enabled | 2260992 |      | redir(destgrp-6) |    src_dst_any(9)    |
|   4278  | 32771  | 49154  | default  |    uni-dir     | enabled | 2260992 |      |      permit      |    src_dst_any(9)    |
+---------+--------+--------+----------+----------------+---------+---------+------+------------------+----------------------+Code language: PHP (php)


Consumer to Provider Direction


Consumer to Provider Direction (from site-2 perspective):

S2-Leaf102# show zoning-rule scope 2424832
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID |   Dir   |  operSt |  Scope  | Name |              Action             |       Priority       |
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+
|   4193  | 32771  | 49154  | default  | uni-dir | enabled | 2424832 |      | redir(destgrp-3),redir_override |    src_dst_any(9)    |
|   4192  | 16388  | 32771  | default  | uni-dir | enabled | 2424832 |      |              permit             |    src_dst_any(9)    |
|   4198  | 49154  | 32771  | default  | uni-dir | enabled | 2424832 |      |         redir(destgrp-3)        |    src_dst_any(9)    |
+---------+--------+--------+----------+---------+---------+---------+------+---------------------------------+----------------------+Code language: PHP (php)

Note: The zoning-rule for EPG-to-EPG contract with PBR is created with action “redir_override”: this is required in the specific PBR deployment with Cisco ACI Multi-Site. With this action, the hardware creates two entries to take different actions depending on whether the destination (provider) is in the local site or not:

  • If the destination is in the local site, the PBR policy is applied.
  • If the destination is NOT in the local site, the traffic is just permitted so that the redirection can instead happen on the leaf in the site where the provider endpoint resides. That’s how to get a provider leaf to always apply PBR policy.

When the consumer EPG-2 endpoint sends traffic toward the EPG-1 endpoints, the consumer leaf just forwards the traffic toward the provider leaf.
The consumer leaf must not apply the PBR policy even if the consumer leaf can resolve the destination class ID of the provider EPG.

In the case of an EPG-to-EPG contract with PBR, the zoning-rule for consumer-to-provider traffic has a special flag called redirect override based on which the leaf avoids applying the policy unless the destination endpoint is locally learned. Thus, the use of the “redirect override” zoning-rule ensures that the provider leaf always applies the PBR policy even for the consumer-to-provider traffic.


Troubleshooting and packet capture

I- Consumer to Provider flow:

we will start ping from VM-2:

1- ELAM on leaf-102 (Site-2), which is the consumer leaf:

module-1(DBG-elam-insel6)# debug platform internal roc elam asic 0
module-1(DBG-elam)# trigger reset
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam-insel6)# set outer  ipv4 src_ip 172.16.31.20 dst_ip 172.16.29.10
module-1(DBG-elam-insel6)# start  
module-1(DBG-elam-insel6)# status 
 ELAM STATUS
===========
Asic 0 Slice 0 Status Triggered

module-1(DBG-elam-insel6)# ereport 
Python available. Continue ELAM decode with LC Pkg
 ELAM REPORT 

======================================================================================================================================================
                                                        Trigger/Basic Information                                                                     
======================================================================================================================================================
ELAM Report File                        : /tmp/logs/elam_2024-09-16-42m-08h-58s.txt
In-Select Trigger                       : Outerl2-outerl3-outerl4( 6 )            
Out-Select Trigger                      : Pktrw-sideband-sb_info( 0 )             
ELAM Captured Device                    : LEAF                                    
Packet Direction                        : ingress                                 
Triggered ASIC type                     : Homewood                                
Triggered ASIC instance                 : 0                                       
Triggered Slice                         : 0                                       
Incoming Interface                      : 0x14( 0x14 )                            
( Slice Source ID(Ss) in "show plat int hal l2 port gpd" )

======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes       : l2uc ipv4 ip ipuc ipv4uc                
Opcode                        : OPCODE_UC                                                       

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
IP Version                    : 4                                       
DSCP                          : 0                                       
IP Packet Length              : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit            : set                                     
TTL                           : 64                                      
IP Protocol Number            : ICMP                                    
IP CheckSum                   : 10156( 0x27AC )                         
Destination IP                : 172.16.29.10                            
Source IP                     : 172.16.31.20                            

We can see that on Consumer leaf, the contract wasn’t applied, meaning that redir_override took effect since destination isn’t local to site-2:

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol                             : ICMP( 0x1 )                   
L4 Src Port                             : 2048( 0x800 )                 
L4 Dst Port                             : 29368( 0x72B8 )               
sclass (src pcTag)                      : 32771( 0x8003 )               
dclass (dst pcTag)                      : 49154( 0xC002 )               
src pcTag is from local table           : yes                           
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet          : no                            
If yes, Contract is not applied here because it is flooded

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop                           : no                            
Contract Logging                        : no                            
Contract Applied                        : no                            
Contract Hit                            : yes                           
Contract Aclqos Stats Index             : 81787                         
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81787" )Code language: JavaScript (javascript)

We can confirm by checking the aclqos rule index:

module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81787"

Extra TCAM resource:
    unit_id: 0
    === Region priority: 2462 (rule prio: 9 entry: 158)===
        sw_index = 151 | hw_index = 133 | stats_idx = 81787 

  Curr TCAM resource:
  =============================
    === SDK Info ===
        Result/Stats Idx: 81787Code language: PHP (php)
S2P1-Leaf102# show zoning-rule rule-id 4193  
Config State
============
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+
| Rule ID | SrcEPG | DstEPG | FilterID |   Dir   |  operSt |  Scope  |   Name   |              Action             |    Priority    |  Intent |
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+
|   4193  | 32771  | 49154  | default  | uni-dir | enabled | 2424832 | ctr-pbr1 | redir(destgrp-3),redir_override | src_dst_any(9) | install |
+---------+--------+--------+----------+---------+---------+---------+----------+---------------------------------+----------------+---------+Code language: PHP (php)

we can see that this rule has two entries:

  • A redirect rule (grp-3) applied when destination is local to site.
  • Extra entry used for override_redir, which will “permit” traffic (policy applied bit NOT set).
  =============================
module-1# show sys int aclqos zoning-rules | grep -A 25 "4193"
Rule ID: 4193 Scope 13 Src EPG: 32771 Dst EPG: 49154 Filter 65535
  Redir group: 3
    unit_id: 0
    === Region priority: 2462 (rule prio: 9 entry: 158)===
        sw_index = 148 | hw_index = 116 | stats_idx = 81804 

  Curr TCAM resource:
  =============================
    === SDK Info ===
        Result/Stats Idx: 81804
        76  
        Tcam Total Entries: 1         
        HW Stats: 3243

Extra TCAM resource:
    unit_id: 0
    === Region priority: 2462 (rule prio: 9 entry: 158)===
        sw_index = 151 | hw_index = 133 | stats_idx = 81787 

  Curr TCAM resource:
  =============================
    === SDK Info ===
        Result/Stats Idx: 81787
        78  
        Tcam Total Entries: 1         
        HW Stats: 1011Code language: PHP (php)


2- ELAM on leaf-101 (Site-1), which is the provider leaf:

We can see that Src Policy Applied Bit is “0” –> contract not applied:

======================================================================================================================================================
                                                            Captured Packet                                                                           

======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x0                                     
TTL                           : 28                                      
IP Protocol Number            : UDP                                     
Destination IP                : 10.0.168.64                             
Source IP                     : 172.16.200.101                          

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x1                                     
TTL                           : 63                                      
IP Protocol Number            : ICMP                                    
Destination IP                : 172.16.29.10                            
Source IP                     : 172.16.31.20                            

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : iVxLAN                                  
Don't Learn Bit               : 0                                       
Src Policy Applied Bit        : 0                                       
Dst Policy Applied Bit        : 0                                       
sclass (src pcTag)            : 0xc002                                  
VRF or BD VNID                : 2260992( 0x228000 )                     Code language: PHP (php)

On the provider leaf, as per design, the contract will be applied and traffic will be redirect to FW in site-1:

======================================================================================================================================================
                                                                Contract Lookup ( FPC )
======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol                             : ICMP( 0x1 )                   
L4 Src Port                             : 2048( 0x800 )                 
L4 Dst Port                             : 46616( 0xB618 )               
sclass (src pcTag)                      : 49154( 0xC002 )               
dclass (dst pcTag)                      : 16387( 0x4003 )               
src pcTag is from local table           : no                            
derived from group-id in iVxLAN header of incoming packet
Unknown Unicast / Flood Packet          : no                            
If yes, Contract is not applied here because it is flooded

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop                           : no                            
Contract Logging                        : no                            
Contract Applied                        : yes                           
Contract Hit                            : yes                           
Contract Aclqos Stats Index             : 81728                         
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81728" )Code language: JavaScript (javascript)


Checking the rule entry, we see it’s pointing to the redirect to group-6:

module-1(DBG-elam-insel14)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81728" 
Rule ID: 4286 Scope 23 Src EPG: 49154 Dst EPG: 16387 Filter 65535
  Redir group: 6
    unit_id: 0
    === Region priority: 2462 (rule prio: 9 entry: 158)===
        sw_index = 232 | hw_index = 192 | stats_idx = 81728 

  Curr TCAM resource:
  =============================
    === SDK Info ===
        Result/Stats Idx: 81728Code language: PHP (php)

Redirect group detail:

S1-Leaf101# show service redir info group 6
===============================================================================================================================================================
LEGEND
TL: Threshold(Low)  |  TH: Threshold(High) |  HP: HashProfile  |  HG: HealthGrp  | BAC: Backup-Dest |  TRA: Tracking | RES: Resiliency | W: Weight
===============================================================================================================================================================
GrpID Name            destination                                                    HG-name                          BAC W   operSt   operStQual      TL  TH  HP  TRA RES
===== ====            ===========                                                    ==============                   === === =======  ============    === === === === ===
6     destgrp-6       dest-[192.168.1.10]-[vxlan-2260992]                            test-1::DeviceCluster1--ndo--imp N   1   enabled  no-oper-grp     0   0   sym yes no Code language: PHP (php)


Next the leaf-101 in site-1 (provider) will apply the redirect rule and send traffic to spine with Service BD VNID to make L2 lookup in spine COOP for the PBR destination MAC:

 List of destinations
Name                                                           bdVnid          vMac                 vrf                       operSt     operStQual      HG-name                         
====                                                           ======          ====                 ====                      =====      =========       =======                         
           enabled    no-oper-dest    Not attached                    
dest-[192.168.1.10]-[vxlan-2260992]                            vxlan-15859680  00:EA:BD:07:**:7C    test1:VRF1               enabled    no-oper-dest    test1::DeviceCluster1--ndo--implctCode language: PHP (php)


ELAM on spine Site-1:

From the spine, we see the packet coming from leaf-101 (site-1) with Service BD VNID 15859680 to perform spine proxy lookup for the destination service node MAC:

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x0                                     
TTL                           : 27                                      
IP Protocol Number            : UDP                                     
Destination IP                : 10.0.128.64                             
Source IP                     : 10.0.168.64                             

------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
DSCP                          : 0                                       
Don't Fragment Bit            : 0x1                                     
TTL                           : 62                                      
IP Protocol Number            : ICMP                                    
Destination IP                : 172.16.29.10                            
Source IP                     : 172.16.31.20                            

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type                       : iVxLAN                                  
Don't Learn Bit               : 1                                       
Src Policy Applied Bit        : 1                                       
Dst Policy Applied Bit        : 1                                       
sclass (src pcTag)            : 0xc002                                  
VRF or BD VNID                : 15859680( 0xF1FFE0 )     

------------------------------------------------------------------------------------------------------------------------------------------------------
FINAL FORWARDING LOOKUP 
------------------------------------------------------------------------------------------------------------------------------------------------------
Bits Set in LUC Forwarding Mode are:    : ISPINE_SC SPINE_PROXY UC INFRA ENCAP BRIDGE HIT 
------------------------------------------------------------------------------------------------------------------------------------------------------Code language: PHP (php)

Then packet is forwarded to the FW and packet will be permitted from FW to VM-1.


I- Provider to Consumer flow:

For the return traffic, it’s not different, but we will illustrate how the Provider leaf (Leaf-101 site-1) will also apply the policy and redirect to FW of site-1, which ensure that no asymmetry is encountered:

1- ELAM on leaf-101 (site-1), Provider:

------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type                       : IPv4                                    
IP Version                    : 4                                       
DSCP                          : 0                                       
IP Packet Length              : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit            : not set                                 
TTL                           : 64                                      
IP Protocol Number            : ICMP                                    
IP CheckSum                   : 13546( 0x34EA )                         
Destination IP                : 172.16.31.20                            
Source IP                     : 172.16.29.10                            

An interesting part to check is the destination IP lookup for Provider to consumer traffic, since we configured the subnet under the consumer EPG, the pctag is tied to consumer EPG subnet:

======================================================================================================================================================
                                                                Forwarding Lookup ( FPB )
======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Destination IP (Lookup Key)
------------------------------------------------------------------------------------------------------------------------------------------------------
Dst IP Lookup was performed             : yes                           
Dst IP Lookup VRF                       : 4656( 0x1230 )                
( Hw VrfId in "show plat int hal l3 vrf pi" )
Dst IP Address                          : 172.16.31.20                  

------------------------------------------------------------------------------------------------------------------------------------------------------
Destination IP (Lookup Result)
------------------------------------------------------------------------------------------------------------------------------------------------------
Dst IP is Hit                           : yes                           
Dst IP Hit Index                        : 24781( 0x60CD )               
( HIT IDX in "show plat int hal l3 routes")Code language: JavaScript (javascript)

Which the provider leaf will rely on it to apply policy (since policy need to be applied on provider leaf) in case the sclass not learn in EPM (from data plane):

module-1(DBG-elam-insel6)# show plat int hal l3 routes | grep 172.16.31
| 4656|                            172.16.31.0/ 24| UC|  ade|       131| TCAM|  812|     0|  812|A|     7591|    801a|   NA|    NA|       NA| NA|      0|c002|
  3|   0| 0| 0| sc, dr, le
| 4656|                           172.16.31.20/ 32| EP|  ae6|      c304| TRIE|   cd| 5/  0| 60cd|A| 80000e81|    88d2| 1e8e| 1/  2|       cd|  0|      0|c002|
  1| 128| 0| 0| sc, le,sneCode language: PHP (php)

Also, we can verify that contract is applied:

======================================================================================================================================================
                                                                Contract Lookup ( FPC )
======================================================================================================================================================

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol                             : ICMP( 0x1 )                   
L4 Src Port                             : 0( 0x0 )                      
L4 Dst Port                             : 50583( 0xC597 )               
sclass (src pcTag)                      : 16387( 0x4003 )               
dclass (dst pcTag)                      : 49154( 0xC002 )               
src pcTag is from local table           : yes                           
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet          : no                            
If yes, Contract is not applied here because it is flooded

------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop                           : no                            
Contract Logging                        : no                            
Contract Applied                        : yes                           
Contract Hit                            : yes                           
Contract Aclqos Stats Index             : 81739                         
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81739" )Code language: JavaScript (javascript)

and redirect rule was hit:

module-1(DBG-elam-insel6)#  show sys int aclqos zoning-rules | grep -B 9 "Idx: 81739"
Rule ID: 4285 Scope 23 Src EPG: 16387 Dst EPG: 49154 Filter 65535
  Redir group: 6
    unit_id: 0
    === Region priority: 2462 (rule prio: 9 entry: 158)===
        sw_index = 229 | hw_index = 181 | stats_idx = 81739 

  Curr TCAM resource:
  =============================
    === SDK Info ===
        Result/Stats Idx: 81739Code language: PHP (php)

Bilel-A

Enthusiastic Network Engineer specializing in Cisco ACI, passionate about solving challenges. A lifelong learner who loves gaining and sharing knowledge. Profile: https://www.linkedin.com/in/bilel-ameur-71116b2b5
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x