Quality of Service (QoS) in Cisco ACI – Overview and Configuration Examples
Any traffic for which the QoS features are enabled undergoes the following stages:
- Classification: identification of the traffic type and assignment of a Cisco ACI QoS Level based on it.
- Policing: control of the traffic based on its classification.
- Marking: tagging of network packets based on the configured policing rules and its behavior.
- Queuing and Scheduling: prioritization and/or isolation of network packets based on their QoS Level and markings.
Contents
I- Classification
Traffic classification is used to partition traffic in your Cisco ACI fabric into QoS Levels based on several criteria:
- Ingress packet headers (DSCP or CoS)
- Source EPGs
- EPG Contracts.
When you configure a QoS Level for a type of traffic, you can specify one or more of these criteria to match, you can choose to exclude a particular criteria, or you can determine the traffic class by matching any or all criteria. Traffic that fails to match any class is assigned to a default class (Level3) of traffic.
When packets first ingress the Cisco ACI fabric, two values can be use to classify the traffic into the proper QoS Level:
- Class of Service (CoS): Also referred to as “dot1p value”, a QoS feature developed by the 802.1p group that uses a 3-bit Priority Code Point (PCP) inside the Layer-2 Ethernet frames to differentiate traffic.
- Differentiated Services Code Point (DSCP): A Layer-3 alternative to CoS that uses a 6-bit value in the IP packet header to classify traffic.
II- Marking
After traffic is classified, the packets are marked by adding the QoS class ID to the outer header of each packet. Traffic classification and marking happens on the ingress leaf switches only; the spine and egress leaf switches only map the packets to proper class of service based on the CoS value.
Class of Service | Traffic Type | Dot1p (CoS) Marking in VXLAN Header | DEI Bit |
| 0 | Level 3 (default) | 0 | 0 |
| 1 | Level 2 | 1 | 0 |
| 2 | Level 1 | 2 | 0 |
| 6 | Level 4 | 2 | 1 |
| 5 | Level 5 | 3 | 1 |
| 4 | Level 6 | 5 | 1 |
| 3 | APIC Controller | 3 | 0 |
| 9 | SPAN | 4 | 0 |
| 8 (SUP) | Control | 5 | 0 |
| 8 (SUP) | Traceroute | 6 | 0 |
| 7 | Copy Service | 7 | 0 |
III- QoS Queuing
After the traffic packets have been classified (or re-classified based on markings) and assigned a QoS Level, they are subject to being queued for transmission.
Multiple queues can be used based on the packet’s priority and a scheduling algorithm is used to determine which queue’s packet is to be transmitted next.
IV- Scheduling and Congestion Avoidance
Cisco ACI uses a Deficit Weighted Round Robin (DWRR) scheduling algorithm. This scheduling algorithm allows packets of variable sizes and provides a deficit counter to dynamically adjust queue priorities. The queuing and scheduling policy is a fabric-wide configuration and applies to all nodes. The same policy is applied within each node whenever packet queuing takes place.
If at any point the network becomes congested, a congestion avoidance algorithm can be used to determine which packets to transmit, queue, or drop.
Cisco APIC deploys two different congestion avoidance algorithms for user-configurable QoS Levels:
- Tail Drop (TD): In case of congestion, any new incoming packets (tail end of a queue) are dropped. Tail Drop uses single threshold per queue. Tail drop is used by default if no tuning done for QoS class configuration.
- Weighted Random Early Detection (WRED): Provides an early detection mechanism, which allows for low priority packets to be preemptively dropped in order to protect higher priority queues from congestion. WRED uses one or more thresholds per queue with each queue associated with DSCP or CoS values.
V- ACI QoS Configuration and Troubleshooting
Let’s put ACI QoS classification and marking into action.
Cisco ACI ruses six different user-configurable qos-groups to prioritize the traffic and four internally reserved qos-groups.
You can check and modify the user configurable QoS classes fromAPIC GUI:
Fabric Access Policies > Policies > Global Policies > QOS Class
If you click on any class on the GUI, you have the option to modify some settings mostly related to queuing and Congestion avoidance:
| Bandwidth allocated (in %) | The percentage of total bandwidth allocated to this QoS Level. The value can be between 0 and 100. The default value is 20. |
| Congestion Algorithm | The congestion algorithm used for this QoS Level. The congestion algorithm can be: – Tail Drop – Weighted random early detection |
For WRED Congestion avoidance:
| Congestion Notification(Weighted random early detection algorithm only) | Indicates the state of Explicit Congestion Notification (ECN) setting. Enabling Congestion Notification causes the packets that would be dropped to be ECN-marked instead. The state can be:EnabledDisabled |
| Min Threshold (percentage)(Weighted random early detection algorithm only) | The minimum queue threshold as a percentage of the maximum queue length for WRED algorithm.If the average queue size is below the minimum threshold value, the arriving packets are queued immediately. |
| Max Threshold (percentage)(Weighted random early detection algorithm only) | The maximum queue threshold as a percentage of the maximum queue length for WRED algorithm.If the average queue size is greater than the maximum threshold value, the arriving packets are dropped. |
| Probability ( percentage) | The Drop probability value for WRED algorithm. The probability determines whether the packet is dropped or queued when the average queue size is between the minimum and the maximum threshold values. |
| Weight(Weighted random early detection algorithm only) | The weight value for WRED algorithm.Weight has a range of 0 to 7 and is used to calculate average queue length. Lower weight prioritizes current queue length, while higher weight prioritizes older queue lengths. |
For Shceduling:
| Scheduling algorithm | The scheduling algorithm used for this QoS Level. The scheduling algorithm can be: – Strict priority – Weighted round robin (Default) |
Also, from leaf CLI, you can check details for each QoS class:
leaf1# show system internal qos classes
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level4
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 0 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level3
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 20 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: span
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 1 meth WRR
Queue: limit 1500 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: control-plane
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 0 meth SP
Queue: limit 1500 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level1
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 20 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level6
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 0 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level2
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 20 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled
-------------------------------------------------------
QOS Class
-------------------------------------------------------
Id: level5
Mtu: 9216
Buffer: min 0
Cong: algo 1 ecn 0 max_thres 100 min_thres 0 drop_prob 0 weight 0 forward_nonecn 0
Sched: bw 0 meth WRR
Queue: limit 1522 meth Dynamic
Uburst: enabled 0 type 1 down_thres 100 up_thres 100
PFC Admin State: Disabled By default, in case there no custom QoS configuration, the traffic from a tenant EPG is mapped to the Level 3 class regardless of the CoS of the original packet.
ACI Qos Classification Rules:
In simple words, the precedence is a below:
1- QoS on contract
2- Custom QoS enabled on source EPG
3- Class ID of the source EPG.
If no QoS class assigned, Level3 class is set by default.
Let’s start with Class ID on source EPG:
I- Classification based on source EPG QoS class:
Under the EPG Policy settings > QoS class, we will specify Level1 as example (which maps to COS 2)
Then, I will start a traffic sourced from this EPG and capture it on the spine:
From the ELAM capture, we can see that COS 2 was assigned in the outer header:
ELAM on Spine:
##############
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 000D.0D0D.0D0D
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : yes
CoS : 2
Access Encap VLAN : 2
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 2
Don't Fragment Bit : 0x0
TTL : 32
IP Protocol Number : UDP
Destination IP : 10.0.128.68
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.31.22
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4003
VRF or BD VNID : 2752518( 0x2A0006 ) Code language: PHP (php)
If we look into diagram to interpret the seen capture:
- Outer L2 header have COS=2 (mapped to Level-1), after traffic was classifed into EPG-1 and EPG-1 had QoS class set to Level-1, then the outer VXLAN ethernet header will have a COS value of 2.
- Outer L2 header DSCP value:
By default, the Outer DSCP will have the 3 most significant bit set to COS value of the actual packet and the other 3 bit are set to assigned QoS class value (if unspecified, level3 is QoS 0)
So, in our example:
– User traffic has Cos value “0” –> three less significant bits: 000
– Assigned class (as per EPG specified class) was “2” (which is Level-1) –> three most significant bits: 010
So, outer header DSCP is 2
When traffic enters the Cisco ACI fabric, each packet is mapped to a Cisco ACI QoS level (default is Level3). These QoS levels are then stored in the CoS field and DEI bit of the packet’s outer L2 header while the original headers are discarded.
You may notice that Inner header COS and DSCP aren’t modified or marked, but we can see later that we can tune it (if needed).
Now, to illustrate the precedence of Custom QoS over EPG QoS class, we will keep the previous configuration and add Custom QoS under the EPG:
II- Classification based on Custom QoS under EPG:
From the Dot1P classifiers section, I’m selecting Level5 and Dot1 COS value as “0”. Meaning, if we receive traffic with COS equal to “0”, it will be classified into Level5.
Verification:
verify that configuration is applied on leaf level:
leaf3# show vlan extended | grep LD-EPG-1
86 LD:LD_App:LD-EPG-1 vlan-2287 Eth1/9, Po1
leaf3# show system internal qos vlan 86
Requested specific VLAN for vlan id
----------------------------------------------------------------
Vlan
-----------------------------------------------------------------
PI id: 86 (0x3f81b3fad0, 0x3f81aceed0) flags 0x0
Type: VLAN Encap: 2287
refcnt: 1 pinst: 106957295 def_nodeid: 106957296
Default Qos Group = 3
Vlan list for this EPG (PI ids): 86
Policies
------------
DOT1P name: (null)
qos_grp: 8 from: 0 to: 0 markDscp: 64Code language: PHP (php)
From ELAM on spine for traffic sourced from the EPG, we can see that Outer L2 header COS changed to value “3” (which is associated to Level-5):
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2uc ipv4 ip ipuc ipv4uc udp ivxlan
Opcode : OPCODE_UC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 000D.0D0D.0D0D
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : yes
CoS : 3
Access Encap VLAN : 2
VN-Tag is valid : no
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 38
Don't Fragment Bit : 0x0
TTL : 32
IP Protocol Number : UDP
Destination IP : 10.0.128.68
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.31.22
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4003
VRF or BD VNID : 2752518( 0x2A0006 ) Code language: PHP (php)Target DSCP marking via EPG Custom QoS:
Note
The DSCP value of the original packet (the inner DSCP value) is normally not modified, and is not mapped to the outer VXLAN header either.
You can remark the DSCP of the original packet by configuring “Custom QoS” under the EPG or as part of the contract configuration by configuring the target CoS or the target DSCP values as part of the Custom QoS configuration.
For example, For the configuration below, once traffic classified into the EPG and it has COS (dot1p value) of 0, ACI will classify it into Level-5 and Remark the Inner DSCP value to Expedited Forwarding “46”:
Let’s confirm from the ELAM capture, we can see that COS value of Outer header is “3” (Level-5) and the Inner DSCP value is set to “46” as expect:
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2uc ipv4 ip ipuc ipv4uc udp ivxlan
Opcode : OPCODE_UC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer iEth Header
------------------------------------------------------------------------------------------------------------------------------------------------------
iEth SUP code : NONE
Packet from CPU : no
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 000D.0D0D.0D0D
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : yes
CoS : 3
Access Encap VLAN : 2
VN-Tag is valid : no
<-- ( FC tells LC via Vn-Tag about which port the packet needs to go to )
Src VIF (in from leaf/IPN) : 0
<-- ( VIF(dec) in ELTMC or LID(hex) in "show plat int hal l2 port pi" )
Dst VIF (out to leaf/IPN) : 0
<-- ( VIF(dec) in ELTMC or LID(hex) in "show plat int hal l2 port pi" )
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 38
Don't Fragment Bit : 0x0
TTL : 32
IP Protocol Number : UDP
Destination IP : 10.0.128.68
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 46
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.31.22
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4003
VRF or BD VNID : 2752518( 0x2A0006 ) Code language: PHP (php)
We can specify the target COS value in order to mark the traffic COS:
Target COS marking via EPG Custom QoS:
In the same way, as using the dot1p as classifier, we can use traffic DSCP to classify in the EPG Custom Qos:
For example: If we receive traffic with DSCP value “0”, it will be classified as “Level-2” and marked with Inner DSCP of CS3 “24”:
leaf3# show system internal qos vlan 86
Requested specific VLAN for vlan id
----------------------------------------------------------------
Vlan
-----------------------------------------------------------------
PI id: 86 (0x3f81b3fad0, 0x3f81aceed0) flags 0x0
Type: VLAN Encap: 2287
refcnt: 1 pinst: 106957295 def_nodeid: 106957296
Default Qos Group = 3
Vlan list for this EPG (PI ids): 86
Policies
------------
DOT1P name: (null)
qos_grp: 8 from: 0 to: 0 markDscp: 46
DSCP name: (null)
qos_grp: 2 from: 0 to: 0 markDscp: 24Code language: PHP (php)Verifying from the spine ELAM, I can see that DSCP classifier takes precedence over dot1p classifier (if we match both, like in our case):
- We can see that Outer header COS value is “1” (associated to Level-2)
- The inner DSCP value remarked as “24” which is DSCP “CS3”
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 000D.0D0D.0D0D
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : yes
CoS : 1
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 1
Don't Fragment Bit : 0x0
TTL : 32
IP Protocol Number : UDP
Destination IP : 10.0.128.68
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 24
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.31.22
Source IP : 192.168.65.10
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L4 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L4 Type : iVxLAN
Don't Learn Bit : 0
Src Policy Applied Bit : 1
Dst Policy Applied Bit : 1
sclass (src pcTag) : 0x4003
VRF or BD VNID : 2752518( 0x2A0006 )
Code language: PHP (php)III- Classification based on Contract:
As per documentation, the QoS specified under the Contract will take precedence, let’s test it:
Verification, we can see that outer header COS value is set to “5” (which is Level6)
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer Packet Attributes : l2uc ipv4 ip ipuc ipv4uc udp ivxlan
Opcode : OPCODE_UC
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 000D.0D0D.0D0D
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : yes
CoS : 5
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Source MAC : 000C.0C0C.0C0C
802.1Q tag is valid : no
CoS : 0
Access Encap VLAN : 0
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 54
Don't Fragment Bit : 0x0
TTL : 32
IP Protocol Number : UDP
Destination IP : 10.0.128.68
Source IP : 10.0.128.67
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
DSCP : 0
Don't Fragment Bit : 0x1
TTL : 63
IP Protocol Number : ICMP
Destination IP : 172.16.31.22
Source IP : 192.168.65.10
Code language: PHP (php)Note
You can also Mark the target DSCP (Inner DSCP) under contract.
QoS configuration can be applied under Contract and Subject level, if both applied Subject will take precedence.
For example if for subject for HTTPS traffic, the was classified into different class than the contract QoS itself. than subject QoS is applied, for other traffic not matched by that subject, contract QoS still apply:
To put all in a diagram, please refer to following (from Cisco White Paper):
Reference: This post consist of notes and lab based on following paper:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/Cisco-APIC-and-QoS.html
