Versa FlexVNF supports the complete gamut of features needed to classify, rate-limit and shape the traffic. These tools can affect the bandwidth, delay, jitter, and packet loss properties of the outgoing traffic. Different types of traffic have different requirements. For example, a video stream is more susceptible to jitter than a voice call, whereas an interactive application might need a consistent response time.
Packet Path
The following figure illustrates how a packet flows through the system.
Classifier
At ingress, during the classification stage, the traffic data is identified and the forwarding profiles are associated with it. The forwarding profile assigns a forwarding class and packet loss priority to an inbound flow or session. You can achieve classification using the following methods:
- L3/L4 Rules
- L7 Rules
L3/L4 Rules
The following rules allow classification of individual flows. Classification is done based on:
- destination zone
- destination port
- dscp
- ether-type
- ether-type-value
- ieee-802.1p
- ip-flags
- working-hours
- Source port
- destination port
- source zone
- source address
- ttl
NOTE: The keyword qos-policies refers to an L3/L4 policy.
Sample configuration:
The following configuration creates an L3/L4 rule that matches ingress flows for dscp 10 and assigns it to qos-profile, profile1.
class-of-service {
qos-policies {
policy1 {
rules {
rule1 {
match {
dscp[10];
}
set {
action allow;
qos-profile profile1;
}
}
}
}
}
} The above configuration creates an L3/L4 rule that matches ingress flows for dscp 10 and assigns it to qos-profile, profile1.
The following rules allow classification of the complete session (in-bound and outbound flows). Classification is done based on:
- application
- destination zone
- destination port
- dscp
- ip-flags
- working-hours
- source port
- destination port
- source zone
- source address
- ttl
- url-category
NOTE: The keyword app-qos-policies refers to an L7 policy.
Sample configuration:
The following configuration creates an L7 rule that matches a session in a tenant that in turn matches any of the predefined applications and assigns it to profile1.
class-of-service {
app-qos-policies {
pol1 {
rules {
rule1 {
match {
application {
predefined-application-list[GOOGLE ICMP SKYPE];
}
}
set {
qos-profile profile1;
}
}
}
}
} Per Packet Evaluation
The classification step mentioned at the beginning of the packet path happens only once per flow. This gives a performance boost as once the session is established, we do not need to evaluate each packet separately. FlexVNF uses 5 tuple to identify a stream (src/dst IP addresses, src/dst ports and protocol type). If these tuples do not change, then the stream will not be reclassified. There might be deployments where this approach might not work. For example, when there is an encryptor or tunnel formed before the traffic enters FlexVNF (host to host GRE tunnels, traffic from a peer active-active node, etc). In such cases, we can enable per packet qos-policy evaluation using the following configuration:
system {
service-options {
continuous-qos-evaluation true;
}
}NOTE: Per packet classification can only be done for qos-policy and not app-qos-policy.
CAUTION:Continuous QoS evaluation can reduce the performance by 10% to 15%.
Gotchas
L7 evaluation happens after L3/L4 rule evaluation. If a flow hits both L7 and L3/L4 rules, the L7 rule overwrites the parameters set by the L3/L4 rule. For example, if there is policer attached to both the qos-profiles, then both the policers will be hit. However, if L3/L4 marks the traffic as fc_ef and L7 marks the traffic as fc_be, L7 would be honored. The final forwarding class of the traffic will be fc_be, and would be policed twice. Note that if there are multiple L3/L4 rules hitting the same flow, then the first rule will be honored (the same applies to L7 rule).
Policer
Policer provides the ability to prevent incoming traffic from overloading the appliance and/or outgoing link. Policer is supported at the following levels:
- Tenant.
- L3/L4 qos-policy (unidirectional).
- L7 app-qos-policy (bi-directional).
Rate limiting based on both kbps (Kilo bits per second) and pps (packets per second) is supported. Based on the examples above, we can device the following configuration for qos-profile profile1:
class-of-service {
qos-profiles {
profile1 {
peak-pps-rate 20000;
forwarding-class fc_ef;
loss-priority high;
dscp-rw-enable yes;
dot1p-rw-enable no;
}
}
}The above configuration limits the flows matching profile1 to 20,000 packets per second. However, the profile also indicates that for the classified flow, the forwarding class is fc_ef, the drop loss-priority is high, and dscp-remarking is enabled. These options are covered in detail in later sections.
Remarking
Remarking provides the ability to alter the cos values in the packet header as the packet leaves the appliance. This conveys the importance of the outbound packet. FlexVNF supports DSCP and 802.1p remarking. This feature provides the ability to mark the traffic belonging to certain forwarding classes and drop loss priority with standard code points.
class-of-service {
rw-rules {
dscp {
rw-1 {
forwarding-class fc_be {
loss-priority low {
code-point be;
}
}
forwarding-class fc_ef {
loss-priority low {
code-point ef;
}
}
forwarding-class fc9 {
loss-priority low {
code-point af41;
}
loss-priority high {
code-point af11;
}
}
forwarding-class fc2 {
loss-priority low {
code-point cs1;
}
}
}
}
}
qos-profiles {
qos-prof-2 {
peak-pps-rate 20000;
forwarding-class fc_ef;
loss-priority high;
dscp-rw-enable yes;
dot1p-rw-enable no;
}
}
}LAN to WAN
- If the DSCP value of the inner-packet needs to be rewritten, then the rule needs to be configured at the tunnel interface level:
class-of-services {
interfaces {
tunnel {
dscp-rewrite-rule v4-tunel-rules;
}
}
}- If the DSCP value of the outer-packet needs to be rewritten, then the rewrite rule needs to be configured on the outgoing WAN interface:
class-of-services {
interfaces {
vni-0/0.0 {
dscp-rewrite-rule v4-wan-rules;
}
}
}- If the DSCP of the outer packet (transport header) needs to be the same as the inner packet, then set copy-from-inner in the rewrite rules.
WAN to LAN
- If the DSCP value of the inner-packet needs to be rewritten, then the rewrite rule needs to be configured on the outgoing LAN interface:
class-of-services {
interfaces {
vni-0/1.0 {
dscp-rewrite-rule v4-lan-rules;
}
}
}- If the DSCP of the inner packet needs to be the same as the outer packet, then set copy-from-outer in the rewrite rules.
NOTE: Rewrite rules cannot be configured on the physical port. It is a property of the logical port.
Hierarchical QoS (shaper)
The hierarchical scheduler block when present is used on the egress to shape the traffic just before the transmission stage. The following block diagram displays the arrangement of hierarchical shapers within Versa FlexVNF:
Schedulers exists at the:
- port level,
- pipe level, and
- traffic class level
All ports have equal priority. All pipes within the ports have equal priority. These pipes represent either VLAN interfaces or a dynamically created IPSec path. Traffic classes within these pipes are handled in strict priority order. Network-control (tc0) > Expedited forwarding (tc1) > Assured Forwarding (tc2) > Best Effort (tc3). Each traffic class contains four queues (only one queue is enabled by default). Each queue within these traffic classes is scheduled using WRR.
This brings us to the concept of forwarding classes. Versa FlexVNF supports 16 forwarding classes, each mapping to a combination for traffic-class and one of the queues within it.
admin@kirk2-vcsn-cli(config)% run show orgs org-services versa class-of-service mapping fc-queue Forwarding-class Queue Mapping ------------------------------ Forwarding Traffic Queue Class Class Number ----------- --------- -------- fc_nc 0 0 fc1 0 1 fc2 0 2 fc3 0 3 fc_ef 1 0 fc5 1 1 fc6 1 2 fc7 1 3 fc_af 2 0 fc9 2 1 fc10 2 2 fc11 2 3 fc_be 3 0 fc13 3 1 fc14 3 2 fc15 3 3
The traffic class scheduling works under the following modes:
- Non-token-bucket mode: In this mode, true token buckets are not used at the traffic class level for shaping. This mode supports only transmit-rate. The guaranteed rate configuration is ignored. This was the default mode in 15.2R4. Please make sure that this mode is not enabled because this is no longer supported.
- Token-bucket-mode (default): In this mode, true token bucket is used at the traffic class level. Both CIR (committed information rate) or guaranteed-rate configuration and traffic rate configuration are honored. This is the default mode in 16.1R1+ version. This can be enabled using the following configuration:
service-options {
shaping {
tc-token-bucket true;
}
} Let's now look at a sample configuration for shaping at various levels of the hierarchy:
org-services org1 {
class-of-service {
interfaces {
vni-0/1 {
shaping-rate {
rate 100000; //in kbps
}
scheduler-map smap; //Allocates 100Mbps of bandwidth to vni-0/1 and associates it with schedule map smap.
}
vni-0/1.10 {
shaping-rate {
rate 5000; //in kbps
}
scheduler-map smap;
}
}
scheduler-maps { //The scheduler map associates schedulers with various traffic classes.
smap {
traffic-class tc0 {
scheduler sc_nc;
}
traffic-class tc1 {
scheduler sc_ef;
}
traffic-class tc2 {
scheduler sc_af;
}
traffic-class tc3 {
scheduler sc_be;
}
}
}
schedulers {
sc_be { //sc_be is associated with tc3 (best-effort). This configuration assigns a guaranteed bandwidth of 20Mbps.
guaranteed-rate {
percentage 20;
}
}
sc_af {
transmit-rate {
percentage 70;
}
guaranteed-rate {
percentage 25;
}
queue 0 {
weight 2; //This queue configuration is used to specify the WRR weights. In this case, the 4 queues are scheduled in the ratio 2:2:1:1. These correspond to fc_af, fc9, fc10 & fc11 respectively.
}
queue 1 {
weight 2;
}
queue 2 {
weight 1;
}
}
sc_ef { //If we only configure the transmit-rate, then this is an indication that both guaranteed-rate and transmit rate are set to the same value. In this case it is set to 60Mbps.
transmit-rate {
percentage 60;
}
}
sc_nc { //Similarly, sc_nc is guaranteed at-least 5 Mbps, but it cannot send more than 10Mbps.
transmit-rate {
percentage 10;
}
guaranteed-rate {
percentage 5;
}
}
}CAUTION: Guaranteed-rate configuration is ignored if tc-token-bucket is disabled in the system configuration.
Network level configuration
Versa added the support for configuring QoS on networks in 16.1R2. In Versa FlexVNF, network is a group of logical interfaces. Adding configuration on a network simply applies the same configuration to each interface within the network. These configurations are still dependent on the constraints of their member interfaces.
Traffic Distribution and Rate Oversubscription
Within a particular pipe, the traffic distribution is governed by the guaranteed-rate or transmit rate and weights allotted per traffic class. However, if the traffic is distributed across pipes on a port, the behavior varies based on the configuration and the traffic pattern. Let’s consider the following cases:
org-services org1 {
class-of-services {
interfaces {
vni-0/1 {
shaping-rate {
rate 60000;
}
}
vni-0/1.10 {
shaping-rate {
rate 10000;
}
}
vni-0/1.20 {
shaping-rate {
rate 20000;
}
}
vni-0/1.30 {
shaping-rate {
rate 30000;
}
}
}
}Case 1: Non-oversubscription, unshared bandwidth
org-services org1 {
class-of-services {
interfaces {
vni-0/1 {
shaping-rate {
rate 60000;
}
}
vni-0/1.10 {
shaping-rate {
rate 10000;
}
}
vni-0/1.20 {
shaping-rate {
rate 20000;
}
}
vni-0/1.30 {
shaping-rate {
rate 30000;
}
}
}
}In this case, units 10, 20 & 30 are allocated a bandwidth of 10 mbps, 20 mbps & 30 mbps respectively. Collectively, they cannot exceed the total bandwidth of 60 mbps on the port. If these pipes (i.e. units) are overloaded, they are not going to eat into the share of other bandwidths. However, if one of these pipes are not using its share of bandwidth, the other pipes are not allowed to use it up. For example, if vni-0/1.30 stopped sending traffic, unit 10 and 20 can still send only 10 mbps and 20 mbps respectively, even though the port can send a maximum of 60 mbps. This mode of operation is the default and preferred mode.
Case 2: Oversubscription
org-services org1 {
class-of-services {
interfaces {
vni-0/1 {
shaping-rate {
rate 60000;
}
}
vni-0/1.10 {
shaping-rate {
rate 60000;
}
}
vni-0/1.20 {
shaping-rate {
rate 60000;
}
}
vni-0/1.30 {
shaping-rate {
rate 60000;
}
}
}
}In this case, units 10, 20 & 30 can each send a maximum of 60 mbps. Collectively, they cannot send more than 60 mbps configured on port vni-0/1. If all these three pipes are overloaded, and traffic is relatively interleaved, we can see an equal distribution of traffic over these three pipes. However, under unfavorable traffic conditions, significant deviations can be observed from an equitable distribution. This problem cannot be solved deterministically at any given point, we cannot predict the traffic that might egress out of a sibling pipe.
Case 3: Non-oversubscription, Shared bandwidth.
NOTE:This feature is available only in versions 20.1+.
org-services org1 {
class-of-services {
interfaces {
vni-0/1 {
shaping-rate {
rate 60000;
bandwidth-sharing true;
}
}
vni-0/1.10 {
shaping-rate {
rate 10000;
}
}
vni-0/1.20 {
shaping-rate {
rate 20000;
}
}
vni-0/1.30 {
shaping-rate {
rate 30000;
}
}
}
}With the introduction of this knob, unused bandwidth can be shared between different pipes.The unused bandwidth is redistributed amongst the pipes that need them in ratio of their configured bandwidths.
In the above example, if unit 30 stopped sending traffic, 30 mbps worth of bandwidth will be redistributed amongst unit 10 and unit 20. Since the ratio of distribution works out to be 1:2, unit 10 gets an extra 10 mbps and unit 20 gets an extra 20 mbps. If unit 20 were to now send only15 mbps of traffic (of the now allotted 40 mbps), unit 10 would be able to send an extra 25 mbps of traffic (for a total of 10+10+25 = 45 mbps).
Dropper
Congestion occurs when there are more packets to be sent out than the interface can handle.Tail drops occur when packet needs to be added to a queue that is already full. FlexVNF supports both WRED (weighted Random Early Detection) and tail-drops. WRED algorithm keeps track of the average queue length and drops packets randomly before the queue is filled.The Versa FlexVNF allows configuration of two drop loss priorities (low & high) and allows different configuration for each of the drop profile.
schedulers {
sc_ef {
drop-profile-map {
loss-priority low {
drop-profile low_drop;
}
loss-priority high {
drop-profile high_drop;
}
}
}
drop-profiles {
low_drop {
minimum 64;
maximum 128; //Starts dropping packets when the average queue length reaches 64 & gradually increase the drop percentage until it hits 128. Do tail drops then.
}
high_drop {
minimum 48;
maximum 96;
}
} CAUTION: Configure the minimum and maximum to the same value to configure tail drop at that value rather than WRED.
The default min/max values for a low loss-priority traffic is 48/64 and the defaults for high loss-priority is 32/64. These numbers are chosen despite a maximum queue size of 256. This is done to keep the latency and jitter at an optimal level for the traffic in majority of our deployments.
Adaptive Shaping
In an SD-WAN deployment, this feature allows a device (branch or hub) to enforce a dynamic egress shaping rate on any other device that is sending it traffic.
On the sender's side, only the traffic destined to the device with adaptive shaping configured is shaped, and this does not affect other traffic.
On a particular WAN interface, adaptive shaping can be enabled using the following configuration:
org org1 {
sd-wan {
site {
wan-interfaces {
vni-0/0.0 {
circuit-name wan-link1;
local-id 1;
shaping-rate {
input-rate { //This configuration advertises a max shaping rate of 500Mbps to be configured on the ipsec tunnel of the remote branches. The minimum-input-rate specifies
kbps 500000; // the minimum rate to which a remote branch may shape its traffic towards this branch.
}
minimum-input-rate {
kbps 1000;
}
}
}
}
}
} CAUTION: If input bandwidth is configured on the interface, the input-rate and minimum-input-rate can be specified as a percentage of that bandwidth.
Since there can be hundreds or thousands of devices in a network and different sets of devices are talking at any given point at various degrees, you must dynamically scale the advertised bandwidth. Refer to the following default configuration for the adaptive shaping subsystem.
system {
sd-wan {
adaptive-shaping {
high-threshold 50;
low-threshold 20;
percent-change 10;
poll-interval 5;
dampening-count 1;
enable true;
}
}
} Adaptive shaping algorithm:
This section explains the above configuration:
- Initialize advertised rate (AR) to the input rate configured on the WAN interface.
- For every poll-interval second, check the link utilization (LU) of the WAN interface on which adaptive shaping is configured.
- If LU > high-threshold for dampening-count times in a row, decrease the AR by percent-change and advertise to peers.
- If LU < low-threshold for dampening-count times in a row, increase the AR by percent-change and advertise to peers.
- AR is always capped by minimum-input-rate and input-rate (configured on the WAN Interface).
- If enable is false, adaptive shaping on the ingress interface is disabled.
- If class-of-service is not configured on the remote branches on the WAN interfaces, then it will not respond to the shaping requests of B1.
Example:
Let’s follow the example in the above figure:
- Branch1 and Branch2 send traffic to the Hub at 10 Mbps each.
- A total of 20 Mbps of traffic can overwhelm the Hub and if the hub has only paid for 10 Mbps, 10 Mbps will be dropped at the provider edge router.
- The Hub, on seeing it’s ingress traffic exceeding a threshold, can ask the other branches to shape the traffic down (to which the branches can choose to comply).
Performance tuning
NOTE: This section is applicable for releases after 16.1R1.S6, 16.1R2.S1 & 20.1.0+.
Please refer to FlexVNF-flow-anchor-white-paper for an in-depth introduction to this feature. By default, all network control traffic is anchored to Core 0. EF and AF traffic can be anchored to any core except Core 0. BE traffic can be anchored to any core, including Core 0. This is done so that system traffic and the traffic that might cause latency (such as doing AV), does not affect the latency sensitive traffic such as Voice. For non-standard test environment, some of the default parameters might need tuning. This tuning might happen because the uneven session distribution might look like a performance issue. You can verify if this is the case using the following command:
vsm-vcsn0> show vsf per-thread nfp stats summary Thr-Id Sess-Active Sess-Created Sess-Closed -------- ------------- ------------------ ------------------ 0 10 564964 564954 1 42 586185 586143
Here, we can see that the sessions are not uniformly distributed between the worker threads. This would be a problem only when doing performance testing. We can verify core anchoring using the following command:
vsm-vcsn0> show vsm anchor core map +--------+--------+---------+--------+---------++----------+----------+ |H-Index | NC Core | EF Core | AF Core | BE Core | +--------+------- +---------+--------+---------++----------+----------+ | 0 | 0 | 1 | 1 | 0 | | 1 | 0 | 1 | 1 | 1 |
This command shows that NC is anchored to Core 0, EF and AF traffic can only go to Core 1, but BE traffic can go on both the cores.
Frequently Asked Questions (FAQs)
- I have shaped the egress port at 50 mbps, which happens to be a WAN interface. However, I am getting traffic rates much below 50 mbps. The traffic rate is worse when I use smaller packet sizes. I am measuring this rate on the remote host/traffic generator.
Answer: The traffic egressing the WAN interface is encapsulated with either VXLAN or ESP headers. The additional headers are responsible for a lower perceived packet rate. However, if you check the CLI on the appliance, you would find that the packets are being shaped at the appropriate rate.
A configuration knob is provided that can be turned on to discount these packet headers for branch to branch traffic.service-options { qos-frame-overhead { auto-adjust true; } } - I have shaped my egress port at 50 mbps, and I am using SCP/iPerf to test shaping of the TCP stream. The traffic output is very bursty and stalls at times. I do not see any issues with UDP traffic though.
Answer: By default, WRED is configured on all the queues. This has empirically proven to give much better amortized performance for a large number of TCP sessions. However, with just one TCP stream running, WRED will drop packets from the middle of the flow. The TCP behavior is then governed by the sender and receiver’s network stack. The receiver can choose not to acknowledge packets until it receives the re-transmitted packet (that was dropped earlier). This can lead to timeouts, re-transmission, and further congestion on the sender's network.A viable solution would be to configure tail drop instead of WRED drops. This works better for a single flow.
drop-profiles { tail_drop { minimum 64; maximum 64; } } - I have a profile that maps a UDP stream and a TCP stream to a forwarding class. The TCP stream adapts itself to the lack of bandwidth, but the UDP can keep on pumping data at a constant rate, much higher than the configured shaping rate. Is there a way I can avoid starvation of TCP flows?
Answer: Yes. If you have TCP and UDP traffic going to the same forwarding class, you can separate out the drop profile. You will need to configure separate QoS policies (based on the L4 protocol along with the other conditions). You can assign a higher drop priority to UDP so that TCP traffic is not unfairly penalized.
You can tune the minimum/maximum variables of the drop profiles associated with TCP and UDP drop priorities until you get the desired effect. - What configuration do you suggest for video or voice traffic?
Answer: Both video and voice traffic need low jitter, low latency, and low drops.
- Jitter increases if there is congestion. We need to assign it to a higher priority forwarding class (fc_ef) and allocate the required amount of bandwidth, so that it does not compete with other traffic.
- Jitter also increases (resulting in choppy video/voice) if packets are dropped from the middle of the flow. This should be avoided by configuring tail drop rather than the default WRED.
- Latency increases if a packet spends more time in the queue. This should be avoided by making the tail drop queue length smaller.
- The above three can be easily be accomplished by simply using the policer instead of a shaper to restrict the traffic.
- show interface statistics shows a higher egress rate than configured on the shaper. Why is that?
Answer: As mentioned earlier, shaper comes after module TX. Please use show class-of-services interfaces set of commands, or show interface port statistics to verify the actual egress rate on the interface.
- I have configured multiple QoS profiles and policies, but I have not configured any shaping on the interface. I do not see EF traffic being prioritized over other traffic. Why is that?
Answer: The various schedulers and their queues are configured only when the shaper is enabled on the interface. By default, the shaping rate is set to the queue's speed. As long as the shaper is configured, the traffic will be prioritized.
- I have configured policer/shaper, but my observed rate is much smaller than the configured rate. What can be happening here?
Answer: There are a few things that can be happening here:
- Traffic rate is higher than the rate supported by the platform:
- If the traffic is not well distributed, or sent at a higher rate, it can overwhelm the traffic processing threads. To figure out if this is the issue, run the following command:
admin@Sridhar-Branch-cli> show debug vsm statistics dropped THRM ERROR STATISTICS ~~~~~~~~~~~~~~~~~~~~~~ Poller PID : 23157 # Drop Packets TX : 3 # Drop Packets RX : 2 vsm-vcsn0> [ok][2018-01-22 11:41:07]
If any of the “Drop Packets TX/RX” goes up, then you are overloading the system. If that is not the case, then please contact the Performance team to debug this further.
- If the traffic is not well distributed, or sent at a higher rate, it can overwhelm the traffic processing threads. To figure out if this is the issue, run the following command:
- Qos-policy or app-qos-policy match conditions are generic:
Make sure that multiple streams are not hitting the same rule and same profile. If you have a bidirectional traffic, both the flows can hit the same rule if based on generic policies. Test this with a single stream to verify if this is the case. Testing should preferably be done with UDP. If you are testing with TCP, make sure that the acks are not hitting the same rule.
- Traffic is bursty:
Traffic burstiness is often a function of end hosts and the application sending traffic. Save the pcap of the received packets on the vnf and generate IO graphs to see the traffic pattern. If you see patterns similar to the following, then the traffic is being sent in bursts:
The workaround here is to increase the burst size on the policer/shaper. Another option on the shaper is to increase the queue size (using drop profiles), so that burst size remains the same, but the packet latency increases. Please choose the approach that suits you best.
- Jumbo frames being sent:
Jumbo frames reduce the performance of FlexVNF because work needs to be done for reassembly and re-fragmentation of these packets. This could cause the packets to drop at the platform level. Please verify if this is the case by looking at the dropped statistics (refer to the first point).
If the drops are in the policer, you might be hitting an older accounting bug that was fixed in 16.1R1S4.
- Traffic rate is higher than the rate supported by the platform: