Problem
SD-WAN is an overlay technology where there is no control on the rate packets that arrive at the receiving branch. In a typical MPLS VPN topologies, the egress interface of the Provider Edge (PE) router shapes the traffic at a particular rate agreed with the customer. However, in SD-WAN the underlay transport network is decoupled for the overlay service. The result on the receiving branch WAN interface gets congested and drops traffic.
Solution
The most elegant solution to address this problem is to have the sending branches reduce their sending rate only to the branch that reports incoming congestion. If those sending sides communicate with the other branches that are not congested, their uplink speed should be restricted. The Versa SD-WAN solution addresses this problem by leveraging the SD-WAN control plane where the receiving branch is able to communicate a throttle message back to the sending branch to adjust its shaper only for sending to the congesting branch.
When adaptive shaping is on, the receiving branch communicates its Input Rate over the SD-WAN control plane to the sending branches. These sending branches tune their PIPE Shaper to the receiving branch to the same input rate. The receiving branch checks the receive utilization at every “poll-interval”. If there is a high threshold, it communicates with the senders to adjust the shaper towards this branch with the rate configured in “percent-change”. This process continues as long as the utilization is above the higher threshold. When the traffic drops below the lower threshold, the receiver communicates with the senders to increase their shapers as per the “percent-change” value. To avoid flapping, a dampening-count can be configured. The link utilization status must be a number of concurrent polling-intervals exceeded before the receiver signals the “present-change”. This number is configured with the “dampening-count”.
Configuring Adaptive Shaping Using GUI
Follow these steps to configure the adaptive shaping using GUI:
- Select Configuration > Networking
> Class of Service > Associate Interface/Network to configure the CoS configuration for an interface. At the minimum, the WAN ports of the system are required to have a QoS shaper configured at the port level. In the below example, all the egress traffic of this branch is shaped at 10 Mbps. This configuration example is for the hub, spokes required to be configured.
- Select Configuration > Services
> SDWAN > System > Adaptive Shaping and click
to enable Adaptive Shaping.
- Refer Adding a Site for Provider Tenant (Versa FlexVNF Advanced SD-WAN Configuration Guide) to configure the Input Rate. This is the receiving rate that is advertised to all other branches. Since this is the maximum rate at which this branch can receive traffic, the other branches default the egress shaper to only that branch to the defined input rate.
- Refer Adding Ethernet (WAN) Interfaces (Versa FlexVNF Advanced SD-WAN Configuration Guide) to configure the maximum bandwidth available on the interface.
Configuring Adaptive Shaping Using CLI
Follow these steps to configure the adaptive shaping using CLI:
- Run these commands to configure the input rate and burst size of a sending interface.
set orgs org-services customer1 class-of-service interfaces vni-0/0 shaping-rate rate 10000 set orgs org-services customer1 class-of-service interfaces vni-0/0 shaping-rate burst-size 312500
- Run these commands to enable the adaptive shaping on the receiving interface.
set system sd-wan adaptive-shaping high-threshold 85 set system sd-wan adaptive-shaping low-threshold 10 set system sd-wan adaptive-shaping percent-change 10 set system sd-wan adaptive-shaping poll-interval 10 set system sd-wan adaptive-shaping dampening-count 1 set system sd-wan adaptive-shaping enable true
- Run these commands on the receiving interface to configure the shaping rates i.e. Input Rate and Minimum Input Rate respectively.
set orgs org customer1 sd-wan site wan-interfaces vni-0/0.0 shaping-rate input-rate kbps 10000 set orgs org customer1 sd-wan site wan-interfaces vni-0/0.0 shaping-rate minimum-input-rate kbps 5000
- Run these commands to configure the bandwidth allocation on an interface.
set interfaces vni-0/0 bandwidth uplink 10000 set interfaces vni-0/0 bandwidth downlink 10000
Adaptive Shaping Scaling Guidelines
Adaptive Shaping requires scaling guidelines. Versa FlexVNF has a total of 4096 pipes per interface (no concept of tenant) and two pipes are created for each adaptive shaping instance. For each interface, Versa FlexVNF requires two separate pipes per adaptive shaper for each branch and tenant. One pipe is used for the non-encrypted path and the other pipe is used for the encrypted path (these two paths are always present in a default Versa SD-WAN deployment).
This means you can run through the 4096 limit by:
- Having 2097 branches with one WAN interface and one tenant.
- Having one branch with a single interface, but 2097 tenants.
It must be noted that for Adaptive Shaping in Versa FlexVNF, these pipes are only relevant to the sending branch since this is where the shaping is done. The SD-WAN control plane “activates” instantiation of the pipe on the sending branch when the receiving branch enables the Input Rate.
Excise caution while configuring Versa FlexVNF with many branches. A typical SD-WAN is made of sites that function more as centralized hub sites and others that function more as decentralized remote sites, Versa recommends enabling such Input Rate only for the centralized hub sites.
Monitoring Adaptive Shaping
Follow these steps to monitor and verify the adaptive shaping on the receiving interface:
- Run the show class-of-services interfaces extensive vni-0/0 CLI command to see the initial shaping rate of the PIPE towards the receiving branch (WAN-101) where 101 is the site ID of the receiving side. The Rate is set to 10000 kbps, which is the input rate.
admin@VM1-cli> show class-of-services interfaces extensive vni-0/0 Interface: vni-0/0 Configuration: Burst Size : 312500 bytes Rate : 10000 kbps Network-Control : 10000-10000 kbps Expedited-Forwarding : 10000-10000 kbps Assured-Forwarding : 10000-10000 kbps Best-Effort : 10000-10000 kbps Traffic Stats: TX Packets : 72 TX PPS : 6 TX Packets Dropped : 0 TX Bytes : 13130 TX bps : 9088 TX Bytes Dropped : 0 Port Stats : Traffic Class TX Pkts TX Dropped TX Bytes Bytes Dropped tc0 network-control 63 0 11930 0 tc1 expedited-fwd 0 0 0 0 tc2 assured-fwd 0 0 0 0 tc3 best-effort 9 0 1200 0 Pipe Stat: Pipe ID : 1 Users : [ WAN-101:17:2:clear ] Type : SDWAN Configuration : Rate : 10000 kbps Traffice Stats: Traffic Queues TX Pkts TX Dropped TX Bytes Bytes Dropped Queue len tc0 network-control: q0 0 0 0 0 0 q1 0 0 0 0 0 q2 0 0 0 0 0 q3 0 0 0 0 0 tc1 expedited-fwd: q0 0 0 0 0 0 q1 0 0 0 0 0 q2 0 0 0 0 0 q3 0 0 0 0 0 tc2 assured-fwd: q0 0 0 0 0 0 q1 0 0 0 0 0 q2 0 0 0 0 0 q3 0 0 0 0 0 tc3 best-effort: q0 0 0 0 0 0 q1 0 0 0 0 0 q2 0 0 0 0 0 q3 0 0 0 0 0
- When congestion occurs you will witness the receiving rate on the receiving branch is higher than the configured rate. This is because adaptive shaping needs to be tuned to the current conditions (polling interval vs percent-change). However, after some time the input rate is stabilized as per the configured input rate.
Run the show interfaces statistics CLI command to check the initial input rate.
admin@Hub1-cli> show interfaces statistics TENANT HOST RX RX RX TX TX TX NAME ID VRF INF PACKETS PPS RX BYTES ERRORS RX BPS PACKETS PPS TX BYTES ERRORS TX BPS ---------------------------------------------------------------------------------------------------------------------------------------- eth-0/0 0 global eth0 8491 0 797311 0 0 2341 0 677849 0 0 ptvi1 3 provider-Control-VR n/a 19 1 1308 0 976 20 1 2235 0 1600 ptvi2 2 cusomter1-Control-VR n/a 20 1 1925 0 984 20 1 1862 0 968 tvi-0/2 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/2.0 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/3 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/3.0 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/4 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/4.0 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/5 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/5.0 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 vni-0/0 3 mpls-1-Transport-VR eth1 26004 1554 37363995 0 17874056 13592 821 2004078 0 970496 vni-0/0.0 3 mpls-1-Transport-VR eth1 25999 1554 37363695 0 17873928 13592 821 2004078 0 970496 vni-0/1 3 internet-1-Transport-VR eth2 0 0 0 0 0 0 0 0 0 0 vni-0/1.0 3 internet-1-Transport-VR eth2 0 0 0 0 0 0 0 0 0 0 vni-0/2 2 cusomter1-LAN-VR eth3 13538 818 905104 0 438744 25894 1548 34956900 0 16724576 vni-0/2.0 2 cusomter1-LAN-VR eth3 13538 818 905104 0 438744 25894 1548 34956900 0 16724576 vni-0/3 0 global eth4 0 0 0 0 0 0 0 0 0 0 [ok][2017-11-30 08:13:19] admin@Hub1-cli>
Run the show interfaces statistics CLI command to check the final input rate.
admin@Hub1-cli> show interfaces statistics TENANT HOST RX RX RX TX TX TX NAME ID VRF INF PACKETS PPS RX BYTES ERRORS RX BPS PACKETS PPS TX BYTES ERRORS TX BPS -------------------------------------------------------------------------------------------------------------------------------------- eth-0/0 0 global eth0 8574 0 804505 0 0 2352 0 681507 0 0 ptvi1 3 provider-Control-VR n/a 322 2 29753 0 1032 368 2 115089 0 1064 ptvi2 2 cusomter1-Control-VR n/a 205 0 19488 0 248 211 0 18362 0 248 tvi-0/2 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/2.0 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/3 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/3.0 3 provider-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/4 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/4.0 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/5 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 tvi-0/5.0 2 cusomter1-Control-VR n/a 0 0 0 0 0 0 0 0 0 0 vni-0/0 3 mpls-1-Transport-VR eth1 219826 832 315397440 0 9549792 116701 447 17335126 0 529784 vni-0/0.0 3 mpls-1-Transport-VR eth1 219761 832 315393794 0 9549664 116701 447 17335126 0 529784 vni-0/1 3 internet-1-Transport-VR eth2 0 0 0 0 0 0 0 0 0 0 vni-0/1.0 3 internet-1-Transport-VR eth2 0 0 0 0 0 0 0 0 0 0 vni-0/2 2 cusomter1-LAN-VR eth3 115753 442 7765790 0 238520 218566 827 295056252 0 8933768 vni-0/2.0 2 cusomter1-LAN-VR eth3 115747 442 7765430 0 238504 218566 827 295056252 0 8933768 vni-0/3 0 global eth4 0 0 0 0 0 0 0 0 0 0 [ok][2017-11-30 08:16:22] admin@Hub1-cli>
- When the congestion extincts the minimum input rate of 5000 Kbps condition is flipped to an input rate of 10000 Kbps. Run the show class-of- services extensive vni-0/0 CLI command to see the stabilized input rate.
admin@VM1-cli> show class-of-services interfaces extensive vni-0/0 Interface: vni-0/0 Configuration: Burst Size : 312500 bytes Rate : 10000 kbps Network-Control : 10000-10000 kbps Expedited-Forwarding : 10000-10000 kbps Assured-Forwarding : 10000-10000 kbps Best-Effort : 10000-10000 kbps Traffic Stats: TX Packets : 168967 TX PPS : 432 TX Packets Dropped : 189 TX Bytes : 241016444 TX bps : 4930312 TX Bytes Dropped : 227438 Port Stats : Traffic Class TX Pkts TX Dropped TX Bytes Bytes Dropped tc0 network-control 1916 36 519920 6812 tc1 expedited-fwd 0 0 0 0 tc2 assured-fwd 0 0 0 0 tc3 best-effort 167051 153 240496524 220626 Pipe Stat: Pipe ID : 1 Users : [ WAN-101:17:2:clear ] Type : SDWAN Configuration : Rate : 5000 kbps Traffice Stats: