Overview

Forward Error Correction (FEC) is a packet loss recovery mechanism built into Versa's SD-WAN engine. It provides error correction for data transmission over unreliable or lossy WAN links.

When a WAN link drops packets, FEC allows the receiving VOS device to mathematically reconstruct the lost packet without requiring retransmission from the sender. Packet loss is therefore hidden from the application, this is useful or UDP which does not have a recovery mechanism and in case of TCP it would be useful if the rtt is high and if we want to hide application loss from the actual application.

 

How FEC Works

The sender assigns a sequence number to every packet in a session and groups packets into windows of size N.

For each window on the Sender:

  1. The sender XORs all packet payloads together, byte-by-byte.
  2. The resulting XOR value becomes a parity packet.
  3. The parity packet is transmitted after the Nth data packet.

On the receiver:

  1. Received data packets are XORed together to build a running parity calculation.
  2. When the parity packet arrives, the receiver determines whether any packet in the window is missing.
  3. If exactly one packet is missing, the receiver XORs the received parity packet with the accumulated parity from the received packets.
  4. The missing packet is reconstructed and reinserted into the packet stream.

 

FEC can recover only when exactly one packet is lost within a window. If two or more packets are lost within the same window, recovery fails. Also, on the receiver side FEC kicks in after the first parity packet is seen, since the fec data is sent in the parity packet only after the first window will the receiver know about the window details and the generate the next 3 Windows(hard coded) for every session. This means if you have “Number of FEC” (explained below) configured as 4 and you send only 4 ping packets the FEC recovery will not be triggered.

 

Configuration

FEC is configured within an SD-WAN traffic-steering forwarding profile and applied through an SD-WAN traffic-steering policy. It is not recommended to enable FEC for all traffic because there are limits to the number of FEC sessions the platform can support.

 

Although receiver-side processing is always available, FEC becomes operational only when the sender option is enabled.

 

 

FEC Parameters

Number of Packets per FEC

Defines the FEC window size i.e how many data packets are grouped before one parity packet is generated. A value of 4 means: for every 4 data packets, 1 parity packet is sent. Smaller values mean more frequent parity packets (lower overhead savings, but faster recovery and less hold time). Larger values mean fewer parity packets (less overhead, but recovery only works if loss is spread thin across windows). This directly controls how often the sender generates and sends a parity packet.

 

FEC Packet (Alternate Circuit)

Determines the path used to send parity packets.

When configured as Alternate Circuit, parity packets are transmitted on a different WAN link than the data packets.This prevents parity packets from being affected by the same congestion or packet loss event impacting the primary data path.

 

Duplicate FEC Packet

When enabled, the parity packet itself is duplicated and sent on a second path as well, so two copies of the parity travel via different routes. This is extra protection for cases where even the parity packet might get dropped — but it adds more bandwidth overhead.

 

Parity Packet Size

The parity packet size is always equal to the largest packet within the FEC window. Smaller packets are padded with zeros before the XOR operation, if there is a mismatch in the lengths then packets larger than the sender value are truncated before XOR. If that packet is lost, then at the receiver the regenerated packet will be corrupted beyond the truncation point. Default of 1400 covers standard Ethernet MTU safely.

 

Stop When / Circuit Utilization

When enabled, parity generation stops once circuit utilization exceeds the configured threshold.This prevents FEC overhead from contributing additional congestion on heavily utilized links.

Recovery (Receiver)

Enables packet reconstruction on the receiving VOS node.

When disabled:

  • Parity packets are received.
  • No recovery attempt is performed.

Default: Enabled

 

Preserve Order (Receiver)

Ensures packets are delivered to applications in the original sequence. When enabled, out-of-order packets are temporarily buffered while the receiver waits for parity-based recovery.

Default: Enabled

 

Important Operational Considerations

 

When FEC is enabled on a VOS node performing packet inspection, the receiving VOS node buffers packets to guarantee in-order delivery whenever packet loss is detected on a link. Under the default FEC configuration, one parity packet is generated for every four data packets, and a parity packet can recover at most one lost packet within its corresponding group. While waiting for parity-based recovery, up to three subsequent packets may be held in the receive buffer. Each buffered packet consumes one packet buffer from the system's statically allocated packet buffer pool.

The packet buffer pool is shared across all gateway functions, including SLA monitoring traffic, control-plane messaging, IPsec processing, NAT processing, and other gateway services. If FEC buffer utilization exceeds 80% of the available packet buffer pool, critical functions such as SLA monitoring may fail to allocate packet buffers. This can result in SLA failures, path flapping, traffic rerouting, and network instability, potentially causing a greater operational impact than the original packet loss event. For this reason, FEC buffer utilization should always remain below 80% of the total packet buffer pool, preserving the remaining 20% for monitoring and control-plane traffic.

 

Maximum Supported FEC Sessions

Packet Buffer Pool Size

Maximum FEC Sessions Supported

64K

42K

128K

84K

256K

168K

512K

336K

1M

672K

 

Logs and Troubleshooting


For FEC-related issues, collect the following:

  • Session extensive output from CLI and VSMD
  • Bidirectional pkttrace
  • VMLH and FEC debug logs

 

Commands

Bidirectional Pkttrace

request debug session filter-create source-prefix 10.1.1.11/32 destination-prefix 10.1.1.5/32 timeout 1200 filter-name t1
 request debug session filter-create destination-prefix 10.1.1.11/32 source-prefix 10.1.1.5/32 timeout 1200 filter-name t2

Enable Debug Logging

admin@branch1-cli(config)% set debug fec all-flags level all send to file name /var/tmp/fec.log
 admin@branch1-cli(config)% set debug vmlh all-flags level all send to file name /var/tmp/vmlh.log

CLI — Session Extensive

show orgs org <tenant> session extensive | select <src_ip> | select <dst_ip> | select match-all

VSMD

vsh connect vsmd
show vsf session all brief | grep <ip> (From this output above, pick the handle in the first column and run:)

show fec session stats handle <handle>

 

Example

 

The following examples use a topology where ping packets are sent from 10.1.1.11 (branch2) to 10.1.1.5 (branch1). The number of packets per FEC is configured as 4, so 4 data packets are sent over MPLS and 1 parity packet is sent over INET (Alternate Circuit). FEC sending is enabled only on branch2. Note: Ping is used here to showcase logs. Since ping packets are sent at higher intervals, the delay in recovery will be higher than with real application traffic.

 

 

Scenario 1: No Packet Loss

 

Session Extensive — Sender (branch2)

admin@branch2-cli> show orgs org Tenant1 sessions extensive | select protocol 1 | select source-ip 10.1.1.11 | match-all
sessions extensive 0 2 115185
source-ip 10.1.1.11
destination-ip 10.1.1.5
protocol 1
multi-link-mode FEC
forward-fec-parity-pkt-tx 1 -- Since the window is 4, only 1 parity packet will be sent
forward-multi-link-total-rx 4 -- 4 packets received; receiver does not have sender option enabled so no parity received from receiver side
 forward-multi-link-total-tx 5    -- 5 packets forwarded: 4 data packets + 1 parity packet

 

 

Session Extensive — Receiver (branch1)

admin@branch1-cli> show orgs org Tenant1 sessions extensive | select protocol 1 | select source-ip 10.1.1.11
sessions extensive 0 2 124810
source-ip 10.1.1.11
destination-ip 10.1.1.5
protocol 1
multi-link-mode FEC
reverse-fec-parity-pkt-rx 1 -- Only 1 parity packet received in this direction
reverse-multi-link-total-rx 5 -- 5 packets received: 4 data packets + 1 parity packet
 reverse-multi-link-total-tx 4    -- 4 ICMP reply packets sent back

 

 

VSMD

No data is shown since FEC per packet is 4 and only 4 packets were sent (FEC recovery is not triggered):

vsm-vcsn0> show fec session stats handle 0x2020bcd
 FEC session not found for 0x2020bcd

 

 

Debug Logs

 

Sender VMLH debug – 4 data packets + 1 Parity Packet , notice the Sending VMLH and Rcvd VMLH

 

2026-06-13 16:49:31.807 DEBUG [0x102] vs_nfp_sess_vmlh_enable:854 Enabling VMLH info for sess 0x7f56ca64c602, dir fwd, in_ifp vni-0/1.0 out_ifp dtvi-0/90

The lines below are important: the first triggers the start of the FEC session by sending Sess_Start with Seq=1 and the site-id:access circuit 1022:11. The received VMLH with Sess_Rebase confirms that the remote side has synced this session:

2026-06-13 16:49:31.808 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x100a07300 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Start:0] Tx:1 info to 102:11
2026-06-13 16:49:31.810 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1075b1cc0 v1 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Rebase:0] Tx:1 from 102:11
2026-06-13 16:49:31.810 DEBUG [0x102] vs_nfp_enter:7030 VMLH processing sess 0x7f56ca64c602 m 0x1075b1cc0 ackt 0x11 m_rmt_branch 102 vmlh_rmt_branch 102 seq 1 base 1 last_rx 0 vmlh_total_rx 1 marker Sess_Rebase
2026-06-13 16:49:31.810 DEBUG [0x102] vs_nfp_vmlh_handle_rmt_branch_change:3348 Peer branch 102 sent marker Sess_Rebase, rebase ctxt m 0x1075b1cc0 seq 1 base 1 last_rx 0
2026-06-13 16:49:31.810 DEBUG [0x102] vs_nfp_sess_vmlh_reset_info:1162 VMLH shdl 0x7f56ca64c602 vmlh r_ctx_base 2 fwd_base 1 tx_reset 0
2026-06-13 16:49:31.810 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3274 FEC processing mbuf 0x1075b1cc0, seq 1 ret Forward
2026-06-13 16:49:32.808 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x105559a00 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 info to 102:11
2026-06-13 16:49:32.810 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10e5e73c0 v1 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 from 102:11
2026-06-13 16:49:32.810 DEBUG [0x102] vs_nfp_enter:7030 VMLH processing sess 0x7f56ca64c602 m 0x10e5e73c0 ackt 0x11 m_rmt_branch 102 vmlh_rmt_branch 102 seq 2 base 2 last_rx 2 vmlh_total_rx 2 marker None
2026-06-13 16:49:32.810 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3274 FEC processing mbuf 0x10e5e73c0, seq 2 ret Forward
2026-06-13 16:49:33.810 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x103ed3240 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 info to 102:11
2026-06-13 16:49:33.812 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1088c1540 v1 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 from 102:11
2026-06-13 16:49:33.812 DEBUG [0x102] vs_nfp_enter:7030 VMLH processing sess 0x7f56ca64c602 m 0x1088c1540 ackt 0x11 m_rmt_branch 102 vmlh_rmt_branch 102 seq 3 base 3 last_rx 2 vmlh_total_rx 3 marker None
2026-06-13 16:49:33.812 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3274 FEC processing mbuf 0x1088c1540, seq 3 ret Forward
2026-06-13 16:49:34.812 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x107f05040 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 info to 102:11
 2026-06-13 16:49:34.812 DEBUG [0x102] vs_tun_encap_vmlh_hdr:110 VMLH FEC OPT start seq:1 len_parity:0 pkts_per_parity:4 flags:0x0

Note the access circuit change below — the parity packet is sent via the alternate circuit (102:22 instead of 102:11):

2026-06-13 16:49:34.812 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x102b40b00 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:1:None:0] Tx:5 info to 102:22
2026-06-13 16:49:34.814 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x105941940 v1 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 from 102:11
2026-06-13 16:49:34.814 DEBUG [0x102] vs_nfp_enter:7030 VMLH processing sess 0x7f56ca64c602 m 0x105941940 ackt 0x11 m_rmt_branch 102 vmlh_rmt_branch 102 seq 4 base 4 last_rx 3 vmlh_total_rx 4 marker None
 2026-06-13 16:49:34.814 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3274 FEC processing mbuf 0x105941940, seq 4 ret Forward

 

Sender FEC debug — Triggered from First Packet

Shows the 4 ping packets sent with their sequence numbers. FEC uses the original ICMP sequence as well:

2026-06-13 16:49:31.807 DEBUG [0x102] vs_fec_create_sess_to_wan_ctx:186 Creating to-wan ext pkts 4 len 1400 gen 1 is_rtp 0
2026-06-13 16:49:31.807 DEBUG [0x102] vs_fec_process_to_wan_pkt:449 m 0x100a07300 seq 1 orig_seq 1
2026-06-13 16:49:31.807 DEBUG [0x102] vs_fec_process_to_wan_pkt:458 Data pkt len 1008, profile max 1400
2026-06-13 16:49:31.807 DEBUG [0x102] vs_fec_process_to_wan_pkt:487 Ext pkt len 1008 Init done seqnum 1
2026-06-13 16:49:31.807 DEBUG [0x102] vs_fec_to_wan_proc_first_pkt:404 Rcvd first pkt 1 next 1 fec at 4
2026-06-13 16:49:32.808 DEBUG [0x102] vs_fec_process_to_wan_pkt:449 m 0x105559a00 seq 2 orig_seq 2
2026-06-13 16:49:32.808 DEBUG [0x102] vs_fec_process_to_wan_pkt:458 Data pkt len 1008, profile max 1400
2026-06-13 16:49:32.808 DEBUG [0x102] vs_fec_to_wan_calc_pkt_parity:215 start 1 seqnum 2 actual 2 offset 1 mask 0x3 parity_len 0 payload_len 1008
2026-06-13 16:49:33.810 DEBUG [0x102] vs_fec_process_to_wan_pkt:449 m 0x103ed3240 seq 3 orig_seq 3
2026-06-13 16:49:33.810 DEBUG [0x102] vs_fec_process_to_wan_pkt:458 Data pkt len 1008, profile max 1400
2026-06-13 16:49:33.810 DEBUG [0x102] vs_fec_to_wan_calc_pkt_parity:215 start 1 seqnum 3 actual 3 offset 2 mask 0x7 parity_len 1008 payload_len 1008
2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_process_to_wan_pkt:449 m 0x107f05040 seq 4 orig_seq 4
2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_process_to_wan_pkt:458 Data pkt len 1008, profile max 1400
 2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_to_wan_calc_pkt_parity:215 start 1 seqnum 4 actual 4 offset 3 mask 0xf parity_len 0 payload_len 1008

 

Parity packet generation — note that seqnum 6 is shown as the start of the next window because seq 5 is assigned to the parity packet:

 

2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_to_wan_gen_fec_pkt:290 To wan FEC generate fec mask 0xf expect 0xf seq 4 cur_fec_len 1008 cur_parity_len 0
2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_to_wan_make_fec_pkt:242 pkt_m 0x107f05040, fec_m 0x102b40b00 Sending fec pkt len 1036 payload 1008
 2026-06-13 16:49:34.812 DEBUG [0x102] vs_fec_to_wan_reinit_parity_norm:144 vs_fec_to_wan_reinit_parity_norm start seqnum 6

 

Receiver VMLH debug- Notice the change in access circuit in the TX 5 which is th parity packet and has the flag P=1

 

 

2026-06-13 16:49:31.819 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10a305e80 v1 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Start:0] Tx:1 from 101:11
2026-06-13 16:49:31.819 DEBUG [0x102] vs_nfp_sess_vmlh_enable:854 Enabling VMLH info for sess 0x7f38952fcd02, dir fwd, in_ifp NULL out_ifp NULL
2026-06-13 16:49:31.819 DEBUG [0x102] vs_nfp_vmlh_handle_rmt_branch_change:3335 Peer branch 101 sent marker Sess_Start, rebase ctxt m 0x10a305e80 seq 1 base 1 last_rx 0
2026-06-13 16:49:31.820 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x103033b80 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Rebase:0] Tx:1 info to 101:11
2026-06-13 16:49:32.820 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1016ffac0 v1 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 from 101:11
2026-06-13 16:49:32.821 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10d578cc0 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 info to 101:11
2026-06-13 16:49:33.821 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10c02f440 v1 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 from 101:11
2026-06-13 16:49:33.822 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10d439100 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 info to 101:11
2026-06-13 16:49:34.823 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x102c42dc0 v1 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 from 101:11
2026-06-13 16:49:34.823 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x104c07300 v1 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:1:None:0] Tx:5 from 101:22
 2026-06-13 16:49:34.823 DEBUG [0x102] vs_tun_decap_vmlh_hdr:315 VMLH FEC OPT start seq:1 len_parity:0 pkts_per_parity:4 flags:0x0

 

 

Receiver FEC debug

 

FEC is triggered after the first parity packet is received. Note how 3 additional windows (hard coded) of 4 packets each are initialized:

 

2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_create_sess_gbl_from_wan_ctx:234 Creating from-wan ext preserve_order 1
2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_process_from_wan_pkt:2529 m 0x104c07300 seq 5 FEC for orig_seq 5-4
2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_process_from_wan_pkt:2589 shdl 0x7f38952fcd02 m 0x104c07300 Init from wan ctx not done seqnum 5 is_fec 1 out_of_range 1 actual_seq 4
2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 1 num_pkt:4 range:(6-9) peer start:1 prev_range:(0-0) pkt_parity_sz 1400
2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 6 num_pkt:4 range:(11-14) peer start:6 prev_range:(0-0) pkt_parity_sz 1400
2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 11 num_pkt:4 range:(16-19) peer start:11 prev_range:(0-0) pkt_parity_sz 1400
 2026-06-13 16:49:34.823 DEBUG [0x102] vs_fec_from_wan_init_ctx_all:941 Init gbl done for num_pkt:4 range:6-19 prev(start:1 fec_seq:5)

 

Scenario 2: Packet Loss — Last Data packet of the window Lost


12 ping packets are sent from 10.1.1.11 to 10.1.1.5. Packet 8 shows very high latency, indicating a packet within that window was lost and the receiver had to wait for the parity packet to arrive before reconstruction. Sender debugs are not helpful here since loss is induced only in the forward direction.

 

root@ubuntu:~# ping 10.1.1.5 -s 1000 -c 12
PING 10.1.1.5 (10.1.1.5) 1000(1028) bytes of data.
1008 bytes from 10.1.1.5: icmp_seq=1 ttl=126 time=103 ms
1008 bytes from 10.1.1.5: icmp_seq=2 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=3 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=4 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=5 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=6 ttl=126 time=103 ms
1008 bytes from 10.1.1.5: icmp_seq=7 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=8 ttl=126 time=1128 ms <-- loss detected here
1008 bytes from 10.1.1.5: icmp_seq=9 ttl=126 time=103 ms
1008 bytes from 10.1.1.5: icmp_seq=10 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=11 ttl=126 time=102 ms
 1008 bytes from 10.1.1.5: icmp_seq=12 ttl=126 time=102 ms

 

admin@branch2-cli> show orgs org Tenant1 sessions extensive | select protocol 1 | select source-ip 10.1.1.11 | match-all
sessions extensive 0 2 115633
source-ip 10.1.1.11
destination-ip 10.1.1.5
protocol 1
multi-link-mode FEC
forward-fec-parity-pkt-tx 3
forward-multi-link-total-rx 12
 forward-multi-link-total-tx 15

Session Extensive — Receiver (branch1)

multi-link-mode                FEC
reverse-fec-parity-pkt-rx 3 -- 12 packets / 4 per FEC = 3 parity packets
reverse-fec-pkt-lost 1 -- 1 data packet was lost
reverse-fec-pkt-recovered 1
reverse-fec-recovery-rate 100.0 -- recovered / lost * 100
reverse-fec-pkt-held-count 2 -- packets held due to missing sequence numbers
reverse-fec-pkt-released-count 2
reverse-multi-link-total-rx 14
reverse-multi-link-total-tx 12
reverse-held-pkt-count 4 -- total out-of-order packets held
 reverse-released-pkt-count     4

 

Debug Logs

Receiver VMLH debug — First 4 data packets + parity received (the P:1 flag in the header indicates the parity packet)

2026-06-13 17:38:13.510 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x105509780 v1 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Start:0] Tx:1 from 101:11
2026-06-13 17:38:13.510 DEBUG [0x102] vs_nfp_sess_vmlh_enable:854 Enabling VMLH info for sess 0x7f385e364002, dir fwd, in_ifp NULL out_ifp NULL
2026-06-13 17:38:13.510 DEBUG [0x102] vs_nfp_vmlh_handle_rmt_branch_change:3335 Peer branch 101 sent marker Sess_Start, rebase ctxt m 0x105509780 seq 1 base 1 last_rx 0
2026-06-13 17:38:13.512 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x103ffd800 [S:R:M:Pr:P:Mr:Rid 1:0:FEC:1:0:Sess_Rebase:0] Tx:1 info to 101:11
2026-06-13 17:38:14.512 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10747af80 v1 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 from 101:11
2026-06-13 17:38:14.513 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x103424940 [S:R:M:Pr:P:Mr:Rid 2:0:FEC:1:0:None:0] Tx:2 info to 101:11
2026-06-13 17:38:15.513 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1064f0880 v1 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 from 101:11
2026-06-13 17:38:15.514 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x1085c9d80 [S:R:M:Pr:P:Mr:Rid 3:0:FEC:1:0:None:0] Tx:3 info to 101:11
2026-06-13 17:38:16.414 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x106d3c3c0 v1 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:1:None:0] Tx:5 from 101:22
2026-06-13 17:38:16.414 DEBUG [0x102] vs_tun_decap_vmlh_hdr:315 VMLH FEC OPT start seq:1 len_parity:0 pkts_per_parity:4 flags:0x0
 2026-06-13 17:38:16.414 DEBUG [0x102] vs_nfp_vmlh_process_higher_seq_pkt:2964 shdl 0x7f385e364002 vmlh base 4: mbuf 0x106d3c3c0, recv_seq 5

 

Receiver FEC debug — Initialized on first parity packet i.e after the 4 data packets are already recieved; global windows created and Parity packet dropped (no loss in first window); seq 6 received (ICMP seq 5)

 

2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_create_sess_gbl_from_wan_ctx:234 Creating from-wan ext preserve_order 1
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_process_from_wan_pkt:2529 m 0x106d3c3c0 seq 5 FEC for orig_seq 5-4
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_process_from_wan_pkt:2589 shdl 0x7f385e364002 m 0x106d3c3c0 Init from wan ctx not done seqnum 5 is_fec 1 out_of_range 1 actual_seq 4
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 1 num_pkt:4 range:(6-9) peer start:1 prev_range:(0-0) pkt_parity_sz 1400
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 6 num_pkt:4 range:(11-14) peer start:6 prev_range:(0-0) pkt_parity_sz 1400
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 11 num_pkt:4 range:(16-19) peer start:11 prev_range:(0-0) pkt_parity_sz 1400
2026-06-13 17:38:16.414 DEBUG [0x102] vs_fec_from_wan_init_ctx_all:941 Init gbl done for num_pkt:4 range:6-19 prev(start:1 fec_seq:5)
2026-06-13 17:38:16.514 DEBUG [0x102] vs_fec_process_from_wan_pkt:2532 m 0x1034511c0 seq 4 orig_seq 4
2026-06-13 17:38:16.514 DEBUG [0x102] vs_fec_process_from_wan_pkt:2589 shdl 0x7f385e364002 m 0x1034511c0 Init from wan ctx done seqnum 4 is_fec 0 out_of_range 1 actual_seq 4
2026-06-13 17:38:16.514 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1785 Received out of range seq 4, gbl win (6-19)
2026-06-13 17:38:16.514 DEBUG [0x102] vs_fec_from_wan_proc_out_of_range_pkts:1621 Gbl win (6-19), Allowing initial old pkt seq 4
 2026-06-13 17:38:16.514 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1791 seq 4 out of range process ret Reordered

2026-06-13 17:38:16.414 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x106d3c3c0, seq 5 ret Drop
2026-06-13 17:38:16.514 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1034511c0 v1 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 from 101:11
2026-06-13 17:38:16.514 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x1034511c0, seq 4 ret Forward
2026-06-13 17:38:16.516 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x1074aa2c0 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 info to 101:11
 2026-06-13 17:38:17.515 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10a168c40 v1 [S:R:M:Pr:P:Mr:Rid 6:0:FEC:1:0:None:0] Tx:6 from 101:11

 

Seq 6, 7, 8 received and forwarded normally

2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_process_from_wan_pkt:2532 m 0x10a168c40 seq 6 orig_seq 5
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_process_from_wan_pkt:2562 seqnum 6, win_idx 0, range [gbl:6-19 win:6-9]
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_process_from_wan_pkt:2589 shdl 0x7f385e364002 m 0x10a168c40 Init from wan ctx done seqnum 6 is_fec 0 out_of_range 0 actual_seq 5
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1828 Received from wan data pkt 6 win(6-9) mask 0x0
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1852 got_all 0 seq 6 win (6-9) pkt_mask 0x1
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_from_wan_check_n_resize_parity:557 Data pkt len 1008, profile max 1400
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_from_wan_calc_parity_data:1705 Calculating parity_from data seq 6 parity_len 1008 payload_len 1008
2026-06-13 17:38:17.515 DEBUG [0x102] vs_fec_from_wan_check_hold:1472 offset 0 fr_state Reorder_None pkt_mask 0x1 offset_calc 0x1
2026-06-13 17:38:17.515 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x10a168c40, seq 6 ret Forward
2026-06-13 17:38:17.517 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x104a15700 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:0:None:0] Tx:5 info to 101:11
2026-06-13 17:38:18.516 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10602c980 v1 [S:R:M:Pr:P:Mr:Rid 7:0:FEC:1:0:None:0] Tx:7 from 101:11
2026-06-13 17:38:18.516 DEBUG [0x102] vs_fec_process_from_wan_pkt:2532 m 0x10602c980 seq 7 orig_seq 6
2026-06-13 17:38:18.516 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1828 Received from wan data pkt 7 win(6-9) mask 0x1
2026-06-13 17:38:18.516 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1852 got_all 0 seq 7 win (6-9) pkt_mask 0x3
2026-06-13 17:38:18.516 DEBUG [0x102] vs_fec_from_wan_check_hold:1472 offset 1 fr_state Reorder_None pkt_mask 0x3 offset_calc 0x3
2026-06-13 17:38:18.517 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x10602c980, seq 7 ret Forward
2026-06-13 17:38:18.518 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x107d70c80 [S:R:M:Pr:P:Mr:Rid 6:0:FEC:1:0:None:0] Tx:6 info to 101:11
2026-06-13 17:38:19.518 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x100d44400 v1 [S:R:M:Pr:P:Mr:Rid 8:0:FEC:1:0:None:0] Tx:8 from 101:11
2026-06-13 17:38:19.518 DEBUG [0x102] vs_fec_process_from_wan_pkt:2532 m 0x100d44400 seq 8 orig_seq 7
2026-06-13 17:38:19.518 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1828 Received from wan data pkt 8 win(6-9) mask 0x3
2026-06-13 17:38:19.518 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1852 got_all 0 seq 8 win (6-9) pkt_mask 0x7
2026-06-13 17:38:19.518 DEBUG [0x102] vs_fec_from_wan_check_hold:1472 offset 2 fr_state Reorder_None pkt_mask 0x7 offset_calc 0x7
2026-06-13 17:38:19.518 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x100d44400, seq 8 ret Forward
 2026-06-13 17:38:19.519 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10573a740 [S:R:M:Pr:P:Mr:Rid 7:0:FEC:1:0:None:0] Tx:7 info to 101:11

 

 

Seq 10 received instead of seq 9 — this is the parity packet (P flag = 1)

After the first 4 packets, one sequence number is assigned to the parity packet. So VMLH seq 9 would be ICMP seq 8, and VMLH seq 10 is the parity packet. In this case the last data packet of the window was lost, so the receiver only had to wait for the parity packet — resulting in impact to just one ping:

 

2026-06-13 17:38:20.420 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10628e900 v1 [S:R:M:Pr:P:Mr:Rid 10:0:FEC:1:1:None:0] Tx:10 from 101:22
2026-06-13 17:38:20.420 DEBUG [0x102] vs_tun_decap_vmlh_hdr:315 VMLH FEC OPT start seq:6 len_parity:0 pkts_per_parity:4 flags:0x0
2026-06-13 17:38:20.420 DEBUG [0x102] vs_nfp_vmlh_process_higher_seq_pkt:2964 shdl 0x7f385e364002 vmlh base 9: mbuf 0x10628e900, recv_seq 10
2026-06-13 17:38:20.420 DEBUG [0x102] vs_fec_process_from_wan_pkt:2529 m 0x10628e900 seq 10 FEC for orig_seq 5-8
2026-06-13 17:38:20.420 DEBUG [0x102] vs_fec_from_wan_is_regen_reqd:1942 seq 10 range,n_pkts [peer:6-9,4 sess:6-9,4] peer_pkt_mask 0xf rcvd_pkts 3
 2026-06-13 17:38:20.420 DEBUG [0x102] vs_fec_from_wan_is_regen_reqd:1994 Only last pkt not recvd, pkt_mask 0x7 holding FEC

 

Seq 10 (parity) is held; when seq 11 arrives from the next window, the lost packet is regenerated and both are released

2026-06-13 17:38:20.420 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x10628e900, seq 10 ret Hold
2026-06-13 17:38:20.461 DEBUG [0x102] vs_nfp_vmlh_adjust_win:2744 shdl 0x7f385e364002 vmlh last_rx_seq 10 shifted win_base 9 => 11, state: In-sync
2026-06-13 17:38:21.545 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x105773740 v1 [S:R:M:Pr:P:Mr:Rid 11:0:FEC:1:0:None:0] Tx:11 from 101:11
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_process_from_wan_pkt:2532 m 0x105773740 seq 11 orig_seq 9
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_proc_data_pkt:1828 Received from wan data pkt 11 win(11-14) mask 0x0
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_proc_cached_fec:2240 Processing FEC m 0x10628e900 again
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_regen_lost_pkt:2203 Regenerated lost pkt m 0x107193840 len 1008 flags 0xc1 seqnum 9 actual_seq 8
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_free_cached_fec_pkt:714 Freeing cached m 0x10628e900 seq 10 from win ctx (6-9)
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_shift_win_base:1154 Shifting win base 1 times, making 0 as last, bitmask 0xf
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_init_ctx:883 Init win done for peer_start 16 num_pkt:4 range:(21-24) peer start:16 prev_range:(6-9) pkt_parity_sz 1400
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_shift_fec_cache_base:676 Shifted fec_cache base 1 times, base 25(0) => 30(1)
2026-06-13 17:38:21.545 DEBUG [0x102] vs_fec_from_wan_check_hold:1472 offset 0 fr_state Reorder_None pkt_mask 0x1 offset_calc 0x1
2026-06-13 17:38:21.545 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x105773740, seq 11 ret Hold
2026-06-13 17:38:21.547 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x105acf940 [S:R:M:Pr:P:Mr:Rid 8:0:FEC:1:0:None:0] Tx:8 info to 101:11
 2026-06-13 17:38:21.547 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x1065ac740 [S:R:M:Pr:P:Mr:Rid 9:0:FEC:1:0:None:0] Tx:9 info to 101:11

 

Scenario 3: Packet Loss — First Data Packet Lost



 

root@ubuntu:~# ping 10.1.1.5 -s 1000 -c 12
PING 10.1.1.5 (10.1.1.5) 1000(1028) bytes of data.
1008 bytes from 10.1.1.5: icmp_seq=1 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=2 ttl=126 time=101 ms
1008 bytes from 10.1.1.5: icmp_seq=3 ttl=126 time=101 ms
1008 bytes from 10.1.1.5: icmp_seq=4 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=5 ttl=126 time=4194 ms <-- loss begins here
1008 bytes from 10.1.1.5: icmp_seq=6 ttl=126 time=3175 ms
1008 bytes from 10.1.1.5: icmp_seq=7 ttl=126 time=2151 ms
1008 bytes from 10.1.1.5: icmp_seq=8 ttl=126 time=1127 ms
1008 bytes from 10.1.1.5: icmp_seq=9 ttl=126 time=103 ms
1008 bytes from 10.1.1.5: icmp_seq=10 ttl=126 time=102 ms
1008 bytes from 10.1.1.5: icmp_seq=11 ttl=126 time=101 ms
 1008 bytes from 10.1.1.5: icmp_seq=12 ttl=126 time=101 ms

 

The impact spans packets 5 through 8 — four consecutive pings. This indicates that ICMP seq 5 (VMLH seq 6) was the dropped packet — the first in its FEC window. Since it was the first packet, all subsequent packets in the window (VMLH seq 7, 8, 9) plus the parity (VMLH seq 10) were held waiting for reconstruction. The parity had to arrive and then seq 11 (from the next window) had to arrive before the FEC module could regenerate the lost packet and release all held packets. With ICMP (1 packet/sec), this caused a ~4-second cascade.Only receiver-side debugs are shown below since loss is induced only in the forward direction.

 

Debug Logs

 

Receiver VMLH debug — Seq 6 skipped; seq 7 arrives and is held

2026-06-13 18:03:59.971 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1050c4000 v1 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:1:None:0] Tx:5 from 101:22
2026-06-13 18:03:59.971 DEBUG [0x102] vs_tun_decap_vmlh_hdr:315 VMLH FEC OPT start seq:1 len_parity:0 pkts_per_parity:4 flags:0x0
2026-06-13 18:03:59.971 DEBUG [0x102] vs_nfp_vmlh_process_higher_seq_pkt:2964 shdl 0x7f38952fe202 vmlh base 4: mbuf 0x1050c4000, recv_seq 5
2026-06-13 18:04:00.071 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x101354480, seq 4 ret Forward
2026-06-13 18:04:00.072 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10b29a240 [S:R:M:Pr:P:Mr:Rid 4:0:FEC:1:0:None:0] Tx:4 info to 101:11
2026-06-13 18:04:02.090 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x107233b80 v1 [S:R:M:Pr:P:Mr:Rid 7:0:FEC:1:0:None:0] Tx:7 from 101:11
2026-06-13 18:04:02.090 DEBUG [0x102] vs_nfp_vmlh_process_higher_seq_pkt:2964 shdl 0x7f38952fe202 vmlh base 6: mbuf 0x107233b80, recv_seq 7
 2026-06-13 18:04:02.090 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x107233b80, seq 7 ret Hold

Seq 8, 9, 10 (parity) all held; eventually seq 11 triggers reconstruction

2026-06-13 18:04:03.114 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10a8a30c0 v1 [S:R:M:Pr:P:Mr:Rid 8:0:FEC:1:0:None:0] Tx:8 from 101:11
2026-06-13 18:04:04.038 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10892dfc0 v1 [S:R:M:Pr:P:Mr:Rid 10:0:FEC:1:1:None:0] Tx:10 from 101:22
2026-06-13 18:04:04.038 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x10892dfc0, seq 10 ret Hold
2026-06-13 18:04:04.138 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x10e60ba40 v1 [S:R:M:Pr:P:Mr:Rid 9:0:FEC:1:0:None:0] Tx:9 from 101:11
2026-06-13 18:04:04.138 DEBUG [0x102] vs_nfp_vmlh_handle_rcv_fec:3261 FEC processing mbuf 0x10e60ba40, seq 9 ret Hold
2026-06-13 18:04:05.162 DEBUG [0x102] vs_tun_decap_vmlh_hdr:264 Rcvd VMLH m 0x1089ab900 v1 [S:R:M:Pr:P:Mr:Rid 11:0:FEC:1:0:None:0] Tx:11 from 101:11
2026-06-13 18:04:05.164 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x1080e6bc0 [S:R:M:Pr:P:Mr:Rid 5:0:FEC:1:0:None:0] Tx:5 info to 101:11
2026-06-13 18:04:05.164 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10694e0c0 [S:R:M:Pr:P:Mr:Rid 6:0:FEC:1:0:None:0] Tx:6 info to 101:11
2026-06-13 18:04:05.164 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x106501740 [S:R:M:Pr:P:Mr:Rid 7:0:FEC:1:0:None:0] Tx:7 info to 101:11
 2026-06-13 18:04:05.164 DEBUG [0x102] vs_tun_encap_vmlh_hdr:162 Sending VMLH pkt m 0x10adcabc0 [S:R:M:Pr:P:Mr:Rid 8:0:FEC:1:0:None:0] Tx:8 info to 101:11

 

Receiver FEC debug — Lost packet regenerated; 3 cached packets released

2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_regen_lost_pkt:2203 Regenerated lost pkt m 0x108d837c0 len 1008 flags 0xc1 seqnum 6 actual_seq 5
2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_send_cached_pkts:992 Sending cached mbuf count 3
2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_send_cached_pkts:1008 Releasing cached pkt m 0x107233b80 from offset 1
2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_send_cached_pkts:1008 Releasing cached pkt m 0x10a8a30c0 from offset 2
2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_send_cached_pkts:1008 Releasing cached pkt m 0x10e60ba40 from offset 3
 2026-06-13 18:04:05.162 DEBUG [0x102] vs_fec_from_wan_free_cached_fec_pkt:714 Freeing cached m 0x10892dfc0 seq 10 from win ctx (6-9)