Table of Contents

Purpose.

1       Connectivity issue between peers

2       Mismatched Pre-shared Key

3       Configuration mismatch

  3.1        IKE parameters

  3.2        IPSEC parameters

4       Tunnel flapping

  4.1        Transport issues

  4.2        Role

  4.3        Rekey timing

  4.4        DPD dropped

  4.5        Tunnel initiate mode

5       The Tunnel is up but traffic is not passing

  5.1        Pushing traffic to tunnel

   5.1.1         Route-based

   5.1.2         Policy-based

  5.2        Traffic selector

  5.3        Make sure there are no duplicate profiles

  5.4        Verify if the traffic is sent/receive on versa box using tcpdump

  5.5        Clearing stale SA information

  5.6        Check the configuration applied to VSMD

6       Please collect below logs

7       Known Interoperability issues

8       Contact Support

Purpose

The purpose of this document is to help troubleshoot IPSEC issues with third-party vendors like Cisco, Fortinet, juniper, etc. As a result, this document provides a checklist of common procedures to try before you begin to troubleshoot a connection and call Versa Technical Support. This document will mainly illustrate and focus on the checks that need to perform on Versa Devices. 

Broadly there is two type of issues seen, one is due to the connectivity problem between peers and another one is due to configuration or compatibility problems.

1. Connectivity issue between peers

Check the reachability between IPSEC peers, use ping test and perform routing checks to isolate any transport issue. In case of and FW device in between to make IPSec work through your firewalls, you should open UDP port 500 and permit IP protocol numbers 50 and 51 on both inbound and outbound firewall filters. UDP Port 500 should be opened to allow Internet Security Association and Key Management Protocol (ISAKMP) traffic to be forwarded through firewalls. IP protocol 50 should be set to allow IPSec Encapsulating Security Protocol (ESP) traffic to be forwarded. Finally, IP protocol 51 should be set to allow Authentication Header (AH) traffic to be forwarded. In addition to this UDP port 4500 for NAT-T functionality. In case such cases normally, IKE will show timed out as below which either the IP or Port reachability is missing.

2. Mismatched Pre-shared Key 

Another common issue is mismatch of pre shared key or CA certificate. The initiation of VPN Tunnel gets disconnected due mismatched pre-shared-key or CA certificate during the phase I negotiations. We usually see authentication failed such scenarios.

In order to resolve this issue, re-enter the pre-shared key on both appliances; the pre-shared-key must be unique and matched. If cert-based authentication is used this requires a CA cert trusted by both parties.

3 Configuration mismatch 

To bring IPSEC up certain parameters must match for IKE and IPSEC on both the ends Validate the IPsec configuration and make sure all Phase1(IKE) and Phase2 (IPsec) parameters match.

For complete configuration please refer KB configuring-ipsec-to-3rd-party-devices available on our support portal which talks about the detailed configuration steps.

3.1 IKE parameters  

All the IKE parameters should match on both ends as shown below. It is recommended using the same identity type both ends if the combination of IP and FQDN used, this also might create issues.

3.2 IPSEC parameters 

Make sure that the IPsec encryption and hash algorithms to be used by the transform set on both ends are the same. Forward secrecy mode if used must match both ends.

Usually, any mismatch here will cause no proposal chosen error as shown below

Note: In Policy-based IPsec separate SA are created to reach traffic selector/policy/rule. We would see “No proposal chosen” IKE history, even if anyone of traffic-selector has a mismatch with the peer, even though SA may be up for another traffic selector which matches on both sides.

4. Tunnel flapping  

Above we have seen conditions that can cause issues to bring the IPsec tunnel up. However, sometimes we encounter scenarios where tunnel came up but it is flapping or going up/down. At this stage, we are discussing a few scenarios which can cause this.

4.1 Transport issues

First and foremost thing to check if there are any transport/underlay issue which is resulting in intermittent packet drops and causing the tunnel to flap. Run multiple ping tests with more packet counts and verify if there are any packet drops

4.2 Role

To bring the tunnel up one peer must be the initiator and the other becomes the responder. In Site to Site mode, Versa does not have the explicit configuration to become “initiator always” or responder always” mode but some vendor has such config knob. So, we have to consider that if another end is responder only then versa the box has to initiate the tunnel. We can set the knob “Automatic” in Tunnel initiate, so we will not wait for traffic to trigger rekey. 

Note:-  This knob "Automatic" is not same as “initiator always” on the vendors.

4.3 Rekey timing

It is good practice to have the same lifetime/rekey timer for IKE/IPsec on both peers. It has been observed different timers can result in tunnel flaps. 

4.4 DPD dropped

Dead Peer Detection (DPD) is a method that allows the detection of unreachable Internet Key Exchange (IKE) peers. IKE peer sends an R-U-THERE query to its peer if it is interested in the liveliness of this peer. Re-transmission of R-U-THERE queries when it fails to receive an ACK after some number of re transmitted messages, peer assumed to be unreachable and delete IPSec and IKE SAs to the peer. This usually happens due to the transport issues, follow step 4.1 to rectify.

Note: These logs are only seen when ipsec debugs is enabled. @/var/log/versa/ versa-ipsec.log.

4.5 Tunnel initiate mode

The tunnel will remain down when tunnel initiate is configured as “traffic” and will only come up once interesting traffic hits the IPsec. When rekey timer is about to expire and there is no traffic locally or from remote, the IPsec will go down. 

4.6 Configuration is not match 

For policy based tunnels , please make sure configuration should match at both end. if one end is having protocol and other end don't have , it will flap and will do rekey.

5. The tunnel is up but traffic is not passing

In some scenario’s where the tunnel will come up but traffic is not passing on the tunnel. In this section, we will discuss some of these scenario’s

5.1 Pushing traffic to tunnel 

There is two way of pushing traffic to IPSEC tunnel Route-based or policy-based. So, if we do not choose the right config we might see some issues around this area which we will discuss going forward.

5.1.1 Route-based 

For route-based tunnel verify that routing is correct on versa device and the route is correctly pointing to right next-hop/tvi/VR in the LAN VR

5.1.2 Policy-based 

For Policy-based tunnel, flexvnf creates a filter where traffic should hit before routing module and forwarded to IPSEC tunnel

All though it’s a policy-based tunnel, technically we do not need to have a route but to hit versa service changing route lookup against the destination must be performed and the session must be created otherwise this will not work. We need a dummy route or default route in the LAN VR matching the destination. 

Note: PBR (using SDWAN or PBF) with Policy-based IPsec in the same VR does not work properly. Make sure to have PBR in LAN-VR and IPsec in Custom-VR(Recommended) or Transport-VR

5.2 Traffic selector 

 If this is the policy-based tunnel, make sure the right policy is configured which is matching the traffic selector.   Make sure if traffic is NAT than traffic selector should be for natted ip’s

5.3 Make sure there are no duplicate profiles 

This has been observed many times that same traffic selector policy has been used in multiple IPSEC VPN profiles. If you have multiple profiles, traffic will hit the first tunnel that matches traffic selectors may not hit the expected tunnel. This is expected behavior so make sure the right policy is configured under the right VPN profile or configured the VPN in different routing-instances.

5.4 Verify if the traffic is sent/receive on versa box using tcpdump

Initiate ping traffic from the host with a packet size of 800 and take a tcpdump on the LAN and WAN interface of the Versa BOX. In the below capture we can see traffic hitting the LAN is clear text and on WAN interface its encrypted. 

IPSEC statistics cmd will also give insight if the packet decap/encap is happening

5.5 Clearing stale SA information 

Sometimes this has been seen that due to some stale SA information, traffic is not passing on tunnels so clearing security association will help. Please collect captures beforehand mentioned in step 6 for the root cause.

request clear ipsec sa vpn-profile <profile name> org <suborg> peer <peer-ip>

5.6 Check the configuration applied to VSMD

When we apply the config and it’s not working please check if the configuration is present in VSMD.

vsh connect vsmd
show ipsec config <tenant-id> <config-type>

 Please try removing and re-applying config 

Please collect captures mentioned at step 7 for root cause

6. Known interoperability issues

N/A

Note:- It is always recommended to go through the known issues list available on the support portal before contacting Versa Tac.

7. Please collect below logs 

From Versa Box

#Cli output

show orgs org-services <org> ipsec vpn-profile <profile name> security-associations detail

show orgs org-services <org> ipsec vpn-profile <profile name> ike history

show orgs org-services <org> ipsec vpn-profile <profile name> ipsec  history

show configuration orgs org-services <ORG> ipsec vpn-profile <profile name> | display set

#Collect ipsec debug

configure

set debug ipsec all-flags level all

set debug ike level debug

show | compare

commit

# Remove debug

configure

rollback

show | compare

commit

# Collect log files

tar -cvzf ipsec-debugs.tar.gz  /var/log/versa/versa-ipsec*

Note: Log the putty/terminal session to capture all outputs when performing requested steps, will be helpful when engaging Support