This article describes how to test the production setup upgrade locally by onboarding the branch. 

Step 1. Create a VMs (Director, Controller and Branch) with the same number of interfaces as of production devices.

Attached the document on how to install versa on KVM platform

For E.g., below is the reference setup for prod local tests.

Step 2. Take the backup of primary director and controller, load it locally in the lab created VMs.

Please follow the below instructions for restoring prod director DB locally. 

When you are restoring customer DB of Versa Director please perform following steps to make sure customer devices and head-ends which are publicly reachable

are not affected during troubleshooting.

1. delete vnmsha    
2. delete nms cmsconnectors cmsconnector <connectorName>      /* Except local you can delete other connectors */
3. set devices device * state admin-state southbound-locked         /* This is to make sure to make sure even appliance (like controller on cloud) is reachable over public ip it is not affected. This is very safe step */

  Please commit in CLI after above operations.

 Also, make sure in /var/versa/vnms/data/conf/vnms.properties there are no references to public-ip of customer versa director.

Step 3: Make sure all the local devices are isolated from public internet while testing upgrade by restoring customer setup backup file.

Add the below subnets.

sudo route add -net 10.240.0.0 netmask 255.255.0.0 gw 10.192.0.1 dev eth0

sudo route add -net 172.17.0.0 netmask 255.255.0.0 gw 10.192.0.1 dev eth0

Delete all default routes pointing towards any gateway.

sudo route del -net 0.0.0.0 netmask 0.0.0.0 gw 10.192.0.1 dev eth0

Try to reach internet from all the devices and make sure it should not have reachability.

admin@VD1:~$ ping 8.8.8.8

connect: Network is unreachable

Step 4: Assign the same mgmt. IP of production devices to the local devices (Director , Controller and Branch) by below command.

sudo ifconfig eth0:0 <ip-address> netmask <netmask> up

E.g.

sudo ifconfig eth0:0 10.49.76.86 netmask 255.255.255.0 up  

Step 5: Remove the route from director pointing towards service-vnf and add the same by pointing next hop as controller southbound IP.

[Administrator@EU-VSA-PROD-001-01: ~] $ route -n

Kernel IP routing table

Destination      Gateway         Genmask         Flags Metric Ref    Use Iface

10.49.76.0       0.0.0.0         255.255.255.0   U      0      0        0 eth0

10.49.77.0       0.0.0.0         255.255.255.0   U      0      0        0 eth3

10.240.0.0      10.192.0.1    255.255.0.0        UG   0      0        0 eth0  

192.168.0.0      10.49.77.86    255.255.0.0     UG     0      0        0 eth3

sudo route del -net 192.168.0.0 netmask 255.255.0.0 gw 10.49.77.86 dev eth3

sudo route add -net 192.168.0.0 netmask 255.255.0.0 gw 10.49.77.82 dev eth3

10.49.77.82 ---> Controller southbound IP

Step 6: Update the southbound interface of the director with service-vnf IP and run the startup script

make sure in /var/versa/vnms/data/conf/vnms.properties there are no references to public-ip of customer versa director.

Step 7: Check the reachability to between the director and controller.

Step 8: Add the Director southbound IP in VNF-Manager of controller.

Appliances > Controller > Configuration > Services > SDWAN > Select Provider Org > Sites > Edit.

Step 9: Check the controller status on director UI, Controller should show reachable and in sync with the device on director UI.

Step 10: Select any existing device template to onboard the device and update the workflow device bind data with the WAN IP and gateway then redeploy the device.

Workflow > Devices > Select device > bind data > update the WAN / LAN Ips

Step 11: Go to branch cli and check the controller reachability.

Step 12: Update the VNF Manager of the particular device template with director southbound IP.

Step 13: Run staging script on branch to stage the branch.

sudo ./staging.py -w 0 -c 3.11.240.214 -s 49.204.84.222/29 -g 49.204.84.217 -l SDWAN-Branch@VERSA.com -r Controller-AWS-staging@VERSA.com

Verify the branch status in cli and director UI.

Step 14: Perform below checks

Below testing, we need to perform.

--> Creating NOC-pre org before upgrade

--> Creating NOC-post org after upgrade

--> onboard the device on both the org after upgrade

Once it is done will try to commit some template and  check it.

Errors / obstacles 

1. If the device / controller is showing out of sync and not able to commit / sync the configuration  or showing below error while committing / sync config.

Run the below script from director.

End.

Thanks.