Purpose:
Purpose of this document is to troubleshoot the issues encountered during downloading and installation of Security packages on FlexVNFs. SPACK contains updated Antivirus Database, IPS/IDS Signatures, GeoIP Database, URL Category/Reputation database, and IP Reputation database. This information is useful for traffic filtering.
There are two categories of SPACK one is sample and other is Premium. By default, versa software comes pre-installed with a sample spack which have limited details. This can later on be upgraded to a premium spack for full-fledged database.
Basic Configuration:
To download spack updates FlexVNF must have below basic config which include URL of the spack server and the routing instance which it will use to reach to spack server. You should also have internet connectivity on your device from one of the routing-instance in order to reach the spack server. In below case it has been achieved using DIA-Transport-VR.
Download Time Out: Specify the time (in minutes) for the download to timeout.
Daily Update: It will schedule daily update of the SPACK. This ensures the SPACK is always up to date to protect the appliances from the latest threats.
Realtime Update: It will allow real-time updates on the appliance. This ensure critical updates are upgraded on the appliance before the next SPACK upgrade is available for download.
Start Time: Select the time to schedule real time update on the appliance.
Interval: Specify a time interval after which the appliance re-attempts to upgrade the SPACK in case of a failure in an earlier attempt.
Download Type: Incremental—Downloads only the incremental update in the SPACK in the previous SPACK installed in Versa Director
Full—Download the complete SPACK and overwrites the last installed SPACK on the Versa Director.
Along with above configuration we also need to configure DNS to resolve the spack server URL.
Reachability to SPACK URL:
Test the reachability of spack server URL.
Issues:
- Spack download failed:
i.) In case with above config and checks the download still fails check the versa-spack.log and versa- spackmgr.log for more information.
Check the reachability of spack server.
Here sharing one example of such issue where spack download was not completed,
Above logs suggest that though the spack server is reachable the system is not able to resolve the URL of spack website.
ii.) Check the content of resolv.conf file under /etc directory if it is correctly updated the name-server.
iii.) Manually download the spack file.
iv.) Check the download status:
2. Spack Installation failed:
Spack installation can fail due to multiple reasons which include low memory, wrong permissions to tmp folder and other software defects.
i.) Check the install status. Install status must be successful.
ii.) Check the free disk space available. In case the device is running low on memory it can cause the spack installation to fail because in such cases device is not able to allocate disk space required to install spack.
If the device is running low on memory you can clear the cache/buffers to free memory in case, they are occupying a lot of memory. Additionally, increasing the memory is also an option.
iii.) Check if any process is utilizing high RAM and CPU making device to be busy during spack installation.
There must be enough resources available during spack installation.
iv.) Check the permissions given to /tmp folder.
While installing security package, versa-spack will extract tar file into /tmp directory and this might fail due to permission issue. After giving correct permissions the security package installation should succeed.
Known Issues:
- Spack installation failed with below error,
This issue is identified as a bug where device is not able to load python module due to improper allocation of memory. This can be verified from versa-spack.log and versa-spackmgr.log.
2020-02-27 10:59:35.315 ERROR [0x201] vs_vparse_load_python_module:1472 PyImport_Import() failed.
2020-02-27 10:59:35.315 ERROR [0x201] vs_vparse_load_python_module:1477 PyImport_Import error: <type 'exceptions.OSError'>
[Errno 12] Cannot allocate memory
<traceback object at 0x7f0ff58746c8>
2020-02-27 10:59:35.315 ERROR [0x201] vs_vparse_scanner_load_python:4626 unable to load python scanner at /opt/versa/etc/spack/installed/versa-security-package-1640/python/peparse.
2020-02-27 10:59:35.315 ERROR [0x201] vs_vparse_spack_load:1181 Unable to load python scanner at python/peparser.py
[0x7f0ff41b8540] Called urlf_global_class deconstructor
2020-02-27 10:59:35.320 ERROR [0x201] spack version 1640; install status 1
2020-02-27 10:59:35.324 ERROR [0x201] spack_revert script failed
As a workaround buffer/cache can be cleaned,
The steps followed to resolve the issue:
(1) Free memory buffer/cached.
sudo su
echo 1 > /proc/sys/vm/drop_caches
(2) Downgrade spack to 1640 (previous version).
(3) Upgrade back to spack 1659.
2. Check-for-update knob returning bad response,
admin@versa-flexvnf-cli(config)% run request security security-package download check-for-updates
Error: bad response from action
[error][2019-12-06 08:56:15]
From versa-spack.log,
2019-09-15 22:55:49.014 DEBUG [30251] flexvnfspack::processCheckUpdates:621: Resolved IP addr 172.217.163.115
2019-09-15 22:55:49.019 INFO [30251] checker::performSpackChecker:168: Performing spack checker
2019-09-15 22:55:49.019 INFO [30251] checker::performSpackChecker:170: Current spack version 1652,premium
2019-09-15 22:55:49.022 DEBUG [30251] downloader::run:1017: Processing check-updates request
2019-09-15 22:55:54.201 INFO [30251] checker::performSpackChecker:212: available new spack version 1653 (but check for updates throwing bad response error)
This is an issue tracked under Bug 43033 - check-for-updates not displaying the latest spack version available in server, will be fixed in R2S11.
Contact Support:
In case the issue is persist please reach out to versa support with below details:
- Command outputs:
i.) From system CLI:
show configuration security security-package | display set | details
show security security-package information
show security security-package security-package-history
request security security-package install status
request security security-package download status
show configuration | match name-server | display set
ping spack.versanetworks.com routing-instance <name of Routing instance used under spack config>
ii.) From system shell:
Free -h
Top -H
2. Log files:
Directory /var/log/versa.
[admin@HUBCPE1: /] $ cd /var/log/versa/
[admin@HUBCPE1: versa] $ ls -la
total 81996
----snip----
-rw-r----- 1 root root 939424 Mar 27 13:30 versa-service.log
-rw-r----- 1 versa versa 0 Oct 11 12:42 versa-spack.log
-rw-r----- 1 root root 110384 Mar 27 14:21 versa-spackmgr.log
Q&A:
1. Versa FlexVNF software is always bundled with only the SAMPLE flavor of SPACK. Why?
• Premium spack has around ~500MB in size compared to ~100MB sample spack. We do not want to increase flexvnf pkg size drastically.
• Premium spack also has full-fledged ips/urlf/av database etc. We do not want to make these available to any customer by default.
Our software is expected to run on 2C/4GB box as well as on high end boxes.
2. Whenever Versa FlexVNF software is upgraded, if the SPACK installed on the box is a PREMIUM flavor will a software upgrade change it?
• If the premium spack installed is not compatible with to-be-installed Flexvnf, they need to manually upgrade premium spack first.
3. Whenever Versa FlexVNF software is upgraded, if the SPACK installed on the box is a SAMPLE flavor and the SAMPLE is *older* than what is In software bin, will it be upgraded?
• Basically for sample spack, let's say Flexvnf installed has version X, Flexvnf to-be-installed has version Y:
a. If X >= Y, SPACK remains unchanged
b. if X < Y, version X will be replaced with version Y as part of Flexvnf upgrade