In 16.1R2S11, 20.2.x and 21.x Director backup process has been enhanced to save an active node backup (in /var/versa/backups/) and a local node backup (in /var/versa/local-backups/).
Files in /backup are rsync from master to the slave node. So essentially both nodes has the same copy of the backup files (Backup of Master node) in /backup directory.
However, both nodes save a local node backup under /local-backups directory. This directory can’t be overwritten by the peer HA node.
Creating a Manual Backup.
To create a backup file.
>>request system recovery backup include-package-dir false
If you wish to include the software image package in the backup file,
>>request system recovery backup include-package-dir true
If you are running the above command on a master node, it will save a copy in /backup and /local-backup directory. Also, it will be rsync to slave node's /backup directory.
However, if you are running it at a slave node, it will be stored only in the slave node’s /local-backup directory.
Restoring Backup
1. If you are restoring a backup file from external storage, you need to copy it to /var/versa/backups/. If you have desired backup file already in this directory, proceed with the following command.
>>request system recovery restore file <backup_file_name>
The command above will use the files from /backup directory only. You can see the list of backup files under /backup directory using the command,
>>request system recovery list
2. If your /backup file is overwritten by peer node due to some incorrect configuration, split-brain situation, or due to other reasons, you may consider /local-backup directory files to restore the configuration. However, you need to copy the desired file from /local-backup to /backup and then run the restore command provided in article-1.
3. If you restore a backup following article-1, at the Slave node, your spring-boot services will not come up after recovery. Because, /backup directory stores the config of the Master node. To restore the services, proceed with the following script,
>>sudo /opt/versa/vnms/scripts/vnms-startup.sh
Precautions
1. If you have one node in good state (if the master is down, but your slave if fully functional or vice versa), you should disable the HA at the healthy node, before proceeding to recovery on peer node. This is to prevent new (recovered) node with older config to become master and wipe out recent changes.
2. Once you have a node recovered, you can enable HA again. Make sure the node with the latest and correct configuration becomes master while enabling HA.
3. As a good practice, when you are going to restore a backup at any node, create a snapshot of both (or working) nodes. So, in case, after restoring a backup, you realize, it's not the right one or if it's worse than the initial setup, you can rollback.
For taking snapshot
>>request system create-snapshot description Pre-Backup
See all snapshots on node
>>show system snapshots
Rollback to a snapshot
>>request system rollback to 20210309T164419
*rollback to is timestamp you see as Taken at in show system snapshot command.
To change the password for UI login (if you don't know the password from the backup file), use the following CLI command:
request nms actions reset-password-by-admin username Administrator
NOTE: To run a backup file from a 14.04 VD setup to 18.04 VD setup (for any use case) , make sure to have the same release and package-id on both the Director setups.
