This article provides step-by-step instructions to replace and re-add a failed node (Analytics or Search) in a Versa cluster.
Prerequisites
Before starting, ensure the following for the replacement node:
Same Versa software version as other nodes in the cluster.
Same number of network interfaces.
Similar CPU, memory, and hyper-threading profile.
IP addresses and routes are identical to the failed node.
Hostname and DNS entries are correctly set in
/etc/hosts.
Step 1: Prepare the Node
Make sure the following files are the same from an existing node of the same personality (Analytics or Search):
/etc/network/interfaces/etc/hosts/opt/versa/scripts/van-scripts/vansetup.confInstall certificates SSL certificates if any. Refer below KB.Perform Manual Hardening for Versa Analytics - Versa Networks
Make sure the below on
vansetup.confon the new node:Update the
rpc addressandlistener addressfields.zookeeper parameter should be set correctly.
Replication factor set as per your cluster configuration.
For Analytics nodes only, set the
seedsparameter to the listener IP of an existing working Analytics node in the cluster, this ensures that the new replacement node communicates with the existing cluster, syncs data, and joins the Cassandra cluster.seeds="a.b.c.d"
Step 2: Perform Pre-Setup Based on Node Type
Analytics Node:
Run the following on any existing Analytics node to remove the old node's host ID:
# nodetool status # Note the host-id of the failednode# nodetool removenode <host-id>
Search Node:
Ensure the Zookeeper cluster is up and has a leader before running the setup. Run below command on node running zookeeper.
# vsh dbstatus
Step 3: Run vansetup.py
Execute the setup script on the new node:
# cd /opt/versa/scripts/van-scripts# sudo ./vansetup.py
Step 4: Sync Certificates from the Director node
# sudo su versa# cd /opt/versa/vnms/scripts/# ./vnms-cert-sync.sh --sync# ./vd-van-cert-upgrade.sh --pull
When prompted for "postpone restart", select y.
Step 5: Restart Directors (HA Setup Only)
On Secondary Director:
vsh stopOn Primary Director:
vsh restartOn Secondary Director:
vsh start
Verify HA sync between directors
Post-Checks
For Analytics Node
Run
nodetool statuson both the new and existing Analytics nodes.New node will initially show
UJ(Joining). It should change toUN(Up/Normal) once data sync completes.Once the newly added node is in
UNstate, run below command on the new node to ensure data consistencies.# nodetool repairOnce the newly added node is in
UNstate, run below command on the existing nodes to removes data that no longer belongs to a node after a topology change.# nodetool cleanup
For Search Node
Check search DB status:
# vsh dbstatusEnsure
live-nodescount is correct.collectionsshould be healthy
Check Solr cluster health:
# sudo /opt/versa/scripts/van-install/cluster_install.sh solr cluster_status
All replicas should show as active, and all data is in sync across the replica.
Confirm the new node is reachable via the Director UI.
Troubleshooting
If the node fails to join properly or services remain inactive:
Review
nodetool status,vsh dbstatus, and Solr status.
If, on the newly added search node, the "Total Documents" field appears blank for any replica of a collection — even though its state is shown as Active — this indicates that the documents have not yet synchronized.
To verify this, navigate in the Analytics GUI to Admin → System Status → Status, and review the "Status per host/replica" table (see reference image below).
If the issue is observed, re-add the affected replica using the commands provided below to initiate document synchronization.

- To remove a replica, run the command below from the CLI of any search node. Please ensure that you are checking the correct hostname (IP of the newly added node), shard_name (highlighted under the Replica column in the image above), coll_name (highlighted under the Collection column in the image above), and replica_name (highlighted under the Core Name column in the image above).
Example:# sudo /opt/versa/scripts/van-install/cluster_install.sh solr delete_replica <coll_name> <shard_name> <replica_name># sudo /opt/versa/scripts/van-install/cluster_install.sh solr delete_replica alarmlogs shard1 core_node8
- To re-add the replica, run the command below from the CLI of any search node.
# sudo /opt/versa/scripts/van-install/cluster_install.sh solr add_replica <coll_name> <shard_name>Example:# sudo /opt/versa/scripts/van-install/cluster_install.sh solr add_replica alarmlogs shard1
If issues persist, open a ticket with Versa TAC and include:
Node type and personality
Output of post-check commands
Capture the tech-support and shell session logs