Published Date: August 5, 2019
1. Symptom
It was reported that after issuing a soft reboot on Advantech FWA-5020(Versa 1000), the appliance starting continuously rebooting every ~5 min.
2. Root Cause Analysis Status
Based on the analysis so far from Advantech, the issue is attributed to the use of ipmitool command.
Calls from ipmitool are causing the interface between BIOS/OS to BMC to go out of sync and when the box soft reboots, next time, the BMC is not able to serve the Keepalive from the BIOS, resulting in continuous reboots.
The rate of occurrence is unknown. When the appliance had a soft reboot, it could sometimes go to continuous reboot status (not always).
The way to recover from continuous reboot is cold-reboot.
Advantech was able to replicate the issue and able to come up with a solution. The solution is the new firmware upgrade for the BMC (Board Management Controller) card.
The attached document (FA report for FWA-5020 BMC Non-Responding Issue_v1.pdf) is the root cause analysis document provided by Advantech.
3. Interim Solution
You could try executing the following script (provided by Advantech) on the appliance [fwa-5020_reboot_bmc.sh]
https://versanetworks.box.com/s/o5qisrd3uzv9yrycli4z6wtgr21rpbe9
- If the BMC is already in a bad state, then this script will try to recover the BMC.
- If the script cannot recover the BMC, then a cold reboot of the box is required.
4. Permanent Solution
If you want to use ipmitool, Advantech recommends a firmware upgrade for the BMC card. Once the firmware is upgraded, ipmitool can be used from 16.1R2S8.1/S9 or above, using the linux shell.
Following is the BMC software update provided by Advantech
https://versanetworks.box.com/s/n8ex9tyiojms7bzl7arx0at10wya8qxn
BMC firmware can be upgraded on a live system.
The attached document (BMC Firmware V1.08 Upgrade SOP 07222019.pdf) is the BMC firmware upgrade procedure provided by Advantech. Please use this procedure to update the BMC firmware of FWA-5020
