NSX-T – Manager upgrade failed

I’ve run into an issue where the NSX manager upgrade failed when I’m upgrading from NSX-T version 3.2.2.0.1 to 4.1.0.0.0. I’ve used this workaround to safe my day.

Symptoms

The third NSX Manager failed to update with the following error message.

Unexpected error while upgrading upgrade unit:
Failed to handover upgrade control to another node.
Please ensure that the other nodes are functioning, and retry the upgrade.

When you retry the upgrade it will fail again, and you seem to get stuck at this point.

Workaround

This is by no means an official workaround, but it worked for me: When it works it can safe you time.

1. Log in to console (VM console or SSH) of the third (failed) NSX-T Manager with the admin account.
2. You will receive a message stating that an upgrade is started and that you should not make any changes until the upgrade has been completed.
3. First run the following command to see where the upgrade is stuck:
   get upgrade progress-status [enter]
4. In my case it was stuck at upgrade step:
   “download_os“
5. Also in my case the upgrade on the manager-node has not been started nor paused, so we have to start the upgrade on the manager-node locally.
   Execute the following command
   start upgrade-bundle VMware-NSX-unified-application-<version>playbook VMware-NSX-manager-<version>-rolling-upgrade-playbook [enter]
6. The manager-node upgrade will start and eventually reboot itself.
   Reboot it yourself if you when to have a bad day!
7. The upgrade-coordinator will resume the upgrade-procedure until finished.

Notes from the author

This workaround worked for me in this particular case, it’s worth trying the procedure in other scenario’s where the NSX manager node upgrade failed: it is an supported solution!

In my particular case the manager-node upgrade wasn’t started nor paused: use step 3 to verify your situation.
If the upgrade has been started/paused in your situation you can execute the Resume upgrade-bundle VMware-NSX-unifies-appliance-<version> playbook and restart service install-upgrade commands to resume the upgrade locally.