During maintenance on a customer’s VMware VMC on AWS environment, I had an upgrade error with the VMware HCX server mesh appliances. And this was something that was not really documented online, so I wanted to blog about it.
During the upgrade of the VMware HCX service mesh, I got an error. The initial HCX Cloud and Enterprise appliance did successfully upgrade without any issue as expected, but with the interconnect and network extension appliances (IX, NE), I got a strange error about a datastore not being available. Deploy and Configuration of Interconnect Appliances Failed. Interconnect Service Workflow DeployAppliance failed. Error: [“The object ‘vim.Datastore:datastore-43129’ has already been deleted or has not been completely created”]
After googling a bit nothing did help me further with troubleshooting the error. Most of the articles I did find had the solution of removing and readding the VM in vCenter. But as the HCX was still working, I did not want to do this. And this deprecated and old VMware KB on vSphere 4.x .
This was strange to me as the HCX appliances were running on a datastore cluster with all datastores still present. But the customer did mention to me that they recently migrated all their datastores from VMFS5 to VMFS6. This meant that all datastore were removed and recreated with the same names. But the datastore-IDs did change.
How did I eventually fix it… After opening the compute profile and validating everything, I noticed the following.
On the deployment resources screen, the previously chosen datastore had disappeared.
And when validating the configuration we did get a similar error.
So, I added the new VMFS6 datastore to the deployment resource and the validation did complete this time.
To be sure that everything was in order, I did a resync of the service mesh.
This caused a storage vMotion of the existing appliances to the newly selected datastore in the compute profile.
After a successfull vmotion, the resync did do an automatic upgrade of the service mesh appliances.
Just make sure that you have a maintenance window as the upgrade of service mesh appliance has a short cutover.
After the upgrade, my entire service mesh was healthy and fully up to date.
I hope this helps!
Interested in getting your VCAP-DTM design certificate, check out my study guide blog series:
VCAP7-DTM Design study guide – part 1