In this article I’m going to demonstrate the steps required to perform an incremental online LCM update from VMware Cloud Foundation 4.0 to 4.0.1, specifically on the management workload domain.
VCF 4.0.1 released 25/06/2020 and this is release notes link
Note that I used the word incremental to identify that LCM is going to apply each and every bundle in between to get to the target version, without skipping in-between versions. This is in contrast with the skip-level upgrade introduced with 3.10 and 4.0+, for which I’m probably going to write a separate article.
The components that are going to be upgraded are:
- SDDC Prechecks: repeat this step before each bundle is applied
- SDDC Manager 188.8.131.52 (bundle ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83)
- from 184.108.40.206 build 16008466 to 220.127.116.11 build 16221345
- vCenter Server 7.0.0a (bundle ID fb468d09-bdb4-4834-a762-953a4d688b23)
- from 18.104.22.16800 build 15952498 to 22.214.171.12400 build 16189094
- SDDC Manager 126.96.36.199 (bundle ID a188cedb-e56f-482d-8a13-7daa36ff7d46)
- from 188.8.131.52 build 16221345 to 184.108.40.206 build 16428904
- SDDC Manager 220.127.116.11 configuration drift (bundle ID f1465cdc-c72d-49f2-976c-f68eca41f1e6)
- same build is kept 18.104.22.168 16428904. This bundle is doing clean-up and under the hood fixes
- NSX-T 3.0.1 (bundle ID 29753b80-2bf4-4604-85c3-86f49d610153)
- from 22.214.171.124.0 build 15946738 to 126.96.36.199.0 build 16404613
- vCenter Server 7.0.0b (bundle ID bb41c565-2d80-47bb-b4ea-8ed532a003b0)
- from 188.8.131.5200 build 16189094 to 184.108.40.20600 build 16386292
- ESXi 7.0.0b (bundle ID 168ee409-d2b7-40e5-9d77-1dc3cec75e07)
- from 7.0.0 build 15843807 to 7.0.0 build 16324942
1.SDDC System Prechecks
Always perform a precheck before you start, it’s going to highlight any problem that needs fixing. Head over to the management workload domain, tab Update/Patches > PRECHECK button
2.SDDC Manager 220.127.116.11
This bundle is going to update all SDDC Manager services
3.vCenter Server 7.0.0a
Because the PSC is embedded on vCenter, while you perform this upgrade you might lose the SDDC Manager UI since since SDDC Manager authenticates against the built-in PSC. That’s perfectly if does happen you simply wait for vCenter Server to be back online and log back in on SDDC Manager.
4.SDDC Manager 18.104.22.168
This bundle is again, going to update all SDDC Manager services
5.SDDC Manager 22.214.171.124 configuration drift
This is a very quick bundle to apply, in fact it’s merely updating the internal SDDC database, even in my nested lab (supposedly slower than a physical VCF setup) it took less than 2 minutes.
Here we’re select the management domain cluster
Note that the NSX-T Manager does not get upgraded on the same run, in my lab I had to re-apply the same NSX-T bundle in order to get the NSX-T Manager upgraded.
7.vCenter Server 7.0.0b
Note the cluster level selection feature is now available 🙂
I’m running nested and ESXi failed to update with the following error on UI
Checking on /var/log/vmware/vcf/lcm/lcm.log would display a more comprehensive error message (bold)
[vcf_lcm,b07a11c6520e3ec3,cd71,upgradeId=955c13e7-c024-4daa-8e09-ece469302edc,resourceType=ESX_HOST,resourceId=0c5b8771-6928-4196-9a67-6d4840553f5c,bundleElementId=64e176ac-11c0-4c10-b91e-455ad1002100] [c.v.e.s.l.p.i.vsan.VsanHealthService,ThreadPoolTaskExecutor-10] VSAN Health: VSAN HCL Info: The controller device vmhba0:VMware Inc. PVSCSI SCSI Controller that is running with driver pvscsi:0.1-2vmw.700.1.0.15843807 does not show up on HCL.
Set to false the following three LCM options to false in /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
Restart lcm and your’e good to go. However the next issue for was me was that the host did not automatically exit maintenance mode, from lcm.log I could see the following (not so useful) error message.
2020-12-24T09:50:06.633+0000 INFO [vcf_lcm,e1f1fe9882e04d53,d0da,upgradeId=9d650475-dca6-4f2f-bdcc-1762384db355,resourceType=ESX_HOST,resourceId=93dc8b07-5cd3-4c23-9fc5-b504d36d584f,bundleElementId=64e176ac-11c0-4c1oryClient,ThreadPoolTaskExecutor-10] logical inventory - update entity status; type ESXI id 93dc8b07-5cd3-4c23-9fc5-b504d36d584f status ERROR 2020-12-24T09:50:06.635+0000 DEBUG [vcf_lcm,e1f1fe9882e04d53,d0da,upgradeId=9d650475-dca6-4f2f-bdcc-1762384db355,resourceType=ESX_HOST,resourceId=93dc8b07-5cd3-4c23-9fc5-b504d36d584f,bundleElementId=64e176ac-11c0-4c1fiedRestTemplate,ThreadPoolTaskExecutor-10] Writing [InventoryEntityUpdateSpec(type=ESXI, status=ERROR, version=null)] as "application/json" "status": "ERROR"
The VUM baseline was correctly attached temporarily to each node
I couldn’t find what was wrong in my setup so I ended up manually exiting maintenance mode and kick-off the ESXi LCM update bundle again.
If I find out what was causing this problem I will update the article; but for now that’s it! We have successfully completed the 4.0 to 4.0.1 VMware Cloud Foundation Update.