Perform an Offline LCM Skip-Level Upgrade from VCF 4.0 to 4.1

VCF Skip-level upgrade is a fancy name and a bit misleading as well, in my opinion. You don’t skip any update, the tool simply takes care of all the bundles that go into SDDC Manager, after which you still have to manually apply the VMware bundles (so NSX, vCenter Server and ESXi). In the 4.0 to 4.1 upgrade there are 6 bundles to be applied on SDDC Manager; without skip-level you’d have to do it manually (schedule each bundle one after another). With this tool it does it for you.

Install the lcm tool on a computer with internet access, in my case it’s the jump host.

Need to download all the bundles listed under 4.0 which is the release I’ll be starting the upgrade from 4.0 so the first column are all the bundles I need to download

This list isn’t really telling me what the bundles contains but the following table I created is going to show you what each bundle is

Bundle ID	Description
bundle-24282	VMware Cloud Foundation Upgrade 4.0.0.0 to 4.0.0.1 contains features, critical bugs and security fixes
bundle-25558	VMware Cloud Foundation Upgrade 4.0.0.1 to 4.0.1.0 contains features, critical bugs and security fixes
bundle-25559	Configuration drift bundle for 4.0.1 update.
bundle-27565	VMware Cloud Foundation Upgrade 4.0.1.0 to 4.0.1.1 contains critical bugs and security fixes
bundle-30441	VMware Cloud Foundation Upgrade 4.0.1.1 to 4.1.0.0 contains features, critical bugs and security fixes
bundle-30442	Configuration drift bundle for 4.1.0 update
bundle-29985	VMware Software Upgrade contains NSX-T 3.0.2 update
bundle-29651	VMware Software Upgrade contains VMware vCenter Server 7.0 Update 1
bundle- 29988	VMware Software Upgrade contains VMware ESXi 7.0 Update 1

1) Download the bundle transfer & skip level utility

from a machine with internet access https://my.vmware.com/en/group/vmware/downloads/details?downloadGroup=VCF410_TOOLS&productId=1061

2) extract it

and create a folder structure similar to the following, in my case I called the root container folder “lcm-tools-skip-level-4.x” because I had other versions at root vcf-bundles

3) download all bundles use the following command:

./lcm-bundle-transfer-util --op f:\local-repo\vcf-bundles\lcm-tools-skip-level-4.x\bundles -d --du VMwareDownload@dell.com -b bundle-24282

The LCM bundle transfer utility will take care of downloading not just the actual bundle itself (.tar file) but also the manifest metadata (.manifest) and the manifest signature (.manifest.sig).

That said, I have experienced many times problems downloading big bundles using the utility. I don’t know what’s causing the problem but the download basically breaks with a connection reset (see here)

If that happens, I download the same bundle from a browser. It’s much faster and does not break with connection reset. The URLs are as following for each bundle:

https://depot.vmware.com/PROD2/evo/vmw/bundles/bundle-30442.tar
https://depot.vmware.com/PROD2/evo/vmw/manifests/bundle-30442.manifest
https://depot.vmware.com/PROD2/evo/vmw/manifests/bundle-30442.manifest.sig

Now repeat this step until for all bundles are downloaded.

In the case of a 4.0 to 4.1 upgrade we’re talking about 53GB in total. Once you’re done move to the next step, grouping all the files together

4) group all the files

that is bundles, manifest files and manifest signature files under the same root folder, such as

Skip Level Update

Now that you have all the bits needed let’s make sure backups and snapshots are in place.

As usual I’m going to snapshot manually the SDDC infrastructure components (SDDC manager, vCenter Server, NSX-T Manager) as well as exporting a backup of NSX-T Manager. I’m then going to skip the backups from LCM (as I did in my previous article VCF 4.0.1 to 4.1 LCM Update) so that I don’t get stuck during the snapshot with memory operation.

5) perform the usual precheck from SDDC Manager

6) Initiate the skip-level upgrade

from the same machine where you downloaded the bundles, connectivity is required to SDDC Manager

tasks running. the script is basically going to upload each bundle from your jump host to SDDC Manager

there is it running automatically, first bundle completed with success and moved to 4.0.1.0

These are the full logs:

PS F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\lcm-tools-prod\bin> .\sddcmanager-skip-level-upgrade -d -u -s
2021-01-23 17:16:13.537+0000 INFO  Welcome to the VMware Cloud Foundation SDDC Skip Level Upgrade Tool, Version: 4.1.0-vcf4100RELEASE-17284351
2021-01-23 17:16:13.538+0000 INFO  It supports SDDC Manager services upgrade from version 4.0.0.x to 4.1.0.0
2021-01-23 17:16:13.541+0000 INFO  Log file: F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\lcm-tools-prod\skiplevelupgrade.log
Please enter the FQDN of SDDC Manager VM: sddc-manager-2.vcf-s1.vlabs.local
Please enter the primary user on SDDC Manager VM: vcf
Please enter the basic auth user on SDDC Manager VM: admin
Please enter password of the primary user:
Please enter root password of SDDC Manager VM:
Please enter basic auth password of SDDC Manager VM:
2021-01-23 17:16:54.519+0000 INFO  Target VCF version is set to 4.1.0.0
2021-01-23 17:16:54.520+0000 INFO  Perform pre-validation before SDDC Manager Skip Level Upgrade
2021-01-23 17:16:54.654+0000 INFO  Pre-validation succeeded
2021-01-23 17:17:09.500+0000 WARN  Download for bundle with ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc is not in success state
2021-01-23 17:17:09.510+0000 WARN  Download for bundle with ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 is not in success state
2021-01-23 17:17:09.517+0000 WARN  Download for bundle with ID a2a2af3a-88f4-4f85-be57-92608e4846a9 is not in success state
Please enter the depot user: VMwareDownload@dell.com
Please enter the depot user password:
Please enter the directory for downloaded bundles: F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\bundles
2021-01-23 17:17:43.446+0000 INFO  Download required bundles and copy them to SDDC Manager VM
2021-01-23 17:17:57.976+0000 INFO  Copy bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 to SDDC Manager VM
2021-01-23 17:23:46.651+0000 INFO  Copy bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 to SDDC Manager VM
2021-01-23 17:28:46.085+0000 INFO  Copy bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 to SDDC Manager VM
2021-01-23 17:28:52.716+0000 INFO  Copy bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 to SDDC Manager VM
2021-01-23 17:33:38.129+0000 INFO  Copy bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d to SDDC Manager VM
2021-01-23 17:42:46.561+0000 INFO  Copy bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd to SDDC Manager VM
2021-01-23 17:42:54.164+0000 INFO  Copy bundle bundle-29985, ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc to SDDC Manager VM
2021-01-23 17:46:04.149+0000 INFO  Copy bundle bundle-29651, ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 to SDDC Manager VM
2021-01-23 17:48:44.535+0000 INFO  Copy bundle bundle-29988, ID a2a2af3a-88f4-4f85-be57-92608e4846a9 to SDDC Manager VM
2021-01-23 17:49:25.614+0000 INFO  Successfully downloaded and copied all required bundles to SDDC Manager VM
2021-01-23 17:49:46.311+0000 INFO  Skip creating SDDC Manager VM snapshot
2021-01-23 17:49:46.312+0000 INFO  Check and upload required cumulative upgrade bundle bundle-29985, ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc
2021-01-23 17:49:46.411+0000 INFO  Uploading bundle bundle-29985 with ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc
2021-01-23 17:58:50.455+0000 INFO  Check and upload required cumulative upgrade bundle bundle-29651, ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8
2021-01-23 17:58:50.471+0000 INFO  Uploading bundle bundle-29651 with ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8
2021-01-23 18:05:26.860+0000 INFO  Check and upload required cumulative upgrade bundle bundle-29988, ID a2a2af3a-88f4-4f85-be57-92608e4846a9
2021-01-23 18:05:26.871+0000 INFO  Uploading bundle bundle-29988 with ID a2a2af3a-88f4-4f85-be57-92608e4846a9
2021-01-23 18:05:57.218+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83
2021-01-23 18:05:57.240+0000 INFO  Uploading bundle bundle-24282 with ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83
2021-01-23 18:27:28.759+0000 WARN  Bundle fb468d09-bdb4-4834-a762-953a4d688b23 will be deleted
2021-01-23 18:28:40.386+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-23 18:28:41.688+0000 INFO  Bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83, for SDDC Manager 4.0.0.0 to 4.0.0.1 upgrade is available for upgrade
2021-01-23 18:28:41.688+0000 INFO  Schedule upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83
2021-01-23 18:28:42.173+0000 INFO  Wait for upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83
2021-01-23 19:08:53.857+0000 INFO  Upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 completed with status: COMPLETED_WITH_SUCCESS
2021-01-23 19:08:53.857+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46
2021-01-23 19:08:53.895+0000 INFO  Uploading bundle bundle-25558 with ID a188cedb-e56f-482d-8a13-7daa36ff7d46
2021-01-23 19:29:54.372+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-23 19:29:55.753+0000 INFO  Bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46, for SDDC Manager 4.0.0.1 to 4.0.1.0 upgrade is available for upgrade
2021-01-23 19:29:55.753+0000 INFO  Schedule upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46
2021-01-23 19:29:56.391+0000 INFO  Wait for upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46
2021-01-23 20:10:05.875+0000 INFO  Upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 completed with status: COMPLETED_WITH_SUCCESS
2021-01-23 20:10:05.875+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6
2021-01-23 20:10:05.907+0000 INFO  Uploading bundle bundle-25559 with ID f1465cdc-c72d-49f2-976c-f68eca41f1e6
2021-01-23 20:12:53.337+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-23 20:15:48.829+0000 INFO  Bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6, for VCF 4.0.0.1 to 4.0.1.0 configuration drift is available for upgrade
2021-01-23 20:15:48.829+0000 INFO  Schedule upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6
2021-01-23 20:15:49.142+0000 INFO  Wait for upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6
2021-01-23 20:17:54.502+0000 INFO  Upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 completed with status: COMPLETED_WITH_SUCCESS
2021-01-24 09:50:02.964+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976
2021-01-24 09:50:03.142+0000 INFO  Uploading bundle bundle-27565 with ID 06970cd8-def6-4337-a99c-a3917f5db976
2021-01-24 10:08:35.952+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-24 10:08:37.361+0000 INFO  Bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976, for SDDC Manager 4.0.1.0 to 4.0.1.1 upgrade is available for upgrade
2021-01-24 10:08:37.361+0000 INFO  Schedule upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976
2021-01-24 10:08:37.916+0000 INFO  Wait for upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976
2021-01-24 10:34:44.369+0000 INFO  Upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 completed with status: COMPLETED_WITH_SUCCESS
2021-01-24 10:34:44.370+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d
2021-01-24 10:34:44.411+0000 INFO  Uploading bundle bundle-30441 with ID 5851343d-a1bb-494b-b9d2-306f9382327d
2021-01-24 10:55:05.996+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-24 10:55:07.882+0000 INFO  Bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d, for SDDC Manager 4.0.1.1 to 4.1.0.0 upgrade is available for upgrade
2021-01-24 10:55:07.882+0000 INFO  Schedule upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d
2021-01-24 10:55:08.467+0000 INFO  Wait for upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d
2021-01-24 11:33:18.415+0000 INFO  Upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d completed with status: COMPLETED_WITH_SUCCESS
2021-01-24 11:33:18.787+0000 INFO  Check and upload required VMware Cloud Foundation bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd
2021-01-24 11:33:18.840+0000 INFO  Uploading bundle bundle-30442 with ID 556e99b8-6863-4038-b9f2-616f6d97c8dd
2021-01-24 11:36:07.697+0000 INFO  Check available VMware Cloud Foundation upgrade
2021-01-24 11:36:10.119+0000 INFO  Bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd, for VCF 4.0.1.1 to 4.1.0.0 configuration drift is available for upgrade
2021-01-24 11:36:10.120+0000 INFO  Schedule upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd
2021-01-24 11:36:10.801+0000 INFO  Wait for upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd
2021-01-24 11:39:16.584+0000 INFO  Upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd completed with status: COMPLETED_WITH_SUCCESS
2021-01-24 11:39:16.585+0000 INFO  Successfully applied all required bundles
2021-01-24 11:39:16.592+0000 INFO  Enable VMWARE_SOFTWARE cumulative upgrade to VCF 4.1.0.0
2021-01-24 11:41:37.738+0000 INFO  Successfully enabled VMWARE_SOFTWARE cumulative upgrade to VCF 4.1.0.0
2021-01-24 11:41:39.037+0000 INFO  Restore bundle manifest polling flag
2021-01-24 11:42:28.380+0000 INFO  Restore bundle manifest polling flag
2021-01-24 11:44:23.843+0000 INFO  Skip deleting SDDC Manager VM snapshot
2021-01-24 11:44:23.843+0000 INFO  If the VCF installation is not connected to VMware depot, please connect to it. If this is a dark site installation, please use bundle transfer utility to download and upload any applicable bundles.
2021-01-24 11:44:23.843+0000 INFO  SDDC Manager skip level upgrade completed with success

Once you’re done with all the SDDC bundles we still need to apply the VMware bundles, and this process is the same as a non-skip level one.
See my previous article VCF 4.0.1 to 4.1 LCM Update

NSX-T 3.0.2

The fist time you apply this NSX-T bundle, the following components will be upgraded:

upgrade coordinator
edge cluster
hosts

you then need to re-apply the bundle in order to upgrade the Manager as last.

My nested enviroment it took 46 mins just for the manager, so have patience!

vCenter Server 7.0 Update 1

Next goes the vCenter Server, that being with embedded PSC makes it pretty straight forward and relatively quick

This time I didn’t have any of the problems mentioned in the VCF 4.0.1 to 4.1 LCM Update

Not only that but also vSphere Cluster Service VMs were successfully deployed.

ESXi 7.0 Update 1

Lastly we need to update the ESXi hosts. Now, unless you make some changes to LCM, performing an ESXi update will fail with the following error:

Running the SDDC Manager SOS health checks will also display the following hardware warnings which are related, it’s the same “problem”

Because we’re running nested, we need to do to bypass the HCL checks. We do this by setting to false the following lcm properties

Please note: I’m not only disabling HCL checks here but also vSAN healthchecks and prechecks, for the same reason that we’re running nested and will create even more unnecessary errors.

and here’s the SOS health check after changing the LCM properties

Once that’s done we can finally get the ESXi upgrade running. However the next problem I encountered was that the host, after successfully updating, would not come out of maintenance mode automatically.

Couldn’t figure out anything wrong with the host; even lcm-debug.log wouldn’t display any error. Next thing I tried was performing the same ESXi upgrade using VUM from vCenter; this way excluding VCF LCM as a potential root cause. And indeed I was getting the same problem: the host would enter maintenance mode automatically, update, reboot and remain in maintenance mode without exiting.

So I was stuck! I asked for help and I went straight to some friends at the VMware VCF Technical Marketing team; my good old buddy Kevin Tebear knew the root cause and how to fix my problem.

It turns out that VMware vSphere Skyline Health is checking a lot of vSAN and vSphere cluster health settings, discovering environmental issues and suggeting best practices by analyzing telemetry data collected from the vSphere environment. Well, long story short, and bear in mind this is only applicable for a nested environment, I had to silence every warning I was getting from Skyline Health in order for the ESXi host to successfully and automatically exit maintenance mode after the upgrade.

Disable all Skyline warnings by silencing every alert.