VCF Skip-level upgrade is a fancy name and a bit misleading as well, in my opinion. You don’t skip any update, the tool simply takes care of all the bundles that go into SDDC Manager, after which you still have to manually apply the VMware bundles (so NSX, vCenter Server and ESXi). In the 4.0 to 4.1 upgrade there are 6 bundles to be applied on SDDC Manager; without skip-level you’d have to do it manually (schedule each bundle one after another). With this tool it does it for you.
Install the lcm tool on a computer with internet access, in my case it’s the jump host.
Need to download all the bundles listed under 4.0 which is the release I’ll be starting the upgrade from 4.0 so the first column are all the bundles I need to download
This list isn’t really telling me what the bundles contains but the following table I created is going to show you what each bundle is
| Bundle ID | Description |
|---|---|
| bundle-24282 | VMware Cloud Foundation Upgrade 4.0.0.0 to 4.0.0.1 contains features, critical bugs and security fixes |
| bundle-25558 | VMware Cloud Foundation Upgrade 4.0.0.1 to 4.0.1.0 contains features, critical bugs and security fixes |
| bundle-25559 | Configuration drift bundle for 4.0.1 update. |
| bundle-27565 | VMware Cloud Foundation Upgrade 4.0.1.0 to 4.0.1.1 contains critical bugs and security fixes |
| bundle-30441 | VMware Cloud Foundation Upgrade 4.0.1.1 to 4.1.0.0 contains features, critical bugs and security fixes |
| bundle-30442 | Configuration drift bundle for 4.1.0 update |
| bundle-29985 | VMware Software Upgrade contains NSX-T 3.0.2 update |
| bundle-29651 | VMware Software Upgrade contains VMware vCenter Server 7.0 Update 1 |
| bundle- 29988 | VMware Software Upgrade contains VMware ESXi 7.0 Update 1 |
1) Download the bundle transfer & skip level utility
from a machine with internet access https://my.vmware.com/en/group/vmware/downloads/details?downloadGroup=VCF410_TOOLS&productId=1061
2) extract it
and create a folder structure similar to the following, in my case I called the root container folder “lcm-tools-skip-level-4.x” because I had other versions at root vcf-bundles
3) download all bundles use the following command:
./lcm-bundle-transfer-util --op f:\local-repo\vcf-bundles\lcm-tools-skip-level-4.x\bundles -d --du VMwareDownload@dell.com -b bundle-24282
The LCM bundle transfer utility will take care of downloading not just the actual bundle itself (.tar file) but also the manifest metadata (.manifest) and the manifest signature (.manifest.sig).
That said, I have experienced many times problems downloading big bundles using the utility. I don’t know what’s causing the problem but the download basically breaks with a connection reset (see here)
If that happens, I download the same bundle from a browser. It’s much faster and does not break with connection reset. The URLs are as following for each bundle:
https://depot.vmware.com/PROD2/evo/vmw/bundles/bundle-30442.tar
https://depot.vmware.com/PROD2/evo/vmw/manifests/bundle-30442.manifest
https://depot.vmware.com/PROD2/evo/vmw/manifests/bundle-30442.manifest.sig
Now repeat this step until for all bundles are downloaded.
In the case of a 4.0 to 4.1 upgrade we’re talking about 53GB in total. Once you’re done move to the next step, grouping all the files together
4) group all the files
that is bundles, manifest files and manifest signature files under the same root folder, such as
Skip Level Update
Now that you have all the bits needed let’s make sure backups and snapshots are in place.
As usual I’m going to snapshot manually the SDDC infrastructure components (SDDC manager, vCenter Server, NSX-T Manager) as well as exporting a backup of NSX-T Manager. I’m then going to skip the backups from LCM (as I did in my previous article VCF 4.0.1 to 4.1 LCM Update) so that I don’t get stuck during the snapshot with memory operation.
5) perform the usual precheck from SDDC Manager
6) Initiate the skip-level upgrade
from the same machine where you downloaded the bundles, connectivity is required to SDDC Manager
tasks running. the script is basically going to upload each bundle from your jump host to SDDC Manager
there is it running automatically, first bundle completed with success and moved to 4.0.1.0
These are the full logs:
PS F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\lcm-tools-prod\bin> .\sddcmanager-skip-level-upgrade -d -u -s 2021-01-23 17:16:13.537+0000 INFO Welcome to the VMware Cloud Foundation SDDC Skip Level Upgrade Tool, Version: 4.1.0-vcf4100RELEASE-17284351 2021-01-23 17:16:13.538+0000 INFO It supports SDDC Manager services upgrade from version 4.0.0.x to 4.1.0.0 2021-01-23 17:16:13.541+0000 INFO Log file: F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\lcm-tools-prod\skiplevelupgrade.log Please enter the FQDN of SDDC Manager VM: sddc-manager-2.vcf-s1.vlabs.local Please enter the primary user on SDDC Manager VM: vcf Please enter the basic auth user on SDDC Manager VM: admin Please enter password of the primary user: Please enter root password of SDDC Manager VM: Please enter basic auth password of SDDC Manager VM: 2021-01-23 17:16:54.519+0000 INFO Target VCF version is set to 4.1.0.0 2021-01-23 17:16:54.520+0000 INFO Perform pre-validation before SDDC Manager Skip Level Upgrade 2021-01-23 17:16:54.654+0000 INFO Pre-validation succeeded 2021-01-23 17:17:09.500+0000 WARN Download for bundle with ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc is not in success state 2021-01-23 17:17:09.510+0000 WARN Download for bundle with ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 is not in success state 2021-01-23 17:17:09.517+0000 WARN Download for bundle with ID a2a2af3a-88f4-4f85-be57-92608e4846a9 is not in success state Please enter the depot user: VMwareDownload@dell.com Please enter the depot user password: Please enter the directory for downloaded bundles: F:\local-repo\vcf-upgrades\lcm-tools-skip-level-4.x\bundles 2021-01-23 17:17:43.446+0000 INFO Download required bundles and copy them to SDDC Manager VM 2021-01-23 17:17:57.976+0000 INFO Copy bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 to SDDC Manager VM 2021-01-23 17:23:46.651+0000 INFO Copy bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 to SDDC Manager VM 2021-01-23 17:28:46.085+0000 INFO Copy bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 to SDDC Manager VM 2021-01-23 17:28:52.716+0000 INFO Copy bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 to SDDC Manager VM 2021-01-23 17:33:38.129+0000 INFO Copy bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d to SDDC Manager VM 2021-01-23 17:42:46.561+0000 INFO Copy bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd to SDDC Manager VM 2021-01-23 17:42:54.164+0000 INFO Copy bundle bundle-29985, ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc to SDDC Manager VM 2021-01-23 17:46:04.149+0000 INFO Copy bundle bundle-29651, ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 to SDDC Manager VM 2021-01-23 17:48:44.535+0000 INFO Copy bundle bundle-29988, ID a2a2af3a-88f4-4f85-be57-92608e4846a9 to SDDC Manager VM 2021-01-23 17:49:25.614+0000 INFO Successfully downloaded and copied all required bundles to SDDC Manager VM 2021-01-23 17:49:46.311+0000 INFO Skip creating SDDC Manager VM snapshot 2021-01-23 17:49:46.312+0000 INFO Check and upload required cumulative upgrade bundle bundle-29985, ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc 2021-01-23 17:49:46.411+0000 INFO Uploading bundle bundle-29985 with ID 4cc5f8fb-bc5f-4fe9-9ce9-bfd3795a44dc 2021-01-23 17:58:50.455+0000 INFO Check and upload required cumulative upgrade bundle bundle-29651, ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 2021-01-23 17:58:50.471+0000 INFO Uploading bundle bundle-29651 with ID ff21d28a-cb1f-447e-91ec-b2861cd43fd8 2021-01-23 18:05:26.860+0000 INFO Check and upload required cumulative upgrade bundle bundle-29988, ID a2a2af3a-88f4-4f85-be57-92608e4846a9 2021-01-23 18:05:26.871+0000 INFO Uploading bundle bundle-29988 with ID a2a2af3a-88f4-4f85-be57-92608e4846a9 2021-01-23 18:05:57.218+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 2021-01-23 18:05:57.240+0000 INFO Uploading bundle bundle-24282 with ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 2021-01-23 18:27:28.759+0000 WARN Bundle fb468d09-bdb4-4834-a762-953a4d688b23 will be deleted 2021-01-23 18:28:40.386+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-23 18:28:41.688+0000 INFO Bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83, for SDDC Manager 4.0.0.0 to 4.0.0.1 upgrade is available for upgrade 2021-01-23 18:28:41.688+0000 INFO Schedule upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 2021-01-23 18:28:42.173+0000 INFO Wait for upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 2021-01-23 19:08:53.857+0000 INFO Upgrade with bundle bundle-24282, ID 6b5d5a0e-5bdc-4679-82c3-860bd660ea83 completed with status: COMPLETED_WITH_SUCCESS 2021-01-23 19:08:53.857+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 2021-01-23 19:08:53.895+0000 INFO Uploading bundle bundle-25558 with ID a188cedb-e56f-482d-8a13-7daa36ff7d46 2021-01-23 19:29:54.372+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-23 19:29:55.753+0000 INFO Bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46, for SDDC Manager 4.0.0.1 to 4.0.1.0 upgrade is available for upgrade 2021-01-23 19:29:55.753+0000 INFO Schedule upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 2021-01-23 19:29:56.391+0000 INFO Wait for upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 2021-01-23 20:10:05.875+0000 INFO Upgrade with bundle bundle-25558, ID a188cedb-e56f-482d-8a13-7daa36ff7d46 completed with status: COMPLETED_WITH_SUCCESS 2021-01-23 20:10:05.875+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 2021-01-23 20:10:05.907+0000 INFO Uploading bundle bundle-25559 with ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 2021-01-23 20:12:53.337+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-23 20:15:48.829+0000 INFO Bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6, for VCF 4.0.0.1 to 4.0.1.0 configuration drift is available for upgrade 2021-01-23 20:15:48.829+0000 INFO Schedule upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 2021-01-23 20:15:49.142+0000 INFO Wait for upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 2021-01-23 20:17:54.502+0000 INFO Upgrade with bundle bundle-25559, ID f1465cdc-c72d-49f2-976c-f68eca41f1e6 completed with status: COMPLETED_WITH_SUCCESS 2021-01-24 09:50:02.964+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 2021-01-24 09:50:03.142+0000 INFO Uploading bundle bundle-27565 with ID 06970cd8-def6-4337-a99c-a3917f5db976 2021-01-24 10:08:35.952+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-24 10:08:37.361+0000 INFO Bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976, for SDDC Manager 4.0.1.0 to 4.0.1.1 upgrade is available for upgrade 2021-01-24 10:08:37.361+0000 INFO Schedule upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 2021-01-24 10:08:37.916+0000 INFO Wait for upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 2021-01-24 10:34:44.369+0000 INFO Upgrade with bundle bundle-27565, ID 06970cd8-def6-4337-a99c-a3917f5db976 completed with status: COMPLETED_WITH_SUCCESS 2021-01-24 10:34:44.370+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d 2021-01-24 10:34:44.411+0000 INFO Uploading bundle bundle-30441 with ID 5851343d-a1bb-494b-b9d2-306f9382327d 2021-01-24 10:55:05.996+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-24 10:55:07.882+0000 INFO Bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d, for SDDC Manager 4.0.1.1 to 4.1.0.0 upgrade is available for upgrade 2021-01-24 10:55:07.882+0000 INFO Schedule upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d 2021-01-24 10:55:08.467+0000 INFO Wait for upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d 2021-01-24 11:33:18.415+0000 INFO Upgrade with bundle bundle-30441, ID 5851343d-a1bb-494b-b9d2-306f9382327d completed with status: COMPLETED_WITH_SUCCESS 2021-01-24 11:33:18.787+0000 INFO Check and upload required VMware Cloud Foundation bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd 2021-01-24 11:33:18.840+0000 INFO Uploading bundle bundle-30442 with ID 556e99b8-6863-4038-b9f2-616f6d97c8dd 2021-01-24 11:36:07.697+0000 INFO Check available VMware Cloud Foundation upgrade 2021-01-24 11:36:10.119+0000 INFO Bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd, for VCF 4.0.1.1 to 4.1.0.0 configuration drift is available for upgrade 2021-01-24 11:36:10.120+0000 INFO Schedule upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd 2021-01-24 11:36:10.801+0000 INFO Wait for upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd 2021-01-24 11:39:16.584+0000 INFO Upgrade with bundle bundle-30442, ID 556e99b8-6863-4038-b9f2-616f6d97c8dd completed with status: COMPLETED_WITH_SUCCESS 2021-01-24 11:39:16.585+0000 INFO Successfully applied all required bundles 2021-01-24 11:39:16.592+0000 INFO Enable VMWARE_SOFTWARE cumulative upgrade to VCF 4.1.0.0 2021-01-24 11:41:37.738+0000 INFO Successfully enabled VMWARE_SOFTWARE cumulative upgrade to VCF 4.1.0.0 2021-01-24 11:41:39.037+0000 INFO Restore bundle manifest polling flag 2021-01-24 11:42:28.380+0000 INFO Restore bundle manifest polling flag 2021-01-24 11:44:23.843+0000 INFO Skip deleting SDDC Manager VM snapshot 2021-01-24 11:44:23.843+0000 INFO If the VCF installation is not connected to VMware depot, please connect to it. If this is a dark site installation, please use bundle transfer utility to download and upload any applicable bundles. 2021-01-24 11:44:23.843+0000 INFO SDDC Manager skip level upgrade completed with success
Once you’re done with all the SDDC bundles we still need to apply the VMware bundles, and this process is the same as a non-skip level one.
See my previous article VCF 4.0.1 to 4.1 LCM Update
NSX-T 3.0.2
The fist time you apply this NSX-T bundle, the following components will be upgraded:
- upgrade coordinator
- edge cluster
- hosts
you then need to re-apply the bundle in order to upgrade the Manager as last.
My nested enviroment it took 46 mins just for the manager, so have patience!
vCenter Server 7.0 Update 1
Next goes the vCenter Server, that being with embedded PSC makes it pretty straight forward and relatively quick
This time I didn’t have any of the problems mentioned in the VCF 4.0.1 to 4.1 LCM Update
Not only that but also vSphere Cluster Service VMs were successfully deployed.
ESXi 7.0 Update 1
Lastly we need to update the ESXi hosts. Now, unless you make some changes to LCM, performing an ESXi update will fail with the following error:
Running the SDDC Manager SOS health checks will also display the following hardware warnings which are related, it’s the same “problem”
Because we’re running nested, we need to do to bypass the HCL checks. We do this by setting to false the following lcm properties
Please note: I’m not only disabling HCL checks here but also vSAN healthchecks and prechecks, for the same reason that we’re running nested and will create even more unnecessary errors.
and here’s the SOS health check after changing the LCM properties
Once that’s done we can finally get the ESXi upgrade running. However the next problem I encountered was that the host, after successfully updating, would not come out of maintenance mode automatically.
Couldn’t figure out anything wrong with the host; even lcm-debug.log wouldn’t display any error. Next thing I tried was performing the same ESXi upgrade using VUM from vCenter; this way excluding VCF LCM as a potential root cause. And indeed I was getting the same problem: the host would enter maintenance mode automatically, update, reboot and remain in maintenance mode without exiting.
So I was stuck! I asked for help and I went straight to some friends at the VMware VCF Technical Marketing team; my good old buddy Kevin Tebear knew the root cause and how to fix my problem.
It turns out that VMware vSphere Skyline Health is checking a lot of vSAN and vSphere cluster health settings, discovering environmental issues and suggeting best practices by analyzing telemetry data collected from the vSphere environment. Well, long story short, and bear in mind this is only applicable for a nested environment, I had to silence every warning I was getting from Skyline Health in order for the ESXi host to successfully and automatically exit maintenance mode after the upgrade.
Disable all Skyline warnings by silencing every alert.
Repear the test until you get no more warnings:
and that was it, after this my ESXi update went smooth
and this completes the 4.0 to 4.1 offline skip-level VCF LCM update for the management workload domain. I hope you find this article useful !
1 Trackback