In the previous article I described what VMware Cloud Foundation 3.9 Multi-Instance is and how to setup the first controller in the federation.
In this article we will see how to configure a second VCF site, first as Member and later on as additional controller.
First VCF Site
From the first site we need to invite the member and generate a token.
- Enter the FQDN for the second site and accept the SSL fingerprint
- here decide if the invited member shall be a Controller or just a Member. I’m going ahead with Member
- Member invite details
Second VCF Site
Moving to the second VCF site, invoke the Join Federation wizard and fill in the form.
Here we go, now the world map view is showing all locations
My second VCF site is (pretending to be) in US and the first one in UK.
Federation Leave
For the sake of knowledge sharing I’m going to show how a Member can leave the federation and I will re-join it as Controller, to see what we see differently.
Head back to the member site, switch the view to grid and click Leave federation
Here you can see the second SDDC Manager has completed the Leave operation
Join Federation as Controller
Head back to the first VCF site and send a new invite for the second site, this time as Controller
Pasting the link will basically auto-complete the FQDN fields as well as the token id
Federation: under the hood
I couldn’t help myself and had to find out where the logs are, as it’s not documented at
https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/com.vmware.vcf.admin.doc_39/GUID-3DA5C89A-DA53-4EB4-91C6-AE39B212F0DC.html
I spotted something I did not notice before called pantheon under /var/log/vmware/vcf and that got my attention.
Looks like that’s the service responsible for the federation and the log file is named pantheon-<SDDC-Manager-FQDN>.log for example:
/var/log/vmware/vcf/pantheon/pantheon-sddc-manager.vcf-s1.vlabs.local.log
In there you will see literally everything the controller cluster members are doing, for example
The system daemon is named (surprise surprise) pantheon
As a quick test shutting down the service on the primary site where the controller member is running will return this message on the UI
API endpoints are also available
For example:
Using the SOS command line utility with option –get-vcf-summary you can see the MULTI_SITE_SERVICE version as well
root@sddc-manager [ /home/vcf ]# /opt/vmware/sddc-support/sos --get-vcf-summary
Welcome to Supportability and Serviceability(SoS) utility!
Summary : /var/log/vmware/vcf/sddc-support/summary-2019-11-02-22-05-02-58618
Summary log : /var/log/vmware/vcf/sddc-support/summary-2019-11-02-22-05-02-58618/sos.log
SDDC Manager
╠══ CEIP
║ ╠══ Id: 658c74d3-7a2e-4ef7-ac25-a4f6db990497
║ ╚══ Status: DISABLED
╠══ Domains
║ ╚══ MGMT
║ ╠══ Clusters
║ ║ ╚══ mgmt01
║ ║ ╠══ # of Hosts: 4
║ ║ ╠══ Datacenter: site1-dc
║ ║ ╠══ Datastore: VSAN
║ ║ ╠══ FTT: 1
║ ║ ╠══ Resource Utilization
║ ║ ║ ╠══ CPU usage in GHz: 21 %
║ ║ ║ ╠══ Memory usage in GB: 52 %
║ ║ ║ ╚══ Storage usage in TB: 36 %
║ ║ ╠══ Status: ACTIVE
║ ║ ╚══ Stretched Cluster: False
║ ╠══ Domain Id: aa867e14-e93c-49a2-baf3-da5974d0863b
║ ╠══ Domain Type: MANAGEMENT
║ ╠══ Esxi Version: 6.7.0-14320388
║ ╠══ NSX Version: 6.4.5-13282012
║ ╠══ NSX-Type: NSX-V
║ ╠══ Name: MGMT
║ ╠══ Status: ACTIVE
║ ╚══ vCENTER Version: 6.7.0-14367737
╠══ Host Details
║ ╠══ Active Hosts: 4
║ ╠══ Dirty Hosts: 0
║ ╚══ Free Hosts: 3
╠══ Licensing Details
║ ╠══ VMware NSX Data Center for vSphere License
║ ║ ╠══ Key: XXXXX-XXXXX-XXXXX-XXXXX-91060
║ ║ ╠══ Product Type: NSXV
║ ║ ╠══ Total: 16
║ ║ ╠══ Used: None
║ ║ ╚══ license Unit: CPU Packages
║ ╠══ VMware vCenter Server License
║ ║ ╠══ Key: XXXXX-XXXXX-XXXXX-XXXXX-CH5HH
║ ║ ╠══ Product Type: VCENTER
║ ║ ╠══ Total: 2
║ ║ ╠══ Used: 1
║ ║ ╚══ license Unit: Server
║ ╠══ VMware vSAN License
║ ║ ╠══ Key: XXXXX-XXXXX-XXXXX-XXXXX-1JJHN
║ ║ ╠══ Product Type: VSAN
║ ║ ╠══ Total: 16
║ ║ ╠══ Used: 4
║ ║ ╚══ license Unit: CPU Packages
║ ╚══ VMware vSphere License
║ ╠══ Key: XXXXX-XXXXX-XXXXX-XXXXX-ATQQQ
║ ╠══ Product Type: ESXI
║ ╠══ Total: 20
║ ╠══ Used: 4
║ ╚══ license Unit: CPU Packages
╠══ Network Pool Details
║ ╚══ NETPOOL-1
║ ╠══ Hosts: 7
║ ╠══ Network Pool ID: 97948de1-6479-4d97-b851-da057e1c9bb4
║ ╚══ Network Type: VMOTION, VSAN
╠══ SDDC Manager Id: 658c74d3-7a2e-4ef7-ac25-a4f6db990497
╠══ SDDC Manager Status: ACTIVE
╠══ SDDC Version: 3.9.0.0-14866160
╠══ Solution
║ ╚══ vRLI
║ ╠══ Domain Association
║ ║ ╚══ 1: MGMT
║ ╠══ Node Size: SMALL
║ ╠══ Status: ACTIVE
║ ╚══ Version: 4.8.0-13036238
╚══ VCF Services
╠══ COMMON_SERVICES
║ ╠══ Memory Utilized: 696.1M
║ ╚══ Version: 3.9.0-RELEASE-14865906
╠══ DOMAIN_MANAGER
║ ╠══ Memory Utilized: 660.8M
║ ╚══ Version: 3.9.0-RELEASE-14865906
╠══ LCM
║ ╠══ Memory Utilized: 1.4G
║ ╚══ Version: 3.9.0-RELEASE-14865906
╠══ MULTI_SITE_SERVICE
║ ╚══ Version: 1.0.38-RELEASE-380
╠══ OPERATIONS_MANAGER
║ ╠══ Memory Utilized: 1.6G
║ ╚══ Version: 3.9.0-RELEASE-14865906
╠══ SDDC_MANAGER_UI
║ ╠══ Memory Utilized: 251.1M
║ ╚══ Version: 3.9.0-RELEASE-14865865
╚══ SOLUTIONS_MANAGER
╠══ Memory Utilized: 1.7G
╚══ Version: 3.9.0-RELEASE-14865906