vSphere Post-Install configuration automation with PowerCLI

The Problem: manual configuration of vSphere with many hosts

Automation-NoThankYouLately in my job it happened that I had to configure a vSphere environment for a customer without the luxury of Host Profiles or Distributed Switch but still had a significant number of hosts to configure, anything more than 2 starts to be annoying, isn’t it 🙂 ?

Let’s face it, not everyone can afford Enterprise Plus! Simple as that right? Now what’s the best way to get rid of annoying repetitive tasks? Alan Renouf would suggest me Automation with PowerCLI it’s a no-brainer so I’ve asked St. Google and immediately found a lot of resources and interesting articles, one of which captured my attention: Back To Basics: Post-Configuration of vCenter 5.5 Install (PowerCLI) by Mike Laverick.

In there Mike is doing a great job creating datacenters, adding existing configured hosts to vCenter reading from a .csv file, as well as licensing them. So I thought: that’s awesome but I want to push it little bit more! Actually I don’t want to manually configure the networking of my ESXi, I want to automate otherwise what’s the point? Now, to give you an example, in a typical FlexPod deployment I’m used to work with, each ESXi has about 10 vmnics, a minimum of 8 ports groups, NFS and iSCSI connectivity, non-default settings including overriding Port Groups and VMkernel port binding. This is a lot of manual operations on a single host, multiply this by say just 8 hosts (1 UCS chassis fully populated) and you’re likely to “lose” a couple of hours just to make sure everything is configured correctly, not to mention how error-prone this process is. But hey, you got the idea right? I needed to automate as much as possible of this process.

My bespoke solution: a PowerCLI script and an Excel spreadsheet

SBespokeo I started to analyse the problem as I wanted the best fit for purpose solution; I really liked the idea of having a single configuration file in which I could put all the deployment settings (as per customer’s environment) so, after spending some good “mumbling time” turned out an Excel spreadsheet was the best option.

The Spreadsheet

The way I’ve organised the spreadsheet is by separating the “global settings” from “specific network setting”. For example, if you have 20 hosts there’s no need to repeat somewhere 20 times that your domain is domain.local so the domain suffix is a global (or common) setting. On the other hand something like the vMotion VMkernel ip address is something unique per host so has to be repeated. So I ended up with the following Global Settings and Network Settings (they are on the same worksheet I just divided the screenshot for easy of reading)

Global SettingsNetwork Settings

As you can see I have defined some delimiters (BeginGlobal, EndGlobal, BeginNetwork, EndNetwork) and the reason for this is simple: I can easily change number of hosts or add another global setting (should I want to) without too much hassle, all I need to do is add or remove rows then shift the delimiter, job done! That means my PowerCLI script is clever enough to know where to start and finish reading the settings. Each line represent a Key-Value pair, the key is marked with bold font, which is then loaded in the code using an hash table array whereby every value can be obtained by referencing the Key.

For example, on the following line the value srvcenter02 is obtained in the script using $GS[“vCenterServer”] where $GS is the hash table array and vCenterServer the actual “index”.

Assumptions and caveats

There are some assumptions and caveats for my script to work:

  • Because I’m leveraging the “Excel Object Model” to open the workbook, Microsoft Excel needs to be installed on the machine where you’re running the script from. If you want to run the script on any machine you’ll need to use ActiveX Data Objects (ADO) on either the classic COM ADO or the most performing ADO.Net
  • ESXi has been installed and a management IP address has been assigned;
  • The root password reflects what’s in the spreadsheet;
  • The default “VM Network” port group has not been deleted;
  • The default “Management Network” port group has not been renamed or deleted;
  • Jumbo frames used on iSCSI, NFS and vMotion traffic;
  • Standard MTU used on Virtual machines traffic and vSphere management;
  • vMotion, FT and Virtual machines traffic is load balancing using “Load Based on IP Hash” algorithm;
  • Management, NFS and iSCSI traffic use “Route Based on the Originating Virtual Port ID” load balancing algorithm;
  • I am booting from local disks;
  • ESXi hosts are not joined the domain;
  • ESXi hostnames DNS resolution is in place, that is DNS server forwards and reverse lookup zones are configured;
  • vCenter Server is already installed and configured but licenses have not been installed yet;
  • Cluster configuration is out of scope at the moment of writing the article;
  • The Excel workbook contains sensible information such as root and domain user’s password in clear text and vSphere licenses; remember it’s meant to be an engineer installation workbook so, although I recognise it’s not ideal re. security, this topic was not meant to be covered originally.

The code explained

Functions

As I don’t like to repeat multiple times the piece of code, I’ve grouped it in as many functions as I could. The following is a summary of the functions

  • FindStringGetPosition given a string and a column where to search, returns the row and column position if matching is found
  • RempveAllSpaces given a string, remove all spaces, whether those are at the beginning, end or inside the string
  • AddQuotes given a string, add quotes at the begin and end of it
  • SplitString given a string, split it using a given delimiter and return the wanted substring
  • GetVMnics given a comma separated list of 2 vmnics, return an array of vmnics
  • ToFQDN given an hostname and suffix, returns the fully qualified domain name
  • AddLicense given a string containing a vSphere license key (serial), add it to vCenter Server
  • Get-LicenseKey given a vSphere license name, return the corresponding license key
  • Get-VMHostId given a string hostname of a host, return the corresponding vCenter API object UUID
  • Set-LicenseKey given an host ID and license key, assigns the license to the host
  • Get-License given an host ID, return license details (key, type and host)

I’m not really a developer so I’m sure the code can be optimised, if you see snippets that should be changed or can be written in a better way please let me know by commenting the article. Thank you!

# Functions =============================================================================

Function FindStringGetPosition {
Param ($StringToFind, $ColumnToSearch)

 $RowMatch = 0
 $ColumnMatch = 0
  if ($ColumnToSearch -eq $null) {
   $ColumnMax = ($Worksheet.UsedRange.Columns).Count
  }
  else { $ColumnMax = $ColumnToSearch }

 for($CurrentRow = 1 ; $CurrentRow -le $RowMax ; $CurrentRow++) {
  for ($CurrentColumn = 1; $CurrentColumn -le $ColumnMax; $CurrentColumn++) {
    if ($WorkSheet.Cells.Item($CurrentRow,$CurrentColumn).Value2 -eq $StringToFind ) {
     $RowMatch = $CurrentRow
     $ColumnMatch = $CurrentColumn
     break
    }
  }
 }

Return $RowMatch, $ColumnMatch
}

Function RemoveAllSpaces {
Param ($InputString)

 Return $InputString -Replace '\s+', ''

}

Function AddQuotes {
Param ($InputString)

 Return '"'+$InputString+'"'

}

Function SplitString {
Param ($InputString,$Delimiter,[int]$Position)

Return ($InputString -split $Delimiter)[$Position]
}

Function GetVMnics {
Param ($InputString)

  $a = SplitString $InputString ',' 0
  $b = SplitString $InputString ',' 1
  $vSwitch_Mgmt_Uplinks = ($a, $b)

Return $vSwitch_Mgmt_Uplinks
}

Function ToFQDN {
Param ($Hostname,$Suffix)

 $FQDN = RemoveAllSpaces ($Hostname+"."+$Suffix)
Return $FQDN
}

Function AddLicense {
Param ([String]$SerialNumber)

 $si = Get-View ServiceInstance
 $LicManRef=$si.Content.LicenseManager
 $LicManView=Get-View $LicManRef
 $License = New-Object VMware.Vim.LicenseManagerLicenseInfo
 $License.LicenseKey = $SerialNumber
 $LicManView.AddLicense($License.LicenseKey,$null)

}

# The following functions by Damian Karlson https://damiankarlson.com
Function Get-LicenseKey {
Param ($LicName)

 $licenses = $LicMgr.Licenses | Where {$_.Name -eq $LicName}
 foreach ($license in $licenses) {
   if ( (($license.Total - $license.Used) -ne "0") -or (($license.Total - $license.Used) -lt "0") ) {
    return $license.LicenseKey
    break
   }
 }
}

Function Get-VMHostId {
Param ($Name)

  $vmhost = Get-VMHost $Name | Get-View

Return $vmhost.Config.Host.Value
}

function Set-LicenseKey {
Param ($VMHostId, $LicKey, $Name)

 $license = New-Object VMware.Vim.LicenseManagerLicenseInfo
 $license.LicenseKey = $LicKey
 $LicAssMgr.UpdateAssignedLicense($VMHostId, $license.LicenseKey, $Name)
}

function Get-License {
Param ($VMHostId)

  $details = @()
  $detail = "" |select LicenseKey,LicenseType,Host
  $license = $LicAssMgr.QueryAssignedLicenses($VMHostId)
  $license = $license.GetValue(0)
  $detail.LicenseKey = $license.AssignedLicense.LicenseKey
  $detail.LicenseType = $license.AssignedLicense.Name
  $detail.Host = $license.EntityDisplayName
  $details += $detail

Return $details
}

# End Functions ============================================================

Variables

I then define my variables, including source xls file and delimiters for begin/end of the sections.

# Variables================================================
$strPath="C:\Deployment\vSphereDeploymentSettings.xls"
$objExcel=New-Object -ComObject Excel.Application
$objExcel.Visible=$False
$TabName = 'HostNetworkConfig'
$WorkBook = $objExcel.Workbooks.Open($strPath)
$Worksheet = $workbook.sheets.item("HostNetworkConfig")
$RowMax = ($worksheet.UsedRange.Rows).Count
$ColumnMax = ($Worksheet.UsedRange.Columns).Count
$BeginGlobalSettingsString = "BeginGlobal"
$BeginNetworkSettingsString = "BeginNetwork"
$EndGlobalSettingsString = "EndGlobal"
$EndNetworkSettingsString = "EndNetwork"
$DefaultMgmtPortGroup = "Management Network"
$DefaultVMNetwork = "VM Network"
# End Variable =============================================

Main

I start by loading into variables the position (row and column) of my delimiters

# Find Cell(Row,Column) position begin/end settings configuration
$BeginGlobalSettingsPosition = FindStringGetPosition $BeginGlobalSettingsString 1
$EndGlobalSettingsPosition = FindStringGetPosition $EndGlobalSettingsString 1
$BeginNetworkSettingsPosition = FindStringGetPosition $BeginNetworkSettingsString 1
$EndNetworkSettingsPosition = FindStringGetPosition $EndNetworkSettingsString 1

I then proceed with the following (the code is commented so it’s easy to follow)

  • read the Global Setting from the spreadsheet into an hash table array
  • build FQDN for vCenter Server and connect to it
  • create the datacenter object
  • add the vSphere licenses
  • initialise the License Manager modules
  • add all the ESXi hosts to vCenter Server and license them
# Load the global settings into hash table array
$GS = @{"Attribute" = "Value"}
 for($CurrentRow = $BeginGlobalSettingsPosition[0] + 1 ; $CurrentRow -le $EndGlobalSettingsPosition[0] - 1; $CurrentRow++) {
   $GS[$WorkSheet.Cells.Item($CurrentRow,1).Value2] = RemoveAllSpaces $WorkSheet.Cells.Item($CurrentRow,2).Value2
 }

# Build FQDN for vCenter
$vCenterServer = ToFQDN $GS["vCenterServer"] $GS["DomainSuffix"]
# Connect to vCenter
Connect-VIServer -Server $vCenterServer -User $GS["DomainUser"] -Password $GS["DomainPassword"] -SaveCredentials

# Create datacenter
New-Datacenter -Location (Get-Folder -NoRecursion) -Name $GS["Datacenter"]
# Add Licenses
AddLicense $GS["vCenterLicense"]
AddLicense $GS["vSphereLicense"]

# Initialise the License Manager modules
$ServInst = Get-View ServiceInstance
$LicMgr = Get-View $ServInst.Content.LicenseManager
$LicAssMgr = Get-View $LicMgr.LicenseAssignmentManager
# Given the license serial number I extract the name, for example "VMware vSphere 5 Enterprise", rather than type it manually
$LicName = $LicMgr.Licenses | Where {$_.LicenseKey -eq $GS["vSphereLicense"]} | % {$_.Name}
$LicKey = Get-LicenseKey -LicName $LicName

# Add ESXi hosts to the datacenter and license them
for($CurrentRow = ($BeginNetworkSettingsPosition[0]+2) ; $CurrentRow -le ($EndNetworkSettingsPosition[0]-1) ; $CurrentRow++) {
  $ESXiHost = ToFQDN $WorkSheet.cells.item($CurrentRow,1).Value2 $GS["DomainSuffix"]
  Add-VMHost $ESXiHost -Location $GS["Datacenter"] -User root -Password $GS["RootPwd"] -Confirm:$false -Force:$true
  #Given -RunAsync doesn't wait for the host to be "Connected" here I wait until it is, otherwise Set-LicenseKey will fail
  While ( (Get-VMHost $ESXiHost).ConnectionState -eq "Disconnected") {"Waiting for host $ESXiHost to be Connected"}
  $VMHostId = Get-VMHostId $ESXiHost
  Set-LicenseKey -LicKey $LicKey -VMHostId $VMHostId -Name $null
}

Once all the hosts have been added to vCenter, I start the configuration, reading through the workbook and configuring each host individually. All the code that follows is within two for loop, the first one iterates through the lines, the second through the columns.

# After all hosts have been added and licensed, I load the network settings from
# the spreadsheet into an array and execute the host configuration

$NS = @("Host Network Settings")
for($CurrentRow = $BeginNetworkSettingsPosition[0]+2 ; $CurrentRow -le $EndNetworkSettingsPosition[0]-- ; $CurrentRow++) {
  for ($CurrentColumn = $BeginNetworkSettingsPosition[1]; $CurrentColumn -le $ColumnMax; $CurrentColumn++) {
   $NS += RemoveAllSpaces $WorkSheet.cells.item($CurrentRow,$CurrentColumn).Value2

    # Host configuration here

  }
}

$objExcel.Quit()
Disconnect-VIServer -Server $vCenterServer -Confirm:$false

This is the expanded section I’ve marked as # Host configuration here, which I’m breaking down in smaller parts for easy of comment.

  • build host FQDN
  • configure DNS settings
  • configure NTP settings
# Build host FQDN
$ESXiHost = ToFQDN $NS[1] $GS["DomainSuffix"]
$VMHostObj = Get-VMHost $ESXiHost

# Configure DNS settings
$DNS_Servers = ($GS["DNS1"], $GS["DNS2"])
Get-VMHostNetwork $ESXiHost | Set-VMHostNetwork -DomainName $GS["DomainSuffix"] -DnsAddress $DNS_Servers -DnsFromDhcp:$false -SearchDomain $GS["DomainSuffix"]

# Configure NTP settings
$NTP_ServersList = ($GS["NTPServer1"], $GS["NTPServer2"])
Get-VMHost -Name $ESXiHost | Get-VMHostService | Where {$_.key -eq 'ntpd'} | Stop-VMHostService -Confirm:$false
$CurrentNTPServerList = Get-VMHostNtpServer -VMHost $ESXiHost
Remove-VMHostNtpServer -VMHost $ESXiHost -NtpServer $CurrentNTPServerList -Confirm:$false
Add-VMHostNtpServer -VMHost $ESXiHost -NtpServer $NTP_ServersList -Confirm:$false
# Start ntpd service
Get-VMHost -Name $ESXiHost | Get-VMHostService | Where {$_.key -eq 'ntpd'} | Start-VMHostService -Confirm:$false
Set-VMHostService -HostService (Get-VMHostservice -VMHost $ESXiHost | Where-Object {$_.key -eq "ntpd"}) -Policy "Automatic"
  • if Enabled, disable IPv6
  • add second vmnic to the management port group
  • create and configure vMotion and FT vSwitches
  • create and configure Virtual Machine traffic vSwitch
# If enabled, disable IPv6 , reboot required!
if (Get-VMHostNetworkAdapter -VMHost $ESXiHost -VMKernel | Select IPv6Enabled) { Get-VMHostNetworkAdapter -VMHost $ESXiHost -VMKernel | Set-VMHostNetworkAdapter -IPv6Enabled:$false -Confirm:$false }
#If still present, remove default port group "VM Network"
if (Get-VirtualPortGroup -VMHost $ESXiHost | Where {$_.Name -eq $DefaultVMNetwork}) { Get-VirtualPortGroup -VMHost $ESXiHost -Name $DefaultVMNetwork | Remove-VirtualPortGroup -Confirm:$false }

# Management Network
#Add second vmnic uplink to the existing vSwitch0, vmnic0 being the first
$Uplinks = GetVMnics $GS["Mgmt_Uplinks"]
Get-VirtualSwitch -VMHost $ESXiHost -Name $GS["Mgmt_vSwitch"] | Set-VirtualSwitch -Nic $Uplinks -Mtu $GS["MTUDefault"] -Confirm:$false
$VMHostObj | Get-VirtualPortGroup -Name $DefaultMgmtPortGroup | Set-VirtualPortGroup -Vlanid $GS["Mgmt_VLAN"] -Name $GS["Mgmt_Label"]

# vMotion
$Uplinks = GetVMnics $GS["vMotion_Uplinks"]
$vSwitch1 = New-VirtualSwitch -VMHost $ESXiHost -Name $GS["vMotion_vSwitch"] -NumPorts $GS["vSwitchPorts"] -Nic $Uplinks -Mtu $GS["MTUJumbo"] -Confirm:$false
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["vMotion_vSwitch"] -PortGroup $GS["vMotion_Label"] -IP $NS[4] -SubnetMask $NS[5] -Mtu $GS["MTUJumbo"] -VMotionEnabled:$true -Confirm:$false
$VMHostObj | Get-VirtualPortGroup -Name $GS["vMotion_Label"] | Set-VirtualPortGroup -Vlanid $GS["vMotion_VLAN"]
Get-NicTeamingPolicy -VirtualSwitch $vSwitch1 | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceIP -Confirm:$false
# FT: in my design I am using the same vSwitch for FT and vMotion, using different VLANS
$Uplinks = GetVMnics $GS["FT_Uplinks"]
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["FT_vSwitch"] -PortGroup $GS["FT_Label"] -IP $NS[6] -SubnetMask $NS[7] -Mtu $GS["MTUJumbo"] -FaultToleranceLoggingEnabled:$true -Confirm:$false
$VMHostObj | Get-VirtualPortGroup -Name $GS["FT_Label"] | Set-VirtualPortGroup -Vlanid $GS["FT_VLAN"]

# Virtual machine traffic
$Uplinks = GetVMnics $GS["VMTraffic_Uplinks"]
$vSwitch2 = New-VirtualSwitch -VMHost $ESXiHost -Name $GS["VMTraffic_vSwitch"] -NumPorts $GS["vSwitchPorts"] -Nic $Uplinks -Mtu $GS["MTUDefault"] -Confirm:$false
Get-NicTeamingPolicy -VirtualSwitch $vSwitch2 | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceIP -Confirm:$false
New-VirtualPortGroup -VirtualSwitch $vSwitch2 -Name $GS["VMTraffic_Label"] -VLanID $GS["VMTraffic_VLAN"] -Confirm:$false

  • add and configure iSCSI vSwitch
  • activate iSCSI software initiator
  • execute iSCSI VMkernel port binding

# iSCSI
$Uplinks = GetVMnics $GS["iSCSI_Uplinks"]
$vSwitch4 = New-VirtualSwitch -VMHost $ESXiHost -Name $GS["iSCSI_vSwitch"] -NumPorts $GS["vSwitchPorts"] -Nic $Uplinks -Mtu $GS["MTUJumbo"] -Confirm:$false
Get-NicTeamingPolicy -VirtualSwitch $vSwitch4 | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -Confirm:$false
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["iSCSI_vSwitch"] -PortGroup $GS["iSCSI1_Label"] -IP $NS[8] -SubnetMask $NS[9] -Mtu $GS["MTUJumbo"] -Confirm:$false
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["iSCSI_vSwitch"] -PortGroup $GS["iSCSI2_Label"] -IP $NS[10] -SubnetMask $NS[11] -Mtu $GS["MTUJumbo"] -Confirm:$false
# Set vmkernel port labels and vlan id
$VMHostObj | Get-VirtualPortGroup -Name $GS["iSCSI1_Label"] | Set-VirtualPortGroup -Vlanid $GS["iSCSI1_VLAN"]
$VMHostObj | Get-VirtualPortGroup -Name $GS["iSCSI2_Label"] | Set-VirtualPortGroup -Vlanid $GS["iSCSI2_VLAN"]
Get-VirtualPortGroup -VMHost $ESXiHost -VirtualSwitch $GS["iSCSI_vSwitch"] -Name $GS["iSCSI1_Label"] | Get-NicTeamingPolicy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -MakeNicActive $Uplinks[0] -MakeNicUnused $Uplinks[1] -Confirm:$false
Get-VirtualPortGroup -VMHost $ESXiHost -VirtualSwitch $GS["iSCSI_vSwitch"] -Name $GS["iSCSI2_Label"] | Get-NicTeamingPolicy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -MakeNicActive $Uplinks[1] -MakeNicUnused $Uplinks[0] -Confirm:$false

# Add iSCSI Software Adapter
Get-VMHostStorage -VMHost $ESXiHost | Set-VMHostStorage -SoftwareIScsiEnabled:$true -Confirm:$false

# iSCSI PortBinding
Get-VMHostNetworkAdapter -VMKernel
$portname = Get-VMHost -Name $ESXiHost | Get-VMHostNetworkAdapter | Where {$_.PortGroupName -match "iSCSI-*"} | %{$_.DeviceName}
$vmhba = Get-VMHostHba -VMHost $ESXiHost -Type iscsi | %{$_.Device}
$esxcli = Get-EsxCli -VMHost $ESXiHost
$esxcli.iscsi.networkportal.add($vmhba, $false, $portname[0])
$esxcli.iscsi.networkportal.add($vmhba, $false, $portname[1])
Get-VMHostStorage -VMHost $ESXiHost -RescanVmfs -RescanAllHba
  • finally add and configure NFS vSwitch

# NFS
$Uplinks = GetVMnics $GS["NFS_Uplinks"]
$vSwitch3 = New-VirtualSwitch -VMHost $ESXiHost -Name $GS["NFS_vSwitch"] -NumPorts $GS["vSwitchPorts"] -Nic $Uplinks -Mtu $GS["MTUJumbo"] -Confirm:$false
Get-NicTeamingPolicy -VirtualSwitch $vSwitch3 | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -Confirm:$false
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["NFS_vSwitch"] -PortGroup $GS["NFS1_Label"] -IP $NS[12] -SubnetMask $NS[13] -Mtu $GS["MTUJumbo"] -Confirm:$false
New-VMHostNetworkAdapter -VMHost $ESXiHost -VirtualSwitch $GS["NFS_vSwitch"] -PortGroup $GS["NFS2_Label"] -IP $NS[14] -SubnetMask $NS[15] -Mtu $GS["MTUJumbo"] -Confirm:$false
# set vmkernel port labels and vlan id
$VMHostObj | Get-VirtualPortGroup -Name $GS["NFS1_Label"] | Set-VirtualPortGroup -Vlanid $GS["NFS1_VLAN"]
$VMHostObj | Get-VirtualPortGroup -Name $GS["NFS2_Label"] | Set-VirtualPortGroup -Vlanid $GS["NFS2_VLAN"]
Get-VirtualPortGroup -VMHost $ESXiHost -VirtualSwitch $GS["NFS_vSwitch"] -Name $GS["NFS1_Label"] | Get-NicTeamingPolicy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -MakeNicActive $Uplinks[0] -MakeNicUnused $Uplinks[1] -Confirm:$false
Get-VirtualPortGroup -VMHost $ESXiHost -VirtualSwitch $GS["NFS_vSwitch"] -Name $GS["NFS2_Label"] | Get-NicTeamingPolicy | Set-NicTeamingPolicy -LoadBalancingPolicy LoadBalanceSrcId -MakeNicActive $Uplinks[1] -MakeNicUnused $Uplinks[0] -Confirm:$false

And that’s it! The more hosts you have the more time you can spend at the coffee machine waiting for it script to complete 🙂

Here attached a copy of:

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

1 Trackback