Ultimate automation guide to deploying VCSA 6.0 Part 0

With vSphere 6.0, there is a new deployment model for vCenter Server which is comprised of following two core components:

  • Platform Services Controller (PSC) Node – Provides VMware Infrastructure services such as vCenter Single Sign-On, vSphere Licensing and VMware Certificate Authority Management (VCMA)
  • vCenter Server Management Node – Provides vCenter Server Service, Inventory Service, vSphere Web Client, vPostgres DB, vSphere Syslog Collector, vSphere Auto Deploy, and vSphere Dump Collector Services

From these two components, there are three deployment types (also shown in the diagrams below):

  1. Embedded Node – Both the Platform Services Controller and vCenter Server Management Node reside on a single system, this is true for both the Windows vCenter Server and the VCSA
  2. External Platform Services Controller Node – You can deploy multiple PSC’s and configure them with independent SSO Domains or have them all joined to a single SSO Domain, replicating between each other
  3. vCenter Server Management Node – This requires that you have deployed an external PSC which the vCenter Server can point to

vcsa-6.0-deployment-options-new-2There are currently two supported methods of deploying the VCSA 6.0 Appliance which is using the new HTML based UI (Supported only on Windows) or a new scripted installer method (supports Windows, Mac & Linux). Both of these methods today require direct access to an ESXi host for deployment, which may not work for everyone. What if you want to deploy the new VCSA 6.0 using an existing vCenter Server or running it on top of VMware Fusion or Workstation? Luckily, I spent quite a bit of time going through all these “alternative” deployment methods and documenting the process so that you have a choice on how you would like to test and evaluate vSphere 6 and the new VCSA in your environment.

Disclaimer: Though these alternative deployment options work, they are however not officially supported by VMware. Please use at your own risk.

In the upcoming days, I will be sharing a 4-part blog series for automating the deployment of the new VCSA 6.0 with the following deployment options:

In each article, I will provide resources on how to deploy to an existing vCenter Server or directly to an ESXi host using ovftool via a shell script as well using PowerCLI, deploying to VMware Fusion and deploying to VMware Workstation. Stay tune for Part 1 …

Reklamlar

Ultimate automation guide to deploying VCSA 6.0 Part 1: Embedded Node

In this article, I will share alternative methods of deploying the new VCSA 6.0 using an Embedded Node configuration. Take a look at the various deployment methods below and their respective instructions for more details.
embedded-vcsa-6.0
Disclaimer: Though these alternative deployment options work, they are however not officially supported by VMware. Please use at your own risk.

Deploying to an existing vCenter Server using ovftool (shell script)

I have created a shell script called deploy_vcsa6_embedded_to_vc.sh which requires using ovftool 4.1 (included in the VCSA ISO) to specify the appropriate OVF “guestinfo” properties for an Embedded configuration. You will need to edit the script and modify several variables based on your environment.

Here is an example of executing the script:

vcsa-6.0-embedded-deployment

Deploying to an ESXi host using ovftool (shell script)

I have created a shell script called deploy_vcsa6_embedded_to_esxi.sh which requires using ovftool 4.0 or greater to specify the appropriate OVF “guestinfo” properties for an Embedded configuration. You will need to edit the script and modify several variables based on your environment. The behavior of this script is similar to the one above, except you are deploying directly to an ESXi host.

Deploying to an existing vCenter Server using ovftool (PowerCLI)

I have created a PowerCLI script called Deployment-Embedded.ps1 which uses ovftool to specify the appropriate OVF “guestinfo” properties for an Embedded configuration. You will need to edit the script and modify several variables based on your environment.

Deploying to VMware Fusion & Workstation

To properly deploy the new VCSA 6.0, the proper OVF properties MUST be set prior to the booting of the VM. Since VMware Fusion and Workstation do not support OVF properties, you will need to manually deploy the VCSA, but not power it on. Once the deployment has finished, you will need to add the following entries to the VCSA’s VMX file and replace it with your environment settings. Once you have saved your changes, you can then power on the VM and the configurations will then be read into the VM for initial setup.

guestinfo.cis.deployment.node.type = “embedded”
guestinfo.cis.vmdir.domain-name = “vghetto.local”
guestinfo.cis.vmdir.site-name = “vghetto”
guestinfo.cis.vmdir.password = “VMware1!”
guestinfo.cis.appliance.net.addr.family = “ipv4”
guestinfo.cis.appliance.net.addr = “192.168.1.54”
guestinfo.cis.appliance.net.pnid = “192.168.1.54”
guestinfo.cis.appliance.net.prefix = “24”
guestinfo.cis.appliance.net.mode = “static”
guestinfo.cis.appliance.net.dns.servers = “192.168.1.1”
guestinfo.cis.appliance.net.gateway = “192.168.1.1”
guestinfo.cis.appliance.root.passwd = “VMware1!”
guestinfo.cis.appliance.ssh.enabled = “true”

For more information, you can take a look at this article here.

Deploying using new scripted install (bonus)

As mentioned earlier, there is also a new scripted installer included inside of the VMware-VCSA ISO under /vcsa-cli-installer which supports Windows, Mac OS X and Linux, but must be connected directly to an ESXi host. There are several templates that are also included within the /vcsa-cli-installer/templates. I thought as a bonus I would also share the template I have been using to deploy an Embedded VCSA 6.0 using a static IP Address which some of you may find useful.

The use the scripted installer, you just need to change into the appropriate OS platform directory (win32,mac or lin64) and there should be a binary called vcsa-deploy. To use this template, you just need to save the JSON to a file and then specify that as the first argument to vcsa-deploy utility.

Here is an example of deploying an Embedded VCSA using the vcsa-deploy scripted installer.

vcsa-6.0-embedded-node-scripted-install

Ultimate automation guide to deploying VCSA 6.0 Part 2: Platform Services Controller Node

In this article, I will share alternative methods of deploying the first Platform Services Controller Node (PSCs) using the VCSA 6.0 appliance. If you are interested in deploying additional PSC instances joined to an existing SSO Domain, stay tune for Part 3 where this will be covered. Take a look at the various deployment methods below and their respective instructions for more details.
psc
Disclaimer: Though these alternative deployment options work, they are however not officially supported by VMware. Please use at your own risk.

Deploying to an existing vCenter Server using ovftool (shell script)

I have created a shell script called deploy_vcsa6_first_psc_to_vc.sh which requires using ovftool 4.1 (included in the VCSA ISO) to specify the appropriate OVF “guestinfo” properties for a PSC deployment. You will need to edit the script and modify several variables based on your environment.

Here is an example of executing the script:

vcsa-6.0-platform-service-controller-node-deployment

Deploying to an ESXi host using ovftool (shell script)

I have created a shell script called deploy_vcsa6_first_psc_to_esxi.sh which requires using ovftool 4.0 or greater to specify the appropriate OVF “guestinfo” properties for a PSC deployment. You will need to edit the script and modify several variables based on your environment. The behavior of this script is similar to the one above, except you are deploying directly to an ESXi host.

Deploying to an existing vCenter Server using ovftool (PowerCLI)

I have created a PowerCLI script called Deployment-PSC.ps1 which uses ovftool and specifies the appropriate OVF “guestinfo” properties for a PSC deployment. You will need to edit the script and modify several variables based on your environment.

Deploying to VMware Fusion & Workstation

To properly deploy the new VCSA 6.0, the proper OVF properties MUST be set prior to the booting of the VM. Since VMware Fusion and Workstation do not support OVF properties, you will need to manually deploy the VCSA, but not power it on. Once the deployment has finished, you will need to add the following entries to the VCSA’s VMX file and replace it with your environment settings. Once you have saved your changes, you can then power on the VM and the configurations will then be read into the VM for initial setup.

guestinfo.cis.deployment.node.type = “infrastructure”
guestinfo.cis.vmdir.domain-name = “vghetto.local”
guestinfo.cis.vmdir.site-name = “vghetto”
guestinfo.cis.vmdir.password = “VMware1!”
guestinfo.cis.appliance.net.addr.family = “ipv4”
guestinfo.cis.appliance.net.addr = “192.168.1.60”
guestinfo.cis.appliance.net.pnid = “192.168.1.60”
guestinfo.cis.appliance.net.prefix = “24”
guestinfo.cis.appliance.net.mode = “static”
guestinfo.cis.appliance.net.dns.servers = “192.168.1.1”
guestinfo.cis.appliance.net.gateway = “192.168.1.1”
guestinfo.cis.appliance.root.passwd = “VMware1!”
guestinfo.cis.appliance.ssh.enabled = “true”

For more information, you can take a look at this article here.

Deploying using new scripted install (bonus)

As mentioned earlier, there is also a new scripted installer included inside of the VMware-VCSA ISO under /vcsa-cli-installer which supports Windows, Mac OS X and Linux, but must be connected directly to an ESXi host. There are several templates that are also included within the /vcsa-cli-installer/templates. I thought as a bonus I would also share the template I have been using to deploy the first PSC using a static IP Address which some of you may find useful.

The use the scripted installer, you just need to change into the appropriate OS platform directory (win32,mac or lin64) and there should be a binary called vcsa-deploy. To use this template, you just need to save the JSON to a file and then specify that as the first argument to vcsa-deploy utility.

Here is an example of deploying a PSC using the vcsa-deploy scripted installer.

vcsa-6.0-first-platform-service-controller-scripted-install

Increasing disk capacity simplified with VCSA 6.0 using LVM autogrow

With previous releases of the VCSA, increasing disk capacity was not a very straight forward process. Even though you could easily increase the size of the underlying VMDK while the VCSA was running, increasing the guestOS filesystem was not as seamless. In fact, the process was to add a new VMDK, format it and then copy the contents from the old disk to the new disk as detailed in VMware KB 2056764. This meant with previous releases of VCSX 5.x, you would need to incur downtime of your environment and it could be also be quite significant depending on your familiarity with the steps mentioned in the KB not to mention the time it took to copy the data.

The reason for this unnecessary complexity is that VCSA did not take advantage of a Logical Volume Manager (LVM)for managing its disks. In VCSA 6.0, LVM is now used to make it extremely easy to increase disk capacity while the VCSA is running. VCSA 6.0 further simplifies this by separating out the various functions into their own disk partitions comprised of 11 VMDKs compared to the monolithic design in previous VCSA releases. This not only allows you to increase capacity for specific a partition but you can also now attach specific storage SLA’s using VM Storage Policies on specific VMDKs such as the Database or Log VMDK for example.

In the example below, I will walk through the process of increasing the DB VMDK from the existing 10GB to 20GB while the vCenter Server is still running.

Step 1 – Verify the existing disk capacity using “df -h”

increase-vmdk-in vcsa-01
Step 2 – Increase the capacity on VMDK 6 which represents the DB partition using the vSphere Web/C# Client.

Step 3 – Once the VMDK has been increased, you will need to run the following command in the VCSA which will automatically expand any Logical Volumes that have had their Physical Volumes increased:

vpxd_servicecfg storage lvm autogrow

increase-vmdk-in vcsa-02
Step 4 – Confirm the newly added capacity has been consumed

increase-vmdk-in vcsa-03
If you would like to learn more about the different VMDK structure in the new VCSA 6.0, I will be sharing more details in a future article

vimtop: esxtop for the VCSA 6.0

A couple of weeks back I learned about a really cool new tool called vimtop located in the new VCSA 6.0 from fellow colleague Nick Marshall. If you have ever used esxtop before with ESXi, then you will feel right at home with vimtop which is purpose built to provide performance information and statistics about VCSA and the applications running under it. This will definitely be a handy a tool to be aware of when needing to troubleshoot performance issues or bottlenecks in the VCSA.

Disclaimer: While testing vimtop, I found that some of the command-line options are not currently functional and probably why the current version is at 0.5 with tag of “Alpha”. I have been told vimtop is still in active development and I suspect Engineering wanted to get something out to customers to try out and get feedback as they continue to iterate and add more features.

To launch vimtop, you will need to SSH to a VCSA 6.0 system and type “vimtop” in either the applianceshell or in a regular bash shell.

vimtop0
At first glance, vimtop looks very similar to esxtop but you will quickly notice there are many cool new UI improvements which really makes navigating the interface much simpler. The first thing that should stand out to you is the use of colors to help improve the readability of all the metrics. You will also notice that you can quickly navigate through current list view by either scrolling up and down or side to side using the directional arrow keys. When a item is selected is also clearly highlighted which is a huge plus in my opinion when needing to troubleshoot and watch for a particular entry or stat.

Here is a screenshot selecting a specific row in vimtop, you can also do this for a column as well:

vimtop-1-up-down
There are three primary views in vimtop: Processes, Disks & Networks statistics which can be toggled using keyboard shortcuts. In fact, all navigation is performed through a series of global keyboard shortcuts similar to esxtop. There is actually a quite a few of them and you can quickly see the list by hitting the “h” key at any time for the help menu.

Here is the complete list of keyboard shortcuts for your reference

Keyboard Key Description
esc Clear existing selection and jump back to Process view
w Write the configure out the current settings goes to a configuration file located in vimtop/vimtop.xml
s Set the refresh interval (seconds)
f Display all available CPUs overview
t Display Tasks currently managed by the appliance
g Expand top 4 physical CPUs currently available to the appliance
h Help menu
u Show/Hide the unit headers
i Show/Hide the top line
o Network view
p Pause the screen
l Select a particular column
delete Remove selected column
PgUp/PgDn Select first and last row and scroll to it
Collapse selected item
+ Expand selected item
home/end Select first and last column and scroll to it
left/right arrow Select column
up/down arrow Select row
enter Display more info about a select item
< Move selected column to the left
> Move selected column to the right
k Disk View
m Display memory overview information
n Show/Hide the name headers
c Add new column
d Add selected column in descending order or to switch column to descending order
x Select optimal column width
z Clear sort order
a Add selected column in ascending order or to switch column to ascending order
q Quit
~ Display vimtop in Back/White mode

If you are more of a visual person, I have also created a visual keyboard layout of all the vimtop commands which might be handy to print out and post on your wall. I actually got this awesome idea from one of our internal Wikis and I have created a new layout to match all the commands that are currently in vimtop.

vimtop-shortcut-keys
For each of the three views, you can also add and remove different columns just like you could with esxtop using the “c” character. You can then select or de-select columns by using the spacebar for the metrics you wish to be displayed in the current view.

add-column
I figure it would also be useful to have a table of all the metrics and their definitions as it is a bit difficult to read while in vimtop itself.

PROCESSES

Metric ID Description
PID Process identifier
CMD Command name used to start the process as it is seen by the underlying system
CMDLINE The full command line of this process used during startup
NAME User readable name of the process
THREADS Number of native threads currently running in the process
%CPU (CPU Usage) Current CPU usage in percent for this process
MHZ Current CPU usage in MHz for this process
CPU Total CPU time used by the process during last measurement cycle (sum of cpu.used.system and cpu.used.user)
SYS CPU time spent by process in the system (kernel) routines
USR CPU time spent by process in the user land
%MEM (Memory Usage) Physical memory usage in percent for this process
MEM Physical (resident) memory used by this process
VIRT Total virtual memory size of this process (the complete working set including resident and swapped memory)
SHR Size of the shared code – these are any shared objects (so or DLL) loaded by the process
TEXT Code segment size of the process without any shared libraries
DATA Data segment size of the process (for managed process like JVM this includes the managed code also)
FD Total number of file descriptors opened by the process
FILS Number of all file objects opened by the process (sum of files directories and links)
FILE Number of regular files currently opened by the process
DIR Number of directories currently opened by the process
LNK Number of symbolic links currently opened by the process
DEVS Number of devices (char or block) opened by the process
CHAR Number of descriptors opened to character devices
BLCK Number of descriptors opened to block devices
CHNS Number of all communication channels opened by the process (either sockets or FIFOs)
SCKS Number of sockets (TCP|UDP|raw) currently opened by the process
FIFO Pipes (named or not) opened by the process

DISKS

Metric ID Description
DISK/PART Storage disk / partition identifier
IOS Number of I/O operations currently in progress on this disk (should go to zero)
IOTIME Milliseconds spent doing I/O operations on this disk / partition (increases for a nonzero number of I/O operations)
LAT disk / partition access latency (in milliseconds) calculated using the total amount of time spend doing I/O divided by the total amount of I/O operations done during last measurement interval
READS Number of reads issued to this disk / partition and completed successfully during last measurement interval
RDMRG Adjacent to each other reads on this disk / partition merged for efficiency
READ Number of reads per second issued to this disk / partition
RDSCTRS Number of sectors read successfully from this disk / partition during last measurement interval
WRITES Number of writes issued to this disk / partition and completed successfully during last measurement interval
WRMRG Adjacent to each other writes on this disk / partition merged for efficiency
WRITE Number of writes per second issued to this disk / partition
WRSCTRS Number of sectors wrote successfully to this disk / partition during last measurement interval

NETWORKS

Metric ID Description
INTF Interface name
TRGPT Total throughput of this interface (Rx + Tx) in kilobytes
RATE The activity of this network interface in kBps
RXED Amount of data (in kilobytes) received during last measurement interval
RXRATE Rate of received data through this interface in kBps
TXED Amount of data (in kilobytes) transmitted during last measurement interval
TXRATE Rate of data transmission through this interface in kBps
RXMCAST Number of multicast packets received on this interface during last measurement interval
RXDROP Number of data rx-packets dropped during last measurement interval
TXDROP Number of data packets dropped upon transmission during last measurement interval
DROPPED Number of dropped packets through this network interface because of running out of buffers during last measurement cycle
ERRS Total number of faults (Tx and Rx) on this interface
RXERRS The sum of receive errors rx-fifo errors and rx-frame errors
TXERRS The sum of transmit errors tx-fifo errors and carrier errors
FIFOERRS FIFO overrun errors on this interface caused by host being busy to serve the NIC hardware
CLLSNS Collisions detected on the transmission medium

There is definitely a lot more to explore in vimtop, but hopefully this provides a good reference point on quickly getting started. I have to say I really like a lot of the UI enhancements to vimtop, especially the ability to select and quickly watch a particular process. Hopefully some of these enhancements can make its way into esxtop to provide the same set of functionality in the future.

Dynamic memory resizing for vCenter Server 6.0

In previous releases of vSphere, scaling up resources such as storage or memory for vCenter Server was a huge pain-point for our customers. Before the various vCenter Server services can consume the new resources, some additional manual steps were required. Though this type of an operation is usually infrequent, there is still an operational overhead which can potentially lead to increased downtime of your vCenter Server.

For example, increasing storage capacity for the VCSA was an offline operation that required adding an additional disk and then copying the existing content to the new disk which can be quite error prone and lead to a significant amount of downtime. In vSphere 6.0, the VCSA now uses LVM which provides the ability for online storage capacity increase without any downtime to vCenter Server. Increasing memory was also challenging because you had to manually adjust several configuration files that manages the JVM heap settings for various vCenter Server services as described in this VMware KB. Having complex workflows to perform basic resource expansion can increase risk of errors, especially when the process is foreign to those performing it for the very first time.

To help solve this problem, in vSphere 6.0 vCenter Server (Windows & VCSA) now includes a built-in dynamic memory reconfiguration process that automatically runs at boot up. This process includes a dynamic algorithm that inspects the current amount of CPU, Memory and Storage that is available to determine the appropriate size to configure the vCenter Server. This means that if you no longer have to tweak individual JVM settings for the various services within vCenter Server, this will happen automatically by analyzing the resources that are available and then calculating the configuration based on the supported maximums for vCenter Server.

Note: In vSphere 6.0, there are additional services going beyond just the core vCenter Server, vSphere Web Client, vCenter SSO and Inventory Services.

The dynamic memory algorithm is configured to understand the minimal amount of resources for running a vCenter Server and is bounded between a “Tiny” configuration which is 2vCPU and 8GB memory and a “Large” configuration which is 16vCPU and 32GB memory. This is important to note because if you try to configure the vCenter Server with less memory than the minimal supported, the algorithm will default to the smallest configuration which could then lead to performance degradation as swapping would occur.

The process which does all the magic is a utility called cloudvm-ram-size and there are several useful options to be aware of. To view the current memory assignment for the various vCenter Server services including the OS, you can run the following command on the VCSA as an example:

cloudvm-ram-size -l

Screen Shot 2015-02-14 at 9.07.52 AM
From the screenshot above, we can see a very simple break down of the current memory assignment for a “Tiny” deployment which has 8GB of memory.

To show that the dynamic memory algorithm is in fact running when more memory is added, the example below is of a VCSA that was initially configured with 8GB of memory. I then capture the running configuration and then shut down the vCenter Server and increased its memory to 10GB. I then power on the VCSA and capture the running state and you can see differences in the screenshot below.

Screen Shot 2015-02-14 at 8.51.16 AM
Another useful command to be aware is being able to see the current memory usage for all services. You can do this by running the following command:

cloudvm-ram-size -S

Screen Shot 2015-02-14 at 9.08.28 AM
As you can see the dynamic memory algorithm is a very much welcome feature for vCenter Server and will greatly simplify the operational tasks when needing to scale up or down resources such as CPU and Memory. I know this is definitely one of the enhancements I have been waiting for and I am glad to see it here in the new vSphere 6.0 release! As of right now, a system reboot is required but who knows maybe in the future we can increase memory while the VCSA is still running and simply reloading the services

Easily manage ESXi & VCSA SSH login banner & MOTD in vSphere 6.0

For customers who have a requirement to configure an SSH login banner and/or message of the day (MOTD) for ESXi or vCenter Server, this usually meant manually editing the /etc/issues (login banner) and /etc/motd (MOTD) configuration files. In vSphere 6.0, this has now been simplified by providing vSphere APIs to allow administrators to easily view and configure SSH login banner and MOTD for both ESXi and VCSA (this does not apply to vCenter Server for Windows).

For ESXi, there are two new ESXi Advanced properties called Config.Etc.issue and Config.Etc.motd and this can also be viewed and edited using the vSphere Web/C# Client as seen in the screenshot below:

esxi-issues-motd
For vCenter Server, there are also two new Advanced Options called etc.issue and etc.motd which can also be viewed and edited using the vSphere Web/C# Client as seen in the screenshot below:

vcsa-motd-issues

What’s Displayed

The /etc/issues displays a banner during an SSH login session, this means that you will see the message during the login as highlighted in the screenshot below:

esxi-issues
The /etc/motd displays a banner after a successful SSH login as highlighted in the screenshot below:

esxi-motd

How to Automate

You can use both the vSphere API/CLI to modify the issues and motd configurations. In the examples below, I will be using PowerCLI to demonstrate retrieving and setting the two properties.

Retrieve /etc/issues for VCSA

Get-AdvancedSetting -Entity $vc -Name “etc.issue” | Format-List

Set /etc/issues for VCSA

$msg = ”
beware
you
are
entering
the
house
of
Cormac
the
grey

Get-AdvancedSetting -Entity $vc -Name “etc.issue” | Set-AdvancedSetting -Value $msg

Retrieve /etc/motd for VCSA

Get-AdvancedSetting -Entity $vc -Name “etc.motd” | Format-List

Set /etc/motd for VCSA

Get-AdvancedSetting -Entity $vc -Name “etc.motd” | Set-AdvancedSetting -Value $msg

Retrieve /etc/issues for ESXi

Get-VmHostAdvancedConfiguration -Name “Config.Etc.issue”

Set /etc/issues for ESXi

$msg = ”
beware
you
are
entering
the
house
of
Cormac
the
grey

Set-VMHostAdvancedConfiguration -Name “Config.Etc.issue” -Value $msg

Retrieve /etc/motd for ESXi

Get-VmHostAdvancedConfiguration -Name “Config.Etc.issue”

Set /etc/motd for ESXi

Set-VMHostAdvancedConfiguration -Name “Config.Etc.motd” -Value $msg