Updated VSAN 6.0 Nested ESXi OVF Templates for 64 Nodes, All-Flash Array & Fault Domain Testing

During the development of vSphere & VSAN 6.0, I built several new OVF templates to be able to help quickly deploy different configurations of VSAN to help test and provide feedback to Engineering. I figure I would share out these new templates as it may also benefit others in the community who wish to quickly setup VSAN 6.0 and give it spin in their own home lab or development environment using Nested ESXi.

Disclaimer: Nested ESXi is not officially supported by VMware, please use at your own risk.

One thing to be aware of when running VSAN 6.0 using Nested ESXi is that you will need a minimum of 6GB of memory for each ESXi VM. If you wish to run additional workloads on top or create larger VSAN Clusters, you may need to add additional memory on each ESXi VM. This is not really a problem for real physical hardware but can be a problem for smaller lab environments when running Nested ESXi.

Here are the four new VSAN 6.0 Nested ESXi OVF Templates:

Note: If you decide to build a 64-Node Cluster, you will need run the following ESXCLI command (reboot is required) to go beyond 32 nodes up to 64 node:

esxcli system settings advanced set -o /VSAN/goto11 -i 1

Here is a few screenshots of running VSAN 6.0 using the above OVF Templates:


How to configure an All-Flash VSAN 6.0 Configuration using Nested ESXi?

There has been a great deal of interest from customers and partners for an All-Flash VSAN configuration, especially as consumer grade SSDs (eMLC) continue to drop in price and the endurance levels of these devices lasting much longer than originally expected as mentioned in this article by Duncan Epping. In fact, last year at VMworld the folks over at Micron and SanDisk built and demoed an All-Flash VSAN configuration proving this was not only cost effective but also quite performant. You can read more about the details here and here. With the announcement of vSphere 6 this week and VMware Partner Exchange taking place the same week, there was a lot of excitement on what VSAN 6.0 might bring.

One of the coolest feature in VSAN 6.0 is the support for an All-Flash configuration. The folks over at Sandisk gave a sneak peak at VMware Partner Exchange couple weeks back on what they were able to accomplish with VSAN 6.0 using an All-Flash configuration. They achieved an impressive 2 Million IOPs, for more details take a look here. I am pretty sure there are going to be plenty more partner announcements as we get closer to the GA of vSphere 6 and there will be a list of supported vendors and devices on the VMware VSAN HCL, so stay tuned.

To easily demonstrate this new feature, I will be using Nested ESXi but the process to configure an All-Flash VSAN configuration is exactly the same for a real physical hardware setup. Nested ESXi is a great learning tool to understand and be able to walk through the exact process but should not be a substituted for actual hardware testing. You will need a minimum of 3 Nested ESXi hosts and they should be configured with at least 6GB of memory or more when working with VSAN 6.0.

Disclaimer: Nested ESXi is not officially supported by VMware, please use at your own risk.

In VSAN 1.0, an All-Flash configuration was not officially supported, the only way to get this working was by “tricking” ESXi into thinking the SSD’s used for capacity tier are MD’s by creating claimrules using ESXCLI. Though this method had worked, VSAN itself was assuming the capacity tier of storage are regular magnetic disks and hence the operations were not really optimized for anything but magnetic disks. With VSAN 6.0, this is now different and VSAN will optimize based on whether are you using using a hybrid or an All-Flash configuration. In VSAN 6.0, there is now a new property called IsCapacityFlash that is exposed and it allows a user to specify whether an SSD is used for the write buffer or for capacity purposes.

Screen Shot 2015-02-10 at 10.01.12 PM
Step 1 – We can easily view the IsCapacityFlash property by using our handy vdq VSAN utility which has now been enhanced to include a few more properties. Run the following command to view your disks:

vdq -q

From the screenshot above, we can see we have two disks eligible for VSAN and that they both are SSDs. We can also see thew new IsCapacityFlash property which is currently set to 0 for both. We will want to select one of the disk(s) and set this property to 1 to enable it for capacity use within VSAN.

Step 2 – Identity the SSD device(s) you wish to use for your capacity tier, a very simple to do this is by using the following ESXCLI snippet:

esxcli storage core device list  | grep -iE ‘(   Display Name: |   Size: )’

We can quickly get a list of the devices and their ID along with their disk capacity. In the example above, I will be using the 8GB device for SSD capacity

Step 3 – Once you have identified the device(s) from the previous step, we now need to add a new option calledenable_capacity_flash to these device(s) using ESXCLI. There are actually three methods of assigning the capacity flash tag to a device and both provide the same end result. Personally, I would go with Option 2 as it is much simpler to remember than syntax for claimrules :) If you have the ESXi hosts connected to your vCenter Server, then Option 3 would be great as you can perform this step from a single location.

Option 1: ESXCLI Claim Rules

Run the following two ESXCLI commands for each device you wish to mark for SSD capacity:

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d naa.6000c295be1e7ac4370e6512a0003edf -o enable_capacity_flash
esxcli storage core claiming reclaim -d naa.6000c295be1e7ac4370e6512a0003edf

Option 2: ESXCLI using new VSAN tagging command

esxcli vsan storage tag add -d naa.6000c295be1e7ac4370e6512a0003edf -t capacityFlash

Option 3: RVC using new vsan.host_claim_disks_differently command

vsan.host_claim_disks_differently –disk naa.6000c295be1e7ac4370e6512a0003edf –claim-type capacity_flash

Step 4 – To verify the changes took effect, we can re-run the vdq -q command and we should now see our device(s) marked for SSD capacity.

Step 5 – You can now create your VSAN Cluster using the vSphere Web Client as you normally would and add the ESXi host into the cluster or you can bootstrap it using ESXCLI if you are trying to run vCenter Server on top of VSAN, for more details take a look here.

One thing that I found interesting is that in the vSphere Web Client when setting up an All-Flash VSAN configuration, the SSD(s) used for capacity will still show up as “HDD”. I am not sure if this is what the final UI will look like before vSphere 6.0 GA’s.

If you want to check the actual device type, you can always go to a specific ESXi host under Manage->Storage->Storage Devices to see get more details. If we look at our NAA* device ID, we can see that both devices are in fact SSDs.

Hopefully for those of you interested in an All-Flash VSAN configuration, you can now quickly get a feel for that running VSAN 6.0 in a Nested ESXi environment. I will be publishing updated OVF templates for various types of VSAN 6.0 testing in the coming weeks so stay tune.

New vSphere 6.0 APIs for VSAN, VVOLs, NFS v4.1 & more!

vSphere 6.0 includes a number of new storage platform capabilities and new features which includes VSAN 6.0, VVOLS and NFS v4.1 to just name a few. For those of you who are interested in consuming these new capabilities programmatically for Automation, here are some of the new vSphere 6.0 APIs to be aware of from a Storage stand point.

Generic Storage Sub-System

  • HostStorageSystem (Existing Managed Object)
    • Methods to manage and configure NFS v4.1
      • ChangeNFSUserPassword
      • ClearNFSUser
      • QueryNFSUser
      • SetNFSUser
    • Methods to mark a Disk device either Local or Remote
      • MarkAsLocal_Task
      • MarkAsNonLocal_Task
    • Methods to mark a Disk device as either an SSD or Magnetic Disk
      • MarkAsNonSsd_Task
      • MarkAsSsd_Task
    • Methods to turn On or Off the LED for supported Disk devices
      • TurnDiskLocatorLedOn_Task
      • TurnDiskLocatorLedOff_Task
    • Method to issue VAAI UNMAP operation
      • UnmapVmfsVolumeEx_Task
VSAN 6.0
  • VsanUpgradeSystem (New Managed Object) – Methods to upgrade to the new VSAN 6.0 on-disk format
    • PerformVsanUpgrade_Task
    • PerformVsanUpgradePreflightCheck
    • QueryVsanUpgradeStatus
  • HostVsanInternalSystem (Existing Managed Object) – Methods to query & upgrade VSAN Objects
    • QueryVsanObjectUuidsByFilter
    • UpgradeVsanObjects
  • HostVsanSystem (Existing Managed Object) – Methods to manage VSAN Node and disks
    • EvacuateVsanNode_Task
    • RecommissionVsanNode_Task
    • UnmountDiskMapping_Task
  • ComputeResource (Existing Managed Object)
    • faultDomainInfo – New Property to configure Fault Domains
  • HostDatastoreSystem (Existing Managed Object) – Methods to create and delete VVOL Datastores
    • CreateVvolDatastore
    • RemoveDatastoreEx_Task