Virtualization Field Day 2 – Symantec Visit

We had two main presenters for this session. George Winter, a Technical Product Manager in the Information Management Group at Symantec in Mountain View, CA and Gareth Fraser-King was also there who is in the Global Technical Field Enablement division in the the United Kingdom.

Both speakers and had a deep understanding of the product lines they referred to and here are some of the highlights on the opening remarks:

  • They have been a partner with VMware for nearly 12 years
  • V-Ray Technologies “best describes the product lines”
  • Visibility into virtual machines and applications
  • Transparent backup and recovery across physical and virtual machines
  • Third generation with granular recovery
  • An entry point into the cloud

Problems that they are solving:

  • Machine restores are placed back on to the original location or alternate location, just like any other restore software.
  • “Design Criteria” – NetBackup Ecosystem? They talked about backing up various OS’s to allow a seamless backup environment.
  • Replication Director coming out in 7.5 as well as vAPP integration with vSphere and vCD.
  • Backup Data Path with the SCSI LUN locking potential if backups consume the pipe and lock the LUN

They showed a few slides on host utilization during a backup and the difference when the products are used, but I was interested to see the memory load on the ESX host in a stateless compute environment.

They said that their Benchmark results – -has only been limited by hardware.

VIP – VMware Intelligent Policy – VMware Protection on Autopilot

  • Alternate method of backing up new and moved vm’s
  • Designed for dynamic environment – vMo, DRS, sDRS, etc
  • Query engine can be taylored to protect VMware environments

VIP will balance the load of backups over the storage path (FC or File Based) – as well as VMware attributes: Folder, resource pool or vApp. I asked the question on backup windows since they are throttling the streams our of each ESX server and how they combat that. They said that they are reliant on aggressive DRS policy setup at the cluster level.

They also had a 1/3 rack in the room to show off the NetBackup appliance. This was the Symantec NetBackup 5220:

I found this link for the product on the web:

Shortly after the noon hour, they talked about the NetBackup Accelerator in NetBackup 7.5

This is a client based backup product that does change block tracking within the OS on the VMware environment. It also provides a single file restore through a catalog that is created on a per vm basis. They say that this is fundamentally different that other solutions that require a full restore and then the file list located. There is a “proprietary technology” (agent type) to do this and must be installed within the OS. Tape restore is done the same way and selected data is pulled from tape in the same fashion.

How is this agent managed? This is usually done through their management utility, but can be done through other facets such as SCCM, etc .

All in all, this was a great meeting / presentation from George Winter and Gareth Fraser-King and a great session to kick off Virtualization Field Day 2!




HP Tech Day 2012

The other week (Jan 26th and 27th to be specific) I made a trip out to the HP Research facility in Ft. Collins Colorado to get a first-hand look at HP’s storage offerings and took a tour of their research datacenter there. In attendance were 10 other bloggers and industry analysts, some that I’ve talked to prior but many whom I’ve met for the first time. Calvin Zito (or otherwise known as @HPStorageGuy on Twitter) was our excellent host who made sure that we stuck to the schedule and prepared us for the presentations. This event was completely paid for by HP and brought all of us in as independent bloggers for an outside view of their products and solutions.

The first presentation was delivered by HP’s Craig Nunes to go over the highlights of  their Converged Storage offerings. During his presentation, he kept driving home that they are investing in the “Elimination of Boundaries”  with the New HP Storage line. This is where the term “Big Data” came up a few times and how products like 3PAR will be an enabler of cloud deployments going forward.

The HP Storage Virtual SAN Appliance (VSA) P4000 is Lefthand running in a virtual machine that is completely oblivious to what is running underneath it. Data is striped across all nodes in the VSA which they say is dissimilar to other technologies that use the other nodes as backups for the primary. A few of the delegates raised the the question a few times regarding support for solid state drives, and they said that “SSD Support in the P4000 is currently being looked at”.

Another solution that was presented was the HP StorageWorks P4800 blade system model that allows one JBOD to serve two nodes. There is current support for up to 70TB in one “Node”. This unit is built as a VDI Solution.

HP’s StoreOnce Family

My favorite session was of course the VMware integration with HP’s product line and was eager to find out support matrix information with these new products and offerings. I did find out that VAAI is supported on every HP array with the exception of the P6000 EVA that has not been released yet. Another interesting note is that VMware’s certification for metro clusters (aka vSphere Metro Storage Cluster) has support for iSCSI targets on the HP P4000. HP also has support for recovery and snapshot solutions with 3PAR ‘s via the “Recovery Manager Software for vSphere” and HP’s P4000 LeftHand solutions through the “Application Aware Snapshot Manager”.

Along the line of vSphere Storage API integration with Storage Awareness (aka VASA), I thought a flowchart was in order here to give a visual representation of how this works:


Guest Speaker:

Another guest speaker on Friday was a fellow named Aboubacar Diare (a master architect) who discussed VMware’s adaptive queue depth algorithm that resolves around response time issues seen on VMware cluster nodes. He referenced KB article 1008113 where 3PAR worked with VMware on this particular issue that resolves SCSI BUSY and QUEUE FULL conditions where ESX becomes unresponsive. This happens when consolidation levels are too high for the number of heavy hitters on any one node.

In ESX 3.5 update 4 and higher, VMware released an “adaptive queue depth” algorithm that dynamically changes the LUN queue depth in the VMkernel I/O stack. Pretty cool.

You can read more about this joint effort  here: KB 1008113

The datacenter tour:

HP also lead us on a tour of the datacenter where they showed us many of the products discussed in the racks. They also had a few engineers showing off the adaptive louvers in the floor tiles so they can control airflow throughout. (top) They also showed us a screenshot of how this is depicted from the management console (bottom) where you can see hot and cold sections of the datacenter.

Jim Richardson (JR) who is one of the founding 3PAR Systems Engineers was also on the tour and gave a first-hand look at some of these arrays in action. Here he is showing off one of the 3PAR systems in the datacenter (top). We also got to do a few labs while we were there and they were showing the HP 3PAR Thin Persistence and optimization with VMware’s eager zero thick (EZT for short). This is best depicted in their slide (bottom).


All in all, this was a great trip and I was glad I got to attend and learn more about these products and how they can supplement VMware and datacenter technology in the SMB and Enterprise space.


Storage I/O Control (SIOC) and the VMware Disk Scheduler

There are a number of fundamentals that are pertinent to discuss when looking at how VMware SIOC really works. The fact that it automatically throttles I/O based on load is a great feature in itself, but what really triggers this event and what mechanisms comprise this feature?

In the storage world, there is one enemy of performance that outweighs all the rest and really causes major issues for data access. This is none other than latency. Quite simply put, Storage I/O Control is the prioritization of I/O on every ESX server that leverages a common datastore with additional detection of SAN bottlenecks. It accomplishes this by utilizing the virtual storage stack and giving priority to those vm’s with higher disk shares.

VMware Virtualized Storage Stack

The virtualized storage stack contained within vSphere has basically two disk scheduler parts to it – the host-level and datastore disk scheduler.

Host-Level Disk Scheduler: Virtual machines that reside on the same ESX node have priorities on the I/O traffic only in the event that the local host bus adapter (HBA) becomes overloaded. This type of scheduler has been around since the ESX 3.x days and has configurable limits for I/O throughput.

Datastore (SAN) Disk Scheduler: vSphere 4.1 added this feature that contains two functions and only functions on block based storage (i.e. Fiber Channel or iSCSI).

1) I/O prioritization across all ESX nodes in the cluster.

2) SAN path contention analysis/calculations.

Again, both of these actions are facilitated by the distributed disk scheduler analysis based on each virtual machine’s share value.

Remember, SIOC kicks in only when the latency threshold has been breached (which has a default value of 30ms). I also put together a very basic flowchart of this operation so you can see where the logic is injected in this process.


As an added benefit, SIOC allows for the maximum density per ESX host while maintaining maximum performance on each cluster node.

SIOC Support:

One thing to consider are the threshold settings on SIOC with automatic tiering arrays. This must be adjusted based on the vendors recommendation so that the SAN is not adversely affected by SIOC.

At the time of this writing, there are three things that are not supported by SIOC:

  1. NAS based arrays (NFS in particular)
  2. RDM’s (Raw Device Mappings)
  3. Datastores that have multiple extents.
  4. Datastores that are managed by multiple vCenter servers.

Look for a subsequent posting on my blog about implementation considerations.