Shellshock / BASH VMware Patches

I’m sure everyone is aware of yet another vulnerability that was discovered just the other week and this time it is targeted at the BASH shell that is on almost every unix system. As this pertains to our VMware environments, there is the good, the bad and the ugly.

The Good:

Ever since VMware went to ESXi for the vSphere platform, they have removed underlying dependency on a full blown Unix environment and vSphere hosts now run the Ash shell (busybox) for commands. This encompasses all versions of ESXi (vSphere). Another good part about this is that many of these patches are very easy to apply and don’t take a lot of work. I patched a number of test virtual appliances within 20-30 minutes.

Since this issue is primarily Unix based, all VMware products that run on Windows systems are not affected.

The Bad:

Older ESX (non-integrated) environments are susceptible to this vulnerability need to be patched and VMware has released an update to fix both ESX 4.0 and 4.1 systems. Two KB articles have been released to address these releases and are 2090853 for ESX 4.0 and 2090859 for ESX 4.1. Both contain a zip file for download that is roughly 2mb in size and do require a reboot to implement. The patch process is done through the ‘esxupdate’ command and has the following syntax:

#esxupdate – update

Alternatively, you can use VUM (VMware Update Manager) to deliver the patch to the host if it is managed by vCenter. This is done in the traditional manner.

*I would seriously consider upgrading to ESXi if your hardware supports it to eliminate these types of issues in the future. I love vSphere 4.x as much as the next person, but there are so many advancements in 5.x that are worth the upgrade – especially if you’re paying for SnS on those systems! Check the VMware support matrix for compatibility.

The Ugly:

Nearly all virtual appliances are affected. The VMware Security Advisories page released VMSA-2014-0010.5 to address all known products and patches. Many of the patches are delivered through a .pak format and can be easily uploaded and applied to those VA’s through the web management console.

Virtual appliances like vCloud Automation Center need to be patched through a more traditional manner. I’m picking on this one since it requires a more lengthy process due to the nature of how the appliance runs. The process for these types of VA’s are:

  1. Take a snapshot of all vm’s associated with the virtual appliance (if they are in a vApp delivery model or not)
  2. Using your favorite SCP program, upload the Zip file to the VA.
  3. Extract the contents to a temp folder on the appliance.
  4. Install the patch through the RPM installer: rpm -Uvh <patch>.rpm
  5. Restart the appliance
  6. Verify that the patch has been installed and that the appliance is functioning correctly.
  7. Remove the snapshot!

Some VA’s re really easy and the patch can be applied simply be logging into the Web UI, navigating to the update tab, clicking on the “check updates” radio button and then selecting “install”. That was easy!


Treating your Virtual Infrastructure like a Physical Datacenter

Many companies have very strict rules on who can enter the datacenter and your VMware infrastructure should not be any different! Sure there are various levels of access in datacenters, I’ve been in quite a few and I sleep better at night knowing that the necessary precautions are taken to¬† secure these facilities. Jason Boche wrote a great article back in 2009 that describes in detail, how the security model works in vCenter and within the article, shows some of the pitfalls of providing too much access for what seems to be minimal rights.

What we need to understand is that the virtual infrastructure should be protected in the same manner that we employ in the physical world. If the wrong person got in as an administrator, it could spell disaster for your entire infrastructure/datacenter. The following recommendations may seem too stringent for some folks, but we should not simply give certain access just to “get things done”. VMware vCenter has some really nice predefined templates that you can use to minimize the attack footprint while allowing different levels of administrators the ability to do their job, but always double-check what permissions are granted with them.

Here are a few guidelines you should follow:

  • Only give a few people full administrator rights to your environment. Treat this like the Enterprise Admin account of your Active Directory forest (if you run Windows).
  • Service accounts and scripted tasks should use an account with the bare minimum they need to carry out their task. Don’t give them admin rights because its easy.
  • Do not allow SSH access as ROOT and turn off SSH when it is not needed. vCenter should be alerting you when this is turned on and should never be suppressed as this widens the attack footprint of your cluster nodes.
  • vCenter best practices are to provide access via groups instead of individual user access.
  • Be careful in granting access at the root level since this gives users access across multiple vCenter’s if you are running in linked mode.
  • Perform a monthly, quarterly audit to ensure security. Especially if you have more than 2 administrators in your environment.

Changes in the authentication mechanism

vSphere 5.1 – along with vCenter 5.1 comes with a requirement to run the environment with single sign-on (or SSO). The implementation has a few components to it and I’m sure many of you have already tested this out in the lab or have deployed some parts of it already. The following diagram is a depiction of the authentication process and how your credentials (tokens) are sent to endpoint.

VMware SSO Login

With the implementation of SSO, VMware is trying to reign in and protect authentication to the core components of the virtual infrastructure and doing your part with these access control guidelines, you will ensure a stable and secure virtual datacenter.