VMware vSphere can virtualize itself + 64-bit nested guests

Running VMware ESXi inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. Virtualization enthusiasts everywhere have benefited from the ability to run ESXi on ESXi, first introduced with the vSphere 4 release.

VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.

Physical Host Setup

The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:

  • Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
  • Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
  • Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
  • Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests
    echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Virtual VMware ESXi Machine (vESXi) Creation

For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.

Create a new VM with the following guidance:

  • Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
  • 2 virtual sockets, 2+ GB RAM
  • 4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
  • Thin provisioned virtual disks work fine
  • Finish creating the VM, then edit the following settings
    • Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
    • CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
  • Don’t power this VM on — keep it to act as a template
  • Clone and install VMware ESXi via ISO image or PXE boot
  • Add to vCenter and configure virtual ESXi hosts for action

Nested 64-bit Guests

With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.

Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.

Nested Hyper-V Virtual Machines

It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!

A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!

Do the following to enable Hyper-V on VMware ESXi:

  • Add hypervisor.cpuid.v0 = FALSE to the VM configuration

  • Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)

For another take, check out William Lam’s post on this topic.

Parting Thoughts

Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.

How are you taking advantage of a virtual vSphere environment?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s