Running VMware ESXi inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. Virtualization enthusiasts everywhere have benefited from the ability to run ESXi on ESXi, first introduced with the vSphere 4 release.
VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.
Physical Host Setup
The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:
- Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
- Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
- Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
- Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests
echo 'vhv.allow = "TRUE"' >> /etc/vmware/config
Virtual VMware ESXi Machine (vESXi) Creation
For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.
Create a new VM with the following guidance:
- Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
- 2 virtual sockets, 2+ GB RAM
- 4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
- Thin provisioned virtual disks work fine
- Finish creating the VM, then edit the following settings
- Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
- CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
- Don’t power this VM on — keep it to act as a template
- Clone and install VMware ESXi via ISO image or PXE boot
- Add to vCenter and configure virtual ESXi hosts for action
Nested 64-bit Guests
With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.
Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.
Nested Hyper-V Virtual Machines
It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!
A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!
Do the following to enable Hyper-V on VMware ESXi:
- Add hypervisor.cpuid.v0 = FALSE to the VM configuration
- Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)
For another take, check out William Lam’s post on this topic.
Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.
How are you taking advantage of a virtual vSphere environment?