Installing Nested XenServer on ESXi 5.1

If you are running the free version of ESXi 5.1 and want to try running nested XenServer hypervisors as vms, please read on!

If you are running a home lab with a limited number of physical servers and resources, then the free version of ESXi 5.1 is a pretty good choice for hypervisor.  I chose it as my hypervisor at home because it gives me memory ballooning (that’s over-provisioning of memory) in the free edition, which XenServer does not  This lets me allocate more memory to vms than I actually have in my physical server.  Note:  it looks like the free version of XenServer 6.2 now offers this, so you might want to think about that.

So, now that you have chosen ESXi 5.1 as your host, but still want to mess about with the excellent Xen hypervisor, here’s how you can do it.

I am assuming here that you already have an ESXi server stood up and that it is indeed version 5.1.  5.1 makes it a bit easier to virtualize other hypervisors, so you want to be using that instead of previous versions.

I am using XenServer 6.1, but 6.2 should work just as well.

Enable Promiscuous Mode on your vSwitches

In the vSphere client, click on your ESXi host, Configuration tab => Networking => Properties of the vSwitch that contains the virtual network that your Xen vm will be sitting in => Security => Change Promiscuous Mode to “Accept”


Build your XenServer VM and Change the Network Adapter Type

Build this just like you would build any other vm, making sure you boot off the Xen ISO.

For Network Adapter Type choose “E1000.”   It doesn’t appear that XenServer supports the more efficient VMXNet3 adapter.

Do not start the vm!

Edit the VMX File to Enable Virtual Hypervisor Functionality

You only need to do this if you don’t have the vsphere web client, in which case you can do this via the web gui on 5.1. if you are running the free version of esxi 5.1 (without vCenter) then you will need to do it as follows:

SSH into your esxi server, find the vmx file for your Xen VM, edit and insert the following line:

vhv.enable = "true"

save the file and exit.

Note:  your vmx file should be somewhere similar to this:


Now you can start the Xen vm and build it from the ISO.


No VMTools?

One thing I couldn’t get working is the vmtools to be installed on the XenServer vm.  So, if anyone has worked this out, I’d be happy to hear in the comments section below.

Build Xen VMs

Now you should be able to install XenCenter on your pc, connect to the XenServer host and start creating Xen-based vms just like your XenServer was installed on a physical server.  Cool!



VMware vSphere can virtualize itself + 64-bit nested guests

Running VMware ESXi inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. Virtualization enthusiasts everywhere have benefited from the ability to run ESXi on ESXi, first introduced with the vSphere 4 release.

VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.

Physical Host Setup

The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:

  • Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
  • Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
  • Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
  • Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests
    echo 'vhv.allow = "TRUE"' >> /etc/vmware/config

Virtual VMware ESXi Machine (vESXi) Creation

For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.

Create a new VM with the following guidance:

  • Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
  • 2 virtual sockets, 2+ GB RAM
  • 4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
  • Thin provisioned virtual disks work fine
  • Finish creating the VM, then edit the following settings
    • Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
    • CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
  • Don’t power this VM on — keep it to act as a template
  • Clone and install VMware ESXi via ISO image or PXE boot
  • Add to vCenter and configure virtual ESXi hosts for action

Nested 64-bit Guests

With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.

Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.

Nested Hyper-V Virtual Machines

It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!

A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!

Do the following to enable Hyper-V on VMware ESXi:

  • Add hypervisor.cpuid.v0 = FALSE to the VM configuration

  • Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)

For another take, check out William Lam’s post on this topic.

Parting Thoughts

Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.

How are you taking advantage of a virtual vSphere environment?

How to run XenServer 6.0 on vSphere 5 – with nested Windows Server 2008 R2 VM

It is possible to install XenServer 6.0 in a virtual machine on vSphere ESXi 5 and then with a few tweaks you can even run a nested Windows Server 2008 R2 VM on the virtual XenServer 6.0.

To install XenServer 6.0 in a VM, first follow this guide to configure ESXi 5.0 (or watch this youtube video).

One important step is to execute the following command from the console:

echo ‘vhv.allow = “TRUE”‘ >> /etc/vmware/config

Otherwise, configure like the guide. Once the custom VM has been created, to be able to choose ESXi 5 as operating system, go to Edit Settings -> Options -> Guest Operating System choose ‘Other’ and then choose VMware ESXi 5.x. This will ensure that you won’t receive the “HVM is required for this operation” error when trying to boot the win2k8R2 vm (it is possible to change this after the install of XenServer as well).

Download the install .iso from
Mount iso and install XenServer
When done, you will get startup screen as below
Download XenCenter from and install

Add the the XenServer to XenCenter

Create a new VM, choose win2k8 R2 64-bit, mount ISO, install.