Updated: Reaction to: “How to Correctly Explain the Architectural Differences Between Hyper-V and ESX”

Oh, goodness! It seems that Greg Shields’ attempt at “How to Correctly Explain the Architectural Differences Between Hyper-V and ESX” isn’t all that correct. Oh, he starts out pretty well, classifying both Hyper-V and ESX as Type-1 hypervisors, which is correct. Where he goes astray is when he claims that Hyper-V utilizes “paravirtualization” and ESX relies on “hardware emulation” – wrong!

As a matter of fact, both Hyper-V and ESX use paravirtualization. Both Hyper-V and ESX, can run non-paravirtualized operating systems because they offer a set of device drivers that emulate some very popular physical devices (VGA adapter, network adapter, disk controller) that are supported – out of the box – by most operating systems. Once the guest operating system is up and running, you have the option to install the paravirtualized drivers – Enlightenments for Hyper-V and VMware Tools for ESX. So, the assertion that Hyper-V derives an advantage over ESX through the use of paravirtualization is totally bogus.

Where Hyper-V does require paravirtualization is in the processor. Hyper-V cannot work without having “virtualization support” (i.e. Intel VT or AMD-V) baked into the CPU. ESX will take advantage of these technologies when they are available, but also has the ability to use binary translation to run VMs on hosts that do not have CPUs that provide hardware assisted virtualization.

Can we please lay this paravirtualization myth to rest?

Next, we have the argument that Hyper-V is “microkernelized” because its device drivers are implemented in the administrative OS. Well, that makes it a distributed OS, not a microkernel  based OS. Take a look at how Wikipedia defines “Microkernel”

In computer science, a microkernel is a computer kernel that provides the mechanisms needed to implement an operating system, such as low-level address space management, thread management, and inter-process communication. If the hardware provides multiple privilege levels, then the microkernel is the only software executing at the most privileged level (generally referred to as supervisor or kernel mode). Actual operating system services, such as device drivers, protocol stacks, file systems and user interface code are contained in user space.

Loading device drivers in the Primary Partition does not make Hyper-V a micro-kernel based OS (I believe it probably is a micro-kernel based OS, but not because of where it loads its device drivers!), nor does loading the device drivers into the vmkernel make ESX a monolithic OS. Both Hyper-V and ESX are modular operating systems, meaning that they extend their functionality through the use of “modules” that interface with the core OS via a well defined set of interfaces.

What Greg is trying to describe (or so I believe) is the difference in how Hyper-V and ESX manage their device drivers. With Hyper-V, the device drivers are loaded in the management partition (primary partition) and with ESX, they are loaded by the vmkernel into usermode space within the vmkernel. I think it’s time for a diagram to illustrate – see Figure 1:

Figure 1. Hyper-V & ESX - Architectural Differences

Figure 1. Hyper-V & ESX - Architectural Differences

Notice that Hyper-V has device drivers living within the Primary Partition and ESX has device drivers that get loaded by the vmkernel. Not a huge deal, unless you consider that, in the case of Hyper-V, if the Primary Partition (Windows 2008) fails, your VMs lose the ability to talk to the outside world (if they survive at all), whereas with ESX, if the service console (modified Red Hat Linux) fails, your VMs continue to operate without problem.

As for the comment:

Ever wonder why it takes VMware forever to add support for new hardware? That’s because any new driver or hardware support has to be specifically encoded, tested, and integrated into that hypervisor.

Again, wrong!  The primary reason it takes “so long” to provide new hardware support is because ESX restricts the HCL to improve reliability. Every component on the HCL has been thoroughly tested and is certified to be compatible with ESX. There is some integration work that has to be done, but no more so than with any other operating system (including Windows!). The difference is that VMware has tighter control over the device driver eco-system than does Microsoft. If you look at Windows hosts, a frequent cause of outages is device drivers developed by third parties that have undergone third party certification. Yes, you get support quickly, but do you get robust support? Sometimes yes, sometimes no.

Then we have this interesting series of statements:

1.  It doesn’t matter how big your administrative OS is.

2. With local disk drives so big these days, your administrative OS (called the “primary partition” in Hyper-V and Xen) can be 10K in size or it can be 10G.

3.  That administrative OS is merely there so you can manage the rest of the box.

4.  Further, because Hyper-V’s administrative OS (“primary partition”) has all this nice and already-existing support for drivers built into the OS, you can argue that it will always have a greater level of driver support than ESX.

Let’s take these one at a time:

1. If it’s truly an administrative OS (as is the case with ESX), I could maybe buy this argument; however, with Hyper-V, the primary partition (that administrative OS) is a key and integral part of the virtualization solution (see item #4).

2. Waste not, want not.

3. That is true for ESX; it is not true for Hyper-V (see item #4)

4. With Hyper-V, the “administrative OS”/”primary partition” provides the device drivers that allow the virtual machines to interact with the underlying hardware. The size and complexity that Greg was disparaging a few paragraphs ago come up in spades in the primary partition. Every new device driver, feature enhancement, user interface modification, etc. that gets incorporated provides a new opportunity for a conflict or incompatibility – not to mention that there’s nothing to stop an administrator from installing whatever application they want in the primary partition – it is, after all, just “another Windows OS instance”.

While we’re on the topic of device drivers, let’s look a little deeper. What are some of the key cornerstones of virtualization? In my book, you have two fundamental aspects:

  • Encapsulation. A virtual machine is encapsulated in a group of files and the boundaries are clearly known.
  • Portability. The ability to move a virtual machine (running or not) from one host to another without concern about the underlying hardware.

Both Hyper-V and ESX do a great job with #1; however, let’s consider #2. In Hyper-V (per Ben Armstrong – Microsoft virtualization Program Manager), “while we do provide high performance drivers, we also provide emulated devices for unsupported operating systems. For systems using the high performance drivers there is clean abstraction at the VMBus layer.” This means that – for supported OSes – there should be no problem with VM portability. What I’m not clear on is whether unsupported VMs have the same degree of portability, or if they are tightly coupled to the underlying hardware…Ben, can you clarify, please?

The current version of Hyper-V is also restricted by its use of NTFS as a filesystem. Since NTFS is not a clustered filesystem (in its current encarnation – supposed to change in W2K8 R2), a single LUN cannot be “owned” by multiple hosts at the same time. This is the lynch pin in the “live migration” issue – since you can’t have two hosts sharing access to the same LUN, Hyper-V is unable to live migrate a running VM. This will change with Windows Server 2008 R2 when Hyper-V will offer live migration support.

So…let’s recap for everyone’s benefit:

  • Hyper-V and ESX are both “Type-1” hypervisors: TRUE
  • Hyper-V is a “microkernelized” hypervisor and ESX is a “monolithic” hypervisor: FALSE
  • ESX drivers exist within its hypervisor: TRUE, but are dynamically loaded and run at a lower privilege level than the core hypervisor functions
  • It doesn’t matter how big your administrative OS is.: TRUE, if it’s really an administrative OS. FALSE in the case of Hyper-V which relies on the primary partition for critical virtualization functionality
  • Further, because Hyper-V’s administrative OS (“primary partition”) has all this nice and already-existing support for drivers built into the OS, you can argue that it will always have a greater level of driver support than ESX.: TRUE – but is that a good thing?
  • That means Hyper-V will always tend to see faster raw performance.: FALSE In both cases (Hyper-V and ESX), there’s a degree of abstraction, which introduces some overhead. In neither case is the overhead “significant”.  In no circumstance would I make the comment that Hyper-V is always faster; nor would I make the assertion that ESX is always faster. The performance for a particular device is dependent upon the device driver implementation as well as the entire software stack that deals with the device. I’d wager that Hyper-V will perform better under some circumstances and ESX will perform better under others.

I hope I’ve cleared up some of the confusion. I know it’s rampant and people are looking for some clear understanding and guidance. To be fair, I am biased toward VMware. I’ve made my living by using and supporting VMware technologies for a long time…


Tags: ,

About Ken Cline

vExpert 2009

7 responses to “Updated: Reaction to: “How to Correctly Explain the Architectural Differences Between Hyper-V and ESX””

  1. Ben Armstrong says :

    Hi Ken,

    Ben Armstrong – Microsoft virtualization Program Manager – here. I am afraid that you have fallen into the same trap as Greg (who in turn fell into the same trap as Michael), and have made some rather false statements here.

    Hyper-V is capable of running operating systems without needing custom drivers installed. All virtualization platforms can do this today – as it is required for operating system installation (where it is too early to get custom drivers in). And all virtualization platforms (ESX included) do this through emulating hardware. Once the operating system is up and running you can either continue to use the slow emulated hardware, or install the virtualization platform specific drivers to get higher performance.

    Also – Hyper-V has no issues with virtual machine portability, you can move between different physical hardware with no problems – in fact this happens all the time for users who use Windows Failover Clustering with Hyper-V. Or for users who deploy virtual machine from the System Center Virtual Machine Manager library.

    You do make some valid points here – but others are a bit off.


    • Ken Cline says :

      Hi Ben,

      Thanks for your reply. Could you please point me to some info on how the virtual device drivers work in Hyper-V, with specifics about virtual machine portability? There is a lot of misinformation out there, and I want to contribute to it as little as possible 🙂


  2. Ben Armstrong says :

    This is a good page to look at: http://msdn.microsoft.com/en-us/library/cc768520.aspx

    While we do provide high performance drivers, we also provide emulated devices for unsupported operating systems. For systems using the high performance drivers there is clean abstraction at the VMBus layer.

    With out this we would never be able to do virtual machine quick migration or live migration.


  3. Michael Naeve says :

    Gentlemen (Ken & Ben),

    Could you please elaborate on the use of portability in regards to Hyper-V? My understanding is that Windows Failover Clustering with Hyper-V is a done at the client O/S level, not the host and therefore would not represent a true portability in the sense that theoretically with VMware VMotion, a client O/S can be ‘moved’ from one hardware platform to another with near zero administrative costs. Furthermore, with VMware Fault Tolerance, a shadow operating system if you will, can be run simultaneously representing a cluster service handled by the hypervisor versus the client O/S. I am not trying to advocate one approach over another, just trying to correctly understand the technical specifics.

    Thank you,

    • Ken Cline says :


      I’ve updated the post to reflect what I hope is a more accurate statement of Hyper-V’s portability status. If I understand correctly, Windows Server 2008 R2 will include an updated Hyper-V that supports a “VMotion-like” live migration capability. As far as I know, Microsoft does not currently have (even in pre-release code) anything that is comparable to VMware FT, but since FT is not available in a currently released product from VMware, I don’t feel that it’s right to use that as a differentiator … I’m happy to wait until there’s a real product before beating up the competition with a feature 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: