<soapbox>
I’m going to let you all in on one of my pet peeves … it seems that too many people who write about technology don’t know the difference between the words “number” and “amount”. A couple of examples:

• What is the amount of NICs in that server?
• The amount of TB of storage is surprising.
• The amount of ways the problem can be solved…
• The number of RAMs in servers today is amazing.

I could go on and on…but you get the idea. Let me make it as simple as I can:

Number

Used for items that are enumerable – i.e. you can count them as 1, 2, 3. In the examples above, you can count NICs, TB, and “ways”, but you can’t count “RAMs”.

Amount

Used for things that do not have a definite quantity. Let’s use RAM as an example. [Q:] How much RAM is in your server? [A:] I have 64GB of RAM in my server.

Notice that you provide a specific number of GB, but “RAM” is not a quantity.

So please, let’s start using the right word in the right place! When you get ready to write a word that indicates a certain quantity of something – consider whether you can COUNT the thing(s) you’re quantifying. And just in case you’re wondering, it doesn’t matter that you can count DIMMs, you can’t count RAM!

</soapbox>

A naming standard is exactly what the title sounds like – a standard for defining the names of things. In my opinion, a naming standard should achieve a couple things:

• Provide a simple, consistent method for assigning names to objects – there is nothing “arbitrary” about a naming standard
• Be flexible enough to accommodate most, if not all, use cases
• Provide an effective means for all parties involved to understand what is being described

While these may seem really obvious and simple to achieve, there can be alligators hiding just beneath the surface!

The first two are, indeed, fairly simple. It’s that third bullet that can cause heartburn. Why? Because there are so many parties involved! First, you have the ESX administrators. They need to be able to accurately connect a virtual machine to the proper port group. Failure to do so could result in a VM that is unable to communicate, or worse, a VM that can communicate on the wrong network! Next, you have the OS administrators who are responsible for administering the guest operating system within the VM. They have to be able to effectively communicate to the ESX admin which network to connect the VM to. This person has absolutely no idea what a “port group” is and they’re not interested in learning…it’s just outside the scope of their responsibility. And if you think the OS admin doesn’t know what a port group is, imagine what the application admin knows!

So, up to this point, we’ve got three different points of view. We’ve got an application administrator who knows that his application needs to be on the SAP network; an OS admin who wants his VM connected to the production network; and an ESX admin who doesn’t care what the functionality is – he just wants to know what port group to connect the vNIC to. Oh yeah, there is one other group who has an interest in this whole thing – the network admins. These folks don’t know about applications, don’t care about operating systems, and stare at you blankly when you mention port groups – all they understand is that a host system needs to be connected to a particular VLAN number!

As you can see, the real challenge of defining a naming standard is developing a common vernacular so that everyone understands what everyone else means when they identify their network. I’ve seen all sorts of standards put into use. Some are better than others, and some are simply miserable!

Virtual switches are actually very simple to name…accept the default value of “vSwitch#” where # is the incremental count of vSwitches that have been created. The first vSwitch to be created will be “vSwitch1”, the second will be “vSwitch2”; I think you get the idea, so I’m not going to spend any more time on vSwitch naming.

Port Groups are another matter all together! By default, when you install ESX, there will be a single vSwitch created (vSwitch1) and it will have two port groups created on it. These port groups will be named “Service Console” and “Virtual Machines” to reflect their intended usage. This is a valid naming convention (i.e. name the port group based on the services that will be connected to it), but it does have some limitations. Think back to our discussion above. The ESX admin will feel right at home, but does “Virtual Machines” mean anything to the application administrator? The OS administrator? Or the network administrator? Probably not.

This is a prime example of a name that doesn’t fulfill our stated objectives. So, what is a good naming standard? Well, let’s consider everyone we have to satisfy:

• Application Admin: We need to be able to let the application administrator communicate the function of the port group
• Server OS Admin: Needs to be able to communicate the environment into which the VM needs to be placed
• ESX Admin: Will be creating the port group and assigning the name. Is also responsible for actually connecting a vNIC to a port group. Needs to be able to understand what his customers are asking him to do
• Network Admin: Needs to be able to communicate and understand where the vNIC needs to reside within the overall network infrastructure

So, a possible option for a naming standard could be something like this:

                SAP Prod VLAN123

This is actually pretty good. We know that the port group is used to support the SAP application in the production environment and that it lives on VLAN 123. There are a couple of issues with this standard, though. The biggest “problem” from my point of view is the use of spaces. Spaces do not cause a problem in a virtual environment; however, they can present an interesting challenge when it comes to automation. Spaces are viewed as delimiters in most scripting languages, so if your script developer (or the script developer of the third-party product or the guy on the Internet who developed a cool script) isn’t careful, you can have some “unexpected behaviors” at some point in the future.

It’s actually a simple matter to protect against this type of scripting error: don’t use spaces in your naming standard! So, you could name your port group something like this:

                SAP_Prod_VLAN123
                SAP_Dev_VLAN224
                Mail_DMZ_VLAN400
                VIAdmin_Admin_VLAN10
                WinAdmin_Admin_VLAN11
                Backup_Backup_VLAN543

Now we have a standard name that is easily understood by all parties involved and protects against poor scripting habits. While this is certainly not the “end-all and be-all” naming standard, it does meet our stated goals. You should look at your environment and develop a standard that meets the requirements identified above plus whatever other requirements you may have. If you can do that – and stick to it – you will have a virtual networking stack that meets your needs and is simple to administer. That’s my definition of goodness.

I’d like to thank you for taking the time to read through this series of articles. It’s been an interesting challenge for me to commit all of this to “paper” and I hope you’ve found it useful. Look forward to more stuff from me in the future!

The HyTrust Appliance Community Edition is a full-featured virtual appliance that allows you to manage up to three ESX hosts. This is a great way for smaller organizations to gain the benefits of centralized authentication, consistent security configuration, and greatly enhanced auditability. It also gives organizations of all sizes the chance to “kick the tires” on the product to see if it fits their needs. All of this in a totally FREE product (well, you do have to register…).

Quoting from the Press Release:

Pricing & Availability

HyTrust Appliance, Community Edition is now available for download now as a pre-built, VMware-compatible virtual appliance to members of HyTrust Community. To join the community free of charge, go to http://www.hytrust.com/community/register. Support for Community Edition is provided by the Community via online forum participation and direct community member interaction.

This is a great opportunity – join the HyTrust Community and download the HyTrust Appliance Community Edition today. It will simplify your life, no matter how small (or large) your environment!

KLC

I wanted to take a minute and let everyone know that I have accepted a position as a Senior Consultant at VMware (actually started 27 April). I am working in the Professional Services Organization (PSO) and will be focusing on customers in the Federal Civil Sector, primarily in the Washington, DC area.

I want to let you know this so that you know where I’m coming from when I write here – although please do understand that this is my PERSONAL blog and anything that shows up here is my PERSONAL opinion. I do not have the authority (nor the desire) to speak on behalf of my employer, so if I say it, it’s because it conforms to “Virtualization according to Ken” – not “Virtualization according to VMware”!

Thanks for stopping by. I’ll try to make the content here worth your time, so stop by every once in a while to see what I’ve cooked up!

KLC

• The Great vSwitch Debate – Part 1
  In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
• The Great vSwitch Debate – Part 2
  In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
• The Great vSwitch Debate – Part 3
  Here I discussed the various load balancing options that are available in a VMware vSwitch.
• The Great vSwitch Debate – Part 4
  In Part 4, I covered fault detection and the Cisco Discovery Protocol.
• The Great vSwitch Debate – Part 5
  In Part 5, I talked about the various networks that you have to contend with in an ESX environment as well as an approach to help in deciding which networks to combine, if you have to.
• The Great vSwitch Debate – Part 6
  I introduced the first host configuration. In this part, I talked about my recommendations for when you have eight pNICs and offered up a couple alternatives, including one for using an iSCSI initiator from within a VM.

In this Part 7, I’m going to discuss configurations for systems with two, four, and six pNICS. The same ground rules I established in Part 6 are going to apply here – for those who are skipping ahead or who have short memories, here they are:

• All networks will have at least two pNICs to provide fault tolerance. While ESX will work just fine without the fault tolerance features, I don’t consider that to be an option for a real live “production” environment. If you’re building out a personal “playground” or a lab environment, feel free to cut the number of pNICs in half, but don’t come crying to me if your one and only network connection fails and your ESX/i host and/or your VMs fall off the face of the network!
• Unless otherwise specified, all configuration options are set to their defaults values. This is in line with my “Don’t change a default value unless you have a good reason” philosophy (and it makes it easier to describe!)
• There is a requirement to support two separate “application zones”. These are logically separate networks that could be separated via VLANs or physical separation.

For each configuration, we’re going to use the same set of Port Groups – regardless of how many pNICs or vSwitches we create. If you remember from the end of Part 5, I made this recommendation for manageability and scaleability, so I’m going to stick to it. So, for each of our host configurations, we’ll have the Port Group configuration shown in Figure 1 and Figure 2. Figure 1 shows the port groups for environments that are using IP Storage (iSCSI and/or NFS):

Figure 1. Port Groups with IP Storage

And Figure 2 shows the port groups for those who are using local storage or fibre channel attached SAN storage:

Figure 2. Port Groups without IP Storage

In all cases, I strongly recommend the use of VLAN tagging if your network infrastructure supports it. Tagging will provide logical separation of traffic (physical is better than logical, logical is better than nothing!). It would be up to your networking team to assign the actual VLAN numbers, so coordination is critical! Now, on with the host configurations!

Configuration with Six pNICs

In Part 6 we talked about the configuration for a host with eight pNICs, so let’s look now at a host with six pNICs. With only six pNICs, you have to start making some choices – especially if you’re using IP Storage. You no longer have enough pNICs to provide each port group with its own fully redundant set of pNICs – you have to combine some networks. This is where you might want to go back and review the interaction matrices back in Part 5. Based on your own unique set of circumstances, you would decide which networks to combine together on a single vSwitch. 

Figure 3 shows a fairly standard configuration for use with six pNICs when IP Storage is used. 

Figure 3. Six pNICs with IP Storage

Notice that I combined the management and VMotion port groups (PG_Mgmt & PG_VMotion) onto a single vSwitch sharing a pair of pNICs. One alternative was to combine Management and IP Storage – I opted for VMotion because there is less traffic there, so there should be less interference between the two networks. The other option was to combine IP Storage and VMotion. I don’t normally like to do that because you then have two high-volume consumers using the same set of pNICs – but it is an option, and if you really need to isolate your management network, it could be the right option.

If you’re not using IP Storage, then six pNICs is pretty straight forward – three vSwitches to support the various functions (see Figure 4):

Figure 4. Six pNICs without IP Storage

Configuration with Four pNICs

With only four pNICs, it begins to get a little more interesting. We are resource constrained to the point where we have no choice other than to mix critical networking services, so the question is – which ones? I opted to mix them up as shown in Figure 5.

Figure 5. Four pNICs with IP Storage

Based on my belief that virtual machine traffic is toxic from a security standpoint, mixing of VM traffic with “infrastructure support” traffic would be a last resort. Taking IP Storage out of the picture yields Figure 6.

Figure 6. Four pNICs without IP Storage

No surprises there! With four pNICs, I don’t see very many alternatives, although I guess you could opt for a single vSwitch with active and standby pNICs, but that adds complexity and little value. All it would do is to provide you with a failover path in the event that you lost both of your pNICs associated with a given function. Since we’ve been satisfied with single pNIC failure tolerance so far, I see no reason to start worrying about it now!

Configuration with Two pNICs

The use of only two pNICs is not recommended. You don’t have the ability to separate virtual machine traffic from infrastructure support traffic and still retain redundancy. To me, that’s not a viable configuration – except, perhaps, in a lab environment. This configuration seems to come up most frequently with some older blade servers and in “build it yourself” systems. If you have any way to add another pair of network adapters into your host, please do so!

So, let’s assume that you’re stuck and you really, really have to use a system with only two pNICs…well, there aren’t many choices! Here’s what I would do (Figure 7):

Figure 7. Two pNICs with IP Storage

Yikes! That’s an ugly diagram! Due to the limited number of pNICs, I have elected to specify active and standby pNICs for each of the port groups. Normally, I don’t like to do this because it adds significantly to the overall complexity of the environment (just look at that diagram!). By overriding the default vSwitch behavior, I’m able to separate my VM traffic from my infrastructure support traffic except under failure conditions. Not an ideal arrangement, but hey, you do what you gotta do, right?

Once again, removing IP Storage only serves to simplify things (one less port group / network type to worry about!) – see Figure 8.

Figure 8. Two pNICs without IP Storage

Configuration with One pNIC

If I don’t recommend the use of a system with only two pNICs, you can probably guess what my opinion of a single pNIC is! There is no way to separate traffic (other than by VLAN tagging – but if you can’t afford at least two pNICs, chances are good your pSwitch doesn’t support 802.1Q!) and there is no way to provide any redundancy. Redundancy was one of the requirements I set out at the beginning of this exercise, and since I can’t satisfy my requirement with a single pNIC, I’m nog even going to draw the picture. 

If you absolutely, postively have to run with only one pNIC, I would still encourage you to create the port groups. It will help you to manage the environment – and if you’re just learning the environment, it will give you the ability to exercise most of the features of the management interface.

Summary

I hope you’ve enjoyed this series on vSwitches and virtual networking in general. I do have one more article that I want to write still – one about naming standards. I hope to get that one out pretty quickly, but the weekend is looking pretty busy and next week is jam packed, so it may take a while. In the meantime, enjoy the articles and please, if you have comments or questions, leave me a note. I’ll try to get back to you as quickly as I can.

As a matter of fact, both Hyper-V and ESX use paravirtualization. Both Hyper-V and ESX, can run non-paravirtualized operating systems because they offer a set of device drivers that emulate some very popular physical devices (VGA adapter, network adapter, disk controller) that are supported – out of the box – by most operating systems. Once the guest operating system is up and running, you have the option to install the paravirtualized drivers – Enlightenments for Hyper-V and VMware Tools for ESX. So, the assertion that Hyper-V derives an advantage over ESX through the use of paravirtualization is totally bogus.

Where Hyper-V does require paravirtualization is in the processor. Hyper-V cannot work without having “virtualization support” (i.e. Intel VT or AMD-V) baked into the CPU. ESX will take advantage of these technologies when they are available, but also has the ability to use binary translation to run VMs on hosts that do not have CPUs that provide hardware assisted virtualization.

Can we please lay this paravirtualization myth to rest?

Next, we have the argument that Hyper-V is “microkernelized” because its device drivers are implemented in the administrative OS. Well, that makes it a distributed OS, not a microkernel  based OS. Take a look at how Wikipedia defines “Microkernel”

In computer science, a microkernel is a computer kernel that provides the mechanisms needed to implement an operating system, such as low-level address space management, thread management, and inter-process communication. If the hardware provides multiple privilege levels, then the microkernel is the only software executing at the most privileged level (generally referred to as supervisor or kernel mode). Actual operating system services, such as device drivers, protocol stacks, file systems and user interface code are contained in user space.

Loading device drivers in the Primary Partition does not make Hyper-V a micro-kernel based OS (I believe it probably is a micro-kernel based OS, but not because of where it loads its device drivers!), nor does loading the device drivers into the vmkernel make ESX a monolithic OS. Both Hyper-V and ESX are modular operating systems, meaning that they extend their functionality through the use of “modules” that interface with the core OS via a well defined set of interfaces.

What Greg is trying to describe (or so I believe) is the difference in how Hyper-V and ESX manage their device drivers. With Hyper-V, the device drivers are loaded in the management partition (primary partition) and with ESX, they are loaded by the vmkernel into usermode space within the vmkernel. I think it’s time for a diagram to illustrate – see Figure 1:

Figure 1. Hyper-V & ESX - Architectural Differences

Notice that Hyper-V has device drivers living within the Primary Partition and ESX has device drivers that get loaded by the vmkernel. Not a huge deal, unless you consider that, in the case of Hyper-V, if the Primary Partition (Windows 2008) fails, your VMs lose the ability to talk to the outside world (if they survive at all), whereas with ESX, if the service console (modified Red Hat Linux) fails, your VMs continue to operate without problem.

As for the comment:

Ever wonder why it takes VMware forever to add support for new hardware? That’s because any new driver or hardware support has to be specifically encoded, tested, and integrated into that hypervisor.

Again, wrong!  The primary reason it takes “so long” to provide new hardware support is because ESX restricts the HCL to improve reliability. Every component on the HCL has been thoroughly tested and is certified to be compatible with ESX. There is some integration work that has to be done, but no more so than with any other operating system (including Windows!). The difference is that VMware has tighter control over the device driver eco-system than does Microsoft. If you look at Windows hosts, a frequent cause of outages is device drivers developed by third parties that have undergone third party certification. Yes, you get support quickly, but do you get robust support? Sometimes yes, sometimes no.

Then we have this interesting series of statements:

1.  It doesn’t matter how big your administrative OS is.

2. With local disk drives so big these days, your administrative OS (called the “primary partition” in Hyper-V and Xen) can be 10K in size or it can be 10G.

3.  That administrative OS is merely there so you can manage the rest of the box.

4.  Further, because Hyper-V’s administrative OS (“primary partition”) has all this nice and already-existing support for drivers built into the OS, you can argue that it will always have a greater level of driver support than ESX.

Let’s take these one at a time:

1. If it’s truly an administrative OS (as is the case with ESX), I could maybe buy this argument; however, with Hyper-V, the primary partition (that administrative OS) is a key and integral part of the virtualization solution (see item #4).

2. Waste not, want not.

3. That is true for ESX; it is not true for Hyper-V (see item #4)

4. With Hyper-V, the “administrative OS”/”primary partition” provides the device drivers that allow the virtual machines to interact with the underlying hardware. The size and complexity that Greg was disparaging a few paragraphs ago come up in spades in the primary partition. Every new device driver, feature enhancement, user interface modification, etc. that gets incorporated provides a new opportunity for a conflict or incompatibility – not to mention that there’s nothing to stop an administrator from installing whatever application they want in the primary partition – it is, after all, just “another Windows OS instance”.

While we’re on the topic of device drivers, let’s look a little deeper. What are some of the key cornerstones of virtualization? In my book, you have two fundamental aspects:

• Encapsulation. A virtual machine is encapsulated in a group of files and the boundaries are clearly known.
• Portability. The ability to move a virtual machine (running or not) from one host to another without concern about the underlying hardware.

Both Hyper-V and ESX do a great job with #1; however, let’s consider #2. In Hyper-V (per Ben Armstrong – Microsoft virtualization Program Manager), “while we do provide high performance drivers, we also provide emulated devices for unsupported operating systems. For systems using the high performance drivers there is clean abstraction at the VMBus layer.” This means that – for supported OSes – there should be no problem with VM portability. What I’m not clear on is whether unsupported VMs have the same degree of portability, or if they are tightly coupled to the underlying hardware…Ben, can you clarify, please?

The current version of Hyper-V is also restricted by its use of NTFS as a filesystem. Since NTFS is not a clustered filesystem (in its current encarnation – supposed to change in W2K8 R2), a single LUN cannot be “owned” by multiple hosts at the same time. This is the lynch pin in the “live migration” issue – since you can’t have two hosts sharing access to the same LUN, Hyper-V is unable to live migrate a running VM. This will change with Windows Server 2008 R2 when Hyper-V will offer live migration support.

So…let’s recap for everyone’s benefit:

• Hyper-V and ESX are both “Type-1″ hypervisors: TRUE
• Hyper-V is a “microkernelized” hypervisor and ESX is a “monolithic” hypervisor: FALSE
• ESX drivers exist within its hypervisor: TRUE, but are dynamically loaded and run at a lower privilege level than the core hypervisor functions
• It doesn’t matter how big your administrative OS is.: TRUE, if it’s really an administrative OS. FALSE in the case of Hyper-V which relies on the primary partition for critical virtualization functionality
• Further, because Hyper-V’s administrative OS (“primary partition”) has all this nice and already-existing support for drivers built into the OS, you can argue that it will always have a greater level of driver support than ESX.: TRUE – but is that a good thing?
• That means Hyper-V will always tend to see faster raw performance.: FALSE In both cases (Hyper-V and ESX), there’s a degree of abstraction, which introduces some overhead. In neither case is the overhead “significant”.  In no circumstance would I make the comment that Hyper-V is always faster; nor would I make the assertion that ESX is always faster. The performance for a particular device is dependent upon the device driver implementation as well as the entire software stack that deals with the device. I’d wager that Hyper-V will perform better under some circumstances and ESX will perform better under others.

I hope I’ve cleared up some of the confusion. I know it’s rampant and people are looking for some clear understanding and guidance. To be fair, I am biased toward VMware. I’ve made my living by using and supporting VMware technologies for a long time…

• The Great vSwitch Debate – Part 1
  In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
• The Great vSwitch Debate – Part 2
  In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
• The Great vSwitch Debate – Part 3
  Here I discussed the various load balancing options that are available in a VMware vSwitch.
• The Great vSwitch Debate – Part 4
  In Part 4, I covered fault detection and the Cisco Discovery Protocol.
• The Great vSwitch Debate – Part 5
  In Part 5, I talked about the various networks that you have to contend with in an ESX environment as well as an approach to help in deciding which networks to combine, if you have to.

Now, in Part 6, we finally start talking about host configurations! I started a thread over on the VMTN Community forums for people to provide input about content they would like to see in this series. VMTN user RobVM asked about a configuration with eight pNICs and iSCSI connectivity, so I’ll tackle that first. But before we do, let me lay some ground rules:

• All networks will have at least two pNICs to provide fault tolerance. While ESX will work just fine without the fault tolerance features, I don’t consider that to be an option for a real live “production” environment. If you’re building out a personal “playground” or a lab environment, feel free to cut the number of pNICs in half, but don’t come crying to me if your one and only network connection fails and your ESX/i host and/or your VMs fall off the face of the network J
• Unless otherwise specified, all configuration options are set to their defaults values. This is in line with my “Don’t change a default value unless you have a good reason” philosophy (and it makes it easier to describe!)
• There is a requirement to support two separate “application zones”. These are logically separate networks that could be separated via VLANs or physical separation.

Now, on with the show!

Configuration with Eight pNICs and IP Storage

Eight is a really good number of pNICs if you plan to use IP-based storage. If you remember from Part 5 of this series, there are a variety of networks that we need to support on ESX:

• VMware Management Network. This is the network that connects vCenter Server to the ESX Service Console. Since ESXi doesn’t have a Service Console, the ESXi Management Network is terminated at the vmkernel.
• VMotion Network. This network interconnects the various ESX/i (reminder, ESX/i is my shorthand notation for ESX and/or ESXi) hosts within a VMware cluster and enables VMotion among those nodes.
• IP Storage (NFS or iSCSI) Network. The Network that provides storage for virtual machine and ancillary support (i.e. .iso files).
• Virtual Machine Network(s). One or more networks that support VM to access and provide services.

Notice that there are four networks – to get the redundancy that I said we were going to implement, multiply four by two and you get eight! (Can you handle this upper level math? It sometimes throws me for a loop!)

So, now we have enough information to throw out a configuration (Figure 1):

Figure 1. Basic Eight pNIC Configuration

As you can see, I’ve laid out four vSwitches, each with two pNIC uplinks configured. The load balancing algorithm is “Route based on virtual switch port ID”. While this is a perfectly sane, robust, and common configuration, there are some negatives associated with this config:

vSwitch0 and vSwitch1 each have two pNICs associated with them; however, there is a single port group and, more importantly, a single service associated with each. This means that each of these two vSwitches will have a pNIC sitting idle, waiting for the active path to fail. In the grand scheme of things, this isn’t such a horrible thing. We’re talking about, what, maybe a thousand dollars worth of “stuff” sitting idle? Between the two pNICs, the associated cabling, and the physical switch ports, that may be a little low, but overall, not a big deal.

vSwitch2 is dedicated to IP Storage. In many cases, this will be either iSCSI or NFS; however, there is nothing to prevent you from running both protocols across this single vSwitch. In fact, many places will do just that – using iSCSI for storing VMFS volumes and NFS for hosting .iso and other “support files”. Notice that if you’re using iSCSI storage, you do need to provide the service console with visibility into that network for authentication purposes.

vSwitch3 is your virtual machine vSwitch. Since we have two application environments, we have defined two different VLANs (by simply adding the VLAN number into the Port Group configuration and ensuring that the pSwitch is configured to trunk the required VLANs). If your policies dictate that these two application environments must not be allowed to comingle on the wire, you would need to add two additional pNICs (to maintain redundancy), and to ensure no comingling, I would recommend a separate vSwitch rather than port groups with active/standby/unused pNIC configurations.

Now, let’s spice it up a little and add in the requirement to support Multipath I/O (MPIO) from within our virtual machines. In order to support MPIO from within a VMware VM, you need to provision the VM with multiple vNICs. In my example, I’m going to configure a VM with three vNICs – one for “regular” network traffic and two to support MPIO access to an iSCSI target. In addition to these direct network connections, the VM will access its system volume as a virtual disk via the vmkernel iSCSI stack (see Figure 2).

Figure 2. Guest OS MPIO iSCSI Configuration

So, we now have a single VM that is connecting to four separate port groups (although the connection to PG_IPStor is totally transparent – abstracted through the guest OS vSCSI controller). What really makes this VM different is the two extra vNICs that are connected to the PG_VMStor1 & PG_VMStor2 port groups. We will be loading an iSCSI Initiator inside the guest operating system to access an iSCSI target. I won’t go into the details of how to configure the iSCSI initiator to support MPIO, but just trust me when I tell you that you can get better load balancing and better overall iSCSI performance using this approach rather than the native ESX iSCSI access. This improved performance comes with a significant price, though. By using this approach, you now have to manage the iSCSI Initiator within the guest OS and you have to manage the allocation of iSCSI targets to initiators. Additionally, the iSCSI Initiator inside the VM will drive your host’s CPU utilization significantly higher. This (higher CPU utilization) is frequently not much of a problem because many environments are not CPU constrained to start with. The additional management overhead is a big concern (to me).

<soapbox>

</soapbox>

Now that I’ve gotten that off my chest…here’s what this solution would look like (Figure 3):

Figure 3. Eight pNIC Configuration with Guest MPIO

Wait! What’s that I see on vSwitch1? Could it be? Is it possible? Did Ken really configure active and passive pNICs? Yep, I did! Why? Well, it’s really fairly simple. The default load balancing algorithm is vSwitch Port ID based. If you remember from Part 3 of this series, there are no guarantees about which pNIC your vNIC will associate with. In this case, we need to make sure that the vNICs wind up on different pNICs. At first blush, it would seem that I could default everything and still be OK, but what happens if connect and disconnect a couple of vNICs during operations? Remember that vSwitch ports are statically mapped to pNICs, so if I were to power up three VMs, the first and third each with two vNICs and the second with a single vNIC connecting into vSwitch1, I would have the following (Figure 4):

Figure 4. Virtual Machine Initial vNIC Mapping

Notice that vSwitch Port #3 is statically mapped to pNIC2. Now, if I power off the second VM (the one with only one vNIC) and power on another VM with two vNICs, I wind up with the configuration shown in Figure 5.

Figure 5. Virtual Machine Secondary vNIC Mapping

Notice now that BOTH vNICs for the newly powered on VM are mapped to pNIC1 – NOT what we wanted! If you could guarantee that every VM connected to this vSwitch would have two vNICs and that, at no point, would any of the vNICs be administratively disabled, you could allow the configuration to default. Personally, that’s too many “ifs” for me to trust!

Wow! Not only did I find a justification to break my “always default” rule, I found a good example of why you would want to use active and standby adapters!

Configuration with Eight pNICs without IP Storage

This one is easy. If you take the first example and keep the requirements the same, just remove the need for IP storage and still use eight pNICs, I would do the following (Figure 6):

Figure 6. Eight pNICs with no IP Storage

All we’ve done is split out the two application networks onto separate vSwitches, providing additional bandwidth (probably not needed) and additional fault tolerance. By doing this, we’ve eliminated the need to configure VLAN trunking on the pSwitch and also no need to specify a VLAN number on the PG_App1 and PG_App2 port groups.

References

Microsoft Multipath I/O: Frequently Asked Questions: http://www.microsoft.com/WindowsServer2003/technologies/storage/mpio/faq.mspx

Microsoft iSCSI Users Guide: http://download.microsoft.com/download/A/E/9/AE91DEA1-66D9-417C-ADE4-92D824B871AF/uGuide.doc

• The Great vSwitch Debate – Part 1
  In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
• The Great vSwitch Debate – Part 2
  In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
• The Great vSwitch Debate – Part 3
  Here I discussed the various load balancing options that are available in a VMware vSwitch.
• The Great vSwitch Debate – Part 4
  In Part 4, I covered fault detection and the Cisco Discovery Protocol.

Now, in Part 5, I’m going to identify the various “networks” that you interact with in a VMware environment and also provide my recommendation for a configuration with only two pNICs. On with the show!

The Various Networks

In the VMware architecture, there are nominally five IP-based networks. I’ll cover each of these below, but in summary, they are listed below and shown in Figure 1:

• VMware Management Network. This is the network that connects vCenter Server to the ESX Service Console. Since ESXi doesn’t have a Service Console, the ESXi Management Network is terminated at the vmkernel.
• VMotion Network. This network interconnects the various ESX/i (reminder, ESX/i is my shorthand notation for ESX and/or ESXi) hosts within a VMware cluster and enables VMotion among those nodes.
• NFS Network. The Network File System (NFS) Network is an IP Storage network that provides the interconnect between ESX/i hosts and one or more NFS servers that provide storage for virtual machine and ancillary support (i.e. .iso files).
• iSCSI Network. The Internet Protocol Small Computer Systems Interface (iSCSI) Network is an IP Storage network that provides the interconnect between ESX/i hosts and one or more iSCSI targets that provide storage for virtual machine and ancillary support (i.e. .iso files).
• Virtual Machine Network(s). One or more networks that support VM to access and provide services.

Figure 1. The Various Networks

You may have additional networks in your environment, and I’ll mention some of them as I get through this article. One of these that I’ll mention right now is the out-of-band (OOB) management network. While not required to make your VMware environment work, this network is present in many, if not most, enterprise VMware environments.

The out-of-band management network provides connectivity to your out-of-band management interface (i.e. HP iLO, Dell DRAC, IBM Director, Sun LOM, etc.). Users who have access to this network essentially have the keys to the kingdom. With proper authentication (or by hacking a username/password), they have direct console access, the ability to connect and disconnect virtual I/O devices (CD-ROM, floppy), the ability to power the host on or off, and pretty much anything else you could do if you were sitting in front of the system. This network needs to be protected at all costs! It should never be placed into a DMZ network. Another way to view the OOB network is as the door to your datacenter. You wouldn’t leave your datacenter door propped open to a back alley, so don’t leave your OOB network exposed to unnecessary risks, either!

VMware Management Network

Much like the Out-of-Band Network, the VMware Management Network is critical to the security of your virtual infrastructure. This network provides the management interface to the vmkernel – either through the service console (for ESX) or directly (with ESXi). This is the network where vCenter Server (a.k.a. Virtual Center) lives, as well as the path for ssh, web access, and third-party tools access, see Figure 2.

Figure 2. VMware Management Network

Again, in much the same manner as your OOB network, the Management Network can be viewed as a door into your datacenter. It needs to be protected very carefully and should never be exposed in a DMZ, and will frequently live behind a firewall with minimal ports exposed.

VMotion Network

The VMotion Network is a special-purpose network with only one use – the live migration of a running virtual machine from one physical ESX/i host to another with no interruption of service to clients of the VM. The VMotion interface is a vmkernel interface that is flagged as being used for VMotion. Figure 3 shows the VMotion network.

Figure 3. VMotion Network

All VMotion Network interfaces should be within the same IP broadcast domain to ensure that the hosts can find each other. No function other than VMotion needs to be able to access the VMotion network – in fact, for a two-node cluster, you can use a direct cable with no intervening switch to support VMotion.

It is important to know that the data sent across the VMotion network (on port TCP 8000) is not encrypted in any manner. This means that anyone who can connect their PC to the VMotion network will be able to listen in and intercept that data, which contains whatever happens to be in the virtual machines vRAM at the time of the VMotion. The information could include usernames, passwords, credit card numbers, you name it…it could be there.

With that knowledge, it should come as no surprise that I’m recommending that you protect the VMotion Network very carefully!

NFS Network

The NFS Network is a vmkernel network that supports access to Network File System v3 (NFSv3) shares over the Transmission Control Protocol (TCP). NFS is a file sharing protocol (much like Server Message Block [SMB] or the Common Internet File System [CIFS], the common Windows file sharing protocols (see http://en.wikipedia.org/wiki/Cifs). NFS was originally developed by Sun Microsystems in 1984 for use with Unix systems (see http://en.wikipedia.org/wiki/Network_File_System_(protocol)). Figure 4 shows a typical NFS network configuration.

Figure 4. NFS Network

Since NFS can be used to store virtual machines and/or utility files such as .iso images, it is very common for the NFS server to have multiple connections to the network. Depending on the NFS server or appliance that you are using, there are a variety of ways that these connections can be aggregated to improve the performance of your NFS network. Regardless of how the NFS server is connected to the network, the ESX host is bound by the rules of the vSwitch that’s used to support NFS traffic, as discussed in Part 3.

From a security perspective, the NFS protocol is not encrypted or otherwise secured on the wire. This means that anyone who has access to the NFS network has the ability to intercept data that represents the on-disk information stored in virtual machine files. Obviously, this is a significant risk that needs to be mitigated with appropriate configuration and management actions.

iSCSI Network

The iSCSI Network is quite similar to the NFS network. The primary difference between NFS & iSCSI is that iSCSI is a “block oriented” protocol whereas NFS is a “file oriented” protocol. What does that mean? Basically, it means that with NFS, it is the NFS server that is responsible for managing the filesystem and individual blocks of data on the disk. The ESX Server doesn’t care if the disks are formatted with NTFS, ZFS, ext3, or Ken’s File System – it never sees the structure on the disk. It is the NFS Server’s responsibility to read from, write to, and manage access of all information on the disk.

Conversely, an iSCSI Target (that’s what a single instance of an iSCSI server process is called), the ESX server is intimately knowledgeable about the on-disk structure (unless you happen to be using a Raw Device Map (RDM)) because ESX will communicate with the iSCSI Target using standard SCSI commands, exposing the actual on-disk blocks of data to the ESX host. In many, if not most, cases, the logical unit number (LUN) exposed by the iSCSI Target will be formatted by the ESX host as a VMware Filesystem (VMFS) volume. Figure 5 shows a typical iSCSI configuration.

Figure 5. iSCSI Network

As with NFS, it is not uncommon to see an iSCSI Target configured with multiple uplink connections into the network. And just like NFS, iSCSI is bound by the rules of the vSwitch for load balancing and failover.

I hate to be redundant, but this is worth stating: From a security perspective, the iSCSI protocol is not encrypted or otherwise secured on the wire. This means that anyone who has access to the iSCSI network has the ability to intercept data that represents the on-disk information stored in virtual machine files. Obviously, this is a significant risk that needs to be mitigated with appropriate configuration and management actions.

Virtual Machine Network(s)

Here’s where things get interesting! When you start talking about virtual machines, you’re talking about all the servers that live in your datacenter. These servers provide and consume services of all types – from other virtual machines, from physical servers in the datacenter, and from resources of all types on the Internet and other external networks. Obviously, this means that there may be the need to connect to more than one network to support all of these different communications paths. Figure 6 provides a view of some of the possible connectivity that needs to be supported by the VM Network(s).

Figure 6. Virtual Machine Networks

Notice that I’ve included an NFS Server and an iSCSI server in this diagram. You might ask “Why?” Well, it’s simple. The guest operating system inside a VM can directly mount an NFS volume and it can also use an iSCSI Initiator to connect to an iSCSI Target.

Some of these networks may need to be separated from each other. Depending on the level of sensitivity of each network, you may be able to use a single network divided only by IP subnets; you may need to use VLANs to provide a logical separation among the networks; and you may need to use totally separate vSwitches with dedicated pNICs to provide the required separation. I can’t help you with these decisions…that’s between you, your network team, your application team, your management team, and your security officer (good luck!).

Also notice that, for the first time in my initial six diagrams (other than the ESX Service Console), the network connections into the ESX/i hosts is not through a vmkernel port. In this case, the connection is via a vSwitch that is configured for Virtual Machine connectivity. That’s really just a technicality, a configuration setting. In reality, all vSwitches are owned and managed by the vmkernel.

Best Practices

There are some general guidelines that I like to use when designing a network architecture. Based on the discussion above, you can see that there are quite a few networks that need to / should be protected or isolated. I recognize that not everyone shares the same views on security and data protection – and that’s perfectly fine (as long as you understand the consequences!). I’ll try to accommodate as many positions as I can…on with the show!

There are two primary considerations when deciding how to carve up your networks: security and performance. The third consideration is manageability, and we’ll talk about it in a moment. The key to picking the optimal configuration based on the number of pNICs you have in your hosts is to understand the level of risk associated with mixing the various networks. I’ve created a couple tables that shows my personal assessment of the risk associated with each pairing. Table 1 shows the security implications while Table 2 shows the performance ramifications.

Table 1. Security Impact of Mixing Networks

My rationale for assigning these levels of security risk are as follows:

• The Management Network should be isolated as much as possible. When it is not possible to give it a dedicated vSwitch and associated pNICs (best case scenario), it is typically a “Medium Risk” to mix it with either VMotion or one of the IP Storage networks. The reason I chose medium risk is that the personnel granted access to the Management network are typically the most trusted in your organization. While it is never a good idea to allow users of any level of trust to access an unsecured storage or VMotion network, if you have to do it due to constraints, do it with your administrative users on your Management network segment.
• For the VMotion Network, there is “Low Risk” associated with the IP Storage networks. The reason for this is that there shouldn’t be any users on any of these three networks (VMotion, NFS, iSCSI), so there’s not much chance of someone intercepting data on the wire.
• The VMotion logic above applies to the NFS and iSCSI networks as well.
• The Virtual Machine Network(s) are always considered “High Risk”. This is because you have unknown/untrusted users connecting via potentially uncontrolled systems. There may be cases where you have specific VM Networks that are not high risk (for example, you may run vCenter Server as a VM. That particular VM would live on the Administrative Network which is a Medium Risk network).

Your assumptions about the level of risk associated with each network may be different than what I’ve suggested here. It doesn’t matter, simply substitute your values into the calculations and you’ll be set.

The second consideration I want to discuss is performance. Similarly to security, there are performance impacts to combining the various networks onto the same set of pNICs. Table 2 shows the matrix of performance impacts from mixing the various networks.

Table 2. Performance Impact of Mixing Networks

If you evaluate each network individually, you can develop an understanding of the traffic patterns that exist for each.

• Management Network: There is typically not a lot of traffic on this network. It is used for management functions such as vCenter Server operations (host configuration and management, virtual machine configuration and management, performance monitoring, etc.) and access by third-party applications (configuration management, resource monitoring, etc.). These are typically low-impact applications. The exceptions to the low-impact “rule” are the deployment of templates and the use of service console-based backup utilities.. Each of these functions has a significant impact on management network utilization.
• VMotion Network: This network sits idle except during a virtual machine migration; however, when a VMotion migration is taking place, you want to have as much bandwidth as possible available to enable the migration to complete as quickly as possible. If you do not use VMotion, you don’t need to worry about this network.
• NFS & iSCSI Networks: These are your IP Storage networks. Their utilization fluctuates wildly depending on what is happening in your virtual environment. During steady-state operations, there is typically a “moderate” level of activity; however, when virtual machines are being powered on, resumed, or backed up across these networks, there is significant activity.
• VM Network(s): This is a total crap shoot. In some environments, these networks sit almost idle, while in others, they are hit very hard. You will have to judge for yourself how significant your workloads are - although I will say that it is the exception rather than the rule where the traffic on these networks is “significant”.

Next, I’ve taken the two considerations and combined them. The result is a matrix that shows the overall “risk” of combining the various networks (see Table 3).

Table 3. Overall Impact of Mixing Networks

Again, these are not set-in-stone, has-to-be-this-way recommendations, but rather a tool to be used to help you make your decisions.

Oh, I nearly forgot - I promised to talk some about manageability. Basically, I recommend that each of these networks be separated into individual port groups, even if you’re not using VLAN tagging. If possible, create separate vSwitches for each of the major networks (Management, VMotion, IP Storage). For the VM Network(s), I recommend at least one vSwitch (depending on separation requirements) with a separate port group for each different network. Figure 7 shows two possible configurations.

Figure 7. vSwitch Options

Notice that, even though there is a different number of vSwitches in the two configurations, the port groups are the same. This logical separation makes it simple to manage your environment - you can even have different hosts with different vSwitch configurations, yet the same port group configuration, that support VMotion among them. It also makes it easy to scale the environment.

OK..that’s it for this time. Next time, in Part 6, I’ll talk about my recommendations for when you have differing numbers of pNICs in your hosts.

So far, we’ve been through three posts on vSwitches. If you’ve not read these posts, I recommend that you go back and do so now (or you can read this post and then go back – there are not many dependencies). The first three posts were:

• The Great vSwitch Debate – Part 1
  In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
• The Great vSwitch Debate – Part 2
  In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
• The Great vSwitch Debate – Part 3
  Here I discussed the various load balancing options that are available in a VMware vSwitch.

So, what does that leave for Part 4? Plenty! In this edition, we’re going to talk about how a vSwitch detects path failures and also dip our toes into the Cisco Discovery Protocol waters. Now, on to the next topic!

Network Path Failure Detection

OK, so we’ve moved the responsibility for fault tolerance from the virtual machine’s guest OS and placed it squarely on the vSwitch. This begs the question – how does a vSwitch know that a failure has occurred, and what does it do about it?

Well, let’s see if we can answer those questions!

ESX supports two types of network failure detection mechanisms, beaconing and link state detection. Let’s investigate beaconing first.

Beacon Probing

Beacon Probing, frequently called simply “beaconing”. Beaconing is intended for use in situations where there are multiple pSwitches between the various pNICs pNICs in a vSwitch. Beaconing is a technique that sends layer two Ethernet broadcast packets from every pNIC in the team to every VLAN to which the vSwitch belongs (yes, that means that if your vSwitch participates in 10 VLANs, beaconing will transmit 10 broadcast packets per pNIC per beacon interval!), although you do have the option of overriding the vSwitch default settings at the Port Group level, if you desire.

See Figure 1 for a depiction of a single pNIC transmitting beacon packets to the broadcast address (in normal operation, every pNIC will transmit beacon packets on every connected VLAN). This packet would be received by all other pNICs in the broadcast domain. An interesting note is that this is the only situation when, during normal operation, you will see the MAC address of the pNIC on the network. The MAC address or the Beacon Initiator pNIC is used as the source MAC address in the beacon frame. It is also of interest that the vSwitch will absorb the beacon frames received by the Beacon Receivers – the virtual machines will never see the beacon frames.

Figure 1. Beacon Probing

If a Beacon Receiver misses three consecutive beacon packets, it will flagged as “bad” and put into a down state. Outbound vSwitch traffic will automatically be routed over surviving interfaces. There are two basic behaviors that the vSwitch will exhibit upon beacon failure (from http://blogs.vmware.com/networking/2008/12/using-beaconing-to-detect-link-failures-or-beaconing-demystified.html):

ESX behavior when a beaconing failure is detected is as follows:

1. If two or more uplinks receive beacons from each other, those uplinks are considered good. We stop using uplinks which do not receive any beacon packets.
2. On ESX 3.5, if no uplink receives beacon packets, traffic is sent to all uplinks (shotgun mode). If a team has two uplinks, any link failure will result in all packets being sent to both uplinks.

As you see, you can wind up in a situation where you are transmitting all packets along every uplink path. This can cause extreme confusion for your pSwitches, especially if you have multiple uplinks connected to the same pSwitch (not recommended when using beaconing)!

There are other issues with Beacon Probing, too. For example http://kb.vmware.com/kb/1004373 reads as follows:

“When configuring networking for an ESX Server host using at least two vmnic as network adapters and VLAN Type 4095, duplicated packets can occur when Beacon Probing is selected in the Network Failover Detection dropdown menu.”

With the following recommended solution:

“Select the Link Status Only option in the Network Failover Detection dropdown menu instead of Beacon Probing.”

Basically, Beacon Probing should not be used as an alternative to a robust Layer Two network implementation. Instead, use Link Status Only as your network failure detection mechanism.

Link Status Only

Link Status Only error detection relies upon the pNIC’s link state detection capabilities to identify when there is a problem with the network path. At first blush, you may think this is not a very robust error detection scheme – it only sees the condition of the connection between the pNIC and the first upstream pSwitch! It also does nothing to verify that the pSwitch port is configured correctly, and it can’t see deeper into the network. While all this is true, there are some things that can be done to help alleviate these problems.

First, use a pSwitch that has the “Link State Tracking” feature. This feature will mirror the link status of the upstream link (pSwitch to pSwitch) down to the downstream link (pSwitch to ESX pNIC). What this means is that if the first upstream pSwitch becomes isolated, the link status indicator for the pNIC will be set to “Down”, indicating to the vSwitch that the associated path is no longer viable.

Figure 2. Link State Tracking

Figure 2 shows an example of how Link State Tracking works. When looking at the Downstream Link (from the first pSwitch’s perspective), everything looks good; however, the Upstream Link between the two pSwitches has failed. With Link State Tracking enabled on the pSwitch, will reflect the state of the Upstream link on the Downstream link, letting ESX know that the path is dead and that the alternate path should be used.

Which to Use?

So, you’ve got two options for managing network failure conditions, which should you use? I recommend the following:

• When you initially configure your vSwitch, use Beacon Probing. This will allow you to test not only the link state, but also ensure that you can talk across all of your configured port groups. Once you’ve validated proper configuration, switch to Link Status Only
• When you add a new port group to an existing vSwitch, set the error detection method for the port group to Beacon Probing to verify correct pSwitch configuration. Once you’ve validated proper configuration, switch to Link Status Only
• Don’t use Beacon Probing if more than one pNIC in the vSwitch is connected to the same pSwitch. This could result in the same MAC address being presented on two or more ports on the pSwitch which is “a very bad thing”.
• Use Link Status Only for network failure detection. If at all possible, use pSwitches that support Link State Tracking to reflect upstream network status back to the vSwitch.
• Implement a robust Layer Two network. If possible, have your first level pSwitch multi-homed to eliminate single points of failure.

Cisco Discovery Protocol (CDP)

The Cisco Discovery Protocol (CDP) is used to obtain pSwitch port configuration from an ESX host. The information returned by CDP can be invaluable when you’re trying to verify or modify your network configuration. Examples of some of the information returned by CDP include:

• Identification of the pSwitch to which a pNIC is connected
• Identification of the pSwitch port to which the pNIC is connected
• Speed & Duplex settings for the pSwitch port
• VLAN number(s) associated with the pSwitch port

The CDP information is available either via the command line (use vmware-vim-cmd) or via vCenter Server (on the Configuration / Networking tab). For more detailed information on the use of CDP, check out the following VMware knowledgebase articles:

http://kb.vmware.com/kb/1007069

http://kb.vmware.com/kb/1003885

OK, this wraps it up for Part 4 of my series on vSwitches. In the next section (Part 5), I’ll start getting into some of my recommended configurations. That’s where the real fun begins – and hopefully we’ll get the “Debate” part of this thing spun up

References:

Beaconing Demystified: Using Beaconing to Detect Link Failures

VMware Virtual Networking Concepts

Duplicated Packets Occur when Beacon Probing Is Selected Using vmnic and VLAN Type 4095

Cisco Discovery Protocol (CDP) network information via command line and VirtualCenter on an ESX host

Configuring the Cisco Discovery Protocol (CDP) with ESX Server

• To enable a single authentication source for all of the different administrative access methods (vCenter Server, ssh, web access, VIC, third-party tools, etc.)
• To enable a granular authorization service for users, regardless of administrative access method
• To provide a template-based security configuration for your hosts, and
• To provide a centralized logging facility to simplify and enhance auditability.

The architecture is relatively simple (which is a good thing). Basically, as shown in Figure 1, the appliance sits between your administrative users and your VMware Infrastructure systems. It can be configured either as an Ethernet bridge or as a proxy. I personally prefer the bridge approach, since it doesn’t require client-side configuration and is much harder to circumvent – intentionally or on purpose.

Figure 1. HyTrust High Level Architecture

Let’s look at the four basic functions individually:

Single Authentication Source

The appliance works in much the same way as vCenter Server – inbound administrative connections are terminated at the HyTrust appliance and HyTrust will use a single privileged account to perform host actions on the user’s behalf. The great thing about this approach is that you now have a single repository for user credentials – your Active Directory or other LDAP v3 repository. You don’t have to worry about keeping multiple authentication sources synchronized, and your users don’t have to remember a bunch of different passwords.

Granular Authorization Service

Because the HyTrust Appliance terminates inbound administrative connections, it has the ability to be very granular in its authorization functions. In addition to being able to authorize users for individual commands, such as esxcfgnas, you have the ability to authorize individual options on each command. For example, you could configure a role that has the rights to execute the command esxcfgnas -l but not esxcfgnas –a. This could be ideal for creating, for example, a role for an auditor who could come in and look at anything they wanted to, but they wouldn’t be able to change anything.

One of the really neat things that was demoed was a user connected to vCenter Server via the VIC attempting to execute a command that they are not authorized for. They were able to issue the command, but it was squashed even before it showed up as a task in the VIC status pane (and the attempt was logged). Ideally, unauthorized commands wouldn’t even show up in the UI, but the ability to prevent the command from even being scheduled for execution is a great first step.

Additionally, there is a single point for logging all actions performed against your environment.

Template-based Host Security Configuration

Another feature that was demonstrated was template-based host security configuration. OK, that sounds ominous – what exactly does it mean? It means that you can define the security settings that you want implemented for your hosts and the HyTrust Appliance can audit your hosts for compliance with your template and it can also remediate any hosts that it finds out of compliance. When the product ships, it will include two pre-defined templates that you can use out of the box or customize for your specific needs. The included templates will be base on the VMware ESX Security Hardening White Paper and the CIS VMware ESX Server Benchmark.

Additional templates may be added, and of course, you are free to create your own by modifying the included ones or starting from scratch.

Centralized Logging Facility

Since all administrative actions will be performed by the HyTrust Appliance, auditable logs can be created in one place. True, if you really have to dig in for forensics purposes, you’ll still have to go back to the source host and do log correlation and piece together an activity trail, but in many cases, you’ll have enough information in the HyTrust logs to satisfy your requirements. This can be a HUGE benefit for organizations that have regulatory obligations (such as HIPAA, GLBA, SARBOX, etc.) or for anyone who cares about security in general.

One of the things that I really liked about the log entries is that they were actually understandable! Here are just a couple examples to give you an idea:

• WARN : AUN0010I The request for {NetworkAdminUser {HT_NetworkAdmin}} to perform PowerOnVM_Task on VM : win2k3 is declined
• INFO : VVM0010O Source: 172.16.1.100 User: NetworkAdminUser Operation: RemoveVirtualSwitch
• INFO : AUN0001I The request for {NetworkAdminUser {HT_NetworkAdmin}} to perform RemoveVirtualSwitch on NETWORK : networkSystem-8 is authorized

There is enough information in the log to recreate the activities that a user has performed, the messages are structured to allow easy parsing by log analyzers, and the messages are clear enough that a human can read them and understand what’s going on.

Summary

I’ve not covered all the features that were presented in the demo, but I’ve hit what I felt were the high points. I’m eagerly awaiting the chance to put my hands on one of these boxes to put it through its paces for real.

In summary, I found the HyTrust Appliance to be a very interesting product that addresses many of the issues that nearly every virtualized environment is facing today. It enables true Role Based Access Control (RBAC) and single-source authentication for all of your VI administrative interfaces.

The only hole that I noticed in the HyTrust armor is the ESX server’s physical console. There is no way for a network-based product to protect against someone who has physical access to a system. This is true for every system and reinforces the best practice of ensuring the integrity of your security policies regarding access to systems.

As always, security is a combination of people, process, and technology. The HyTrust Appliance should be looked at by anyone who is serious about security in their environment.

Thanks Eric & Ken – I look forward to a product release in the near future!

More opinions can be found here:

Hytrust, virtualization under control @ Yellow Bricks / Duncan Epping
HyTrust Launches Security Appliance @ Scott Lowe
HyTrust: An Elegant Solution To a Messy Problem @ Rational Survivability / Chris Hoff
Virtually Secure – HyTrust Launches Virtual Security Appliance @ Professional VMware / Cody Bunch