OK, so the count is up to five posts on vSwitches. If you’ve not read these posts, I recommend that you go back and do so now. The first five posts were:
- The Great vSwitch Debate – Part 1
In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
- The Great vSwitch Debate – Part 2
In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
- The Great vSwitch Debate – Part 3
Here I discussed the various load balancing options that are available in a VMware vSwitch.
- The Great vSwitch Debate – Part 4
In Part 4, I covered fault detection and the Cisco Discovery Protocol.
- The Great vSwitch Debate – Part 5
In Part 5, I talked about the various networks that you have to contend with in an ESX environment as well as an approach to help in deciding which networks to combine, if you have to.
Now, in Part 6, we finally start talking about host configurations! I started a thread over on the VMTN Community forums for people to provide input about content they would like to see in this series. VMTN user RobVM asked about a configuration with eight pNICs and iSCSI connectivity, so I’ll tackle that first. But before we do, let me lay some ground rules:
- All networks will have at least two pNICs to provide fault tolerance. While ESX will work just fine without the fault tolerance features, I don’t consider that to be an option for a real live “production” environment. If you’re building out a personal “playground” or a lab environment, feel free to cut the number of pNICs in half, but don’t come crying to me if your one and only network connection fails and your ESX/i host and/or your VMs fall off the face of the network J
- Unless otherwise specified, all configuration options are set to their defaults values. This is in line with my “Don’t change a default value unless you have a good reason” philosophy (and it makes it easier to describe!)
- There is a requirement to support two separate “application zones”. These are logically separate networks that could be separated via VLANs or physical separation.
Now, on with the show!
Configuration with Eight pNICs and IP Storage
Eight is a really good number of pNICs if you plan to use IP-based storage. If you remember from Part 5 of this series, there are a variety of networks that we need to support on ESX:
- VMware Management Network. This is the network that connects vCenter Server to the ESX Service Console. Since ESXi doesn’t have a Service Console, the ESXi Management Network is terminated at the vmkernel.
- VMotion Network. This network interconnects the various ESX/i (reminder, ESX/i is my shorthand notation for ESX and/or ESXi) hosts within a VMware cluster and enables VMotion among those nodes.
- IP Storage (NFS or iSCSI) Network. The Network that provides storage for virtual machine and ancillary support (i.e. .iso files).
- Virtual Machine Network(s). One or more networks that support VM to access and provide services.
Notice that there are four networks – to get the redundancy that I said we were going to implement, multiply four by two and you get eight! (Can you handle this upper level math? It sometimes throws me for a loop!)
So, now we have enough information to throw out a configuration (Figure 1):
Figure 1. Basic Eight pNIC Configuration
As you can see, I’ve laid out four vSwitches, each with two pNIC uplinks configured. The load balancing algorithm is “Route based on virtual switch port ID”. While this is a perfectly sane, robust, and common configuration, there are some negatives associated with this config:
vSwitch0 and vSwitch1 each have two pNICs associated with them; however, there is a single port group and, more importantly, a single service associated with each. This means that each of these two vSwitches will have a pNIC sitting idle, waiting for the active path to fail. In the grand scheme of things, this isn’t such a horrible thing. We’re talking about, what, maybe a thousand dollars worth of “stuff” sitting idle? Between the two pNICs, the associated cabling, and the physical switch ports, that may be a little low, but overall, not a big deal.
vSwitch2 is dedicated to IP Storage. In many cases, this will be either iSCSI or NFS; however, there is nothing to prevent you from running both protocols across this single vSwitch. In fact, many places will do just that – using iSCSI for storing VMFS volumes and NFS for hosting .iso and other “support files”. Notice that if you’re using iSCSI storage, you do need to provide the service console with visibility into that network for authentication purposes.
vSwitch3 is your virtual machine vSwitch. Since we have two application environments, we have defined two different VLANs (by simply adding the VLAN number into the Port Group configuration and ensuring that the pSwitch is configured to trunk the required VLANs). If your policies dictate that these two application environments must not be allowed to comingle on the wire, you would need to add two additional pNICs (to maintain redundancy), and to ensure no comingling, I would recommend a separate vSwitch rather than port groups with active/standby/unused pNIC configurations.
Now, let’s spice it up a little and add in the requirement to support Multipath I/O (MPIO) from within our virtual machines. In order to support MPIO from within a VMware VM, you need to provision the VM with multiple vNICs. In my example, I’m going to configure a VM with three vNICs – one for “regular” network traffic and two to support MPIO access to an iSCSI target. In addition to these direct network connections, the VM will access its system volume as a virtual disk via the vmkernel iSCSI stack (see Figure 2).
Figure 2. Guest OS MPIO iSCSI Configuration
So, we now have a single VM that is connecting to four separate port groups (although the connection to PG_IPStor is totally transparent – abstracted through the guest OS vSCSI controller). What really makes this VM different is the two extra vNICs that are connected to the PG_VMStor1 & PG_VMStor2 port groups. We will be loading an iSCSI Initiator inside the guest operating system to access an iSCSI target. I won’t go into the details of how to configure the iSCSI initiator to support MPIO, but just trust me when I tell you that you can get better load balancing and better overall iSCSI performance using this approach rather than the native ESX iSCSI access. This improved performance comes with a significant price, though. By using this approach, you now have to manage the iSCSI Initiator within the guest OS and you have to manage the allocation of iSCSI targets to initiators. Additionally, the iSCSI Initiator inside the VM will drive your host’s CPU utilization significantly higher. This (higher CPU utilization) is frequently not much of a problem because many environments are not CPU constrained to start with. The additional management overhead is a big concern (to me).
|I strongly encourage you to use this solution ONLY when you find that the standard mechanism (connection via the vSCSI interface) for accessing iSCSI storage does not provide the levels of performance that you need. Remember, just because something is “faster” doesn’t make it “better”! In many cases, I’ve seen people go through the pain of implementing this solution only to find that storage throughput was not their bottleneck, and giving a fatter pipe didn’t help overall performance at all. In many (most?) cases, the performance bottleneck lives in the application and not in the supporting infrastructure. Remember that most “modern” applications were designed years ago when Pentium III and Pentium 4 were rulers of the roost, when 500MHz RAM was blazingly fast, when 100Mbps networks were the norm, and when U160 SCSI on 10,000 RPM drives provided more than enough “speed”.
Now that I’ve gotten that off my chest…here’s what this solution would look like (Figure 3):
Figure 3. Eight pNIC Configuration with Guest MPIO
Wait! What’s that I see on vSwitch1? Could it be? Is it possible? Did Ken really configure active and passive pNICs? Yep, I did! Why? Well, it’s really fairly simple. The default load balancing algorithm is vSwitch Port ID based. If you remember from Part 3 of this series, there are no guarantees about which pNIC your vNIC will associate with. In this case, we need to make sure that the vNICs wind up on different pNICs. At first blush, it would seem that I could default everything and still be OK, but what happens if connect and disconnect a couple of vNICs during operations? Remember that vSwitch ports are statically mapped to pNICs, so if I were to power up three VMs, the first and third each with two vNICs and the second with a single vNIC connecting into vSwitch1, I would have the following (Figure 4):
Figure 4. Virtual Machine Initial vNIC Mapping
Notice that vSwitch Port #3 is statically mapped to pNIC2. Now, if I power off the second VM (the one with only one vNIC) and power on another VM with two vNICs, I wind up with the configuration shown in Figure 5.
Figure 5. Virtual Machine Secondary vNIC Mapping
Notice now that BOTH vNICs for the newly powered on VM are mapped to pNIC1 – NOT what we wanted! If you could guarantee that every VM connected to this vSwitch would have two vNICs and that, at no point, would any of the vNICs be administratively disabled, you could allow the configuration to default. Personally, that’s too many “ifs” for me to trust!
Wow! Not only did I find a justification to break my “always default” rule, I found a good example of why you would want to use active and standby adapters!
Configuration with Eight pNICs without IP Storage
This one is easy. If you take the first example and keep the requirements the same, just remove the need for IP storage and still use eight pNICs, I would do the following (Figure 6):
Figure 6. Eight pNICs with no IP Storage
All we’ve done is split out the two application networks onto separate vSwitches, providing additional bandwidth (probably not needed) and additional fault tolerance. By doing this, we’ve eliminated the need to configure VLAN trunking on the pSwitch and also no need to specify a VLAN number on the PG_App1 and PG_App2 port groups.
Microsoft Multipath I/O: Frequently Asked Questions: http://www.microsoft.com/WindowsServer2003/technologies/storage/mpio/faq.mspx
Microsoft iSCSI Users Guide: http://download.microsoft.com/download/A/E/9/AE91DEA1-66D9-417C-ADE4-92D824B871AF/uGuide.doc