The Great vSwitch Debate – Part 7

OK…if you’ve followed along this far, you’re either 1) enjoying what you’re reading, 2) a glutton for punishmnet, or 3) really, really bored. Hopefully, it’s #1 and you’re here because you’ve read the first six posts in this series and you just can’t wait for me to add #7! If you’ve not read the first six posts, I recommend that you go back and do so now. The first six posts were:

  • The Great vSwitch Debate – Part 1
    In this post, I discussed vSwitch functions, Port Groups, VLAN tagging/trunking, valid communications paths, and some other basic vSwitch information.
  • The Great vSwitch Debate – Part 2
    In Part 2, I covered the vSwitch security features (Promiscuous Mode, MAC Address Change, and Forged Transmits) as well as network traffic shaping options.
  • The Great vSwitch Debate – Part 3
    Here I discussed the various load balancing options that are available in a VMware vSwitch.
  • The Great vSwitch Debate – Part 4
    In Part 4, I covered fault detection and the Cisco Discovery Protocol.
  • The Great vSwitch Debate – Part 5
    In Part 5, I talked about the various networks that you have to contend with in an ESX environment as well as an approach to help in deciding which networks to combine, if you have to.
  • The Great vSwitch Debate – Part 6
    I introduced the first host configuration. In this part, I talked about my recommendations for when you have eight pNICs and offered up a couple alternatives, including one for using an iSCSI initiator from within a VM.

In this Part 7, I’m going to discuss configurations for systems with two, four, and six pNICS. The same ground rules I established in Part 6 are going to apply here – for those who are skipping ahead or who have short memories, here they are:

  • All networks will have at least two pNICs to provide fault tolerance. While ESX will work just fine without the fault tolerance features, I don’t consider that to be an option for a real live “production” environment. If you’re building out a personal “playground” or a lab environment, feel free to cut the number of pNICs in half, but don’t come crying to me if your one and only network connection fails and your ESX/i host and/or your VMs fall off the face of the network!
  • Unless otherwise specified, all configuration options are set to their defaults values. This is in line with my “Don’t change a default value unless you have a good reason” philosophy (and it makes it easier to describe!)
  • There is a requirement to support two separate “application zones”. These are logically separate networks that could be separated via VLANs or physical separation.

For each configuration, we’re going to use the same set of Port Groups – regardless of how many pNICs or vSwitches we create. If you remember from the end of Part 5, I made this recommendation for manageability and scaleability, so I’m going to stick to it. So, for each of our host configurations, we’ll have the Port Group configuration shown in Figure 1 and Figure 2. Figure 1 shows the port groups for environments that are using IP Storage (iSCSI and/or NFS):

Figure 1. Port Groups with IP Storage

Figure 1. Port Groups with IP Storage

And Figure 2 shows the port groups for those who are using local storage or fibre channel attached SAN storage:

Figure 2. Port Groups without IP Storage

Figure 2. Port Groups without IP Storage

In all cases, I strongly recommend the use of VLAN tagging if your network infrastructure supports it. Tagging will provide logical separation of traffic (physical is better than logical, logical is better than nothing!). It would be up to your networking team to assign the actual VLAN numbers, so coordination is critical! Now, on with the host configurations!

Configuration with Six pNICs

In Part 6 we talked about the configuration for a host with eight pNICs, so let’s look now at a host with six pNICs. With only six pNICs, you have to start making some choices – especially if you’re using IP Storage. You no longer have enough pNICs to provide each port group with its own fully redundant set of pNICs – you have to combine some networks. This is where you might want to go back and review the interaction matrices back in Part 5. Based on your own unique set of circumstances, you would decide which networks to combine together on a single vSwitch. 

Figure 3 shows a fairly standard configuration for use with six pNICs when IP Storage is used. 

Figure 3. Six pNICs with IP Storage

Figure 3. Six pNICs with IP Storage

Notice that I combined the management and VMotion port groups (PG_Mgmt & PG_VMotion) onto a single vSwitch sharing a pair of pNICs. One alternative was to combine Management and IP Storage – I opted for VMotion because there is less traffic there, so there should be less interference between the two networks. The other option was to combine IP Storage and VMotion. I don’t normally like to do that because you then have two high-volume consumers using the same set of pNICs – but it is an option, and if you really need to isolate your management network, it could be the right option.

If you’re not using IP Storage, then six pNICs is pretty straight forward – three vSwitches to support the various functions (see Figure 4):

Figure 4. Six pNICs without IP Storage

Figure 4. Six pNICs without IP Storage

Configuration with Four pNICs

With only four pNICs, it begins to get a little more interesting. We are resource constrained to the point where we have no choice other than to mix critical networking services, so the question is – which ones? I opted to mix them up as shown in Figure 5.

Figure 5. Four pNICs with IP Storage

Figure 5. Four pNICs with IP Storage

Based on my belief that virtual machine traffic is toxic from a security standpoint, mixing of VM traffic with “infrastructure support” traffic would be a last resort. Taking IP Storage out of the picture yields Figure 6.

Figure 6. Four pNICs without IP Storage

Figure 6. Four pNICs without IP Storage

No surprises there! With four pNICs, I don’t see very many alternatives, although I guess you could opt for a single vSwitch with active and standby pNICs, but that adds complexity and little value. All it would do is to provide you with a failover path in the event that you lost both of your pNICs associated with a given function. Since we’ve been satisfied with single pNIC failure tolerance so far, I see no reason to start worrying about it now!

Configuration with Two pNICs

The use of only two pNICs is not recommended. You don’t have the ability to separate virtual machine traffic from infrastructure support traffic and still retain redundancy. To me, that’s not a viable configuration – except, perhaps, in a lab environment. This configuration seems to come up most frequently with some older blade servers and in “build it yourself” systems. If you have any way to add another pair of network adapters into your host, please do so!

So, let’s assume that you’re stuck and you really, really have to use a system with only two pNICs…well, there aren’t many choices! Here’s what I would do (Figure 7):

Figure 7. Two pNICs with IP Storage

Figure 7. Two pNICs with IP Storage

Yikes! That’s an ugly diagram! Due to the limited number of pNICs, I have elected to specify active and standby pNICs for each of the port groups. Normally, I don’t like to do this because it adds significantly to the overall complexity of the environment (just look at that diagram!). By overriding the default vSwitch behavior, I’m able to separate my VM traffic from my infrastructure support traffic except under failure conditions. Not an ideal arrangement, but hey, you do what you gotta do, right?

Once again, removing IP Storage only serves to simplify things (one less port group / network type to worry about!) – see Figure 8.

Figure 8. Two pNICs without IP Storage

Figure 8. Two pNICs without IP Storage

Configuration with One pNIC

If I don’t recommend the use of a system with only two pNICs, you can probably guess what my opinion of a single pNIC is! There is no way to separate traffic (other than by VLAN tagging – but if you can’t afford at least two pNICs, chances are good your pSwitch doesn’t support 802.1Q!) and there is no way to provide any redundancy. Redundancy was one of the requirements I set out at the beginning of this exercise, and since I can’t satisfy my requirement with a single pNIC, I’m nog even going to draw the picture. 

If you absolutely, postively have to run with only one pNIC, I would still encourage you to create the port groups. It will help you to manage the environment – and if you’re just learning the environment, it will give you the ability to exercise most of the features of the management interface.

Summary

I hope you’ve enjoyed this series on vSwitches and virtual networking in general. I do have one more article that I want to write still – one about naming standards. I hope to get that one out pretty quickly, but the weekend is looking pretty busy and next week is jam packed, so it may take a while. In the meantime, enjoy the articles and please, if you have comments or questions, leave me a note. I’ll try to get back to you as quickly as I can.

About these ads

Tags: , ,

About Ken Cline

vExpert 2009

30 responses to “The Great vSwitch Debate – Part 7”

  1. Rob D. says :

    Ken,

    I’m back to bother you again. In light of the changes in vSphere/ESX 4, what do you see as the best use of eight pNICs with IP storage where you want to use MPIO? It looks to me like it will be built-in with version 4, but I could be wrong. Of course there will still be some reasons to use a dedicated pNICs for iSCSI from the guest OS, but MPIO in the vmkernel might remove the need for many situations.

    Thanks again for the great articles!

    -Rob

    • Ken Cline says :

      Rob,

      Don’t have the answer for you yet…but I will get it and update here as soon as I know better.

      Thanks!
      KLC

    • Ken Cline says :

      Hi Rob,

      OK…the load balancing with vSphere4 will only be available with the vds (Virtual Distributed Switch) which is in the Enterprise+ license tier. From what I’m hearing, it will provide “better” performance than you currently get with a standard vSwitch and IP Hash, but it’s still unclear how much better. The overall networking and storage stacks in vSphere have been significantly improved, so I would recommend that you test the “standard” approach with a “standard” vSwitch to see if the performance meets your needs. If not, and you don’t have access to Enterprise+, then stick with the MPIO in the guest.

      HTH,
      KLC

      • Rob D. says :

        Ken, I believe I will take your advice and test the standard approach first. I’m not really sure where our license will end up…we have VI3 now, which maps to vSphere Enterprise, and we would have to pay for Enterprise Plus. As our shop is mid-sized I’m not sure if it’ll make sense to upgrade to Enterprise Plus. I need more information on the new features and performance that isn’t out there yet.

        Thanks again!

        -Rob

  2. Andrea C. says :

    i have a very simple question for you: in figure 8 you have one nic for vmotion and service console and another one for the rest; the first nic is obviously marked as a vmotion enabled nic but what about the other one? If it’s not marked for vmotion it won’t be able to take care of that in case of a failure of the first nic and if it’s marked as a vmotion nic how do we know it’s not used for vmotion even in normal conditions?

    Hope i made my question clear and congrats for the excellent articles! Looking forward for more!

    Andrea

    • Andrea C. says :

      I’ve just realized that it’s the port group that is vmotion enabled, not the single nic. This is how and why you always know what nic is used for vmotion in that scenario.

      Sorry to waste your time :)

      • Ken Cline says :

        No problem, Andrea! Glad you were able to figure it out on your own.

  3. JR says :

    Hi Ken,

    I have 2 ESX servers coming that each have 6 pNICs (Onboard 2-port Broadcom and 4-port Intel Quad Gig)..

    I was thinking of creating 2 vSwitches:
    1) Management + vMotion (2 pNICs)
    2) VM Network (4 pNICs).

    Simple reason (I thought) is that there’s double the pNICs for the VMNetwork and hence better load balancing? Am I mistaken in doing it this way?

    I also though of adding a VMNetwork port group on vSwitch0 in order to plug a VM into my management LAN (the management LAN is only used for bladecenter/SAN management so the traffic is nil on it).

    Any pros/cons to this setup?

    • Ken Cline says :

      JR,

      You could do that, but to be honest, I seriously doubt that you’re going to be pushing enough traffic to worry about better load balancing. In most cases, networking is far from the bottleneck of the system – you’ll probably wind up with four GbE interfaces running at 20Mbps rather than having two running at 40Mbps — in either case, not enough traffic to break a sweat. If you’ve modeled your workloads and can honestly say that you expect to have more than 1Gbps of sustained network traffic, then I’d say go for it, otherwise, I would recommend going with the three vSwitch config to maintain separation (assuming you care about security).

      KLC

      • Jason T. says :

        In my experience with nic failures is that the ASIC dies or begins to spew enough crap to convince the switch to shut off its port. Depending on your NICs, an ASIC may control two GIGe nic ports.

        I make it my own best practice to split redundant links over multiple ASICs. For example 0/1 and 0/2 are onboard nic ports, 1/1 – 1/4 are your four PCIe NIC ports.

        IPStor – 0/1, 1/1
        Mgmt – 0/2, 1/4
        VM Network(s) – 1/2, 1/3

        I also do the same thing for my pSwitch ports – different blades in our Nexus 7k or 6509 gige cards – or at the very minimum on different port clusters (0-12, 13-24, etc).

        Belts and suspenders are no good if they have a single point of failure somewhere.

      • Ken Cline says :

        Great point, Jason. Now we just need a good way to determine the ASIC per port. I normally go with the PCI device ID and try to split across buses – that

          should

        span ASICS as well.

        Thanks for the comment!

  4. Jay Rubenstein says :

    Hello Ken,

    excellent description of how vSwitches work thanks.

    My question may have more to do with basic networking than vSwitches but thought this was still a good forum to ask in.

    Will you cause problems by having multiple vSwitches assigned to the same subnet?

    For example an ESX server has 6 nics:
    3 vSwitches,
    2 pNics per vSwitch,
    1 vSwitch for the VMs
    2 vSwitches for iscsi traffic both vSwitches are with in the same subnet.

    From my general reading I would expect this to cause problems because the 2 vSwitches used for Iscsi traffic are not bonded.

    Thanks Jay

    • Ken Cline says :

      Hi Jay,

      Multiple vSwitches in the same subnet, in and of itself, will not cause a problem. It would be the same as putting two physical switches on the same subnet.

      I have never, personally, tried to create two vSwitches on the same subnet for vmkernel iSCSI use, the vmkernel may complain. If vCenter Server (VirtualCenter) will allow you to allocate a second vSwitch for vmkernel use, then you’re probably OK.

      From my perspective, it creates far too much complexity. My guess is this is a case of “oh my gosh, I need lots of bandwidth for my iSCSI connection because my VMs will be driving LOTS of I/O” – when in reality, there isn’t that much I/O happening and the setup would be just fine with the two pNIC configuration (one for redundancy).

      KLC

  5. J.J. McDaniel says :

    Hi Ken.

    Awesome articles. In regards to figure 8 above, the 2 pnic configuration.
    What if you were using something like Flex-10 on some HP BL460c G6 blades… using the onboard nics with VLAN tagging? That way you’d have the bandwidth, redudancy and some security / seperation.

    I’m trying to balance cost with security.

    Or would you highly recommend another 2 nics in the mez to physically seperate the VM and management traffic?

    Thanks!

    • Ken Cline says :

      Hey JJ,

      Welcome to my blog! Hope you’re enjoying life post-HP, I see you’re still working with HP gear :)

      The G6 blades are nice – I really like having 10GbE on-board. It does raise some interesting configuration questions, though. You certainly do have enough bandwidth to support pretty much anything you want to do, but the security question looms large. In reality, it comes down to how comfortable you/your security team is/are with the separation provided by VLAN tags. Ask lots of security folks and they’ll tell you (correctly) that the 802.1Q standard does not address network security…it is an administrative division of traffic only. In truth, as long as your infrastructure addresses the four or five known VLAN vulnerabilities, you’re probably OK.

      Personally, I like to physically separate the different types of traffic (administrative, storage, production, VMotion, etc.) just because that’s the way I’ve always done it. It also makes it easier to scale the environment in case you ever decide to build out physically separate infrastructures for the different traffic types. In the project I’m currently working, we have BL680g5 blades with the Flex-10 & FlexNICs. We’re splitting things up as follows:

      – 1Gb Service Console
      – 7Gb VMotion
      – 2Gb Production

      As you can see, we’re using FC storage, so don’t need IP storage. Some may ask why the bandwidth split — well it’s actually pretty simple. We decided to split on the Gb boundary “just because” which is why the service console has a full Gb allocated. We’re allocating 2Gb to the virtual machine traffic, because there are no real network heavy-hitters in the mix (and even if there were, 2Gb is a lot of available bandwidth!). We gave 7Gb to VMotion because we want to be able to evacuate a system quickly in the event we need to perform maintenance activities.

      Everything is, of course, redundant – wouldn’t do it any other way.

      Thanks for stopping by – hope to run into you soon…
      KLC

  6. Mark Harre says :

    Hi Ken,

    Very interesting articles balancing security and number of pNics. I see you generally recommend putting different VLANs for VM traffic on the same vSwitch separated by port groups. We would also like to introduce a DMZ VLAN but don’t feel comfortable using the same vSwitch as the other PG_Apps – are there security considerations in doing this?. If you had 8 pNics how would you distribute them across the vSwitches/port groups (incl iSCSI) bearing security in mind.

    Thanks for your insights
    Mark

    • Ken Cline says :

      Hi Mark,

      The question of DMZ separated by VLANs only really comes down to this: How much do you trust VLANs?

      Even though VLANs were never implemented for security purposes (they came into being for network simplification), they do provide a degree of separation that many organizations are comfortable with. As long as your physical network infrastructure is protected against the few known VLAN exploits, then there’s really minimal risk. With that said, my customers who have implemented DMZs on the same hosts as production workloads have all opted to use a separate vSwitch to host DMZ traffic. I think this stems from a “that’s how it’s always been done” mentality and an abundance of risk avoidance.

      As long as your security team is comfortable with using port groups to maintain separation, I say go for it. Here’s a SANS article that talks to the vulnerabilities and ways to mitigate: http://www.sans.org/security-resources/idfaq/vlan.php and here is a Cisco VLAN Security White Paper to help you understand the implications: http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml

      If it were me – I think I’d probably create four vSwitches, each with 2 pNICs, as follows:

      vSwitch0: Service Console & VMotion
      vSwitch1: IP Storage
      vSwitch2: Production VMs
      vSwitch3: DMZ VMs
      HTH,
      KLC

  7. Chris says :

    Hi Ken,

    Thanks for this blog. I have a question regarding Virtual switches implemented on the NIC. it would seem to me that there should be a NIC card that offloads the resource intensive task of performing the Switching between VMs. Would you be able to comment on this idea, the savings advantage with respect to performance and security and whether anyone builds this?

    thanks,
    Chris.

    • Ken Cline says :

      Hi Chris,

      No, I have never heard of this – and honestly, I don’t think this is something you are likely to see implemented. Not that it’s a bad idea, but there are a couple issues with it. The first is purely economic from the perspective of the manufacturer. There aren’t that many systems that would use such a device, so they would have to be priced fairly high to make them profitable. Second, and perhaps more importantly, is that you would be introducing another dependency into the environment. How would this vSwitch on a NIC be upgraded? Would it prevent you from moving to the next version of your favorite hypervisor? Lots of things to consider.

      I think you are more likely to see things like the Cisco Nexus 1000v used to improve networking. By using the vNetwork APIs third party vendors (such as Cisco) can implement their own optimized vSwitches.

      Thanks for stopping by!
      KLC

  8. Dustin U says :

    Hi Ken,
    First of all GREAT blog and thanks for putting it together. Secondly sorry for the long post and any insight would be greatly appreciated.

    Thanks!

    Currently our ESXi environment is as follows (config is that same on 3 hosts):

    2 1gig nics
    1 10gig nic
    1 10gig not connected

    3 VLAN’s – A,B,C

    vSwitch0
    vmnic1 – 1000 full
    – vmk0 – Net management VLAN A

    vSwitch1
    vmnic2 – 1000 full
    – vmk1 – VMotion VLAN C

    vSwitch2
    vmnic4 – 10000 full
    – vmk2 – IP Storage VLAN A
    – PG_VLAN_B (Public VM’s)
    – PG_VLAN_A (Private VM’s)

    The biggest problem I am having is that my NFS traffic is hitting the NFS server on the Management IP address, can’t figure that one out at all. Can having so much traffic on the same VLAN, even if it is on different vSwitches be causing this problem?

    Also after reading your blog I am thinking of a networking redesign. Here is what I am thinking:

    vSwitch0
    vmnic1 – 1000 full
    – vmk0 – IP Storage VLAN A
    Redundant for Vmotion and Mgmt Net (vSwitch2)

    vSwitch1
    vmnic2 – 1000 full
    – PG_VLAN_B (Public VM’s)
    – PG_VLAN_A (Private VM’s)

    vSwitch2
    vmnic4 – 10000 full
    – vmk1 – VMotion VLAN C
    – vmk0 – Net management VLAN A
    Redundant for vSwitch0 and vSwitch1

    Thanks again!

    Dustin

  9. Tuganologia says :

    serious five star content! thanks

  10. stephan says :

    Awesome and very helpful article.
    Keep up the good work!

  11. Andrew Miller says :

    One revision on the 6 NIC configuration (as I implement that one a lot).

    On vSwitch0, you can set the vSwitch to use both pNICs but then prioritize the Service Console to be Active on one pNIC and standby on the other. Then for the vMotion VMkernel you do the same thing but reserve it.

    This gives you traffic isolation/guaranteed bandwidth during regular operation but still provides resilience.

    Thanks for the great set of articles…reading through just to see if I can pick up any extra nuggets (as I do quite a bit implementing ESX right now).

  12. Tongers says :

    If its not too late to join :-

    We are a newbie to ESXi 4 using Vsphere essentials and 3xHP servers each with 4 physical NICS in each. Plus we have 2 physical stacked switches.

    We are propsoing to run 3 or 4 VM’s on each physical server. Any suggestions as to what a typical network configuration would be.

    There seems to be little around in network configuration for the beginner and any advice would be appreciated

  13. Matt says :

    So here’s what I’ve got going on. I inherited 3ea. 1U servers each configured as follows:

    2ea. Xeon – 6 Core
    60GB RAM
    1ea. 4 port 1Gb NiC on the Mobo
    1ea. 4 port 1Gb NiC on a PCI riser

    1ea. FAS2020 with 2ea 1Gb Nics configured as a “Vif” (basically teamed in NetApp-speak)

    Here’s the rub. I’ve got ONE CAT-3750 (48-ports) to use as the ONLY physical switch for ALL these connections….oh…and maybe 12 ports free (4 ports for each ESX 5.0 host to use)…I can scare up a lot more 1Gb ports as I P-to-V existing hosts….but NOT before.

    I was thinking on each server:

    One vswitch–>one pNic—>management network
    One vswitch–>two pNics–>NFS storage network—>FAS2020
    One vswitch–>one pNic—>VM Network—>client IP network

    Is two pNiCs per server more inportant for storage or network access for the clients to reach my VMs? I don’t believe I can VLAN or LACP anything at this time…so I’ll be reliant on VMware exclusively for load balancing/traffic control, etc.

  14. Matt says :

    I meant ESXi 5.0 with VCenter 5.0 as the clustering OS….not ESX 5.0…..sorry….

    Also…it looks like NFS on the NetApps is just thin-provisioned by default…so one NFS mount/datastore for all the hosts?….I can’t sVmotion anyway…it’s a single filer…with 1 Proc.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 33 other followers

%d bloggers like this: