Home > Hyper-V Server > Jumbo Frames on Hyper-V Server

Jumbo Frames on Hyper-V Server

Enabling Jumbo Frame support in Hyper-V Server 2008 R2 (or Windows Server Core) has proven to be a bit of an adventure.  It really just involves setting the MTU size, but it has to be done in the OS (to affect the TCP/IP stack) as well as the network cards’ driver.  Since Core versions of Windows do not have a network control, setting the MTU on the cards proves to be a bit of a trick.  This is what I had to do to enable Jumbo Frames on several iSCSI nics, and since it differs for Intel vs Broadcom adapters, there are two procedures.

I should point out that this does not address configuring the network switch that these nics are attached to.  That is a whole ‘nother can of worms, but suffice it to say that the switch must not only support Jumbo Frames but have that support enabled, along with a whole host of other settings.

Enable Jumbo Frames on the OS

The first thing you need to do is make sure that your server will allow jumbo frames.  You do this by setting the MTU on your adapters to 9000.  The easiest way to do this is by running a netsh command on each adapter you want to use Jumbo Frames.

Get a list of interface names by running “netsh int show int

Admin State    State          Type             Interface Name
-------------------------------------------------------------------------
Enabled        Disconnected   Dedicated        Local Area Connection 2
Enabled        Connected      Dedicated        Bcom-GB3-iSCSI-A
Enabled        Connected      Dedicated        Local Area Connection
Enabled        Connected      Dedicated        Local Area Connection 3
Enabled        Connected      Dedicated        Local Area Connection 4
Enabled        Connected      Dedicated        Bcom-GB4-iSCSI-B
Enabled        Connected      Dedicated        Intel-GB1-Guest-B
Enabled        Connected      Dedicated        Bcom-GB2-Guest-A
Enabled        Connected      Dedicated        Intel-GB2-Guest-C
Enabled        Connected      Dedicated        Bcom-GB1-Mgmnt
Enabled        Connected      Dedicated        Intel-GB3-iSCSI-C
Enabled        Connected      Dedicated        Intel-GB4-Migration

In this case I have already re-named the Interfaces that I intend to use for iSCSI.  You might just see a whole list of “Local Area Connection” interfaces.  You can use ipconfig or netsh to further identify which ones you want to use.

Now for each interface you want jumbo frames enabled, run this command:

netsh int ipv4 set subint “<Interface Name Goes Here>” mtu=9000 store=persistent

Now you have to configure Jumbo Frames in the driver for each interface.

Enable Jumbo Frames on Intel cards

The Intel driver stores it’s “Jumbo Frame” settings in the registry.  Thankfully, Hyper-V Server (and Windows Core) comes with Regedit, so you can just launch that from command line (regedit.exe) and browse to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\Interfaces

Here you will see all the network interfaces listed by GUID.  I have found that the easiest way to determine which GUID is which adapter is by finding the IP address and being able to correlate it to the right Interface name.

At this point you should start making a list to help keep things straight.  Copy the GUID into notepad and list the IP address next to it and do this for each card you want to configure.  So for this server, my list looks like this:

SERVERNAME {7A310D71-217C-4E4A-9DA7-43299A76CBD5} 172.16.0.9
SERVERNAME {7BC7F3B9-B245-4579-82CB-C94161BFDBC1} 172.16.0.7
SERVERNAME {8BA5076E-0FC3-4D20-9609-654F228EE6BD} 172.16.0.6
SERVERNAME {98ABBECA-B8A2-41D2-9550-8B571E50F49A} 172.16.0.8

Now we have to navigate to a new registry key to configure the driver.  Go here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}

Here you will again see a list of all network interfaces, only this time they are under 4 digit identifiers.  From here, search for the GUID that you copied to your list and you should find it as the “NetCfgInstanceId” key of one of the adapters.  Once found, it’s not a bad idea to update your list to keep track of what’s what.  Mine looks like this now:

SERVERNAME {7A310D71-217C-4E4A-9DA7-43299A76CBD5} 172.16.0.9  0009 Intel
SERVERNAME {7BC7F3B9-B245-4579-82CB-C94161BFDBC1} 172.16.0.7  0005 Broadcom
SERVERNAME {8BA5076E-0FC3-4D20-9609-654F228EE6BD} 172.16.0.6  0004 Broadcom
SERVERNAME {98ABBECA-B8A2-41D2-9550-8B571E50F49A} 172.16.0.8  0008 Intel

Scroll up to find the “*JumboPacket” key and double click it to change the default value of 1514 to 9014.  Note the extra 14 bytes here represents packet headers that normally are not counted in MTU size.

Repeat this for each Intel adapter you need to configure, and then reboot the server for the setting to take effect.

Enable Jumbo Frames on Broadcom cards

First make sure you have the latest Broadcom drivers.  Make sure you get the 2008 R2 x64 set.

If you haven’t already, then download and install the driver and then reboot the host.  Note: Make sure you migrate any existing guest servers off the host before you install the drivers.  The temporary outage of the card due to the update seems to make a failover cluster angry.

Now get the Broadcom Management Application suite.  Again, get the x64 set from the same page.

Install the management app.  I opt’d not to install the BASP component (see screenshot below) since we do not want failover or teaming in this scenario.  It’ll likely warn you that you need the dotNet Framework 2.0 and you should be able to ignore this because the installer just does not recognize the “Core” framework, but the application still runs.  To make sure you do in fact have the framework installed, run “oclist | findstr /i netfx” and look for a line stating that NetFx is installed.  For example, “Installed:NetFx2-ServerCore”.  If not, you can install it by running “start /w ocsetup NetFx2-ServerCore” or instead you can install dotNet 3.0 and 3.5 by running “start /w ocsetup NetFx3-ServerCore”.

From C:\Program Files\Broadcom\BACS run “BACSCLi” to run in interactive mode.  It will show you a list of all network adapter drivers installed.  You only care about the “NDIS” adapters so enter “list ndis” and you’ll see something like this:

C  MAC          Dev Type Name
-  ------------ -------- ----------------------------------------------------
0  001B214285B8 NDIS     [0000] Intel(R) Gigabit ET Quad Port Server Adapter
1  001B214285B9 NDIS     [0007] Intel(R) Gigabit ET Quad Port Server Adapter #2
2  001B214285BC NDIS     [0008] Intel(R) Gigabit ET Quad Port Server Adapter #3
3  001B214285BD NDIS     [0009] Intel(R) Gigabit ET Quad Port Server Adapter #4
4  0026B9429866 NDIS     [0002] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient)
5  0026B9429868 NDIS     [0003] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #2
6  0026B942986A NDIS     [0004] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #3
7  0026B942986C NDIS     [0005] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #4

If you did the Intel configuration you’ll notice the four digit number in square braces of the Name field matches the Control\Class registry key.

Use some combination of “ipconfig /all” in another window or CtxAdmTools’ Visual Core Configurator 2008 or the four digit registry code to identify the adapter that you want to configure.  In this example we want Connection #6.  Select it by using “select 6” or whatever number is in the “C” column that matches your adapter.  Now validate that you have selected the correct adapter by reviewing some of its details.  Run “info” to see it’s MAC/IP, etc.

Vital Signs
-----------
MAC Address:              : 00-26-B9-42-98-6A
Permanent MAC Address:    : 00-26-B9-42-98-6A
IPV4 Address              : 172.16.0.6
Link Status               : UP
Duplex:                   : Full
Speed(in Mbps):           : 1000
Offload Capabilities      : TOE,LSO,CO,RSS
Mtu                       : 1500

Driver Information
-----------
Driver Status:            : Loaded
Driver Name:              : bxnd60a.sys
Driver Version:           : 5.0.13.0
Driver Date:              : 07/30/2009

Notice the MTU setting is set to 1500 by default.  Now run “cfg advanced” to list its advanced properties.

Advanced
--------
Ethernet@WireSpeed:                     Enable (Default)
Flow Control:                           Disable
IPv4 Checksum Offload:                  Tx/Rx enabled (Default)
IPv4 Large Send Offload:                Enable (Default)
IPv6 Checksum Offload:                  Tx/Rx enabled (Default)
IPv6 Large Send Offload:                Enable (Default)
Interrupt Moderation:                   Enable (Default)
Jumbo MTU:                              1500 (Default)
Locally Administered Address:           Not Present (Default)
Number Of RSS Queues:                   8 (Default)
Priority & VLAN:                        Priority & VLAN enabled (Default)
Receive Buffers:                        750 (Default)
Receive Side Scaling:                   Enable (Default)
Speed & Duplex:                         Auto (Default)
TCP Connection Offload (IPv4):          Enable (Default)
TCP Connection Offload (IPv6):          Enable (Default)
Transmit Buffers:                       1500 (Default)
VLAN ID:                                0 (Default)
Wake Up Capabilities:                   Both (Default)

Run “cfg advanced “Jumbo MTU”=9000” to set Jumbo frames to 9000 bytes.  Note that you do not have to account for the 14 bytes of header data here.  It’ll take a few seconds to apply the change but you should not need to reboot (yay!).  You can now run “cfg advanced” and “info” to list the settings and ensure that the MTU is in fact set to 9000.

You should also enable Flow Control for Transmit (Tx) and Receive (Rx).  With the correct adapter already selected, run “cfg advanced “Flow Control”=”Rx & Tx enabled””.

Once that is complete you can enter “q” to exit BACScli or start over using “list ndis” and select another interface to configure.  You can also use this utility to select non-Broadcom adapters to display some of their info like MTU size.

Testing Jumbo Frames

To test if Jumbo Frames are working you can ping another host target that also supports Jumbo Frames.  The easiest way that I have found to do this was to just change the IP of your test NIC and your test target NIC to something that no other adapter has.  This is because there is no way to tell windows specifically what NIC to send traffic over, so setting the NIC’s to their own network ip space is the only way to ensure that the ping traverses a particular adapter.

For example, I changed the source test nic to 172.16.1.4 and the target to 172.16.1.8 and no other adapters on either host saw set in the 172.16.1.* range.

First try a normal “ping 172.16.1.8” and it should work fine.  Then use “ping -f -l 6000 172.16.1.8” to test jumbo frames and it should also work, only this time you’ll see it sending 6000 bytes instead of 32.

So that about covers it.  I had to do this for each of the 32 iSCSI nics spread across the 8 host servers, but it works!  You should be aware that if you do a driver update or if you share a NIC with a virtual network (as a Hyper-V Host) your settings may be lost and you’ll have to go through this again.

Advertisements
  1. 2010-Jun-10 at 12:49 pm

    Hi
    I’ve been struggling with iscsi/hyper-v server R2 and jumbo frames for over a week now and your blog may have just saved my sanity and job, thank you so much.
    Simon

  2. 2010-Jun-11 at 05:40 am

    I am glad to hear that it helped! And don’t feel bad, I struggled with it too. Now that you have jumbo frames enabled you get to do the fun part: Creating iSCSI Favorite Targets to your SAN. Another blog post I should have written by now.

  3. helen
    2010-Oct-29 at 07:09 am

    Could you tell me how to set flow control on the intel cards?

    Also i have windows 2008 (pre R2) and cannot install .Net 2, is there another way to set jumbo frames for broadcom?

    • helen
      2010-Oct-29 at 08:40 am

      My network card drivers didn’t need .net to install BACSCLi after all, jumbo frames are now working.

      Thank you so much

    • 2010-Oct-30 at 09:09 am

      I have found that by default, Flow Control on the Intel NICs is enabled by default. If you want to validate that is is enabled, look under the GUID key located under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\ where the “*JumboPacket” setting is that you found earlier. There you will find a REG_SZ item named “*FlowControl” and you can set it to 0, 1, 2 or 3. Here’s what they mean:

      0: Disabled
      1: Tx Enabled
      2: Rx Enabled
      3: Rx & Tx Enabled (This is the one you want)

      The only place you might need to make a change is on the Switch Port that it is connected to and/or the SAN that it is destined for.

  4. Muhammad Gurwara
    2011-Feb-28 at 11:25 am

    We have are having an issue with Jumbo Frames on Hyper-V where we are running Hyper-V role in Windows 2008 R2. It is a blade Chassis with blades running Window 2k8 R2 with Hyper-V role. I have enabled jumbo frames on the chassis switch, the physical NIC on the blade dedicated to iSCSI VLAN, the parent W2k8 OS virtual adapter (for this VLAN), and the VMs. My iSCSI storage is attached directly to the chassis switch. Using a dedicated VLAN for iSCSI and has jumbo frames enabled i.e mtu=9000.

    The standard ping works fine as well as this ping without the -f option. I perform the following jumbo frame test

    ping -l 8000 -f -n 1

    The test fails with Request timeout, instead of getting a message that “Packet needs to be fragmented but DF bit is set”

    The above test succeeds if the target IP is within the same blade i.e. I can use above ping successfully by pinging a VM from the Hyper-V host or the Hyper-V host from the VM or from one VM to the other VM (on the same Hyper-V host). However, it does not work for anything outside the blade.

    I have tried the above command by enabling jumbo frame support on my laptop gbit NIC and connecting to a port configured for iSCSI VLAN on the same chassis switch and it works. So it leaves me to think that something is not right in Hyper-V.

    Appreciate any help

    • 2011-Mar-01 at 06:57 am

      Do not test jumbo frames by pinging from the host to a VM (or from a VM to the Host).

      Instead, test by pinging from the Host to the SAN or from one Host to another Host. This will send the packet out one of the iSCSI NICs, over your switch to another device that should be enabled for Jumbo frames. If you are pinging from a Host to a VM it’ll traverse the Host NIC and switch and will not be testing you iSCSI path.

      You could also try pinging the iSCSI switch if it has a management IP, however I have seen some switches that do not reply to jumbo framed pings even though they do pass jumbo frames successfully to other network devices.

      Also, do not try pinging an IP address that is on the same Host (like “iSCSI NIC A” to “iSCSI NIC B”) since that will never traverse the adapter either. It would be like pinging the local loopback IP and will always reply, so it’s not a very useful test.

      • Muhammad Gurwara
        2011-Mar-01 at 08:07 pm

        Thanks Shannon. I forgot to mention in my earlier reply that I did ping the iSCSI SAN from my VMs as well as the Base Hyper-V machines and the result was this, when:

        JF=0 and DF=0 —- Ping succeeds
        JF=0 and DF=1 —- Packet fragmentation message
        JF=1 and DF=0 —- Ping succeeds
        JF=1 and DF=1 —- Request Time Out

  5. cher cher
    2011-Aug-23 at 10:27 pm

    legend, thanks!

  6. AJBernard
    2011-Aug-31 at 09:49 am

    Muhammad Gurwara, Some blade enclosures or interconnect modules will nto support an MTU of 9000. Can you provide some model information for you setup?

  7. Slawek
    2012-Jul-18 at 01:14 am

    Intel with network drivers deliver also a commandline tool prosetcl to configure its NICs. For example – for setting Jumbo Packet (Jumbo Frames):
    1. we need to enumerate nics
    PROSetCL Adapter_Enumerate
    Number of adapters currently present: 6

    1) Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #36
    2) Intel(R) PRO/1000 PT Server Adapter #2
    3) Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #8
    4) Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #30
    5) Broadcom BCM5709C NetXtreme II GigE (NDIS VBD Client) #42
    6) Intel(R) PRO/1000 PT Server Adapter
    2. then we can enumerate setting for a specific card – I chose nic #2
    PROSetCL Adapter_EnumerateSettings 2
    2) Intel(R) PRO/1000 PT Server Adapter #2

    Settings:

    *ReceiveBuffers – 256
    *TransmitBuffers – 512
    *FlowControl – Rx & Tx Enabled
    *InterruptModeration – Enabled
    *IPChecksumOffloadIPv4 – Rx & Tx Enabled
    *JumboPacket – 9014 Bytes
    *LsoV2IPv4 – Enabled
    *LsoV2IPv6 – Enabled
    *PriorityVLANTag – Priority & VLAN Enabled
    *RSS – Enabled
    *SpeedDuplex – 1.0 Gbps Full Duplex
    *TCPChecksumOffloadIPv4 – Rx & Tx Enabled
    *TCPChecksumOffloadIPv6 – Rx & Tx Enabled
    *UDPChecksumOffloadIPv4 – Rx & Tx Enabled
    *UDPChecksumOffloadIPv6 – Rx & Tx Enabled
    AdaptiveIFS – Disabled
    EnablePME – Disabled
    ITR – Adaptive
    LogLinkStateEvent – Enabled
    MasterSlave – Auto Detect
    SPDEnabled – Disabled
    WaitAutoNegComplete – Off
    WakeOn – Wake on Magic & Directed
    WakeOnLink – Disabled
    NetworkAddress –

    3. to check a values accepted for JumboPacket
    PROSetCL Adapter_GetSetting 2 *JumboPacket

    2) Intel(R) PRO/1000 PT Server Adapter #2

    *JumboPacket – 9014 Bytes

    Valid Values: “Disabled”
    “4088 Bytes”
    “9014 Bytes”
    4. to set JumboPacket to 9014 bytes
    PROSetCL Adapter_SetSetting 2 *JumboPacket “9014 Bytes”

    2) Intel(R) PRO/1000 PT Server Adapter #2

    Setting value is already set to ‘9014 Bytes’

    I set this value before, so I got this message as above.

  8. Thomas
    2015-Jan-23 at 10:37 am

    What about Hyper-V 2012 R2?
    Are there the same chnages required?
    If a ping -f -l 8000 172.16.1.8 is working, is there everything ready for Jumbo Frames?

    Best regards
    Thomas

  1. 2012-Oct-20 at 12:12 pm
  2. 2014-Jan-26 at 04:36 am
  3. 2014-Jan-28 at 06:17 am
  4. 2014-Oct-03 at 08:38 am
  5. 2015-May-18 at 08:32 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: