Archive

Posts Tagged ‘iSCSI’

Jumbo Frames on Hyper-V Server

2010-Jan-13 17 comments

Enabling Jumbo Frame support in Hyper-V Server 2008 R2 (or Windows Server Core) has proven to be a bit of an adventure.  It really just involves setting the MTU size, but it has to be done in the OS (to affect the TCP/IP stack) as well as the network cards’ driver.  Since Core versions of Windows do not have a network control, setting the MTU on the cards proves to be a bit of a trick.  This is what I had to do to enable Jumbo Frames on several iSCSI nics, and since it differs for Intel vs Broadcom adapters, there are two procedures.

I should point out that this does not address configuring the network switch that these nics are attached to.  That is a whole ‘nother can of worms, but suffice it to say that the switch must not only support Jumbo Frames but have that support enabled, along with a whole host of other settings.

Enable Jumbo Frames on the OS

The first thing you need to do is make sure that your server will allow jumbo frames.  You do this by setting the MTU on your adapters to 9000.  The easiest way to do this is by running a netsh command on each adapter you want to use Jumbo Frames.

Get a list of interface names by running “netsh int show int

Admin State    State          Type             Interface Name
-------------------------------------------------------------------------
Enabled        Disconnected   Dedicated        Local Area Connection 2
Enabled        Connected      Dedicated        Bcom-GB3-iSCSI-A
Enabled        Connected      Dedicated        Local Area Connection
Enabled        Connected      Dedicated        Local Area Connection 3
Enabled        Connected      Dedicated        Local Area Connection 4
Enabled        Connected      Dedicated        Bcom-GB4-iSCSI-B
Enabled        Connected      Dedicated        Intel-GB1-Guest-B
Enabled        Connected      Dedicated        Bcom-GB2-Guest-A
Enabled        Connected      Dedicated        Intel-GB2-Guest-C
Enabled        Connected      Dedicated        Bcom-GB1-Mgmnt
Enabled        Connected      Dedicated        Intel-GB3-iSCSI-C
Enabled        Connected      Dedicated        Intel-GB4-Migration

In this case I have already re-named the Interfaces that I intend to use for iSCSI.  You might just see a whole list of “Local Area Connection” interfaces.  You can use ipconfig or netsh to further identify which ones you want to use.

Now for each interface you want jumbo frames enabled, run this command:

netsh int ipv4 set subint “<Interface Name Goes Here>” mtu=9000 store=persistent

Now you have to configure Jumbo Frames in the driver for each interface.

Enable Jumbo Frames on Intel cards

The Intel driver stores it’s “Jumbo Frame” settings in the registry.  Thankfully, Hyper-V Server (and Windows Core) comes with Regedit, so you can just launch that from command line (regedit.exe) and browse to the following key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\Interfaces

Here you will see all the network interfaces listed by GUID.  I have found that the easiest way to determine which GUID is which adapter is by finding the IP address and being able to correlate it to the right Interface name.

At this point you should start making a list to help keep things straight.  Copy the GUID into notepad and list the IP address next to it and do this for each card you want to configure.  So for this server, my list looks like this:

SERVERNAME {7A310D71-217C-4E4A-9DA7-43299A76CBD5} 172.16.0.9
SERVERNAME {7BC7F3B9-B245-4579-82CB-C94161BFDBC1} 172.16.0.7
SERVERNAME {8BA5076E-0FC3-4D20-9609-654F228EE6BD} 172.16.0.6
SERVERNAME {98ABBECA-B8A2-41D2-9550-8B571E50F49A} 172.16.0.8

Now we have to navigate to a new registry key to configure the driver.  Go here:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}

Here you will again see a list of all network interfaces, only this time they are under 4 digit identifiers.  From here, search for the GUID that you copied to your list and you should find it as the “NetCfgInstanceId” key of one of the adapters.  Once found, it’s not a bad idea to update your list to keep track of what’s what.  Mine looks like this now:

SERVERNAME {7A310D71-217C-4E4A-9DA7-43299A76CBD5} 172.16.0.9  0009 Intel
SERVERNAME {7BC7F3B9-B245-4579-82CB-C94161BFDBC1} 172.16.0.7  0005 Broadcom
SERVERNAME {8BA5076E-0FC3-4D20-9609-654F228EE6BD} 172.16.0.6  0004 Broadcom
SERVERNAME {98ABBECA-B8A2-41D2-9550-8B571E50F49A} 172.16.0.8  0008 Intel

Scroll up to find the “*JumboPacket” key and double click it to change the default value of 1514 to 9014.  Note the extra 14 bytes here represents packet headers that normally are not counted in MTU size.

Repeat this for each Intel adapter you need to configure, and then reboot the server for the setting to take effect.

Enable Jumbo Frames on Broadcom cards

First make sure you have the latest Broadcom drivers.  Make sure you get the 2008 R2 x64 set.

If you haven’t already, then download and install the driver and then reboot the host.  Note: Make sure you migrate any existing guest servers off the host before you install the drivers.  The temporary outage of the card due to the update seems to make a failover cluster angry.

Now get the Broadcom Management Application suite.  Again, get the x64 set from the same page.

Install the management app.  I opt’d not to install the BASP component (see screenshot below) since we do not want failover or teaming in this scenario.  It’ll likely warn you that you need the dotNet Framework 2.0 and you should be able to ignore this because the installer just does not recognize the “Core” framework, but the application still runs.  To make sure you do in fact have the framework installed, run “oclist | findstr /i netfx” and look for a line stating that NetFx is installed.  For example, “Installed:NetFx2-ServerCore”.  If not, you can install it by running “start /w ocsetup NetFx2-ServerCore” or instead you can install dotNet 3.0 and 3.5 by running “start /w ocsetup NetFx3-ServerCore”.

From C:\Program Files\Broadcom\BACS run “BACSCLi” to run in interactive mode.  It will show you a list of all network adapter drivers installed.  You only care about the “NDIS” adapters so enter “list ndis” and you’ll see something like this:

C  MAC          Dev Type Name
-  ------------ -------- ----------------------------------------------------
0  001B214285B8 NDIS     [0000] Intel(R) Gigabit ET Quad Port Server Adapter
1  001B214285B9 NDIS     [0007] Intel(R) Gigabit ET Quad Port Server Adapter #2
2  001B214285BC NDIS     [0008] Intel(R) Gigabit ET Quad Port Server Adapter #3
3  001B214285BD NDIS     [0009] Intel(R) Gigabit ET Quad Port Server Adapter #4
4  0026B9429866 NDIS     [0002] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient)
5  0026B9429868 NDIS     [0003] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #2
6  0026B942986A NDIS     [0004] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #3
7  0026B942986C NDIS     [0005] Broadcom BCM5709C NetXtreme II GigE (NDIS VBDClient) #4

If you did the Intel configuration you’ll notice the four digit number in square braces of the Name field matches the Control\Class registry key.

Use some combination of “ipconfig /all” in another window or CtxAdmTools’ Visual Core Configurator 2008 or the four digit registry code to identify the adapter that you want to configure.  In this example we want Connection #6.  Select it by using “select 6” or whatever number is in the “C” column that matches your adapter.  Now validate that you have selected the correct adapter by reviewing some of its details.  Run “info” to see it’s MAC/IP, etc.

Vital Signs
-----------
MAC Address:              : 00-26-B9-42-98-6A
Permanent MAC Address:    : 00-26-B9-42-98-6A
IPV4 Address              : 172.16.0.6
Link Status               : UP
Duplex:                   : Full
Speed(in Mbps):           : 1000
Offload Capabilities      : TOE,LSO,CO,RSS
Mtu                       : 1500

Driver Information
-----------
Driver Status:            : Loaded
Driver Name:              : bxnd60a.sys
Driver Version:           : 5.0.13.0
Driver Date:              : 07/30/2009

Notice the MTU setting is set to 1500 by default.  Now run “cfg advanced” to list its advanced properties.

Advanced
--------
Ethernet@WireSpeed:                     Enable (Default)
Flow Control:                           Disable
IPv4 Checksum Offload:                  Tx/Rx enabled (Default)
IPv4 Large Send Offload:                Enable (Default)
IPv6 Checksum Offload:                  Tx/Rx enabled (Default)
IPv6 Large Send Offload:                Enable (Default)
Interrupt Moderation:                   Enable (Default)
Jumbo MTU:                              1500 (Default)
Locally Administered Address:           Not Present (Default)
Number Of RSS Queues:                   8 (Default)
Priority & VLAN:                        Priority & VLAN enabled (Default)
Receive Buffers:                        750 (Default)
Receive Side Scaling:                   Enable (Default)
Speed & Duplex:                         Auto (Default)
TCP Connection Offload (IPv4):          Enable (Default)
TCP Connection Offload (IPv6):          Enable (Default)
Transmit Buffers:                       1500 (Default)
VLAN ID:                                0 (Default)
Wake Up Capabilities:                   Both (Default)

Run “cfg advanced “Jumbo MTU”=9000” to set Jumbo frames to 9000 bytes.  Note that you do not have to account for the 14 bytes of header data here.  It’ll take a few seconds to apply the change but you should not need to reboot (yay!).  You can now run “cfg advanced” and “info” to list the settings and ensure that the MTU is in fact set to 9000.

You should also enable Flow Control for Transmit (Tx) and Receive (Rx).  With the correct adapter already selected, run “cfg advanced “Flow Control”=”Rx & Tx enabled””.

Once that is complete you can enter “q” to exit BACScli or start over using “list ndis” and select another interface to configure.  You can also use this utility to select non-Broadcom adapters to display some of their info like MTU size.

Testing Jumbo Frames

To test if Jumbo Frames are working you can ping another host target that also supports Jumbo Frames.  The easiest way that I have found to do this was to just change the IP of your test NIC and your test target NIC to something that no other adapter has.  This is because there is no way to tell windows specifically what NIC to send traffic over, so setting the NIC’s to their own network ip space is the only way to ensure that the ping traverses a particular adapter.

For example, I changed the source test nic to 172.16.1.4 and the target to 172.16.1.8 and no other adapters on either host saw set in the 172.16.1.* range.

First try a normal “ping 172.16.1.8” and it should work fine.  Then use “ping -f -l 6000 172.16.1.8” to test jumbo frames and it should also work, only this time you’ll see it sending 6000 bytes instead of 32.

So that about covers it.  I had to do this for each of the 32 iSCSI nics spread across the 8 host servers, but it works!  You should be aware that if you do a driver update or if you share a NIC with a virtual network (as a Hyper-V Host) your settings may be lost and you’ll have to go through this again.

Advertisements

Making iSCSI Targets via CMD

Adding iSCSI targets can be cumbersome when you have several volumes on the SAN and several network cards in several hosts.  At Orion there were 8 hosts with 4 NICs and 8 volumes to connect too.  That’s 256 iSCSI connections, and if you use the GUI to make these connections, then you’re talking about doing at least 15 clicks per connection (3840 total) and then try to keep everything straight.  Good luck!   

Rather than clicking 4,000 times (taking into account a few mistakes) I wrote a script that uses a command that Nate gave me to make a Multipath IO connection with iscsicli.exe.  It’s easy to use and should work on any flavor of Windows that uses Microsoft’s iSCSI Initiator (2008 Standard, Core, Hyper-V Server R2, Enterprise, Windows 7, etc).  You just need to discover what “position” your network iSCSI cards are in to prepare the script.   

First, log in to the server and from command line run “iscsicpl” to open the control panel.  Select any target, click Connect and then Advanced.  From here select the Microsoft iSCSI Initiator and then pull down the Initiator IP drop down list.  Use the screenshot below to determine what “Position ID” your iSCSI NICs use.  Then hit Cancel a couple times to back out without making changes.   

    

Now you can edit iSCSINICs.txt to list only the ID’s for each NIC that you want to connect.  Put one number per line for each NIC.  In this example, because the 172.16.0.* NIC’s are being used for iSCSI, you would use:  

5
6
7
8

I do not understand where the order for this list comes from, so I can’t explain why it is listed that way, but I can tell you that if you make changes to your network hardware (like a driver update or share the nic with a virtual network) the order can and likely will change and you’ll have to re-create the iSCSI connections.  So try to get that stuff settled before bothering with this too much.  

Next you can edit iSCSITargets.txt to list each of the iSCSI targets that you want to connect those NICs to.  For example:  

iqn.2001-05.com.equallogic:0-8a0906-d7899e704-8ba0000000b4b437-hv-hio-csv02
iqn.2001-05.com.equallogic:0-8a0906-f5b99e704-1860000000e4b437-hv-mio-csv02
iqn.2001-05.com.equallogic:0-8a0906-fc599e704-20a000000104b437-hv-mio-csv03
iqn.2001-05.com.equallogic:0-8a0906-ff199e704-bbc000000124b437-hv-mio-csv04
iqn.2001-05.com.equallogic:0-8a0906-01999e704-002000000144b438-hv-mio-csv05
iqn.2001-05.com.equallogic:0-8a0906-06799e704-ade000000164b438-hv-mio-csv06

You can pull the target names from the “Discovered Targets” list in iscsicpl by copying it from the “Connect to Target” window or copying it from the Properties page.  

Finally, save the txt files and run the makeTargets.cmd script.  It’ll loop through the list of targets and make a Multi-Path (MPIO) enabled connection for each NIC to it.  Repeat this process on each host.  Make sure to check the NIC ID’s, but you’ll likely only need to get the Targets once. 

@echo off
ECHO.
ECHO Create MPIO "Favorite Targets" for each Nic and Target in the txt files...
Pause
ECHO.</pre>
FOR /F %%T IN (iSCSITargets.txt) DO (
 echo %%T
 FOR /F %%N IN (iSCSINICs.txt) DO (
  echo Targeting Network Interface %%N
  iscsicli persistentlogintarget %%T T * * * %%N * 0x00000002 * * * * * * * * * 0
  REM Sleep for one second, just to let things sort of settle...
  PING -n 2 127.0.0.1>nul
 )
) 

start iscsicpl

Now that they are connected, your next task it to initialize the volume and create a partition…

Categories: Scripts Tags: ,