VMware iSCSI Multipathing: Setting Up with Distributed vSwitches

Sorry the title is a bit of a mouthful. I have written some posts on iSCSI in the past, around setup:

Setting up iSCSI with VMware ESXi and the FlashArray

Configuring iSCSI CHAP in VMware with the FlashArray

Another look at ESXi iSCSI Multipathing (or a Lack Thereof)

These have been on various parts, but primarily the setup around standard vSwitches, which generally, in at least in larger environments, is not the norm. Distributed vSwitches are. I have seen a few posts on doing this with the old C# client, but not the vSphere Web Client. Reference those posts here:

Creating and Configuring an iSCSI Distributed Switch for VMware Multipathing

So with the amount of questions I have received on it, it is probably worth putting pen to paper on it. Nothing profound here, basically a walkthrough.

This is of course assuming you are doing port binding. If you are not, then just the standard software iSCSI setup (as described in the above 1st post) is needed.

Creating the Distributed vSwitch

The first step, if you haven’t already done so is creating the distributed vSwitch. Under the networking tab, right-click on your datacenter, choose distributed virtual switches and then “New…”

Name it whatever. I am going to use iSCSI to make it easier to identify from my others.

The next screen choose your version, basically whatever the minimum version of the hosts participating in your DvSwitch is the version you should choose. I’ll go with 6.5.0.

Now the number of uplinks. This is how many physical NICs each host will be contributing to this distributed switch. For redundancy, I will use 2. I will give the port group the name iSCSI-PG1.

Finish the wizard.

Now you will see a new DvSwitch created, with an uplink object and your port group(s).

Now depending on how many physical NICs you mean to add to your iSCSI initiator configuration dictates how many port groups you need. This is due to the fact that your vmkernel adapters, when used with iSCSI, must be connected to one and only one physical NIC. In DvSwitches, this vmkernel port inherits the physical NIC configuration of the port group. Therefore, for each NIC, add one port group. I plan on using two NICs, so I need to create one more port group (since the first wizard creating.

Right-click the DvSwitch and choose “Distributed Port Group”, then “New Distributed Port Group”…

Give the PG a name and accept the defaults (unless you know you need changes):

If you have a special VLAN you can add it here.

And done:

Add Host Networking to a DvSwitch

Now right-click on the DvSwitch and choose “Add and Manage Hosts…”

Next we will add the hosts. Note that this step and the next can be done in one wizard, but for the sake of understanding I will split them out.

Click on “Add hosts”.

Click on the “New hosts…” button…

Choose your hosts…

If all of your hosts are the same (hopefully they are) you can select the template option below. For large environments (lots of hosts) I highly recommend using this option. I will follow this. If your hosts have different configurations then you will need to not use the template mode. The process is basically the same, except you need to repeat wha

Template-based host configuration

If you choose the template option, follow these steps:

First choose a template host:

Now choose the defaults to add physical adapters and vmkernel ports:

On the template host, choose your NIC and then click “Assign Uplink”:

Now either choose auto-assign uplink, to have vCenter choose which uplink to assign to that physical adapter, or pick one yourself. Do this for each NIC/uplink pair on that first host.

Now that the template host is done, choose the “Apply to all” to the rest of the hosts:

This will apply the same NIC/uplink config to each host.

The next step, is vmkernel creation. Since I am using template mode, I will do this on one host and then use the wizard to propagate the configuration to the other hosts.

So select your host and choose “New adapter”. This takes you through a familiar wizard for creating standard vmkernel adapters.

You will now want to choose browse and choose a port group from the DvSwitch to associate the vmkernel port to. If one does not show up, it means you deselected that option when creating the DvSwitch. Go back and create one if so.

Remember I named my first PG “iSCSI-PG”.

Click next. If you want additional services to use this vmk, choose them. Otherwise, click Next. The vmkernel ports will be named vmk1, vmk2 etc. automatically.

In the next screen, choose the IP info. I will choose the default DHCP. Enter as appropriate.

Finish the vmkernel adapter wizard. Now repeat for each PG for each host. I have two PGs, so I need to create two vmkernel ports. Like so:

Once the template host is done, choose “Apply to all” to copy the config to all other selected hosts.

Confirm the copy and the config will be copied:

You probably understand now why using the “template” mode is a good idea. If your hosts are all the same, you just have to do this once, instead of once per host. Saves A LOT of time.

Now analyze the impact. If nothing is being moved from something in use, you should be green. If something is impacted, triple check what is going on.

So now let’s take a look at our adapters. If we go to the host and clusters view in vCenter, click on one of my hosts and then go to Configure, and then vmkernel adapters, we see the new ones:

But no IPs. They didn’t get a DHCP address. In my case, this is due to not having a VLAN configured, as my DHCP server uses the VLAN 2202. Furthermore, my port groups access all of my physical NICs (and therefore its vmkernel adapters), but software iSCSI requires that all of the vmkernel adapters only have 1 active NIC and zero standby. (see info on this requirement here) Any others must be unused. So that needs to be changed on the port group.

Both of these things are configured on the port group.

NOTE: Of course, if you are not using VLANs you can skip the VLAN part, but you do need to follow the steps for NIC binding.

So back in my distributed switch, right-click on the port group and choose “Edit Settings…”:

So first, click on the VLAN pane and add your VLAN:

Then click on teaming and failover:

This has two uplinks (physical NICs). Move all but one down to unused. Each port group should have its own active uplink. So my iSCSI-PG1 will use uplink 1. iSCSI-PG2 will use uplink 2. Do the same for your environment, ensuring each port group intended for iSCSI uses just one uplink and it is uniquely used by that port group (in other words, it is not set to active for another port group also intended for iSCSI).