I jumped on a call the other day to talk about iSCSI setup for a new FlashArray and the main reason for the discussion had to do with co-existence of a pre-existing array from another vendor. They were following my blog post on iSCSI setup and things didn’t quite match up.
To setup multi-pathing (the recommended way) for Software iSCSI is to configure more than one vmkernel port that each have exactly one active host adapter (physical NIC). You then add those vmkernel ports to the iSCSI software adapter and the iSCSI adapter will then use those specific NICs for I/O transmission and load-balance across those ports.
The issue appeared at the step of configuring the vmkernel ports. As I mentioned above, each vmkernel port must have only one active NIC and the rest must be designated as in-active (not even standby). If they are not configured as such, they will be filtered out of the available vmkernel ports that can be added to the iSCSI adapter:
Or if it is added because it is configured properly and then it is changed later (let’s say a new active adapter was added) it will become non-compliant and marked as not usable by the iSCSI adapter until it is fixed.
In this case the customer had one pre-existing vmkernel port with two active adapters AND were still using the Software iSCSI Adapter. There were no network port bindings in the iSCSI adapter though but it still saw storage fine, when they got to the step in my blog post to create the network port bindings it couldn’t be done because the vmkernel port was non-compliant.
The iSCSI adapter was using their previous storage fine, saw the paths and presented the volumes. So this begs the questions–what NIC/vmkernel ports was it using and how was it multipathed? Can you then actually use vmkernel ports with active-active host adapters? Is everything a lie? Santa…?
So the answer to all of this is “kinda, but not really.”
Network port bindings in a Software iSCSI Adapter serves two main purposes:
- Specifically indicate what vmkernel ports and by association, physical NICs (host adapters), iSCSI should use
- Provide for the iSCSI Adapter to manage multi-pathing
So essentially, you do not have to create network port bindings to get the Software iSCSI Adapter to work and access storage. If no port bindings exist, the iSCSI Adapter will
simply look for the first vmkernel port it finds (vmk0, then 1, then 2 etc.) that can communicate with the configured targets in the static target list in the iSCSI adapter refer to the vmkernel route table for the specified connection.
So while it is predictable to what vmkernel adapter it will choose, it is not obviously apparent by looking at the vmkernel port or the iSCSI adapter. There is no configured relationship between them really.
***UPDATE: The choice of vmk ports is more sophisticated than I mention above. It has to do with the vmkernel routing table as Andy Banta notes in the comments. Thanks so much Andy!***
Furthermore, the multipathing is not really active/active–even though there are two active adapters for the vmkernel port. Instead a single vmkernel port is chosen and only one of the host adapters/NICs is used at a time. So basically this turns the active-active host adapters into active/standby–so their configuration state is somewhat misleading. There is no load balancing across the adapters–they are just failed over when there is a failure.
So below we have zero network port bindings but we do have a vmkernel port with two active-active physical adapters (as indicated by the two orange lines).
If I were to run I/O to an iSCSI device we will see in esxtop that only one NIC is being used but the I/O is still spread across the array front end. So it is multipathed on the array but not on the host–all of the I/O is going through one physical adapter. The first image shows esxtop and the fact that only one vmkernel adapter (vmk1) is being used and only vmnic2. The second image shows the balance on the FlashArray across both controllers. You can see they are pretty well balanced.
Now what if we configure network port mappings in the Software iSCSI Adapter? In this case I have two vmkernel ports each with their own physical adapter (vmk1 and vmk2, with vmnic2 and vmnic3 respectively).
Let’s restart the I/O. You can see in esxtop that both vmkernel ports and their respective physical adapters are bring used and quite evenly. Also the FlashArray controllers remained balanced. Lastly note the increase in throughput by leveraging two physical adapters.
Besides improvements in multipathing, assigning network port bindings help make sure you do not accidentally remove access for the software iSCSI adapter by deleting the wrong vmkernel port. If you don’t have network port bindings and instead are just having the iSCSI adapter pick one, there is no protection against deletion because the Web Client or CLI or whatever doesn’t know about the relationship.
If you try to remove a vmkernel port that is not bound to the iSCSI adapter the “Analyze impact” check doesn’t warn you:
If it is bound to an iSCSI adapter you do get a warning:
Okay, so I think you get the point. Let’s all use network port binding from now on.
Before I go–a few recommendations:
- Do have Round Robin enabled and the IO Operations Limit set to one.
- DO set the iSCSI “LoginTimeout” to 30 seconds and “DelayedAck” to disabled. Set these on the dynamic target and let it cascade to the static targets. DO NOT set them on the iSCSI adapter itself–this will provide for the possibility of different targets that might have different recommendations down the road.
- DO use network port bindings to more than one physical adapter if possible. Redundancy and performance.
- DO NOT configure the physical adapter teaming and failover only at the virtual switch layer and then let the vmkernel ports inherit them. If you add a new active adapter at some time later the vmkernel ports will inherit them and become non-compliant. Always override the teaming and failover settings on the vmkernel ports themselves and choose just one active physical adapter.
- OPTIONALLY use Jumbo Frames–this is at your discretion.