This is part 3 of this 7 part series. The series being:
- Mounting an unresolved VMFS
- Why not force mount?
- Why might a VMFS resignature operation fail?
- How to correlate a VMFS and a FlashArray volume
- How to snapshot a VMFS on the FlashArray
- How to mount a VMFS FlashArray snapshot
- Restoring a single VM from a FlashArray snapshot
This post is really the genesis of this entire series. I was asked to help troubleshoot at a customer site why the resignaturing and mounting of a VMFS volume wasn’t possible. Turns out it was because they had a few unresolved identical volumes present. So I went to put a post together and it turned into seven parts. Here is part three.
If you want to know what an unresolved volume is check out part one.
Reason 1: ESXi does not see the unresolved VMFS
This is a simple one but is an easy one to fall victim to. Most common issue is that the host has not been rescanned. This is an easy one to miss in a script. The other times is that if ESXi has already seen that exact SCSI device and it has been removed, it might be detached from the host, so you would then need to re-attach it first.
Reason 2: There are two unresolved VMFS volumes that share a source VMFS
This is the most annoying one. If you have two or more unresolved VMFS volumes presented but not mounted to the same host you cannot resignature or even force mount either of them. The important note here too is that they both also have to share the same source VMFS. So to explain an example:
- I have a VMFS “X” on SCSI device “A”
- I copy VMFS “X” to SCSI device “B” and SCSI device “C”. So there are two copies
- I connect/present/mask “B” and “C” to the ESXi host.
- Both “B” and “C” have the same same VMFS label and UUID
This situation will prevent you from mounting either one. The reason for this is that the resignature process takes in a VMFS label or a VMFS UUID to start. Since these identifiers in this situation are not unique it cannot be known which one is meant to be resignatured, so all you can do is reformat them. That is the only option allowed.
If you see these options grayed out, this is likely the issue. You can verify this by running the following command:
esxcli storage vmfs snapshot list
The results will return that the extent is not resignaturable and the reason why:
So you need to remove one of the devices from ESXi, either on the array or by running a detach operation.
Reason 3: A force mounted copy is present somewhere in the vCenter
This is another common one. If you have an unresolved VMFS that is force mounted on another host, anywhere in that vCenter, you will be unable to resignature additional copies of that original VMFS until that force-mounted VMFS is removed or resignatured.
vCenter will allow you to finish the resignature wizard, but the operation itself will fail.
Why is this? Well it is for you protection. Two reasons. First is that as discussed in my previous post on “why never force mount”. When you force mount it only appears force-mounted on one host when attempted in vCenter. vCenter will not allow you to force mount it on any other host because that UUID is already present in it in an “unresolved” mode. So if you want to force mount the same volume on more than one ESXi host at once you need to login to them individually. Since ESXi does not have the knowledge of hosts this vCenter check is not present and the process is allowed. If the VMFS is force mount on that ESXi host the ESXi host will still allow it because it can compare the NAA.
vCenter does not do NAA comparisons to see if the UUID that is force mounted is on a different volume. I suspect this is a safety feature too, because there are situations where the same SCSI device can have a different NAA or identifier based on some settings on the array. In my VMAX days, SPC-2 compliancy was a setting that could be configured on a port by port basis or even on a host-by-host basis and if it was off on one and on on another, the device was seen differently even though it was the same device.
Therefore you could accidentally resignature a device that is online and in use in a force-mounted fashion.
ESXi does allow you to do this by itself, even if it is force-mounted locally. I suppose they think it is extremely unlikely for the same device to be presented to the same host in two different ways (it is very unlikely).
So basically, if vCenter sees a UUID that is force mounted, you will not be able to resignature additional copies of that source VMFS from vCenter. You need to go to an individual host to do this to skip this check. I just recommend NEVER force-mounting, so you never run into this problem.