Enhanced UNMAP script using with PowerCLI and RESTful API

***********UPDATE PLEASE REFER TO THE POST AT THIS LINK FOR UPDATED INFORMATION ON THIS SCRIPT***********************

The most common request I get for scripts here at Pure Storage is an UNMAP script using PowerCLI. I have a basic one here that does the trick–UNMAPs Pure Storage volumes in a vCenter. That being said it is pretty dumb–doesn’t tell you much about what happened other than what volumes it is reclaiming (or not reclaiming) and moves on. A few requests have come in recently for something a little more in-depth. Most notably the ability to see how much space has been reclaimed. This information cannot be gathered from the VMware side of things–it has to come from the FlashArray.

There are two options here–either use our REST APIs or use our PowerShell toolkit to get this information (which just wraps the REST calls). For this script I chose to use the REST API directly from within PowerShell. What this script does is:

  1. Connects to the vCenter and FlashArray
  2. Finds all of the datastores and counts how many are actually Pure Storage volumes (NAA comparison)
  3. Iterates through all of the datastores
  4. Skips it if it is not Pure
  5. If it is, the current data reduction ratio is reported and so the is current physical written capacity on the FlashArray.
  6. Runs UNMAP on the datastore
  7. Reports the new data reduction and physical space after UNMAP completes and how much was reclaimed.
  8. Repeats for the rest of the volumes.

The script reports all of this to the console window, but it always throws it in a log file through add-content. If you don’t want it to return the info to the console, simply delete the write-host lines. If you don’t want it to log, delete the add-content lines.

There are a few required parameters–vCenter information (IP, username, password), FlashArray info (IP, username, password), UNMAP block count and a log file location. These are hard-coded parameters, but that can easily be changed by altering it to a read-host.

You may also note that after each UNMAP the script sleeps for 60 seconds–I do this so I make sure the FlashArray has time to update its information right after the UNMAP. 60 seconds is VERY conservative–probably 10 or so is fine, so feel free to mess with that number if you don’t like waiting. I also have another sleep at the end of each datastore operation to give a quick chance to review the latest results before it starts spewing the next datastore information on the screen (note this update didn’t make it into the video demo below–it doesn’t wait after each datastore).

See the script in action below. Essentially I am deleting a bunch of VMs across 4 datastores and then running the UNMAP. You can see the space get reclaimed on the FlashArray.

Note: You need particular access (see a blog post about that here) to vCenter to run UNMAP. For the FlashArray only Read Only is needed (higher of course is fine too).

Get the script here:

https://github.com/codyhosterman/powercli/blob/master/unmapsdk.ps1

34 Replies to “Enhanced UNMAP script using with PowerCLI and RESTful API”

  1. Cody,

    We’ve tried to utilize your script, put in our site variables, went to run it, and we get the following. We are we missing? Is there a prior step to enable the REST APIs or anything?

    Invoke-RestMethod : {“msg”: “invalid credentials”}
    At line:46 char:13
    + $ApiToken = Invoke-RestMethod -Method Post -Uri “https://${purevip}/api/1.2/auth …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
    Invoke-RestMethod : {“msg”: “Invalid api_token”}
    At line:50 char:1
    + Invoke-RestMethod -Method Post -Uri “https://${purevip}/api/1.2/auth/session” -B …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

      1. I was using AD credentials. Is there specific credentials to Pure this user would need? I assumed to Pure it only needed RO, and to VCenter a bit more rights, so in that case I was using my own VCenter login.

      2. This script doesn’t make any change calls so read only should be fine. I have’t tested it out with read only though. Let me run a test and see if I have the same issue.

        1. I’m almost positive ours is the same issue as the poster above, dealing with self signed certs. I’m going to try with your new line of the script tomorrow.

  2. Would still using Pure self signed certs have anything to do with it? I have the same issue with the REST Snap Script. and it doesn’t matter whether I use AD credentials or pureuser. I am trying to run it against Purity 4.1.2

  3. Ok Cody I think we got it. We were making a mistake with credentials all along. But of course we hit something else. Should we be worried this message came up/and is taking a while to go away/run:

    Connection to FlashArray successful

    Connection to vCenter successful

    Initiating VMFS UNMAP for all Pure Storage volumes in the vCenter
    Searching for VMFS volumes to reclaim (UNMAP)
    Found 174 VMFS volume(s).
    get-scsilun : 3/17/2015 9:02:16 AM Get-ScsiLun ScsiLun info is available only for Vmfs datstores.
    At line:49 char:26
    + $purevols = $datastores |get-scsilun |where-object {$_.CanonicalName -like “naa. …
    + ~~~~~~~~~~~
    + CategoryInfo : InvalidArgument: (:) [Get-ScsiLun], VimException
    + FullyQualifiedErrorId : Client20_StorageServiceImpl_GetScsiLun_InvalidDatastoreType,VMware.VimAutomation.ViCore.Cmdlets.Commands.Host.GetScsiLun

    1. Nah this probably means you have NFS in your environment. I have updated the script so that this error doesn’t appear anymore (it now can detect and skip NFS) in the latest version.

      1. We have 174 VMFS volumes, 18 total Pure datastores. Would you recommend any other tweaks to the script based on my site’s details?

      2. Looks like it runs solidly, Cody. I mentioned our datastore count just because, as I am sure you’re aware, the checks on whether they are Pure or not, just take time. If there’s a way to skip them quicker, great, if not, that’s fine. Before your script we had nothing but a manual process anyway, and this definitely beats that. Thanks again.

      3. and I spoke to soon. We ran the script against a test cluster of ours, it found the Pure datastore as well as ID’d the local datastore as not Pure, but it then spit this error at us:

        This volume has a data reduction ratio of 9.872 to 1 prior to reclamation.
        This volume has 12.066 GB of data physically written to the SSDs on the FlashArray prior to reclamation.

        Initiating reclaim…Operation time will vary depending on block count, size of volume and other factors.
        Permission to perform this operation was denied. Required privilege ‘Host.Config.Storage’ on managed object with id ‘ReflectManagedMethodExecuter-ManagedMethodExecuter-34’.
        At line:132 char:13
        + $esxcli.storage.vmfs.unmap($UNMAPBlockCount, $datastore.Name, $null) …
        + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo : OperationStopped: (:) [], NoPermission
        + FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.NoPermission

  4. The latest version of the script removes that first sort which takes forever (starts with $purevols =) and it isnt really necessary. That will save a lot of time. Remove the subsequent line too that has $purevols in it. Or just download my latest version.

    For the error, how long did it take to get to that step? Maybe the vCenter connection timed out?

      1. Just to clarify, the vcuser and pureuser I’m using has RO access to VCenter and the Pure array. That is enough, correct?

      2. Ah that will do it. You only need RO on the FlashArray but the UNMAP process in ESXi is a change process so you need admin rights to that object. I will add that in the post.

      1. Not a pest at all! Sorry my lab was being moved and I just got it up but then went on vacation for a week and a half and just got back. I plan on looking into this in the next few days. Apologize for the delay.

  5. Vacation? What? Who ever heard of vacations…..

    Thanks again Cody. Looking forward to automating this process.

  6. Hi Cody, great script, thank you. However, we have a vcenter infrastructure spread across multiple sites, with a PURE array at each site. Any way to allow for it working through more than one pure array or should I just have a script for each site and just connect to a single VM host, rather than vcenter server, and specify the PURE array for that site?
    Kind regards,
    Neil

    1. Yeah you can create connections for multiple arrays. You will just need to create multiple session /connection variables, one for each array. You will just need to run the connect/session create commands multiple times, storing them in differently named variables.

  7. We have deleted vm in vcenter, which was using pure datastore, now in vcenter, we are getting proper free space, but in pure storage still volume is showing used space 157 GB, so how can we reclaim free space on pure storage, our purity version is 6.1.16 & we are using vcenter7.0.3.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.