Introduction
During my day-to-day profession I encounter a lot of scripts. Some of them are good, most of them suck (from my point of view). I would not say that I’m an excellent script writer, but I’ve found my way of doing things (scripting-wise) which are making my (scripting) life much easier.
Scripts are mostly used to solve a particular problem, a repeating problem: problems that need to be resolved multiple times on different objects. And yes, scripting are ideal for those situations.
I still see a lot of admins stay away from scripts, as they are “too complex”, “too much work”, or i hear other excuses as “It’s too much work” . These are the same admins that have to do the same task over and over again, not having enough time available to develop or implement new technologies.
I believe that with general scripting knowledge you will become a far better administrator for your company, as you are building consistent environments which lowers the amount of errors/failures.
When I have to execute changes, the first thing that I do is open the PowerShell Integrated Scripting Environment (ISE) (or better Visual Code with PowerShell integration enabled) and start scripting my changes to overcome simple to make, manual errors, as scripting forces me to re-think some of the configuration changes.
This blog is all about sharing a collection of methods that I use when I’m creating PowerShell (POSH) scripts including some examples.
Most of my time I’m working with PowerCLI (for vSphere), PowerNSX (for NSXv) or my own modules. This blog will not cover those modules specifically, it’s more sharing my lessons learned so you can become a better script-writer.
Shifting from “problem solving scripting”, to “desired state scripting”
When I’m scripting, the majority of the time is not consumed by actually solving a particular problem, but rather getting the script to be as adaptive/foolproof as possible. A lot of the scripts that I find online are written for one environment only (not your environment). They lack validations built into them, making them prone for error. Those online available scripts are usually excellent in configuring objects in bulk, overwriting everything when they run and not checking any previous configuration. Be aware that these scripts can be dangerous when used improperly.
Here’s an example of how-not-to-script:
Get-VMhost | get-Portgroup -name "MGMT" | Set-PortGroup -vlanid 1001
This script will reconfigure portgroup “MGMT” on all ESXi host to vlan id 1001.
Yes, this will work and there is nothing wrong with it when configuring in bulk for the writers-environment. But this script will not work in a dual datacenter environment where each datacenter has its own management network: This script will cause a datacenter outage. It’s a good advice to make manual selection for the input of your script, making it adaptable to any environment.
Using Out-Gridview for quick manual selections.
I’m using Out-Gridview in all of my script where I need to make a quick manual selection. It’s an easy-to-use data grid viewer, which enables me to filter data based on a manual selection. Here is an example which enhances the previous example:
$clusterVMs = Get-Cluster | Out-GridView -PassThru | Get-VM $clusterVMs | get-Portgroup -name "MGMT" | Set-PortGroup -vlanid 1001
Now only the ESXi hosts in a manual selected cluster will be changed, safeguarding against unwanted changes in any environment.
Transform your script in to a desired state script
Remember that configuring stuff usually takes more time than querying and filtering stuff.
That’s why I recommend to filter the input BEFORE configuring stuff, instead of reconfiguring everything and has a additional benefit.
Let’s first start with a example:
$VlanID = "1001"
$clusterVMs = Get-Cluster | Out-GridView -PassThru | Get-VM
$clusterVMs | get-Portgroup -name "TEST-VLAN" | Where-Object {$_.vlanid -ne $VlanID} | Set-PortGroup -vlanid $VlanID
The change in this line of code is the Where-Object cmdlet: It will filter out the port groups that already have the proper VLAN id configured. If only 50% of the portgroups already have the proper VlanID configured, the script will run 50% faster!
This addition may look like an unnecessary step, but in fact it enables you to run a script over-and-over again without a bulk of changes: it only changes the configuration were needed. Changing this from a bulk-change script to a validation -or- desired state script.
You can even schedule the execution of this script on a daily basis, making your environment always compliant to your configuration.
I’m aware there are a ton of tools available that can do the same, but some admins have never ever touched a script and they have to start somewhere.
But the next question is: how to build the Where-Object command yourself?
The Where-Object cmdlet is used here to filter the “inline” data. I’m using the “|”- pipe command to take the output of the previous command and use it as an input for the following command. This is a commonly used method within PowerShell, called “piping“. In this example the “$clusterVMs| Get-PortGroup -name “TEST-VLAN” command will return a variable including all available “TEST-VLAN” portgroups of selected ESXi host.
The Where-Object cmdlet is used to filter out the portgroups that do not have VLAN ID 1001 configured for those ESXi hosts.
The Where-object command is followed by a scriptblock (surrounded by the curly brackets). In this example it is “{$_.vlanid -ne $VlanID}” which filters the data. At first this can be daunting to interpret, after a while it almost like the English language. Here is a small explanation.
"Give me all object where portgroup's vlanid is not equal to my VlanID variable."
Where { $_.vlanid -ne $VlanID}
The $_ used in this scriptblock refers to the output variable of the previous command through piping..
If you are using a good editor (like Visual Code), the formatting is done almost automatically for you; including adding the curly brackets “{ }” for the scriptblock and when pressing TAB it will autofill any available variables. Writing this statement is is just a matter of practice, practice and practice… and be aware Google is your friend.
If .. Then .. Else use case #1
Another method I’m using to make my script foolproof is by using the “If .. then.. else” command.
I’m sure you already know this command, but I can also bet that you don’t use it as much as you should.
Let start with an example (including a trick question for you):
$input = "2" $output = $input + 2
What do you think the output is of this script is?
If you thought that the answer is “22” then you are correct. If you thought the answer is 4 you are wrong.
You would expect that the input has a numeric value of 2, instead of a text value of “2”, which changes the outcome of this script.
There are 2 options to solve this problem:
- Use a variable type indicator (example: [int]$input) to make it a numeric value. But, as it is recommended to always use indicators, there are situation where you cannot use this method as it will result in an error/stopping the execution of the script. Making your script unreliable and non-reusable.
- Use a “If ..Then ..Else” statement to validate the variable.
Regardless of the input value, using the “If .. Then .. Else” command gives you multiple options based on the input data. Which can be used to rectify data (or use it to start a plan B script when rectifying is not possible).
The example below shows you an example of how you can use the “If .. Then.. Else” command in your favor.
$input = "2"
if ($input -is [int]) {
write-host "Do nothing"
} elseif ($input -is [string]) {
write-host 'Convert $input to Integer'
$input = $input -as [int]
}
$output = $input + 2
In this simple example the script only rectifies my variable (a rather simple task) based on the IF-statement. But think of a situation where you need to rely heavily on a variable, as it steers the outcome of your script: You need to be 100% sure that your script can handle all possible situations. In that situation you may have to use multiple IF statements to rectify your problem.
If .. Then .. Else use case #2
Another thing I observer frequently is the use of only the “IF” command, without the ELSE or ELSEIF commands. Which is a missed opportunity: Always use a “IF” with an accompanied “ELSE” command! It’s even beneficial if you only use it for logging or monitoring purposes (as shown in the above example). It gives you control over the execution of your script and enforces you to think about a plan B, making you script foolproof.
Another example of the usage of the “If .. Then .. Else”command:
if (!$input) {$input = Read-Host "Enter Input"} else {write-host '$input is already filled'}
I’m using this method constantly when developing my script. This enables me to execute (and test) the whole script instead of pieces of the script, without having to re-enter my input over-and-over again. It checks if the $input variable is empty, if it is empty it will ask for my input. Making my script easier to re-use.
So building filters and using the “If .. Then .. Else” commands inside your script can be called “desired state scripting”: only changing the things that need to be changed, based on the validated input and making is it as foolproof as possible.
I used simple examples in this blog , but this can be make as comprehensive as you want.
Check out my VSAN Cluster build script: I used the above methods extensively in that script. It provides a foolproof script, for enabling VSAN on any existing vSphere cluster, no matter what the current (mis-)configuration is. It will (re-)build a VSAN cluster (including the network configuration) taking VMware best practices into consideration.
Gather all information at the start before filtering
Needless to say, I also find much scripts online that do not use variables (see my first example). This will make your scripts static and only functional for one goal in a particular environment. Resulting in a non-reusable script.
Therefor I recommend to use variables as much as possible. Maybe it’s not needed for your problem now, but when needed you can re-use an existing script later.
When I’m starting the creation of a script, I start with collecting as much as data into variables as needed and put this in the very top of my script. For example:
$AllVMs = Get-VM $AllIpSetObjects = Get-NSXIpSet $AllNSXSecurityGroups = Get-NSXSecurityGroup
As you can see, I’m not filtering anything at this point. I’m just try to retrieve as much as data into variables as possible. The biggest advantage of this is, that all data is available in-memory. Filtering in-memory data is (much) faster compared to retrieve your specific piece of data directly each and every time. This is especially true when you have to retrieve multiple items or you want to make your script reusable.
So as an example:
$SGs = Get-NSXSecurityGroup
$SGa = $SGs | Where-Object {$_name -match "Domain Controller"}
$SGb = $SGs | Where-Object {$_name -match "Exchange Servers"}
is much (in this case 2x) faster than:
$SGa = Get-NSXSecurityGroup -name "Domain Controller" $SGb = Get-NSXSecurityGroup -name "Exchange Servers"
The end result will be the same, but the time to get the results is 2 times faster in the first example. Think about this when you have to retrieve hundreds or thousands of objects. This will save you a lot of waiting time.
To explain this in more technical details: The first example will only execute one API call and the data is filtered in the following lines (based on in-memory data). In the second example two API calls are needed to get the required data.
Be aware that an API call always takes a lot of time, as each API call must be authenticated. Authentication will happen in the background, but it still requires unwanted CPU cycles and waiting time.
Using variable types and methods to structure your data
A small introduction:
As with each scripting language, multiple variable types are available. Each variable type dictates what piece of information can be store into the variable.
Again, if you already have some scripting experience you are aware of variables and variable types, including:
- Booleans – 1/0 (TRUE/FALSE)
- Integers – numbers
- Strings – “text”
- Arrays – a collection of items (strings, integers, etc).
- Hashtables – a collection of key/value pairs.
A lot more variable types are available.
As this is not a how-to-script-crash-course, I will only cover the types that I use often and will describe how I use them to structure my data.
Using PSCustomObjects to you advantage
Booleans, integers and strings are types that everyone uses and I will be the last person to say that you should’t use them: They are very useful.
But, my mostly used variable types are the PSCUSTOMOBJECT and array variable-types:
The PSCUSTOMOBJECT type allows me to create Excel-like data structures inside a variable, which can be placed inside other array-type variables (and visa-versa). Which maybe need a little bit of explaining.
The PSCUSTOMOBJECT type is related to the hashtable variable, which has key/value pairs. But PPCUSTOMOBJECTS are extended with named columns and can hold tons (megabytes) of data. Each “cell” (item/object) can also include other variables (including arrays or hashtables), which allow me to create integrated variables: a excelsheet inside a excelsheet-cell.
This sounds complex, but in fact it isn’t: This can be used on many occasions.
So for this example I want to create an overview of all VM’s and their related NSX Security groups.
Think about this in an Excel format:
| VM | Security Group |
| VM-A | SecurityGroup-A, SecurityGroup-B |
| VM-B | SecurityGroup-A, SecurityGroup-C |
In Excel sheet it looks very easy, right? This is exactly the same as a PSCUSTOMOBJECT.
This is how I should script this:
$VMs = get-vm # <- retrieve all data (one API CALL)
$report = @() # <- create a empty variable which will filled with data
foreach ($VM in $VMs) { # <- needed to collect NSX Security Group data per VM
$reportitem = [pscustomobject]@{ # <- Create PSCUSTOMOBJECT variable
VM = $VM # <- Create VM "row"-line containing VM Data
SecurityGroup = ($VM | Get-nsxsecuritygroup) # <- Create a SG "row" containing SG data
}
$report += $reportitem # <- Add PSCUSTOMOBJECT variable to $report variable.
}
Side note: In this example I’m unable to retrieve all NSX-Security Group data first: this is by (PowerNSX) design. I’ve to use the Get-NSXSecurityGroup cmdlet for each VM individually making it a rather slow script.
In the above example the $report variable holds the top-level Excel-like array variable, containing multiple PSCUSTOMOBJECT variables which hold the VM- and NSX Security Group data. One thing to remember is that it holds ALL VM and NSX Security Group detailed data in a tree like manner.
You can find the variable type by using the Get-Member cmdlet (example: “$report | Get-Member” to show the variable type for $report) and will also show you the available columns (Properties) and methods, which I will explain in further detail below.
In this example the $report variable contains 2 properties (columns): “VM” and “SecurityGroup” (just as the excel example). Each item (object) in these variable contain other variables with more underlying (sibling) data.
Think about it as a tree of data:
$report - VM - Name - NumCPU - etc. - SecurityGroup - Name - Description - etc.
This allows me to query or filter the available data very easily as it is all in-memory data:
If (for an example) I want to show only the VM details, I simply use $report.vm which will show the VM details (not the NSX Security Group details). But I’ve want to see only the configured vCPU’s for all VMs I can use $report.vm.NumCpu. to view them. The dot represent a leaf (layer) inside the variable.
There is a small problem with this when editing, as you cannot use the autofill of your script editor to scroll through the sibling leafs/layers. To overcome this problem type “[0]” behind the variable (for example “$report[0].“).
The [0] indicates that you only want to show row 0 (the first row), the dot “.” will indicate that you want to show the sibling properties. An example is shown in the figure below:
Be aware that, in this example, there is a 1:1 relation between the rows in $report and the underlying ‘VM’ property.
After finding the correct property “column”, remove the [0] to continue your script with all results (rows).
We can also use the same method for underlying variabels, for example “$report[0].SecurityGroup[0].” which will only return the first (row 0 of) security group from the first row (VM) of the variable $report.
now you should have a global understanding how to work with PSCUSTOMOBJECTS.
Again .. practice makes perfect.
Filtering PSCustomObjects
Now let’s combine PSCUSTOMOBJECTS with filters, by starting with an example:
($report | Where-Object {$_.vm.name -eq "SRV001"}).SecurityGroup.name
This small script will show all NSX Security Group names which belong to VM with the name “SRV001”. Be aware that for this query no API call is needed anymore, only raw CPU processing power and the in-memory data. This filtered data is retrieved fast (within several milliseconds).
On my computer this query only took 14 milliseconds for filtering 400 VM’s.
Let’s break this script down:
- We start with the $report variable and use this as an input (“$report | ..”).
- The Where-Object filters the $report data only for VM with name “SRV001”, including the VM and SecurityGroup data (for that particular VM).
- The first part will return the rows from $report, including VM and SecurityGroup detailed data (for server “SRV001”). Which will return too much data: I only want the NSX Security group names.
To resolve this, the first part is surrounded by rounded brackets “( )” and attached by “.SecurityGroup.name” properties. This will return only the Security Group names for VM named “SRV001”.
As this is a very fast way of scripting, there is still room for some improvement.
Each PSCustomObject has builtin methods, like .foreach() and .where(). These methods are much faster compared to their cmdlet-equivelants as they do not rely on the piping“|” command.
Let me re-write of the above code
($report.where({$_.vm.name -eq "SRV001"})).SecurityGroup.name
Here I use .where-method instead of the Where-Object cmdlets, which enables me to complete the same query within only 2(!) milliseconds instead of 14 milliseconds.
I’ve seen scripts which initially run for several hours in large environment, going down to several minutes using the above improvements.
Using Sort-Object and Group-Object to de-duplicate data
There is nothing more annoying than having to wait on a script which is executing redundant input data. AARGH! Here a quick solution that can solve that problem:
$report | Sort-Object VM -Unique
As this command works, I’ve found some cases where it didn’t work as expected and returned some redundant data: I’ve not figured it out why this is happening. In those cases I had to use the Group-Object to deduplicate the data.
($report | Group-Object VM).foreach({$_.group[0]})
This script is more complex, but gets the job done every time.
Using foreach-loops
Every scripter uses foreach-loops, and that’s a good thing!
I only have a small tip: if you need to test your foreach-loop, execute it without a full script inside the scriptblok.
foreach ($VM in $VMs) {}
This will fill the $VM variable with the last item from $VMs, making you ready to test your script
Do not create screenlogs, create real logs
Big shoutout to: https://gallery.technet.microsoft.com/scriptcenter/Write-Log-PowerShell-999c32d0
Don’t use write-host, instead use write-log every time! Do not ask questions, just use this .. if you want to keep your current job!
It provides a track record for everything you do. If something bad happens, you have the logs which can support your actions. Logs are your friend! Logs are gooood.
Code formatting / quick tips
- Always use a TAB (or 4 spaces) when indenting your code, so you can backtrack where you are in the script.
- Do not make long lines of code (rule of thumb is a maximum of 115 characters in a single line)
- Watch out for trailing spaces (these can really ruin your day).
- Create #remarks everywhere, describe what your script does.
- Want to use variable inside a string variable? Use the following “$($variable.name)” format.
- Use functions as much as possible.
Ok there will be more tips and tricks added to this blog in the future. If you have any PowerShell tips for me, please reach out!
