OrchestratorServiceModule.psm1 occasionally not getting runbooks

I've had an interesting issue where the executerunbook function I wrote occasionally doesn't return the runbook.

I can't manage to reproduce the issue reliably yet enough that I am able to fully diagnose the issue, so as a work around I have added a retry option to the script.. where it tries to get the runbook again.

This will slow you down if you type the runbook name in wrong, but that's tough, and your own fault. I've uploaded the new version here (OrchestratorServiceModule.psm1) A quick screenshot of the fix, which now throws a warning rather than a terminating error:

One other thing is I have added a parameter for the poll interval when executing runbooks, which will help with slow running jobs and stopping them returning excessive amounts of verbose logging.

401 Errors and Powershell..

I've spent all morning trying to figure out why connecting to a web service using powershell kept giving me 401 errors from a remote machine but worked fine on my own machine..

I was trying to do:

$credentials = New-Object System.Net.NetworkCredential -ArgumentList @($username, (ConvertTo-SecureString -String $password -AsPlainText -Force))

$request = [System.Net.HttpWebRequest]::Create($Urlstring)
$request.Credentials = $credentials
$response = [System.Net.HttpWebResponse] $Request.GetResponse()

I've tried messing with IIS configuration, Application Pools, SPN's etc..

I then found This page, which had a useful packet capture filter, I followed this but wasn't seeing the errors I wanted! Eventually after playing with the options I began to see kerberos errors for the account being locked out. This then let me realize that we were getting closer, so I tried resetting the password to something known without a mass of random characters in. It was still getting locked out.

I then changed from using a secure string to use a string as below and it immediately started working.

$credentials = new-object System.Net.NetworkCredential("AccountName", $password, "Domain")
$request = [System.Net.HttpWebRequest]::Create($Urlstring)
$request.Credentials = $credentials
$response = [System.Net.HttpWebResponse] $Request.GetResponse()

It seems that checking the CLR version with

$PSVersionTable.CLRVersion

shows the problem as apparently SecureString wasn't introduced until .Net v4.0

So I now need to either do plain text passwords - Not.. or come up with a way of testing this!

Also pay attention to pre-windows 2000 account names as they may also cause issues if not using the UPN, but I need to do further testing around this.

Invoking System Center Orchestrator Runbooks from Powershell

I've recently been trying to get a neater way of executing System Center Orchestrator runbooks from Powershell.

I found a number of different ways of doing this on the internet, but none seemed to quite do what I wanted, which was the ability to provide an easy way of non orchestrator users to be able to invoke our runbooks, for tasks such as automating vm snapshots, managing load balancer pools etc.

The closest I got was the following: System Center Orchestrator Web Service PowerShell

This was a good starting point, I then added additional functionality in the form of:

WaitForRunbook
Get-ReturnedData
ExecuteRunbook
New-ParameterValue

Wait for runbook does exactly what you would think, as does get-ReturnedData, new parameterValue allows you to do runbook parameters in a 'neater' one liner. - not strictly necessary but still..

Execute-Runbook is the one where I use all of the pre-existing an new functions, so I will try and explain what it does if called with the below command:

ExecuteRunbook -Name "\DavidW\LoadBalancer\Get-PoolMembers" -OrchestratorServer ServerName.blah.local -Parameters $RunbookparamArray -wait -verbose

So first we get the runbook by its path \folder\folder\runbookname

# create the base url to the service
$url = Get-OrchestratorServiceUrl -server $OrchestratorServer

# Get Runbook by RunbookPath
$runbook = Get-OrchestratorRunbook -ServiceUrl $url -credentials $credentials -runbookpath $Name

We then loop through all the available parameters on the runbook with the direction set to in and see if the array passed in ($RunbookparamArray) contains a matching name value pair, if it does then we add it to a hashtable ready to pass to Start-OrchestratorRunbook.

NB: I haven't done any checking / testing around these being case sensitive as yet, its on the list of things to do, but I needed something up and working quickly.

# Sort Runbook Parameters
$ParamsWithValues = $null
 
if ($Parameters -ne $null){
 Write-Verbose "Parameters supplied for Runbook: $($runbook.Name) $($runbook.Id)"
 # Create Empty Hashtable
 $ParamsWithValues = @{}

 # Loop through Runbook Parameters adding values where we have them.
 foreach ($Parameter in $runbook.Parameters) {
  if ($Parameter.Direction -eq "In"){
   Write-Debug "Parameter Name: $($Parameter.Name)"
   # Add to hashtable so we have an entry for each runbook param, regardless of whether we have a value.
   $ParamsWithValues.Add($Parameter.ID, $null) 
   # Now look through passed param's and add a value where we have one..
   foreach ($item in $Parameters){
    Write-Debug "Item: $($item.Name) Param Name: $($Parameter.Name)"
    if ($item.Name -eq $Parameter.Name) {
     Write-Verbose "Processing Parameter: $($Parameter.Name) ID: $($Parameter.Id)"
     # They both match so add the param value
     $ParamsWithValues.Set_Item($Parameter.Id, $item.Value)
     # Exit For Each... no need to continue within this loop
     break
    }
   }
  }
 }
}

So the function now starts the runbook with the hash table passed through with the parameters, this saves us from having to know the GUID's for the param's and allows admins to change the initialize data step without scripts having to be hard coded with the parameter GUID.

# Start The Runbook
write-verbose "Starting Runbook: $($runbook.Name) $($runbook.Id)"

$job = Start-OrchestratorRunbook -runbook $runbook -credentials $credentials -Parameters $ParamsWithValues 

As we have specified the -wait switch the job will wait for the job to complete, timeout or fail.
(need to do the timeout bit iirc)

again as we specified -wait we can also then try and get the returned data from the runbook (if there is any)

$ReturnedData = $null
if ($Wait -eq $true){
 # Monitor job (Runbook) status 
 Write-Verbose "Job: $($job.id) Waiting for execution, please wait"
 $jobStatus = WaitForRunbook -Job $job -credentials $credentials
 
 # TODO: Implement this functionality.
 If ($jobStatus -eq "Timedout"){
  Write-Verbose "Job: $($job.id) Status: Timeout whilst polling runbook status"
  Throw "Timeout waiting for job to complete. The job will continue to run on the server."
 }
 
 # get ReturnedData if there is any.
 Write-Verbose "Job: $($job.id) Status: Getting returned data"
 $ReturnedData = Get-ReturnedData -Job $job -Credentials $credentials

 if ($ReturnedData -ne $null){
  Write-verbose "Job: $($job.Id) Status: Retrieved returned data"
 }else{
  $ReturnedData = "Nothing to return"
 }
}else{
 Write-Verbose "Job: $($job.Id) Status: Not waiting for runbook execution to complete."
 $ReturnedData = "Runbook Started"
}
Write-Verbose "Finished executing Runbook: $($runbook.Name) Runbook: $($runbook.Id) Job: $($job.id) "

# Return Data
return $ReturnedData

I'm not convinced on the way I am returning the data at the moment (ps-object within hashtable) I think this can be improved on, but haven't really looked at this yet.

An example of calling a runbook is as below:

# Build Runbook Parameters

$RunbookparamArray = @()

$RunbookparamArray += New-ParameterValue -name "LoadBalancer" -Value "nlb.blah.local"

$RunbookparamArray += New-ParameterValue -name "Server" -Value "AServerName"

$RunbookparamArray += New-ParameterValue -name "Object" -Value "Blah Blah"

$RunbookparamArray += New-ParameterValue -name "TargetUserId" -Value "$($env:USERDOMAIN)\$($env:USERNAME)"



#Execute Runbook

ExecuteRunbook -Name "\DavidW\Test\Runbook2" -OrchestratorServer server.blah.local -Parameters $RunbookparamArray -wait -verbose # -Credentials $Credentials

There is also a fix within function getRunbookObject as when calling the sendHttpGetRequest it doesn't pass a credential object. This is fine when running as a user with appropriate credentials but we want to be able to call this from applications such as Octopus deploy where the tentacles run as local system by default. This was meaning that when we passed a credential object in to ExecuteRunbook it was unable to add the runbook parameters as it wasn't getting the available parameters from orchestrator.

Its a simple fix:

[xml] $xmlDoc = sendHttpGetRequest -url $pmurl

Needs changing to:

[xml] $xmlDoc = sendHttpGetRequest -url $pmurl -credentials $Credentials

Whilst I remember I have also changed some of the write-host entries to be either write-debug or write-verbose so that there is no unnecessary noise from the functions.

The new version of the module is available for download from here OrchestratorServiceModule.psm1 and I am currently speaking to a couple of people to see if we can get the codeplex site updated with the minor fixes.

Usual disclaimer, test in your own environment and if in doubt use a file comparison utility / visual studio to view the changes.

You can invoke devenv.exe /diff list1.txt list2.txt from the command prompt or, if a Visual Studio instance is already running, you can type Tools.DiffFiles in the Command window, with a handy file name completion: (Copied From here)