Getting all automation connections from within SMA of a given type

I'm currently looking at how I can migrate some runbooks from System Center Orchestrator 2008 R2 to Service Management Automation (SMA). One issue I have in Orchestrator is that it isn't possible to subscribe to a configuration activity if you package it up into an Orchestrator Integration pack (OIP). To get around this, you can use the invoke.net activity and call the Class directly from the DLL.

Why does it matter your thinking?  One scenario we have is that we want to be able to have a common runbook, and execute it against multiple load balancers. Each of these connections is then stored in Orchestrator.. and the caller passes the configuration name in (which is actually the lb hostname) and then it connects with the correct credentials.

Anyway I was thinking about how I could do the same or achieve the same in SMA, and realised that it was a non issue, I then started migrating some other runbooks, and one task requires running against all instances of an env.. for example VMWare - Go get all VM's across all vcenters..

I thought I'll just get all the connections of a given type and easy done, however Get-AutomationConnection has a mandatory parameter of the name.. and wildcards don't work.

I had a quick nosy in the database and realised it should be easy to do what I needed, so what I have done is create a workflow that queries the relevant table in the DB for the connections and then returns the name so you can use that to return the credential / connection pair required. This maybe achievable in other ways but I am new to SMA :)

The scripts are available here: Get-AutomationConnections.ps1 and an example of calling it here: UseMultipleConnections.ps1

Executing SMA runbooks from linux

I've been trying to work out a way of executing SMA runbooks from some of our linux systems.

so I have knocked together a crude python script that can be further extended for doing such a task..

python isn't my language of choice - this is probably quite apparent in the lack of classes etc.

typically SMA uses ODATA for the interface via REST - this was proving too much of an issue, so I resorted to regex and extracting the data from the XML that I needed.. Initially I was going to use XPATH to do it, but things like elementtree didn't seem to like the 'XML'.

Below is a pic of the output:

Anyway I have uploaded the script to my codeplex site located here: http://smaworkflows.codeplex.com/SourceControl/latest#Python/SMARunbooks.py


Also a thanks should be mentioned to Laurie Rhodes for the information provided here:
http://www.laurierhodes.info/?q=node/105

SMA Workflows

I'm trying to get to grips with SMA and what it will take for us to migrate existing integration packs and runbooks that we have authored in C#

I'm going to try and keep everything we have done open, So I am uploading all the scripts to a new codeplex site I created located here: smaworkflows.codeplex.com

I've just created one for sending text messages via a provider called AQL for more details see here: www.aql.com/sms/

The above script is located here: http://smaworkflows.codeplex.com/SourceControl/latest#Misc/Send-SMS.ps1 if anyone is interested.

Creating DHCP option 121 or 249 string via powershell

We have a number of networks that addresses were statically assigned.. we are now moving these to DHCP to aid with the DevOps principles and destroying machines and rebuilding from code. In order to facilitate this I needed to deliver the static routes via DHCP.. It turns out its a bit of a pain in the arse to do this.. From a stack overflow post:

You're seeing these because option 249 must be specified as as BINARY value, not as an IPADDRESS value. If you can't set this with the GUI then you'll have to convert your desired route into a hexadecimal string yourself. An example would be as follows: 10.1.1.0/24 accessible via 10.1.1.1 converts into "180a01010a010101". The first octet, "18", is the number of bits of subnet mask (0x18 = 24 decimal). The next octets are the network ID (0a = 10, 01 = 1, 01 = 1, for "10.1.1"), padded by zeros on the right if the subnet mask doesn't end on an even octet boundary. The last four octets are the IP address of the gateway. Set the value in the GUI and you'll be happier.

Well I had too many to do, so I couldn't do it with the GUI really and it was too prone to error with fat fingers.. so I knocked up a quick script to accomplish this.. a word of warning is that I have only done /20 /23 and /24 networks with this.. so test it and verify in the GUI!!

 Here's the powershell script with an example at the end of how its run for multiple routes...

function ConvertTo-MaskLength {
  <#
    .Synopsis
      Returns the length of a subnet mask.
    .Description
      ConvertTo-MaskLength accepts any IPv4 address as input, however the output value 
      only makes sense when using a subnet mask.
    .Parameter SubnetMask
      A subnet mask to convert into length
   
 Function pinched from http://www.indented.co.uk/2010/01/23/powershell-subnet-math/
  #>

  [CmdLetBinding()]
  param(
    [Parameter(Mandatory = $True, Position = 0, ValueFromPipeline = $True)]
    [Alias("Mask")]
    [Net.IPAddress]$SubnetMask
  )

  process {
    $Bits = "$( $SubnetMask.GetAddressBytes() | ForEach-Object { [Convert]::ToString($_, 2) } )" -replace '[\s0]'
    return $Bits.Length
  }
}

function GenerateHex {
  <#
    .Synopsis
      Returns the hex value for creating a static route (DHCP option 121 / 249) - RFC3442
    .Description
      GenerateHex creates the hex value for creating a static route.. untested with anything other than /23 /24
    .Parameter SubnetMask
     EG 255.255.255.0 or /24
    .Parameter NetworkAddress
      The Network address IE 192.168.1.0
    .Parameter Gateway Address
      The IP Address of the gateway to send the traffic to IE 192.168.1.254    
  #>

 [CmdLetBinding()]
 param(
     [Parameter(Mandatory = $True, Position = 0, ValueFromPipeline = $True)]
     [String]$NetworkAddress,
  
  [Parameter(Mandatory = $True, Position = 1, ValueFromPipeline = $True)]
     [String]$SubnetMask,
  
  [Parameter(Mandatory = $True, Position = 2, ValueFromPipeline = $True)]
     [String]$GatewayAddress
 )

 # Convert subnet mask to correct format
 if ($SubnetMask -like "/*") {
  $SubnetMask = $SubnetMask.Replace("/","")
 }elseif ($SubnetMask -like "*.*") {
  $SubnetMask = ConvertTo-MaskLength -SubnetMask $SubnetMask
 }

 # Split network into an object by octet
 $NetAddress = $NetworkAddress.split('.')

 # Step through the correct number of significant octets (mask / 8 then rounded up) and convert to hex
 foreach($i in 1..([math]::Ceiling($SubnetMask / 8))){
  $Network += "{0:X2}" -f [convert]::ToInt32($NetAddress[$i-1])
 }

 # Convert Gateway Address to Hex
 Foreach ($octet in $GatewayAddress.split('.')) {
  $Gateway += "{0:X2}" -f [convert]::ToInt32($octet)
 }

 # Convert Mask to Hex
 $maskHex = "{0:X2}" -f [convert]::ToInt32($SubnetMask)

 # return calculated output
 $value =  $maskHex + $Network + $Gateway
 return $value
}

$String = ""
$String += GenerateHex -NetworkAddress 192.168.1.0 -SubnetMask 255.255.255.0 -GatewayAddress 10.111.111.1
$String += GenerateHex -NetworkAddress 10.1.2.0 -SubnetMask 255.255.254.0 -GatewayAddress 10.111.111.1

# Display command rather than execute it
Write-Output "netsh dhcp server 192.168.1.1 scope 10.111.111.0 set optionvalue 121 binary $($string)"

I then needed to convert some existing boxes from having lots of local persistent routes and was planning on using something like this as the basis of starting a script..

Get-WmiObject Win32_IP4PersistedRouteTable | Select-Object Destination, Mask, Nexthop

however I quickly discovered the machine I was looking at doesn't have powershell installed.. Given I'm in a rush I resorted to VBS as it was my scripting language of choice prior to powershell.. so the following script was used to export the routes and generate the netsh command.. Crude I know, but not as crude as multiple static routes on a server, rather than using a Router!

' Export Routes.vbs
On Error Resume Next
Set objWMIService = GetObject("winmgmts:\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * from Win32_IP4PersistedRouteTable",,48)

For Each objItem in colItems   
    wscript.echo "$String += GenerateHex -NetworkAddress " & objItem.Destination & " -SubnetMask " & objItem.Mask & " -GatewayAddress " & objItem.NextHop
Next

Naming VMWare Guest (Windows) NIC's

One of the things that annoy's me a little bit (A lot!) is when the adapters on a server have no meaningful name and you are only left with the network address to try and work out what the hell the network is, on some systems where you may have multiple adapters then this is only harder.

We talked over at work how best to identify what network the machine was on and work from there, we use Cisco Nexus 1000v switches on some of our vmware estate, so I initially thought of using CDP to determine the VLAN ID or name.. however I dismissed it as soon as I thought of it. The key was in the word 'some', so this potentially means this process wouldn't work for test and dev environments where an enterprise plus licence wasn't / isn't currently present.

So the option we came up with is using the MAC address to identify the correct adapter and then set the name based on the vmware port group.

To achieve the rename in the guest I am using the

Invoke-VMScript

command, which allows you to execute a command in the guest providing you have the correct credentials, vmware tools running and tcp port 902 open in the correct places (or is it 901? - 902 I think, but I've had a beer now!)

In order to do the rename, I currently use

Get-Netadapter

and

Rename-NetAdapter

commands which are only available in 2012 onwards, I will over time amend this to work with 2008 (prob using wmi and netsh), but this is only an initial draft, so the usual caveat's apply!

The script builds a scriptblock object to pass through to command, which took a little bit of fiddling to ensure that some variables weren't expanded such as $_.MacAddress in the where clause, but then the other variables were expanded, I'm sure when I revisit this I will change my mind of how to do it, but for now it works, so function over form.

FWIW, I have made it fail back to the administrator username if the supplied username fails auth.. but modify this to suit your own environment.

Script:

# Name VM Guest Adapters
# David Wallis 12/05/2015 blog.wallis2000.co.uk

# Variables
$viServer = "vmware.blah.local"
$ServerName = "Server001"
$Username = "Admin123"
$Password = "Password"

# Add Snapin if needed
If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) {
    Add-PSSnapin VMware.VimAutomation.Core | Out-Null 
}

# Connect to Vsphere
$null = Connect-VIServer $viServer 

# Hide Progress Bars as System Center Orchestrator might not like these 
$ProgressPreference='SilentlyContinue'

# get VM Object
$vm = Get-VM -Name $ServerName -ErrorAction stop
 
function RenameNic
{
    [CmdletBinding()]
    param (
     [Parameter(Mandatory=$true)]
     [String] $User,
  
  [Parameter(Mandatory=$true)]
     [String] $GuestPw,
  
  [Parameter(Mandatory=$true)]
     [String] $MacAddress,
  
  [Parameter(Mandatory=$true)]
     [String] $NewInterfaceName
 )

  # Replace colons with dashes in the MAC address so that we can use it to match with the get-NetAdapter cmdlet.
  $macAddress = $macAddress.Replace(":","-")

  # Construct Scriptblock to execute the command
  $command = 'Get-Netadapter | where {$_.MacAddress -eq ' +  [char]34 + $macAddress + [char]34 + "} | Rename-NetAdapter -NewName " + [char]34 + $NewInterfaceName + [char]34
  Write-Verbose "Attempting to execute command within the VM ($($VM)): $($command)"
  $ScriptBlock = [scriptblock]::Create($command)
  
  # Try and invoke the command within VM
 try
 {  
  $output = Invoke-VMScript -vm $vm -guestuser $Username -guestpassword $Password -scripttype powershell -scripttext $ScriptBlock -ErrorAction stop
  # Nothing should be returned.. but the output would be in the below
  # $output.ScriptOutput
 }
 catch [VMware.VimAutomation.ViCore.Types.V1.ErrorHandling.InvalidGuestLogin]
 {
  # Invalid Guest Login
  Write-Verbose 'Invalid credentials trying the username "Administrator"'
  $output = Invoke-VMScript -vm $vm -guestuser "Administrator" -guestpassword $Password -scripttype powershell -scripttext $ScriptBlock -ErrorAction stop
  write-output $output.ScriptOutput
  
 }
 catch
 {
  throw
 }
 
}

# Get all the Adapters assigned to the VM and rename the nic to the vm portgroup name using the MAC
# to identify the correct adapters - this will only work on 2012 onwards currently
$Adapters = $vm | Get-NetworkAdapter | Select NetworkName, MacAddress
foreach ($adapter in $Adapters){
 write-verbose "Renaming Adapter with MAC Address ($($adapter.MacAddress))"
 RenameNic -User $Username -GuestPw $Password -MacAddress $($adapter.MacAddress) -NewInterfaceName $($adapter.NetworkName) -verbose
}

Auto generating server names

So we spent a fair old time working out a naming convention at our place. I wont go into the details of that here, but suffice to say when picking a naming convention, make sure you write some regex so that you can parse it, and then make use of capturing groups to determine OS, location, role, environment etc.

I can thoroughly recommend Debuggex for visualizing and testing regular expressions if you aren't fluent in them! There are also cheat sheets available and don't worry that there isn't a drop down for powershell the default is good enough to get you where you need to be.

So now we are automating the build it made sense to automate the naming of servers, and given the vast size of our estate and the number of non joined up systems, then checking of names is a bit error prone and time consuming. Given we can do it manually, how hard could it be to automate it?

I did a bit of googling and found this post on stackoverflow which was a nice starting point. It didn't fully work and you had to read between the lines.

After a bit of tweaking I got it working so we could fill in the gaps in sequences, IE 1,2,3,5,6,7 would produce the next server as 4. However 3,4,5,6,7 would produce the next server number as 8. A quick fix to the code to make it think a server 0 already exists fixes that issue. (NB: our server numbers start at 001, not 000 - if your's don't then you may need to add -1 instead - Untested!)

The next issue is that, not every server is domain joined so you need to query more than just AD.. For us I wanted to query a couple of SCCM environments (2007 and 2012 r2) SCOM, Multiple VMware environments, and also DNS.

So the code I have produced (which can be re-factored and probably will be at some point) queries each environment getting just the digits of the server name and add's them to the array.

At the end of the script we combine all of the array's and add the 0 previously mentioned and then strip duplicates and sort into numerical order.. this gives us our definitive list of numbers in use. (we work on worse case, if we see it anywhere then assume its in use)

Once we find the name we create an entry in DNS so that subsequent script use wont issue the same name - we deal with concurrency issues by running this in System Center Orchestrator and setting job concurrency on the runbook to be 1.

Script below, change values to suit your environment.

# Add this script can support -verbose (Convert to function later)
[CmdletBinding()]
param()

# ********************************************************
$startOfName = "xxxYYYZZWEB"
# ********************************************************

# VMWare Details
$ADVIServers = @("vsphere1.blah.local","vsphere2.blah.local","vsphere3.blah.local","vsphere4.blah.local")
$StandAloneHosts = @()

# DNS Details
$DNSServer = "xxxxxx.blah.local"

# SCCM 2012 Details
$SCCM2012SiteServer = "sccm2012.blah.local"
$SCCM2012SiteCode = 'SiteCode' 

# SCCM 2007 Details
$SCCM2007SiteServer = "sccm2007.blah.local"
$SCCM2007SiteCode = 'SiteCode2' 

# SCOM 2007 Details
$SCOMServer = "scom.blah.local"

# Create Empty Arrays
$VMNumbers = @()
$ADnumbers = @()
$DNSNumbers = @()
$SCCM2012Numbers = @()
$SCCM2007Numbers = @()
$SCOM2007Numbers = @()


# VMWare

 Write-Verbose "Processing VMware"
 Add-PSSnapin vmware.vimautomation.core -ErrorAction SilentlyContinue

 # Set options for certificates and connecting to multiple enviroments
 $null = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$False
 $null = Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Scope User -Confirm:$False

 # Connect to each AD Authenticated viServer
 foreach ($VIServer in $ADVIServers){$null = Connect-VIServer $VIServer -verbose:$false} 
 
 # Connect to standalone host
 foreach ($Host in $StandAloneHosts){$null = Connect-VIServer $Host -User 'usernamehere' -Password 'passwordhere' -verbose:$false} 
 
 # get next available number in a range of numbers.
 $VMNames = Get-VM -Name "$($startOfName)*" -verbose:$false |select Name
 $VMNames |select Name | Foreach-Object {Write-Verbose $_.Name} | Sort-Object
 $VMNumbers = $VMNames |select Name | Foreach-Object {[int]($_.Name -replace '\D').Trim() } | Sort-Object
 Write-Verbose "$($VMNumbers.Count) Matching entries found"
 
# Active Directory

 Write-Verbose "Processing Active Directory"
 
 # Issue Query
 $searcher = [ADSISearcher]"(&(objectCategory=computer)(name=$($StartOfName)*))"
 $searcher.PageSize = 1000

 # get next available number in a range of numbers. returns 5 for 1,2,3,4,6,7,9 From AD
 $ADNames = $searcher.FindAll() | Foreach-Object {[string]$_.Properties.name} | Sort-Object
 $ADNames | Foreach-Object {Write-Verbose $_} | Sort-Object  
 $ADnumbers = $ADNames | Foreach-Object {[int]($_ -replace '\D').Trim() } | Sort-Object  
 Write-Verbose "$($ADnumbers.Count) Matching entries found"
 
# Search DNS

 Write-Verbose "Processing DNS"
 
 # Import DNS module
 Import-Module dnsShell -Verbose:$false
 $DNSNames = get-dnsRecord -server $DNSServer -RecordType A -Zone blah.local | select Name |where {$_.Name -like "$($startOfName)*"} 
 $DNSNames | Foreach-Object {Write-Verbose $_.Name} | Sort-Object -Unique
 $DNSNumbers = $DNSNames | Foreach-Object {[int]($_.Name -replace '\D').Trim() } | Sort-Object -Unique
 Write-Verbose "$($DNSNumbers.Count) Matching entries found"
 
# Search SCCM

 Write-Verbose "Processing SCCM 2012"
 
 # Query SCCM2012 Env
 $SCCM2012Members = Get-WmiObject -ComputerName $SCCM2012SiteServer -Namespace  "ROOT\SMS\site_$SCCM2012SiteCode" -Query "SELECT * FROM SMS_FullCollectionMembership WHERE CollectionID='SMS00001' AND Name LIKE '$($startOfName)%' order by name" | select Name -Unique
 $SCCM2012Members |select Name | Foreach-Object {Write-Verbose $_.Name} | Sort-Object
 $SCCM2012Numbers = $SCCM2012Members |select Name | Foreach-Object {[int]($_.Name -replace '\D').Trim() } | Sort-Object
 Write-Verbose "$($SCCM2012Numbers.Count) Matching entries found"
 
 Write-Verbose "Processing SCCM 2007"
 
 # Query SCCM2007 Env
 $SCCM2007Names = Get-WMIObject -ComputerName $SCCM2007SiteServer -Namespace "root\sms\site_$SCCM2007SiteCode" -class "SMS_R_System" -filter "Name LIKE `"$startOfName%`"" |select Name | Sort-Object -Property Name -Unique
 $SCCM2007Names |select Name | Foreach-Object {Write-Verbose $_.Name} | Sort-Object
 $SCCM2007Numbers = $SCCM2007Names |select Name | Foreach-Object {[int]($_.Name -replace '\D').Trim() } | Sort-Object
 Write-Verbose "$($SCCM2007Numbers.Count) Matching entries found"
 
# Search Production SCOM 2007

 Write-Verbose "Processing SCOM 2007"
 
 #Initialize SCOM SnapIn
 Add-PSSnapin Microsoft.EnterpriseManagement.OperationsManager.Client -ErrorAction SilentlyContinue -verbose:$false

 #Connect to Production SCOM 2007 Env.
 $null = New-ManagementGroupConnection -ConnectionString $SCOMServer
  
 #Connect to SCOM Provider
 Push-Location 'OperationsManagerMonitoring::'

 # Get Agents Matching Name
 $SCOM2007Names = Get-ManagementServer |Get-Agent |Where {$_.Name -like "$($startOfName)*"}
 $SCOM2007Names | Foreach-Object {Write-Verbose $_.Name} | Sort-Object
 $SCOM2007Numbers = $SCOM2007Names | Foreach-Object {[int]($_.Name -replace '\D').Trim() } | Sort-Object
 Write-Verbose "$($SCOM2007Numbers.Count) Matching entries found"
 
 # Return to previous location
 Pop-Location
 
# Merge arrays adding a zero so we allways start issuing numbers from the beginning (ie 001)
$list = @(0) + $VMNumbers + $ADnumbers + $DNSNumbers + $SCCM2012Numbers + $SCCM2007Numbers + $SCOM2007Numbers

# Remove Duplicates numbers from the array and sort into numerical order
$list = $list | Sort-Object -Unique

Write-Verbose "Used numbers after sorting: $($list)"

# Determine if next server name is a gap in the sequence in the array
for($i=0; $i -lt $list.length; $i++) {
 if( $list[$i+1]-$list[$i] -gt 1) {
  # The gap between the current server number and the next element in the array is greater than 1
  # So we have an available number we can use.
  # TODO: - Add support for consecutive numbers IE build 6 servers with consecutive numbers.
  $num = "{0:000}" -f ($list[$i]+1)
  break
 }
}

# If no gap found in the sequence then use the next number from the sequence in the array
if ($num -eq $null) {
 $num = "{0:000}" -f (($list[-1]+1))
}

# Construct new name
$NewComputerName = "{0}{1}" -f $startOfName,$num

# Create DNS Record to 'reserve / mark the name as in use'
Write-Verbose "Creating DNS Reservation"
New-DnsRecord -Name $NewComputerName -IPAddress "127.0.0.1" -Zone blah.local -Type A -Server $DNSServer

write-output $NewComputerName

Status messages from SCCM task sequences via SQL.

I've got an automated server build process working well now using System Center Orchestrator and System Center Configuration Manager (SCCM), currently we poll the status messages via custom c# code within an Integration pack to determine the task sequence status. What I wanted to do was replicate the status message viewer output within our email step.

I've knocked together a crude SQL query to get this using the provided view. Please bear in mind SQL isn't my strong point.. I can get by with the help of Google! So I just thought I would provide a copy of the query here in case it is of use to anyone else..

You just need to remove or edit the WHERE clauses for the date and the machinename values where appropriate.

SELECT
 CASE [Severity] 
  WHEN '1073741824' THEN 'Informational' 
  WHEN '-1073741824' THEN 'Error' 
  WHEN '-2147483648' THEN 'Warning' 
 END AS Severity
  ,[SiteCode]
  ,[Time]
  ,[MachineName]
  ,[Component]
  ,[MessageID],
 CASE [MessageID] 
  WHEN '11124' THEN ('The task sequence execution engine started the group (' + [InsStrValue3] + ').')
  WHEN '11127' THEN ('The task sequence execution engine successfully completed the group (' + [InsStrValue3] + ').') 
  WHEN '11128' THEN ('The task sequence execution engine skipped the disabled action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ').') 
  WHEN '11130' THEN ('The task sequence execution engine skipped the action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ').')
  WHEN '11134' THEN ('The task sequence execution engine successfully completed the action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ') with exit code ' + [InsStrValue4] + ' Action output: ' + (COALESCE([InsStrValue5], '') + '' + COALESCE([InsStrValue6], '') + '' + COALESCE([InsStrValue7],'')+ COALESCE([InsStrValue8],'')+ COALESCE([InsStrValue9],'')+ COALESCE([InsStrValue10],''))) 
  WHEN '11135' THEN ('The task sequence execution engine failed execuiting the action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ') with exit code ' + [InsStrValue4] + ' Action output: ' + (COALESCE([InsStrValue5], '') + '' + COALESCE([InsStrValue6], '') + '' + COALESCE([InsStrValue7],'')+ COALESCE([InsStrValue8],'')+ COALESCE([InsStrValue9],'')+ COALESCE([InsStrValue10],'')))  
  WHEN '11138' THEN ('The task sequence execution engine ignored execution failure of the action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ').')  
  WHEN '11140' THEN ('The task sequence execution engine started execution of a task sequence.')  
  WHEN '11142' THEN ('The task sequence execution engine performed a system reboot initiated by the action (' + [InsStrValue2] + ') in the group (' + [InsStrValue3] + ').')  
  WHEN '11144' THEN ('The task sequence execution engine from a non-client started execution of a task sequence.')
 END AS Description 
FROM [CM_SiteCodeHere].[dbo].[vStatusMessagesWithStrings] (NOLOCK) 
WHERE MachineName = 'MyServerNameHere'
 AND Component in ('Task Sequence Engine','Task Sequence Manager','Task Sequence Action')
 AND Time BETWEEN '2015-04-02 08:30' AND GETDATE() 
ORDER BY Time DESC
  

Change the site code in the query.. CM_SiteCodeHere and apollogies for the formatting here.

Getting errors from System Center Orchestrator Runbooks without subscribing multiple times.

It’s been quite annoying trying to get an error summary from a single runbook, or even a collection of runbooks, I didn’t want to resort to using a sql database for doing this at this stage. I came up with a quick proof of concept for our automated server build that would allow me to log errors and informational messages and get them all with one activity and not having to subscribe to hundreds of previous activities.

What I came up with was writing an event to the eventlog on the runbook server which contains the runbook job ID, the message text and the eventId of 100 or 200 based on info or error status.
Firstly we get the runbook job id using a sql query, this allows us to filter specifically on that instance of the runbook, rather than getting the previous attempts error messages.

This SQL query step contains the following query:

SELECT     POLICYINSTANCES.JobID 
FROM         POLICYINSTANCES INNER JOIN
                      ACTIONSERVERS ON POLICYINSTANCES.ActionServer = ACTIONSERVERS.UniqueID
WHERE     (POLICYINSTANCES.ProcessID = ) AND (ACTIONSERVERS.Computer = '') AND (POLICYINSTANCES.Status IS NULL)

Note that in the above you will need to subscribe to runbook server name and the Activity Process ID from the initialise data step.

I found this https://social.technet.microsoft.com/Forums/en-US/d87a3ec4-073a-448c-b3eb-cd707cd70a53/job-id-inside-runbook?forum=scogeneral

The next step is to add a log Error step into your Runbook, or Log Info step as per below.

This run .net step contains the following Powershell script:

$JobID = 'Subscribe_To_SQL_Query_Results'
$DateTime = 'Subscribe_To_Activity_End_Time_From_Previous_Step'
$message = 'Subscibe_To_Previous_Activity_Name Failed. Subscibe_To_Previous_Activity_ErrrorSummaryText'

# Check eventlog exists, creating if not
if (! [System.Diagnostics.EventLog]::Exists("ServerRunbook1234")) {
 New-EventLog –LogName "ServerRunbook1234" –Source "Orchestrator"
}

# Check source exists, creating if not
if (![System.Diagnostics.EventLog]::SourceExists("Orchestrator")){
 [System.Diagnostics.EventLog]::CreateEventSource("Orchestrator", "ServerRunbook1234")
}

$null = Write-Eventlog -Logname "ServerRunbook1234" `
-Source Orchestrator `
-EventID 100 `
-EntryType Information `
-Message "$($JobID)`n$($DateTime) $($message)"

For example, Something like this:

Then at the end of your error rail you have a final step that gets all the errors and sends an email or whatever you choose to do.

The get all errors step is a little crude currently, but to minimise the query on the eventlog we query on events since the Runbook started, for this I take the initialise data step start time, and convert the utc format string to a datetime object.. I know this could be better but I was rushing.

It then turns out there is an issue when using en-GB culture to get messages – so if your outside of the UK – you may need to refine this!

Once we have the messages since the runbook started, we then select only the messages containing the runbook GUID, The guid and new line is then replaced with nothing to remove this from the log string that we output.

This crude “get all errors step” contains the following powershell script:

$activityStarttime = 'Subscribe_To_Activity_start_time_of_Initialise_Data_Step'
$jobID = 'Subscribe_To_SQL_Query_Results'

# Error 100, Info 200

# Create Constant for EventType
Set-Variable id -option Constant -value 100

# Crudely Convert Orchestrator date time to something we can search with
$activityStarttime = $activityStarttime.Replace("UTC ","")
[System.Globalization.CultureInfo]$Culture = New-Object "System.Globalization.CultureInfo" "en-US"
[DateTime]$converted = [DateTime]::ParseExact($activityStarttime,'s',$Culture)

# Change locale, as bug prevents getting localaised fields in some locales (like en-GB)
[System.Threading.Thread]::CurrentThread.CurrentCulture = New-Object "System.Globalization.CultureInfo" "en-US"

# Get corresponding Eventlog entries
$LogEntries = Get-WinEvent -FilterHashtable @{logname='ServerRunbook1234'; id=$id; StartTime=$Converted} -ErrorAction SilentlyContinue | where { $_.Message.Contains($jobID) -eq "true" }

if (!$LogEntries){throw "No events found"}

# Process relevant entries
$Errors
foreach ($entry in $LogEntries){
    $message = $entry.Message.Replace("$($jobID)`n","")
    $Errors += "$($message)`n"
}

The activity then publishes the $errors variable as per below:

This can then be simply subscribed to on an email activity.

OrchestratorServiceModule.psm1 occasionally not getting runbooks

I've had an interesting issue where the executerunbook function I wrote occasionally doesn't return the runbook.

I can't manage to reproduce the issue reliably yet enough that I am able to fully diagnose the issue, so as a work around I have added a retry option to the script.. where it tries to get the runbook again.

This will slow you down if you type the runbook name in wrong, but that's tough, and your own fault. I've uploaded the new version here (OrchestratorServiceModule.psm1) A quick screenshot of the fix, which now throws a warning rather than a terminating error:

One other thing is I have added a parameter for the poll interval when executing runbooks, which will help with slow running jobs and stopping them returning excessive amounts of verbose logging.

401 Errors and Powershell..

I've spent all morning trying to figure out why connecting to a web service using powershell kept giving me 401 errors from a remote machine but worked fine on my own machine..

I was trying to do:

$credentials = New-Object System.Net.NetworkCredential -ArgumentList @($username, (ConvertTo-SecureString -String $password -AsPlainText -Force))

$request = [System.Net.HttpWebRequest]::Create($Urlstring)
$request.Credentials = $credentials
$response = [System.Net.HttpWebResponse] $Request.GetResponse()

I've tried messing with IIS configuration, Application Pools, SPN's etc..

I then found This page, which had a useful packet capture filter, I followed this but wasn't seeing the errors I wanted! Eventually after playing with the options I began to see kerberos errors for the account being locked out. This then let me realize that we were getting closer, so I tried resetting the password to something known without a mass of random characters in. It was still getting locked out.

I then changed from using a secure string to use a string as below and it immediately started working.

$credentials = new-object System.Net.NetworkCredential("AccountName", $password, "Domain")
$request = [System.Net.HttpWebRequest]::Create($Urlstring)
$request.Credentials = $credentials
$response = [System.Net.HttpWebResponse] $Request.GetResponse()

It seems that checking the CLR version with

$PSVersionTable.CLRVersion

shows the problem as apparently SecureString wasn't introduced until .Net v4.0

So I now need to either do plain text passwords - Not.. or come up with a way of testing this!

Also pay attention to pre-windows 2000 account names as they may also cause issues if not using the UPN, but I need to do further testing around this.

Invoking System Center Orchestrator Runbooks from Powershell

I've recently been trying to get a neater way of executing System Center Orchestrator runbooks from Powershell.

I found a number of different ways of doing this on the internet, but none seemed to quite do what I wanted, which was the ability to provide an easy way of non orchestrator users to be able to invoke our runbooks, for tasks such as automating vm snapshots, managing load balancer pools etc.

The closest I got was the following: System Center Orchestrator Web Service PowerShell

This was a good starting point, I then added additional functionality in the form of:

WaitForRunbook
Get-ReturnedData
ExecuteRunbook
New-ParameterValue

Wait for runbook does exactly what you would think, as does get-ReturnedData, new parameterValue allows you to do runbook parameters in a 'neater' one liner. - not strictly necessary but still..

Execute-Runbook is the one where I use all of the pre-existing an new functions, so I will try and explain what it does if called with the below command:

ExecuteRunbook -Name "\DavidW\LoadBalancer\Get-PoolMembers" -OrchestratorServer ServerName.blah.local -Parameters $RunbookparamArray -wait -verbose

So first we get the runbook by its path \folder\folder\runbookname

# create the base url to the service
$url = Get-OrchestratorServiceUrl -server $OrchestratorServer

# Get Runbook by RunbookPath
$runbook = Get-OrchestratorRunbook -ServiceUrl $url -credentials $credentials -runbookpath $Name

We then loop through all the available parameters on the runbook with the direction set to in and see if the array passed in ($RunbookparamArray) contains a matching name value pair, if it does then we add it to a hashtable ready to pass to Start-OrchestratorRunbook.

NB: I haven't done any checking / testing around these being case sensitive as yet, its on the list of things to do, but I needed something up and working quickly.

# Sort Runbook Parameters
$ParamsWithValues = $null
 
if ($Parameters -ne $null){
 Write-Verbose "Parameters supplied for Runbook: $($runbook.Name) $($runbook.Id)"
 # Create Empty Hashtable
 $ParamsWithValues = @{}

 # Loop through Runbook Parameters adding values where we have them.
 foreach ($Parameter in $runbook.Parameters) {
  if ($Parameter.Direction -eq "In"){
   Write-Debug "Parameter Name: $($Parameter.Name)"
   # Add to hashtable so we have an entry for each runbook param, regardless of whether we have a value.
   $ParamsWithValues.Add($Parameter.ID, $null) 
   # Now look through passed param's and add a value where we have one..
   foreach ($item in $Parameters){
    Write-Debug "Item: $($item.Name) Param Name: $($Parameter.Name)"
    if ($item.Name -eq $Parameter.Name) {
     Write-Verbose "Processing Parameter: $($Parameter.Name) ID: $($Parameter.Id)"
     # They both match so add the param value
     $ParamsWithValues.Set_Item($Parameter.Id, $item.Value)
     # Exit For Each... no need to continue within this loop
     break
    }
   }
  }
 }
}

So the function now starts the runbook with the hash table passed through with the parameters, this saves us from having to know the GUID's for the param's and allows admins to change the initialize data step without scripts having to be hard coded with the parameter GUID.

# Start The Runbook
write-verbose "Starting Runbook: $($runbook.Name) $($runbook.Id)"

$job = Start-OrchestratorRunbook -runbook $runbook -credentials $credentials -Parameters $ParamsWithValues 

As we have specified the -wait switch the job will wait for the job to complete, timeout or fail.
(need to do the timeout bit iirc)

again as we specified -wait we can also then try and get the returned data from the runbook (if there is any)

$ReturnedData = $null
if ($Wait -eq $true){
 # Monitor job (Runbook) status 
 Write-Verbose "Job: $($job.id) Waiting for execution, please wait"
 $jobStatus = WaitForRunbook -Job $job -credentials $credentials
 
 # TODO: Implement this functionality.
 If ($jobStatus -eq "Timedout"){
  Write-Verbose "Job: $($job.id) Status: Timeout whilst polling runbook status"
  Throw "Timeout waiting for job to complete. The job will continue to run on the server."
 }
 
 # get ReturnedData if there is any.
 Write-Verbose "Job: $($job.id) Status: Getting returned data"
 $ReturnedData = Get-ReturnedData -Job $job -Credentials $credentials

 if ($ReturnedData -ne $null){
  Write-verbose "Job: $($job.Id) Status: Retrieved returned data"
 }else{
  $ReturnedData = "Nothing to return"
 }
}else{
 Write-Verbose "Job: $($job.Id) Status: Not waiting for runbook execution to complete."
 $ReturnedData = "Runbook Started"
}
Write-Verbose "Finished executing Runbook: $($runbook.Name) Runbook: $($runbook.Id) Job: $($job.id) "

# Return Data
return $ReturnedData

I'm not convinced on the way I am returning the data at the moment (ps-object within hashtable) I think this can be improved on, but haven't really looked at this yet.

An example of calling a runbook is as below:

# Build Runbook Parameters

$RunbookparamArray = @()

$RunbookparamArray += New-ParameterValue -name "LoadBalancer" -Value "nlb.blah.local"

$RunbookparamArray += New-ParameterValue -name "Server" -Value "AServerName"

$RunbookparamArray += New-ParameterValue -name "Object" -Value "Blah Blah"

$RunbookparamArray += New-ParameterValue -name "TargetUserId" -Value "$($env:USERDOMAIN)\$($env:USERNAME)"



#Execute Runbook

ExecuteRunbook -Name "\DavidW\Test\Runbook2" -OrchestratorServer server.blah.local -Parameters $RunbookparamArray -wait -verbose # -Credentials $Credentials

There is also a fix within function getRunbookObject as when calling the sendHttpGetRequest it doesn't pass a credential object. This is fine when running as a user with appropriate credentials but we want to be able to call this from applications such as Octopus deploy where the tentacles run as local system by default. This was meaning that when we passed a credential object in to ExecuteRunbook it was unable to add the runbook parameters as it wasn't getting the available parameters from orchestrator.

Its a simple fix:

[xml] $xmlDoc = sendHttpGetRequest -url $pmurl

Needs changing to:

[xml] $xmlDoc = sendHttpGetRequest -url $pmurl -credentials $Credentials

Whilst I remember I have also changed some of the write-host entries to be either write-debug or write-verbose so that there is no unnecessary noise from the functions.

The new version of the module is available for download from here OrchestratorServiceModule.psm1 and I am currently speaking to a couple of people to see if we can get the codeplex site updated with the minor fixes.

Usual disclaimer, test in your own environment and if in doubt use a file comparison utility / visual studio to view the changes.

You can invoke devenv.exe /diff list1.txt list2.txt from the command prompt or, if a Visual Studio instance is already running, you can type Tools.DiffFiles in the Command window, with a handy file name completion: (Copied From here)