Sunday, December 12, 2010

Use PowerShell Jobs to Run Management Agents in Parallel

It is pretty easy with PowerShell to execute management agents in FIM.  It is also pretty easy to do it in parallel using PowerShell jobs.
The sample below shows how to use the Start-Job cmdlet to run Management Agents in FIM.  In this sample I am using Get-WmiObject to get all of the Management Agents, then I apply a Where filter to return just the ADMAs.  I could have specified a similar filter using Get-WmiObject, and would have done so if there were a large number of management agents on the server.
The Start-Job cmdlet runs the script block in another PowerShell session.  If you’re watching Task Manager while the script runs you’ll see another PowerShell.exe process start up.  Also, if you’re watching Identity Manager, you’ll see in the Operations pane that all of the ADMAs start at about the same time (cool!).
Keep in mind that THIS IS NOT RECOMMENDED FOR SYNCHRONIZATION RUN PROFILES!!!  The sync engine does not like this one bit, and will likely grind to a halt dealing with SQL deadlocks.
### Run ADMAs - Full Import - In Parallel

$admas = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer | where {$_.Type -eq 'Active Directory'}

foreach ($ma in $admas)
    Write-host "Starting Full Import on" $ma.Name"..." 
    Start-Job -ArgumentList $ma.Name -ScriptBlock `
            param($maName)$ma = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer -Filter "name='$maName'";
            $result = $ma.Execute('Full Import').ReturnValue;
            Write-Host ("{0} result: {1}" -f $maName, $result
        } | Out-Null

### Wait for the imports to complete before returning
Get-Job | Wait-Job

### Review the output from each job
Get-Job | Receive-Job

### Remove the jobs
Get-Job | Remove-Job


Thomas Lee said...

Two minor points.

First, in the ForEach loop, you will be running a separate instance of PowerShell for each job. If there are a low number of MAs in the collection, this is no big deal. But as you get more, memory useage rises and depending on the RAM on your system, you could get into a large amount of paging, killing performance on that system. This problem is excerbated if the import is particularly memory hungary or takes a while.

Secondly, the colouring on the script (as viewed from IE8) leaves some text missing. If you hightlight the script, you can see the missing chars.

Craig Martin said...

Thanks for the feedback Thomas.

Most FIM servers have a small number of MAs, and the jobs do not produce a large amount of output or incur much processing in PowerShell, it is mostly done in the FIM server process.

I've been caught by the issue you mention though. In another script using PowerShell jobs I was doing a lot of processing in PowerShell and the scripts in the jobs did produce a lot of output. The issue I ran into was that the jobs did not automagically clean themselves up once finished. If I didn't do Remove-Job on the finished jobs then the script would eat through all of the memory keeping the results of those jobs around.

PowerShell jobs are awesome for cheap multi-threading, but I'm hoping for improvements in passing data between the jobs. It was a bummer that I couldn't easily use a pipeline to stuff objects into the child jobs.

Fixed the formatting too, thanks for catching that!

Thomas Lee said...

Thanks for the respons!

With only a few MAs to import,
then I suspect there is not much if any performance issue. But it never hurts to at least ask the question.

The problem, as you allude, comes when there are a lot of objects either being processed or being returned. In those cases, memory usage can rise pretty dramatically to the point where the host ends up paging-bound (not good).

Like you, I hope to see improvements in the Job object when MS reveal what's going to be in PowerShell Version 3. I have a bunch of stuff I'd like jobs to do, not least of which is persist across instances of PowerShell. Pipelinine between job instances is another thing I'd also like to see! We'll have to wait for a while though.

Thanks for fixing the formating too.