Friday, September 24, 2010

Sync Configurations and Source Control

Synchronization configurations can fit nicely into source control, which is a great way to track the changes of your ILM or FIM changes.  Checking them in is pretty easy, but lately I checked in a change that broke my configuration.
The scenario was:
  1. Export the configuration from the Sync server (File—> Export Server Configuration)
  2. Check in the modified files (I only changed rules on one MA)
  3. Test the configuration on another server (File –> Import Server Configuration)
  4. BANG
The error message returned by Identity Manager was:
File header validation starting...
    'FILE:C:\SyncConfig\New Folder\MA-TargetWebService.xml' contains an invalid header.
File header validation FAILED.

Turns out when you export a server configuration, it stamps the ‘server’ and ‘export-date’ into all of the files.  When you import a server configuration, the import process validates that ALL of the XML files have the same ‘server’ and ‘export-date’.

In my case I tried to add only the changed files to my source control system.  This caused the import process to fail when I tested the changes I’d checked in.
The moral of the story is to keep these header details consistent, either by hacking it manually, or always checking in the complete server configuration as a set.  The more supportable way is to always check the files in as a set because as the XML snippet above reminds us; the XML files should not be edited, even if such edits prevent errors during import ;-)

Monday, September 20, 2010

Using PowerShell Jobs to Run Management Agents Concurrently

There are three basic operations performed by Management Agents in the synchronization engine:

  1. Import (get stuff from the connected system)
  2. Sync (run the Sync rules)
  3. Export (put stuff into the connected system)

The Sync operation should only ever be run by one Management Agent at a time, but the Import and Export operations can (and usually should) be run concurrently.

PowerShell makes easy work of this thanks to the Start-Job cmdlet.  The Start-Job cmdlet starts another PowerShell session and executes a script block in that session.  If you watch Task Manager you will see ‘PowerShell.exe’ processes start up every time you run Start-Job.

The script sample below uses Start-Job to run each Management Agent in its own job.  Here are the highlights:

Get the list of management agents using WMI. 

In my script I want to run all of the AD Management Agents, so I filter the list of Management Agents based on the ‘Type’ property.  This is done using the Where cmdlet, but I could have also done it with a WMI filter.

Loop through the Management Agents

Loop through the Management Agents to execute a Run Profile.  In my example I want to run a Run Profile named ‘Full Import (Stage Only).  That Run Profile has to exist on the Management Agent or the call to the ‘Execute’ method will fail.

Here is the script:

$admas = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer | where {$_.Type -eq 'Active Directory'}

foreach ($ma in $admas)
    Write-host "Starting Full Import on" $ma.Name"..." 

    Start-Job -ArgumentList $ma.Name -ScriptBlock {param($maName)$ma = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer -Filter "name='$maName'";$ma.Execute('Full Import (Stage Only)');}    

Tuesday, September 14, 2010

Multi-Threaded FIM MA Export

The performance of the FIM MA on Export can present quite the deployment challenge.  To review, the FIM MA is very fast on import because it reads the FIM Service database directly.  On export the FIM MA acts like any other FIM Service client, whereby it submits Requests to the FIM Service, which are then processed by MPRs.

You’d think the Request processing and all the associated workflows would be the bottleneck here, and of course a system can be configured as such, but the real problem with FIM MA exports is that the MA processes exports one at a time.

So if you have a large number of pending exports on your FIM MA you can suck it up, run the export run profile then take an extended coffee break, or you could hack around a little bit to push the bottleneck back onto the FIM Service’s SQL database.

PowerShell is my favourite tool lately, and makes for pretty simple multi-threading thanks to the Start-Job cmdlet.  At a high level the approach does the following:

  1. Use CSExport.exe to dump the Pending Exports to a giant XML file
  2. Use an XmlReader to rip through the XML file
  3. Every X number of cs-objects, use Start-Job to process the exports
  4. Inside your Start-Job script, use the FIM PowerShell cmdlets to send the updates to FIM

Benefits of this approach are:

  1. The SQL Server that hosts my FIM Service is pegged, which I take to mean the bottleneck is no longer FIM Sync or FIM Service
  2. Deploying in large environments takes a lot less time

The challenges I’ve hit so far include:

  1. The script is a deployment tool, works great in my hands but tends to fail miserably when handed off to somebody without PowerShell and FIM skills
  2. Passing data to PowerShell jobs in Start-Job is nowhere near as cool as simply piping them in, which is where my expectation is set.  I hope improvements are eventually made here so it becomes as easy as pipeline processing in PowerShell functions.
  3. Some attributes are not easy (or possible) to handle with the FIM PowerShell cmdlets, such as DateTime attributes (no matter how well you format them).  An alternative here would be to use the Quest PowerShell cmdlets for FIM, or to just use the FIM Service web service using the CodePlex client or roll your own.
  4. The FIM MA is able to set the CSEntry anchor on export.  Any approach you take here will not be able to do the same.  The workaround is to immediately import the changes from the FIM Service then run a delta sync.  This will fix up the pending exports. Getting this wrong can result in unnecessary exports that will look like duplicates.
  5. The parent script (the one that creates all the child jobs using Start-Job) will take a lot of memory if the jobs are not removed.  by default they stick around in memory until you remove them, so if the job produced a lot of output then you’ll see a lot of memory used by the parent script.

My plan is to demo this script next month at TEC, and hopefully polish it up to share it afterwards.  FIM MA export performance is an issue that the FIM team at Microsoft is well aware of, so maybe they’ll ship an update to address this before I get to post my script ;-)  Yay optimism!