Wednesday, December 29, 2010

Querying for FIM CM Profiles and Requests

The FIM CM MA uses SQL to query for the CM Profile and Request objects.  Turns out this isn’t simple to do with the FIM CM Provision API, so going to the database seems justifiable in this case.

The PowerShell snippets below uses the .NET SQL Client to query the FIM CM database using queries similar to the full import from the FIM CM MA.

Some differences include:

Delegation

The FIM CM MA does not execute the SQL from the FIM Sync box.  Instead it asks a proxy object on the FIM CM server to do this.  While this is elegant from a security POV (we are just doing what the CM server already does, instead of pulling the SQL back to the FIM Sync box) it is a pain in production environments where best practice dictates we don’t run FIM CM and SQL on the same server, introducing a kerberos delegation scenario.

The script below initiates the query from ‘localhost’ but could easily be modified to run from a remote server, eliminating the kerberos delegation issue.

Finding the SQL Server

The FIM CM MA by default asks FIM CM where the CM database is located.  This data is stored in the registry on the CM server, and the FIM CM MA queries for it.  The script below simply hard codes the database name and database server name.

Query for CM Requests

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033



###
### Query for CLM Requests
### Filter out all completed (8), denied(4), failed(9) and canceled(17) requests
###

$sqlQuery = 
@"
SELECT Requests.* , UserNameCache.unc_user_nt4_name AS req_target_user_name
FROM Requests WITH (NOLOCK)
LEFT OUTER JOIN UserNameCache
    ON Requests.req_target_user_uuid = UserNameCache.unc_user_uuid
WHERE req_status <> 8
AND req_status <> 4
AND req_status <> 9
AND req_status <> 17
"@


$sqlServer = "localhost"
$sqlCatalog = "FIMCertificateManagement"
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = "Server = $SqlServer; Database =$SqlCatalog; Integrated Security = True"
$sqlCmd = New-Object System.Data.SqlClient.SqlCommand
$sqlCmd.CommandText = 
$sqlQuery
$sqlCmd
.Connection = 
$sqlConnection
$sqlAdapter
 = New-Object System.Data.SqlClient.SqlDataAdapter
$sqlAdapter.SelectCommand = 
$sqlCmd
$dataSet
 = New-Object System.Data.DataSet
$sqlAdapter.Fill($dataSet)
$sqlConnection.Close()

$dataSet.Tables[0].
Rows


Query for CM Profiles

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026

###
### Query for CLM Profiles
###

$sqlQuery = 
@"
Select Profiles.* , UserNameCache.unc_user_nt4_name AS pr_assigned_user_name, sc_status, sc_serial_number, sc_permanent_sc_uuid
FROM Profiles with (NOLOCK)
LEFT OUTER JOIN UserNameCache
    ON Profiles.pr_assigned_user_uuid = UserNameCache.unc_user_uuid
LEFT OUTER JOIN Smartcards
    ON Profiles.pr_sc_uuid = Smartcards.sc_uuid
"@


$sqlServer = "localhost"
$sqlCatalog = "FIMCertificateManagement"
$sqlConnection = New-Object System.Data.SqlClient.SqlConnection
$sqlConnection.ConnectionString = "Server = $SqlServer; Database =$SqlCatalog; Integrated Security = True"
$sqlCmd = New-Object System.Data.SqlClient.SqlCommand
$sqlCmd.CommandText = 
$sqlQuery
$sqlCmd
.Connection = 
$sqlConnection
$sqlAdapter
 = New-Object System.Data.SqlClient.SqlDataAdapter
$sqlAdapter.SelectCommand = 
$sqlCmd
$dataSet
 = New-Object System.Data.DataSet
$sqlAdapter.Fill($dataSet)
$sqlConnection.Close()

$dataSet.Tables[0].Rows

Dirty SQL Query to Get MV Attribute Details

First a warning: querying the sync database directly is a worst practice.  The following script was run on a test server, and will never see my production servers.  If you really want to run this on a product server the copy off the database to another server and experiment there.

DO NOT USE THIS ON PRODUCTION SERVERS

Okay, on to business.  I recently had to answer the question: how many times will MA X be precedent for a given attribute?  Unfortunately the consulting answer (it depends) did not fly so I went to the database to find out.  The dirty little script below shows how to dig out the answer.  It is only interesting because it requires the UNPIVOT statement which is not straight forward unless you happen to be David Lundell.

The script below hard codes the MA GUID, as well as the attribute I’m interested in.  There may be a way more elegant script for this but I couldn’t find one so hopefully the next person that needs to answer this question will find either my post or be David ;-)

BTW – this is one of the few questions lately where I did NOT have a good PowerShell answer (at least not a supported one).  PowerShell is awesome for slicing and dicing data with Select-Object and Group-Object but in its infancy just doesn’t have an Unpivot-Object cmdlet yet.  This is easily forgivable since it is way cooler than TSQL will ever be, at least for an IT Pro like me.

   1: ---



   2: ---Objects that have UserPrincipalName attribute contributed by Exchange



   3: ---



   4: SELECT mailNickname



   5: FROM mms_metaverse 



   6: WHERE object_id IN



   7: (



   8:     SELECT Object_id



   9:     FROM 



  10:     (



  11:         SELECT Object_id, userprincipalname



  12:         FROM mms_metaverse_lineageguid



  13:     ) p



  14:     UNPIVOT



  15:     (



  16:         LineageID FOR Attribute IN 



  17:         (



  18:             userprincipalname



  19:         )



  20:     )



  21:     AS unpvt



  22:     JOIN mms_lineage_cross_reference on mms_lineage_cross_reference.lineage_id = unpvt.LineageID



  23:     WHERE 



  24:     (



  25:         mms_lineage_cross_reference.ma_id = 'AAAAAAAA-DDDD-4444-8B6A-CD6ACEA5CB49' 



  26:         and 



  27:         attribute = 'userprincipalname'



  28:     )



  29: )


More FIM Awards

FIM seems to be quietly accumulating quite a few accolades…  Does this mean it is on the verge of becoming mainstream?  Am I still cool if I don’t work on obscure and niche products?  Time will tell, but I doubt it will be another decade :-|

Microsoft STB Products Recognized as Industry Winners in 2010

Using the FIM WMI Provider While Syncs Are Running

It’s always been a worst-practice to make external calls from rules extensions in the sync engine, but a bad idea of mine illustrated the reason why today.

In my scenario, I really want my MV code to fire on objects ONLY when they have already been exported and confirmed by the outbound MA.  The right way to do this is to use a constant Import Attribute Flow (IAF) rule on that MA, in which case the scenario looks like this:

  1. MV code creates a new CSEntry in the outbound MA
  2. Outbound MA runs Export
  3. Outbound MA runs Import
  4. Outbound MA runs Sync – this is where our IAF rule flows a value to the MVEntry

If the attribute on the MVEntry has a value, then we know the outbound MA has completed the Export and Import successfully.

The rub in my scenario is that I cannot modify the MV schema, so I can’t use this approach.  Grasping at straws I thought I could just look at the CSEntry using the MIIS_CSObject class in WMI.  This is where the wheels came off.

When I put the code in place and ran Preview, the sync service hung.  In my sync cycle, WMI was being called from my MV code which seems harmless enough since WMI is only reading objects, but I knew I was being less than smart, this was just prototyping, really, I’m normally a lot smarter, at least the second or third time.

Anyhow, this exact scenario created what is most likely a deadlock in SQL but the usual MIIS deadlock error didn’t happen.  Unfortunately the service just hung and had to be killed (killed several times in fact, as I was troubleshooting the issue).  Hard to call this a fault of the product since I should know better, but I would have liked the sync cycle to fail with a deadlock error, or the WMI call to fail with something similar.

The moral of the story is simple; follow the best practices as much as possible; don’t make external calls from rules extension code, and don’t query the sync database (even with WMI I guess) while the sync engine is running.

Speaking at TechReady12 in February!

Got lucky with session submissions and will be speaking at TechReady12 in February.  I’ll be talking about PowerShell integration with FIM, and another session about integrating FIM CM with FIM Sync and the FIM Service and Portal.

Hope to see you there!

Sunday, December 12, 2010

In Search of Export-Not-Reimported Errors

Lately I’ve been scripted the heck out of the Run-History.  This facility has been in the product forever but now PowerShell makes it a lot easier to slice and dice thanks to the way PowerShell makes WMI and XML very easy.

The oddity here is that the CSObject in the Sync engine contains a lot of useful error detail, as well as the holograms.  There’s a lot of data packed into those CSObjects, and pretty much the only facilities for getting at them are WMI and CSExport.exe.

The MIIS_CSObject WMI class allows easy access to the CSObjects but unfortunately the query functionality is very limited.  You need to know the DN or the GUID of the CSObject in order to find it with WMI.  My guess is that this is because the rest of the attributes in the CSObject are packed away as binary blobs in the underlying SQL database.

Anyhow, my point is that there is a lot of detail on the CSObject but the Export-Not-Reimported details are stored in RunHistory, not on the CSObject.

The script below is a sample for getting the run history from the server, then getting the RunDetails XML.  The RunDetails XML can then be parsed to dig out the export-not-reimported errors, which I’ll show in another post.

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023

### Use WMI to get the Management Agent
$managementAgent = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer -Filter "name='HoofHearted'"

### Construct a filter to get the management agent's last run
$filter = ("RunNumber='{0}'" -F $managementAgent.RunNumber().ReturnValue)

### Use WMI to get the RunHistory
$theRun = Get-WmiObject -Class MIIS_RunHistory -Namespace root/MicrosoftIdentityIntegrationServer -Filter $filter

### Call the RunDetails() method to get the RunHistory RunDetails XML
[xml]$runHistoryDetails = $theRun.RunDetails().ReturnValue
if ($runHistoryDetails -ne $null)
{
    Write-Host " ...Success" -ForegroundColor Green
}
else
{
    ### There have been bugs with the WMI provider for ILM/FIM
    ### When it fails it usually returns null
    ### But on rare occasions I've also seen it fail with out-of-memory errors (even though the box had 128GB, and over 100GB free)
    Write-Host " ...Error" -ForegroundColor Red
    Continue
}

Use PowerShell Jobs to Run Management Agents in Parallel

It is pretty easy with PowerShell to execute management agents in FIM.  It is also pretty easy to do it in parallel using PowerShell jobs.
The sample below shows how to use the Start-Job cmdlet to run Management Agents in FIM.  In this sample I am using Get-WmiObject to get all of the Management Agents, then I apply a Where filter to return just the ADMAs.  I could have specified a similar filter using Get-WmiObject, and would have done so if there were a large number of management agents on the server.
The Start-Job cmdlet runs the script block in another PowerShell session.  If you’re watching Task Manager while the script runs you’ll see another PowerShell.exe process start up.  Also, if you’re watching Identity Manager, you’ll see in the Operations pane that all of the ADMAs start at about the same time (cool!).
Keep in mind that THIS IS NOT RECOMMENDED FOR SYNCHRONIZATION RUN PROFILES!!!  The sync engine does not like this one bit, and will likely grind to a halt dealing with SQL deadlocks.
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
###
### Run ADMAs - Full Import - In Parallel
###


$admas = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer | where {$_.Type -eq 'Active Directory'}

foreach ($ma in $admas)
{
    Write-host "Starting Full Import on" $ma.Name"..." 
    Start-Job -ArgumentList $ma.Name -ScriptBlock `
        {
            param($maName)$ma = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer -Filter "name='$maName'";
            $result = $ma.Execute('Full Import').ReturnValue;
            Write-Host ("{0} result: {1}" -f $maName, $result
        } | Out-Null
}

### Wait for the imports to complete before returning
Get-Job | Wait-Job

### Review the output from each job
Get-Job | Receive-Job

### Remove the jobs
Get-Job | Remove-Job

Monday, November 15, 2010

Connection Strings in the FIM CM Lab Documents

The procedure in the FIM CM lab document for creating the connection string is really neat, but it did not work for me because FIM CM not like the resulting connection string. 
In the "FIM Certificate Management" logs in Event Viewer I got an Error followed by a useful warning.

The same issue applies to both documents:

In short:

The FIM CM Exit Module does not like the "provider" keyword, so just omit it from the connection string and voila!

Error Detail:

"2010-11-03 05:42:21.07 -07" "Microsoft.Clm.ExitModule.CertExit" "Void RegisterCA()" "" "NT AUTHORITY\SYSTEM" 0x00000BFC 0x00000001
[DatabaseConnection] configuration setting is not configured correctly. Use FIM CM Exit Module settings dialog to set it to a valid value.

Warning Detail:

1) Exception Information
*********************************************
Exception Type: System.ArgumentException
Message: Keyword not supported: 'provider'.
ParamName: NULL
Data: System.Collections.ListDictionaryInternal
TargetSite: System.Data.Common.NameValuePair ParseInternal(System.Collections.Hashtable, System.String, Boolean, System.Collections.Hashtable, Boolean)
HelpLink: NULL
Source: System.Data

Posting PowerShell Snippets Using Windows Live Writer

Lately I’ve been trying to find the best method of posting PowerShell snippets using LiveWriter.  So far the best method I’ve found is a script from PowerShell guru Lee Holmes.

The only tricks to it are:

  1. Run the script from PowerShell ISE (instead of the command prompt)
  2. Use ‘Paste Special’ in Windows Live Writer (by default it removes the formatting)

Thanks to Lee my blog should start looking just a little bit prettier!

Tuesday, November 09, 2010

ILM Sync Engine Configuration PowerShell Commandlet Update

ILM (the precursor to FIM) has had some PowerShell coverage for a while now, allowing the automation of configuration for ADMA and XMA management agent types.
The cmdlets don’t ship with the product as they do in FIM.  They can be downloaded here:
Identity Lifecycle Manager 2007 FP1 Sync Engine Configuration PowerShell Commandlets
Until recently there were three cmdlets available, but with a recent update they have added another cmdlet that allows you to toggle the provisioning extension.
PS C:\> Add-PSSnapin MIIS.MA.Config
PS C:\> Get-Command -Module MIIS.MA.Config
CommandType     Name
-----------     ----
Cmdlet          Import-MIISServerConfig
Cmdlet          Set-MIISADMAConfiguration
Cmdlet          Set-MIISExtMAConfiguration
Cmdlet          Set-ProvisioningRulesExtension
Here’s the contents from the cmdlet help to illustrate how the cmdlet is used:
--------------  Example 1 --------------
C:\PS>Set-ProvisioningRulesExtension true
This command will enable the provisioning rules extension.
Note: The provisioning rules extension assembly must be specified.

--------------  Example 2 --------------
C:\PS>Set-ProvisioningRulesExtension false
This command will disable the provisioning rules extension.
The cmdlet is toggling the checkbox for “Enable Provisioning Rules Extension”, as shown below.  It accepts a String instead of a Boolean, but specifying anything other than ‘true’ or ‘false’ results in an error.

Thursday, November 04, 2010

TechNet Guide for Building a FIM CM Lab

Pretty good timing, given that I’m posting on CM lately… Microsoft just posted a lab guide for demonstrating FIM CM 2010.

Test Lab Guide: Demonstrate FIM CM 2010

If you’re looking to learn more about FIM CM then this will be a great resource in addition to all the other FIM CM documentation available on TechNet.

Use the FIM CM Provision API from PowerShell

In my previous example I showed how to call the CM MA’s proxy from PowerShell for the purposes of CM MA troubleshooting.

In this example I show how to use PowerShell to call the CM Provision API.

The Provision API is the main extensibility point into FIM Certificate Management.  It offers an API for the following:

  • Working with Profile Templates (shown below)
  • Working with Requests
  • Permission Operations
  • Working with Profiles and Certificates

There are a number of scenarios where it makes sense to use the CM Management Agent, but in some cases it is overkill.  If a small piece of functionality can be accomplished using just the Provision API then it probably makes sense to just use code/script against the Provision API, as shown in the sample below.

.NET Remoting

FIM CM employs .NET Remoting, and the bulk of the sample script is dedicated to setting up the .NET Remoting connection to the FIM CM server.  It’s only the last two lines of the script that do anything fun really.

This is supposed to be easier with the ‘New-WebServiceProxy’ cmdlet in PowerShell V2 but I haven’t had any luck with it yet.

Enabling the Provision API in the CM Service

The Provision API is not enabled by default.  The CM web.config file needs to be modified before you can access the Provision API.  Follow the instructions here to make the web.config file modification.  Specifically you want to following the instructions under “Server Configuration”.

The Sample

###

### Load the CLM Provision Assembly, and the .NET Remoting Assembly

###

[reflection.Assembly]::LoadFrom("C:\Program Files\Microsoft Forefront Identity Manager\2010\Certificate Management\web\bin\Microsoft.Clm.Provision.dll")

[reflection.Assembly]::LoadFrom("C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Runtime.Remoting.dll")

 

###

### Set up the remoting infrastructure

###

$clmUrl = "http://localhost/certificatemanagement/remoterequests3.rem"

$binaryClientFormatterSinkProvider = new-object System.Runtime.Remoting.Channels.BinaryClientFormatterSinkProvider

$httpClientChannel = New-Object System.Runtime.Remoting.Channels.Http.HttpClientChannel("ClmHttpChannel", $binaryClientFormatterSinkProvider)

[System.Runtime.Remoting.Channels.ChannelServices]::RegisterChannel($httpClientChannel, $true)

[System.Runtime.Remoting.RemotingConfiguration]::RegisterWellKnownClientType([Microsoft.Clm.Provision.FindOperationsByCulture], $clmUrl)

 

$FindOperationsByCulture = New-Object Microsoft.Clm.Provision.FindOperationsByCulture

$channelProperties = [System.Runtime.Remoting.Channels.ChannelServices]::GetChannelSinkProperties($FindOperationsByCulture)

$clmUri = [System.Runtime.Remoting.RemotingServices]::Marshal($FindOperationsByCulture).URI

 

###

### Supply the credentials for connecting to CLM

###

$networkCredentials = New-Object System.Net.NetworkCredential("administrator",'hoofhearted',"icemelted")

$credentialCache = New-Object System.Net.CredentialCache

$credentialCache.Add($clmUri,'ntlm',$networkCredentials)

$channelProperties.credentials = [System.Net.CredentialCache]$credentialCache

 

###

### Get the Profile Templates

###

$profileTemplates = $FindOperationsByCulture.FindAllProfileTemplates([System.Globalization.CultureInfo]::InvariantCulture,[System.Globalization.CultureInfo]::InvariantCulture)

 

$profileTemplates | ft DisplayName

Output from the above script should look like this:

DisplayName                                                    
-----------                              
FIM CM Sample Profile Template                             
FIM CM Sample Smart Card Logon Profile Template 

Wednesday, November 03, 2010

Certificate Management MA Troubleshooting

Working on a new CM MA at the moment, so of course am using PowerShell to verify connectivity from the Sync box to the CM box.  This is the quick little script that mimics what the CM MA does when it connects to the CM service:

###

### Load the CLM MA Proxy Assembly, and the .NET Remoting Assembly

###

[reflection.Assembly]::LoadFrom("C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\Microsoft.Clm.ClmMaProxy.dll")

[reflection.Assembly]::LoadFrom("C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Runtime.Remoting.dll")

 

###

### Set up the remoting infrastructure

###

$clmUrl = "http://localhost/certificatemanagement/clmManagementAgent.rem"

$binaryClientFormatterSinkProvider = new-object System.Runtime.Remoting.Channels.BinaryClientFormatterSinkProvider

$httpClientChannel = New-Object System.Runtime.Remoting.Channels.Http.HttpClientChannel("ClmHttpChannel", $binaryClientFormatterSinkProvider)

[System.Runtime.Remoting.Channels.ChannelServices]::RegisterChannel($httpClientChannel, $true)

[System.Runtime.Remoting.RemotingConfiguration]::RegisterWellKnownClientType([ExtensibleWfMA.ClmMaProxy], $clmUrl)

$clmMaProxy = New-Object ExtensibleWfMA.ClmMaProxy

$channelProperties = [System.Runtime.Remoting.Channels.ChannelServices]::GetChannelSinkProperties($clmMaProxy)

 

 

###

### Supply the credentials for connecting to CLM

###

$clmUri = [System.Runtime.Remoting.RemotingServices]::Marshal($clmMaProxy).URI

$networkCredentials = New-Object System.Net.NetworkCredential("administrator",’hoofhearted’,"icemelted")

$credentialCache = New-Object System.Net.CredentialCache

$credentialCache.Add($clmUri,'ntlm',$networkCredentials)

$channelProperties.credentials = [System.Net.CredentialCache]$credentialCache

 

###

### Call the ConnectionTest method

###

$clmMaProxy.ConnectionTest()

The commands in the above script use .NET Remoting to connect to the ClmMaProxy which sits on the CM server.  Everything but the last line is setting up the .NET Remoting infrastructure, then finally the last line calls a method “ConnectionTest()” on the ClmMaProxy to uh, test the connection.

If all is good I expect it to return “True”, otherwise an error from the CM service.

Tuesday, November 02, 2010

Starting work on a New FIM MA for Certificate Management

Been thinking about doing this for a couple years now, so finally getting a start and should have something to post in a couple weeks.

The idea is to create a new MA and post it to CodePlex for people that want to integrate FIM CM with the FIM Service and the FIM Sync Engine.

Both FIM CM and the FIM Service have workflow engines.  In the FIM Service we have Management Policy Rules, and in FIM Certificate Management we have Profile Templates.  Before FIM the only integration story was really to create a management agent, which was of course delivered in ILM 2007.  Now with FIM we have the opportunity for even more integration, which I hope to facilitate with this effort.

My development environment (building it now) will have the following machines:

DC – Windows Server 2008 R2 x64 DC and CertSrv

CM – Windows Server 2008 R2 X64 with SQL and FIM CM

FIM – Windows Server 2008 R2 x64 with SQL and FIM Service and Portal

SYNC – Windows Server 2008 R2 x64 with SQL and FIM Sync

At the moment I’m not planning on building or testing this with ILM 2007 FP1 unless I get a lot of feedback to do so.

Simple Reporting in FIM 2010 with SSRS

I’ve been working on an extension to SQL Server Reporting Services (SSRS) to facilitate simple reporting of data in the FIM 2010 service.

SSRS provides the Data Processing Extension interface which allows other data sources to feed data into reports.  This little extension acts as a gateway to the features and functionality of SSRS.  In other words, it helps close the feature gap in FIM since FIM 2010 does not ship with any reporting functionality yet the FIM service itself tends to contain mountains of identity data that customers tend to want to report on.  FIM requires SQL Server anyhow, so why not use a little bit more of that box?

So far I’ve been using the FIM Data Processing Extension (FIM DPE) to produce reports of data in the FIM Service, but also to address some of the shortcomings of the FIM notifications.  For example, with SSRS data-driven subscriptions you can use the FIM DPE to deliver notifications based on events, or just on a schedule.  And instead of using FIM email templates you can use the SSRS report designers to produce the notifications.

This effort in no way attempts to replace what partners such as Quest, Omada and bHold have in their offerings.  I consider this a low cost DIY approach to FIM reporting.  If you have the data you need in the FIM Service and are willing to design your reports using the SSRS report designers (they are pretty easy to use) then this project is worth exploring (and contributing feedback to!).  If you are in the market for a full featured partner solution to FIM then of course skip this project and go right into evaluating the other solutions.

At the moment the code is posted to CodePlex.  A release isn’t carved out yet since I want to add more documentation and instrumentation.  In the interim, feel free to poke at the code, give it a run, and provide feedback and code reviews!

Monday, October 18, 2010

Magic Quadrant for User Provisioning

Gartner released the Magic Quadrant for User Provisioning recently.  The Microsoft product in this category is of course our beloved FIM 2010. 

At a glance Microsoft looks to be in the wrong quadrant (yikes!) but the optimist in me sees Microsoft moving from Challengers into Leaders.  My optimism is based on the following:

  • FIM 2010 adds a lot of functionality to the sturdy sync engine, and this new functionality has room to improve
  • Future integration with Sentillion
  • Improved integration with Certificate Management (it wasn’t even mentioned)
  • The increase in community participation

At TEC2010 in LA it was revealed that Microsoft re-org’d the FIM team.  At TEC2010 in Germany we saw a glimpse of what is possible when FIM is combined with Microsoft System Center (System Center Identity Manager perhaps?).  I mention TEC a lot here because it is a nice focused venue to get these tidbits face-to-face.

If Microsoft can nail the ‘better together’ story then I think it will hard to keep them out of the Leaders quadrant.  Will it happen by next year?  It’s a good question to ask at TEC 2011 in Vegas Baby!

Friday, September 24, 2010

Sync Configurations and Source Control

Synchronization configurations can fit nicely into source control, which is a great way to track the changes of your ILM or FIM changes.  Checking them in is pretty easy, but lately I checked in a change that broke my configuration.
The scenario was:
  1. Export the configuration from the Sync server (File—> Export Server Configuration)
  2. Check in the modified files (I only changed rules on one MA)
  3. Test the configuration on another server (File –> Import Server Configuration)
  4. BANG
The error message returned by Identity Manager was:
File header validation starting...
    'FILE:C:\SyncConfig\New Folder\MA-TargetWebService.xml' contains an invalid header.
File header validation FAILED.











Turns out when you export a server configuration, it stamps the ‘server’ and ‘export-date’ into all of the files.  When you import a server configuration, the import process validates that ALL of the XML files have the same ‘server’ and ‘export-date’.











In my case I tried to add only the changed files to my source control system.  This caused the import process to fail when I tested the changes I’d checked in.
The moral of the story is to keep these header details consistent, either by hacking it manually, or always checking in the complete server configuration as a set.  The more supportable way is to always check the files in as a set because as the XML snippet above reminds us; the XML files should not be edited, even if such edits prevent errors during import ;-)

Monday, September 20, 2010

Using PowerShell Jobs to Run Management Agents Concurrently

There are three basic operations performed by Management Agents in the synchronization engine:

  1. Import (get stuff from the connected system)
  2. Sync (run the Sync rules)
  3. Export (put stuff into the connected system)

The Sync operation should only ever be run by one Management Agent at a time, but the Import and Export operations can (and usually should) be run concurrently.

PowerShell makes easy work of this thanks to the Start-Job cmdlet.  The Start-Job cmdlet starts another PowerShell session and executes a script block in that session.  If you watch Task Manager you will see ‘PowerShell.exe’ processes start up every time you run Start-Job.

The script sample below uses Start-Job to run each Management Agent in its own job.  Here are the highlights:

Get the list of management agents using WMI. 

In my script I want to run all of the AD Management Agents, so I filter the list of Management Agents based on the ‘Type’ property.  This is done using the Where cmdlet, but I could have also done it with a WMI filter.

Loop through the Management Agents

Loop through the Management Agents to execute a Run Profile.  In my example I want to run a Run Profile named ‘Full Import (Stage Only).  That Run Profile has to exist on the Management Agent or the call to the ‘Execute’ method will fail.

Here is the script:

$admas = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer | where {$_.Type -eq 'Active Directory'}

foreach ($ma in $admas)
{
    Write-host "Starting Full Import on" $ma.Name"..." 


    Start-Job -ArgumentList $ma.Name -ScriptBlock {param($maName)$ma = Get-WmiObject -Class MIIS_ManagementAgent -Namespace root/MicrosoftIdentityIntegrationServer -Filter "name='$maName'";$ma.Execute('Full Import (Stage Only)');}    
}

Tuesday, September 14, 2010

Multi-Threaded FIM MA Export

The performance of the FIM MA on Export can present quite the deployment challenge.  To review, the FIM MA is very fast on import because it reads the FIM Service database directly.  On export the FIM MA acts like any other FIM Service client, whereby it submits Requests to the FIM Service, which are then processed by MPRs.

You’d think the Request processing and all the associated workflows would be the bottleneck here, and of course a system can be configured as such, but the real problem with FIM MA exports is that the MA processes exports one at a time.

So if you have a large number of pending exports on your FIM MA you can suck it up, run the export run profile then take an extended coffee break, or you could hack around a little bit to push the bottleneck back onto the FIM Service’s SQL database.

PowerShell is my favourite tool lately, and makes for pretty simple multi-threading thanks to the Start-Job cmdlet.  At a high level the approach does the following:

  1. Use CSExport.exe to dump the Pending Exports to a giant XML file
  2. Use an XmlReader to rip through the XML file
  3. Every X number of cs-objects, use Start-Job to process the exports
  4. Inside your Start-Job script, use the FIM PowerShell cmdlets to send the updates to FIM

Benefits of this approach are:

  1. The SQL Server that hosts my FIM Service is pegged, which I take to mean the bottleneck is no longer FIM Sync or FIM Service
  2. Deploying in large environments takes a lot less time

The challenges I’ve hit so far include:

  1. The script is a deployment tool, works great in my hands but tends to fail miserably when handed off to somebody without PowerShell and FIM skills
  2. Passing data to PowerShell jobs in Start-Job is nowhere near as cool as simply piping them in, which is where my expectation is set.  I hope improvements are eventually made here so it becomes as easy as pipeline processing in PowerShell functions.
  3. Some attributes are not easy (or possible) to handle with the FIM PowerShell cmdlets, such as DateTime attributes (no matter how well you format them).  An alternative here would be to use the Quest PowerShell cmdlets for FIM, or to just use the FIM Service web service using the CodePlex client or roll your own.
  4. The FIM MA is able to set the CSEntry anchor on export.  Any approach you take here will not be able to do the same.  The workaround is to immediately import the changes from the FIM Service then run a delta sync.  This will fix up the pending exports. Getting this wrong can result in unnecessary exports that will look like duplicates.
  5. The parent script (the one that creates all the child jobs using Start-Job) will take a lot of memory if the jobs are not removed.  by default they stick around in memory until you remove them, so if the job produced a lot of output then you’ll see a lot of memory used by the parent script.

My plan is to demo this script next month at TEC, and hopefully polish it up to share it afterwards.  FIM MA export performance is an issue that the FIM team at Microsoft is well aware of, so maybe they’ll ship an update to address this before I get to post my script ;-)  Yay optimism!