Wednesday, November 08, 2006

Taming My Memory Hungry XMA

XMA performance can be a really fun challenge. Sometimes the simple solution doesn't prevail, and in my case the simple design to an LDAP XMA didn't make very efficient use of memory on the server. It actually ran the server dry out of memory. Now I'm trying to improve memory usage but I'm seeing some odd behaviour; the memory I'm freeing to garbage collection doesn't seem to be released. If I run the XMA in the miiserver.exe process then it doesn't get released until the service restarts. If I run it in its own process then it gets released but only after the MA finishes running.

UPDATE - November 20
Smaller Queries
Garbage collection issues aside, the XMA needs to use a lot of memory because it is trying to do a full import from an LDAP server that doesn't support paging. To reduce the memory usage on full import I employed a query option that breaks one big query (objectClass=*) into smaller queries, (sn>=m) (sn<=l). This significantly reduced memory usage but I still had a large memory footprint due to another problem.

The XMA for this solution needs to construct attributes based on other attributes, sometimes from other objects. These attributes are very easy to construct if you just query the DS for the ingredients, it even performs well if you only have a modest number of object. It turns out a full import on 1/2 million objects just didn't perform to expectations. To reduce the number of queries to the DS I decided to cache the ingredients into memory. This worked fine, but of course introduced a large memory footprint. I initially decided on using a SearchResultCollection to store the entries in memory but later decided on using a DataTable because it offered me more flexibility and used less memory.

So in the end, the XMA has a lot of work to do and it performs reasonably well on full imports. Luckily it does implement delta imports so full imports will not be a regular activity.

No comments: