Tuesday, September 04, 2012

Does Code Coverage Really Matter for FIM Deployments?

There is tremendous value in treating a FIM deployment like software development project but there is a balance to strike between dev and ops (devops anyone?).  Keep in mind, I qualify myself as more ops than dev!

Code coverage is good at measuring how much of a project’s code is covered by tests.  FIM as a product aims to be highly declarative, in theory making the product easier to deploy without writing much code.   A FIM deployment (even one with lots of code) consists mostly of configuration.  Unfortunately there is no ‘configuration coverage’ tool for FIM, which makes it difficult to measure how much of a deployment’s configuration is covered by tests.

Code Coverage Doesn’t Provide Enough Value for the Typical FIM Deployment

My opinion is that code coverage by itself doesn’t provide enough value or raise quality enough for the average FIM deployment because most of the functionality is accomplished through configuration supported by some code.

Test Plans Become More Important

With or without code coverage, we’re supposed to have a nice little document that describes all of the tests we will perform on our deployment, and how we are going to do them.  The test plan needs to have tests for all of the functionality that is supposed to be in the solution.  The lack of code coverage puts more pressure on this document (and the resulting tests) to be more complete.  My suspicion here is that very few deployments see quality measured this way, but I bet most get a rubber stamp on testing!

Code Coverage for Finding Dead Code

A FIM deployment with lots of code has probably been produced and maintained by several developers (and even non-developers) over time.  Code coverage can be used in this scenario to simply show how much code is executed.  For example you can instrument your DLLs, deploy them to your server and let it run for a day or two then produce the code coverage report.  It will show code that is most likely no used, and can likely be removed.  Some of it will be obvious (properties and methods that are never called) but some of it could be code for use cases that rarely happen in production.

Can a Configuration Coverage Tool Be Built?

I thought about doing this for the Sync Engine, and for the FIM Service a while ago but decided against actually building it because there are probably very few people that would actually use it.  Maybe as FIM integrators move more towards devops it will be interesting.  Hopefully it will since configuration coverage would be a really fun tool to build. 


ITNotes said...

Build it and they will come.

ITNotes said...

Build it and they will come.

ITNotes said...

Sorry about the multiple posts. I didn't read the very top. :)

Craig Martin said...

Would love to build it, but there's a long list of tools I wish I had time build and this one is just not high enough on the list yet. It would be a great tool to support a service offering around FIM maintenance though...