continuous integration

What is code coverage?

Code coverage, in short, is all about how thoroughly your tests exercise your code base. The intent of tests, of course, is to verify that your code does what it’s expected to, but also to document what the code is expected to do. Taken further, code coverage can be considered as an indirect measure of quality — indirect because we’re talking about the degree to what our tests cover our code, or simply, the quality of tests. In other words, code coverage is not about verifying the end product’s quality.

So, how does code coverage differ from other types of testing techniques? Code coverage can be classified as white-box testing or structural testing because the “assertions” are made against the internals of our classes, not against the system’s interfaces or contracts. Specifically, code coverage helps to identify paths in your program that are not getting tested. It is because of this distinction that code coverage shows its usefulness when testing logic-intensive applications with a lot of decision points.

Needs for Code Coverage

To be able to use a code coverage tool we need a number of requirements, they are:

Command line driven
Xml reports
Usable within any unit testing framework
Support for up to .Net 4.0 framework
Stable and supported

Tools

There are a number of tools available for code coverage, mainly in the Java world however here are a few in the .NET world

PartCover http://partcover.blogspot.com/

NCover http://www.ncover.com/

TestDriver.NET http://www.testdriven.net/Default/?ReferrerId=7393

dotCover http://confluence.jetbrains.net/display/DCVR/dotCover+Early+Access+Program

Question is which tool and why?

The quickest answer would be dotCover as it is from JetBrains, however dotCover is currently only in Early Access Program, and perhaps in the future this would fill our needs.

PartCover, does not appear to be well supported and a little tricky to get working.

TestDriver.NET is perfect for checking out your code coverage from within Visual Studio and works well with Gallio too, however is not well supported and the documentation is very sparse.

Which leaves us with NCover, although it used to be free, the owners have now produced a commercial version, which is very well supported, supporting all .NET languages, 32 and 64 bit support, no special builds required, Sequence point coverage reporting and Branch coverage reporting, XML coverage data output for integration to CI servers, any many more. It would be nice for each developer to have NCover, however the cost is quite high ($199.00 per user), and if budgets are restricted then a single version would be sufficient for the CI server.

Summary

I have not gone in to great detail about how to perform code coverage, as there is ample information on the internet, what I have tried to do is provide a simple justification to why code coverage is an integral part of the development process.

My recommendation would be NCover, mainly because it appear to be the only fully supported Code Coverage tool for .NET

Another Code Coverage tool, which is built in to Visual Studio Team edition, I have not covered here, as it was not available for me at the time of writing. Thanks Nathan Gloyn for point this out.

Task today is to decide and implement an Continuous Integration build process, I will be looking at two options CruseControl and Team City. The are a lot more but for the purpose of the current .NET environment these are the two I’ll be looking at.

When working in a team, the developer needs to know that any changes he makes and stores in to your repository will work for others. When the developer submits code to the repository they must first update their code to reflect the changes in the repository since they took their copy. The more changes there are to the repository, the more work the developer must do before submitting their own changes.

Eventually, the repository may become so different from the developer’s baseline that they enter what is sometimes called, “integration hell,” where the time it takes to integrate is greater than the time it took to make their original changes. In a worse case scenario, the changes the developer is making may have to be discarded and the work redone.

Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of “integration hell”. The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals.

After speaking and Googling for each of the products, I quickly came to the conclusion that although a number of clients I have work with before have been using CruseControl I have found it quite hard to setup and the configuration of XML configuration files can be a bit of a nightmare, and as such we just need a tool to work and not have to maintain. So for the rest of this article I will be focusing on Team City by JetBrains.

I will be using version 4.5.1. TeamCity-4.5.1.exe (231 mb)

What I like about this product is the licensing, currently the Professional version is free for up to 20 users and 20 build configurations, and when the project get larger you purchase an enterprise license, currently £1,560, ideal way of working and getting you to use the produce.

It’s worth watching this short introduction video from Dime Cast, to find out how easy it is to create and use Team City.

The Server automates the integration process by monitoring the team’s source control repository directly. Every time a developer commits a new set of modifications, the server will automatically launch an integration build to validate the changes. When the build is complete, the server notifies the developer whether the changes that they committed integrated successfully or not.

Effectively, integration becomes as easy as checking in code. Using an automated integration server not only makes integration easy, it also guarantees that an integration build will happen. There is no danger of developers forgetting to validate their changes after checking in.

The Team City offers several key features:

• Integration with a variety of Source Control systems

• Integration with other external tools, such as NAnt and Visual Studio

• Can build multiple projects on one server

• Remote management and reporting

So down to the installation, once downloaded (231mb file), I installed using all the default settings and it worked, very easy to setup.

Next was to setup a project, again quite easy, I was attaching to a Visual SourceSafe (VSS) solution, I ran in to my first issue as I was getting directory authentication failed. This was because the running “TeamCity Web Server” was using my System login and it did not have permissions to the network drive for the VSS. A quick change to the TeamCity Build Server service login and it all worked.

I ran the build on a simple project and it worked without a hitch.

Next was to install the System Tray Notifier, quite simple, going to the “My Settings & Tools” tab brings a list of download, after a simple download and run, then pointing the HTTP location of the Team City installation, it worked.

What can I say it’s all just too easy to setup.

Pre-tested Commit

One issue I had, but not with Team City, is that if you are using MS Test, the built in Visual Studio 2008 testing tool, then you will need to run MS Test. But to run MS Test it is tightly coupled with Visual Studio. Which means to run MS test you’ll need to install a version of Visual Studio on your Continous Intergration machine. Therefore it might be worth using NUnit for testing.

One very nice feature of Team City is the ability to perform a Pre-tested Commit, this is possible when you install the Visual Studio Addin

An approach which prevents committing defective code into a build, so the entire team’s process is not affected.

Submitted code changes first go through testing. If it passes all of the tests, TeamCity can automatically submit the changes to version control. From there, it will automatically be integrated into the next build. If any test fails, the code is not committed, and the submitting developer is notified.

The TeamCity plugins for IntelliJ IDEA, Microsoft Visual Studio 2005/2008 and Eclipse extend the respective IDE with the remote run and pre-tested commit features. Developers test their changes by performing a Remote Run. A pre-tested commit is enabled when commit changes if successful option is selected.