Metrics and Code Coverage in Eclipse 3.3 Europa

This is actually something that I spent two days on trying to get to work properly. There were a couple of variables that made this harder than necessary: the documentation on getting the latest version of Photran to compile was a bit out-dated (we need the latest version of the CDT from HEAD for the tests to run) and the fact that we are switching to Eclipse 3.3 which might not work well with all the plug-ins out there.

The plan was to run some metrics and code coverage measurements on the Photran code base. Using the data that we gathered we could evaluate sections of code that needed refactoring and/or testing. We could only use free tools since we wanted the students to be able to run them on their own computers. Naturally, if we did not have this major restriction we would have chosen some better commercial tools.

The State of Flow Metrics was the first one I tried but it does not do a good job. The UI and visual feedback were so bad that I did not even know if it was working. So, I ran the tool on a smaller project; it completed the analysis. However the information it provides is not presented intuitively. Instead it adds a little icon on the left margin of the editor that does nothing more than indicate that it had analyzed that method. Then the data is presented in the Problems view along with all the other errors and warnings. For small projects, this is fine but for larger ones with 100s (or 1000s like the CDT) of warnings, you cannot even determine which are warnings from the metrics tool (vs warnings from the Java compiler). So this is pretty much useless for large projects or workspaces with 1000s of warnings. It has an export feature that is supposed to export the data to HTML and CSV forms but it refuses to complete for me.

The second metrics plug-in I tested was much better. It had failed to work the first time I used it because it was not able to access the data that it had collected from the analysis. However, after upgrading to the latest version of Eclipse 3.3 (Version: 3.3.1 Build id: M20070921-1145) this problem vanished.

Metrics view using the Metrics Eclipse plug-in

So, now we needed a code coverage tool or more specifically one that could analyze the tests that have been written for Photran. This is slightly more tricky. We need code coverage tools that can measure coverage on JUnit plug-in tests (vs the normal JUnit tests). JUnit plug-in tests must launch an instance of Eclipse to successfully execute to completion. So a majority of the test coverage tools mentioned here cannot do a decent job.

For instance, in Photran, if you just ran the normal JUnit tests there are only 125 such tests vs 1566 if you run the JUnit plug-in tests. So the coverage is going to be low for the former execution.

There was only one free tool that I could find that offered support for measuring JUnit plug-in tests: EclEmma. The first time I installed EclEmma it corrupted my entire workspace and I had to restore everything from the repository. I installed EclEmma again today on the latest version of Eclipse 3.3 and I think I know the proper way to invoke it so that it will not corrupt the workspace: always create two different run configurations and never mix them. So, when running the Photran, you should always create 2 configurations — one that EclEmma will run and one that normal Eclipse will execute without coverage. And the workspace should also always be cleaned after running EclEmma.

EclEmma code coverage in action.

I believe that the Eclipse update fixed something to make EclEmma work properly on Mac OS X. Yesterday, there was no clean separation of an EclEmma run-time from a normal run-time. In fact, both of them were always merged in the same dialog. This could be why the workspace was being corrupted since EclEmma's instrumentation was replacing the normal ones.

Bottom line, if you want to a good metrics and code coverage tool, go with the Metrics and EclEmma plug-ins. Of course, if you can afford it, go for a commercial tool like Clover.