The past few days I've been working with the code coverage extensions to SUnitToo in VW (SUnitToo(verage) and SUnitToo(lsoverage). It is an impressive tool. And the metrics accurately reflected the way I developed some packages. Our in-house issue tracker and patch manager had tests added after the fact, with a resulting code coverage of 30%. The Report4PDF package, the PDF4Smalltalk based report writer I'm working on, was developed from the start with SUnit tests, with a code coverage of 65%.
At first I thought that 65% seemed a bit low, since all the code has associated tests. But the code coverage is smart: it does not just measure method hits, but which paths within a method are exercised. In many cases these were exception and error message branches; the paths less taken. But some cases should have been tested. These were very narrow special cases that I had simply not thought of. The code looks OK, and I'm confident that it will work, but I won't release it without an SUnit test.
If feels like I've been given a flashlight to see the dark nooks and crannies of my code.
I also like the idea of using the path count (the denominator in the code coverage number) as an application complexity metric. It could work well when used with number of classes, number of methods, and average method size. I wonder if 'number of paths per method' would be useful.
It is an impressive tool. Thanks to all who made it available.
Simple things should be simple. Complex things should be possible.
The "number of paths per method" metric that you refer to sounds similar to how many developers think of the Cyclomatic Complexity metric. I've seen multiple ways that such a metric is calculated in the various Smalltalk dialects and in Java tools. Sometimes they just count the number of return points or the number of branch conditions. It definitely can be helpful to highlight potential problem areas of the code.
ReplyDelete