Coverage isn’t everything

September 30, 2008 at 11:09 am 3 comments

A few posts ago, I mentioned Emma and code coverage, and how it could be useful. Code coverage, in layman’s terms, is the lines of code covered by your tests. So generally, you want a high code coverage, implying that most of your code is well tested.

But since my post, I have talked with a few friends and colleagues, and one point has been brought up again and again. First, it was a discussion with my friend on his homework assignment. It was a hypothetical situation about a manager wanting 100 % code coverage before he launches his product. That discussion quickly devolved into a rant about why code coverage isn’t the alpha and the omega.

And then, just one or two days ago, I had another discussion with a colleague, who was telling / reminding me of the fact that a project having great code coverage doesn’t mean that they don’t need to test it anymore. I politely agreed and nodded my head, and both of us ended up repeating the same facts to each other. Which was that code coverage, like a lot of other things, can be faked.

The trick with code coverage is that a high code coverage number doesn’t actually tell you anything in and by itself. For it is just as easy to get a high code coverage with a single line of test which exercises the entire system as it is to do it the hard way and write a lot of small unit tests. A single unit test which executes main will most likely result in a high code coverage number, even without having any assertions.

Similarly, consider the following example :

void someMethod(int a, String b) {

if (a < someThreshold || isValid(b)) {
// Do first thing
} else if (a == someThreshold && isNumber(b)) {
// DO second thing
}
}

Now, it is possible to to write just two unit tests, and still attain 100% coverage. Here, if I call this test with a < someThreshold and a == someThreshold, I get 100% coverage. That does not mean though, that I have covered all possible cases. I mean, in my example, I didn’t even touch b.

And these are exactly the type of things which can demean the value of code coverage. Any manager worth his salt should understand this aspect of code coverage. It is possible and not too difficult to attain a very high code coverage even with a sub par quality product. The trick is in understanding how the tests are written, and how comprehensive they are.

Code coverage is a great tool to find out spots which are completely untested, and projects which develop their code in a Test Driven fashion often end up with high code coverage. But that does not imply that projects with a high code coverage are of the greatest quality. Careful consideration of their testing practices and comprehensiveness is essential in these cases. Track stuff like bugs, amount of test code written compared to amount of production code, etc to get a overall picture, rather than relying on just one metric.

Advertisements

Entry filed under: code, code coverage, emma, java, testing. Tags: , , .

My code’s untestable !! Two heads are better than one

3 Comments Add your own

  • 1. Shaft  |  September 30, 2008 at 3:00 pm

    I have to agree and add that in testing, there are very few absolutes. So while 100% coverage does not ensure a bug free program it does show there is an effort to mitigate easy to spot errors.

    Reply
  • 2. shyamseshadri  |  October 1, 2008 at 10:01 am

    Definitely. Increasing coverage is a great trend, especially with increasing amount of test code. Its when the coverage increases with no increase in test or just by modifying a few lines of test is when you should watch out. But as usual, any coverage is always better than no coverage.

    Reply
  • 3. Kunal Thakar  |  October 7, 2008 at 9:16 am

    In my program analysis class, we learned about the different types of code coverages. Amongst them, path coverage is the strongest, followed by branch coverage followed by statement coverage followed by function coverage. Emma and other such tools provide statement coverage which is obviously not sufficient.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


OmniscientOne

Blog Stats

  • 5,590 hits

Feeds

Pages

September 2008
M T W T F S S
« Aug   Jun »
1234567
891011121314
15161718192021
22232425262728
2930  

%d bloggers like this: