Code Coverage vs Test Coverage

When organizations mandate “code coverage” numbers, like 95%, they’re barking up the wrong tree. Just because a line of code has been touched during testing, doesn’t mean it’s been tested; you’re only fooling yourself. Every time I tell someone the last product I created had 93% test coverage, they say “wow, that’s great!”. Then my follow up statement is “Yeah, it sounds great, but bugs still crept into covered areas”.

My weapon of choice in the Ruby world is simplecov. It’s dead simple (I guess that’s why it’s part of the name) and does exactly what it advertises. You can see very clearly how much of your code is exercised when your tests are finished executing, and with that a simple percentage of exercised vs not reached. That number is where people lose sight of the spirit of testing.
The goal of testing is to validate expectations while a given function executes under different conditions. A problem with code coverage is a single function can light up green from a single test without testing all the conditions that can occur. Exceptions and error handling were the number one culprit for people feeling good about their coverage numbers without adding a lot of value from their tests. What if that Integer was actually a String? Or nil and throws “no implicit conversion from nil to integer“? How about if the service returned an error instead of a value?  Or doesn’t return at all?Net::HTTPServiceUnavailable? What type of errors can it return?  All of these things need to be considered, especially in todays distributed computing climate.
Simply having code coverage does not mean you have test coverage. Code coverage numbers are a false security blanket and too many people value a higher number over a qualitative number like permutations of execution.
Long story short, do not confuse code coverage with test coverage. Code coverage is a single number that says “N percentage of my code is executed during test execution”. Test coverage says “N permutations of a function are exercised during execution”. The later is far more important in my book.

Leave a Reply

Your email address will not be published. Required fields are marked *