Andy manages programmers for Follett Library Resources in McHenry, IL. In his spare time, he works on his CPAN modules and does technical writing and editing. Andy is also the maintainer of Test::Harness and can be contacted at [email protected].
Coverage testing lets you automatically find out what parts of your code are covered by tests or by documentation. It's an extension of automated testing that I've written about before. For someone who's releasing code to CPAN, documentation coverage is important for making sure that you have documented everything.
Code coverage lets you make sure that your test suite actually exercises all the options and paths through which the code can travel. I used it to find some functions that had never been tested because they were never actually used anywhere, much less in the test suite.
Starting with Documentation Coverage
The easiest way to think about coverage is by looking at documentation coverage. A simple rule to follow for your code is: "Every subroutine must have a block of POD that describes it." Subroutines that are documented are said to be covered, and those that aren't are uncovered or naked.
The following program, pcover, matches up POD sections to subroutines. It's pretty simple and works well for very regular, nontricky code in a single file.
#!/usr/bin/perl -w
use strict;
my %subs;
my %docs;
while ( <> ) {
chomp;
# Assume a =head1, =head2 or =item is the start
# of some documentation.
if ( /^=(head[12]|item [\d*]+)\s+([^({]+)/ ) {
my $item = $2;
$item =~ /([a-zA-Z0-9_]+)$/ or next;
$docs{ $1 } = $.;
next;
}
# Find subroutine declarations and stash the line
# number where it appears.
if ( /^\s*sub\s+([a-zA-Z0-9_]+)/ ) {
my $sub = $1;
$subs{ $sub } = $.;
next;
}
} # while
my $nerr = 0;
for my $sub ( sort keys %subs ) {
if ( !$docs{ $sub } ) {
print "The following subroutines have no docs:\n"
if ++$nerr == 1;
print "$sub, line $subs{$sub}\n";
} # if
} # for
printf "%d sub%s found without docs.\n", $nerr, $nerr == 1 ? "" : "s";
Now, when I run pcover against WWW::Mechanize,
$ pcover 'perldoc -l WWW::Mechanize'
or
$ pcover /usr/local/lib/perl5/site_perl/5.8.1/WWW/Mechanize.pm
I get the following:
The following subroutines have no docs: _die, line 1397 _pop_page_stack, line 1350 _warn, line 1390 die, line 1380 res, line 808 warn, line 1370 6 subs found without docs.
(Note: If you haven't seen perldoc -l Module::Name before, start using it now. It prints out the full path of any module, so long as it contains some POD. It will save you much typing and hunting for module files.)
So I have six functions that aren't documented, at least according to my simple program. The first three, with the underscores, are for internal use only, so I don't mind. I have docs for res(), but the heuristic didn't find it. That leaves me only two that I actually need to document.
Aside from the problems described earlier, there's a big, unfixable problem with pcover. The heuristic for finding subroutine names relies on a very rudimentary parsing of Perl and doesn't handle the possibility that documentation for a package might be in a different module. Still, for simple coverage checking of simple modules, pcover may be all you need, and without having to install any new modules.
Pod::Coverage to the Rescue
For flexibility and accuracy, we turn to Pod::Coverage, which takes a different approach to handling the code. As Tom Christiansen said, "Nothing but Perl can parse Perl," so instead of trying to parse the contents of a source file itself, Pod::Coverage loads up the module into Perl, then takes a peek at the Perl internals. The documentation is still found with a simple set of heuristics, but they're far more flexible than pcover's.
Pod::Coverage was cowritten and is maintained by the wily Richard Clamp, most notable for his File::Find::Rule. Like File::Find::Rule, Pod::Coverage is clever and flexible, with three different ways to use the module. The easiest is right from the command line:
$ perl -MPod::Coverage=WWW::Mechanize -e1
The -M flag tells the Perl executable to load Pod::Coverage as a module, passing MARC::Record as a parameter. The -e1 is just a dummy program that does nothing, since Pod::Coverage does all its magic at load time.
When I run that command line, I get output similar to pcover:
WWW::Mechanize has a Pod::Coverage rating of 0.948717948717949 The following are uncovered: die, warn
The coverage rating of 0.948... means that 95 percent of my subroutines are documented. Note that Pod::Coverage took care of the problems that pcover didn't handle: It ignored functions prepended with underscores, and it found the documentation for res(). As an author, I'm glad that Pod::Coverage found these because I realized that their differences from the core die and warn need to be noted.
Note that when we invoke Pod::Coverage this way, it expects that the module has already been installed on your system, and you can't pass a filename. If you've got a module you're working on, but haven't installed it on your system, you'll need to use a different approach. This command-line version also only works with a single module, not multiple modules in a distribution. For this extra flexibility, the pod_cover script, installed for you by Pod::Coverage as of Version 0.13, is a good place to start.
pod_cover assumes that you have a lib/ directory that contains the modules you want to check, and checks them all for you. When I run this on my working directory for the WWW-Mechanize distribution, I find that at least the other module is covered:
Pod coverage analysis v1.00 (C) by Tels 2001. Using Pod::Coverage v0.13 Sun Dec 28 22:14:52 2003 Starting analysis: WWW::Mechanize has a doc coverage of 94.87%. Uncovered routines are: die warn WWW::Mechanize::Link has a doc coverage of 100%. Summary: sub routines total : 47 sub routines covered : 45 sub routines uncovered: 2 total coverage : 95.74%
This is very convenient for a distribution with many modules. Just imagine having to maintain Dave Rolsky's DateTime-TimeZone!
Modifying the Rules
If the default settings for pod_cover don't fit your needs, you can quickly whip up a customized script. As part of the Phalanx project (http://qa.perl.org/phalanx/), I'm checking coverage of Test::Reporter, which has a few quirks. The documentation for it is kept in a .pod file, and there are a number of constant functions that need not be documented. Fortunately, the constant functions are all named with all capital letters, so it's easy to write a regular expression to match them. This little script gives me the accurate coverage results:
use Pod::Coverage;
use lib 'lib';
my $pc =
Pod::Coverage->new(
package => 'Test::Reporter',
also_private => [ qr/^[A-Z_]+$/ ],
pod_from => 'lib/Test/Reporter.pod',
);
print "Coverage = ", $pc->coverage, "\n";
print "Uncovered: ", join( ", ", $pc->uncovered ), "\n";
pod_from tells the constructor where to find the POD for the module, and also_private is a reference to an array of regular expressions that match subroutines that don't need to be documented. I only needed one regex to match the functions in question, but I could have passed as many as necessary.
Devel::Cover
Documentation coverage is helpful to ensure that no undocumented subroutines slip through the cracks. Code coverage is the measure of whether or not your tests are exercising all the parts of the code that they should.
Paul Johnson's Devel::Cover works by installing itself as a debugger hook, running your code, and then saving its metrics to a special database directory, called cover_db by default. Then, when you run the cover program, you'll be given two sets of output: a plain text summary of the coverage stats for each file and a set of HTML files that give in-depth, line-by-line code coverage analysis of your program. Plus, as an added bonus, if you have Pod::Coverage installed, Devel::Cover does documentation coverage analysis and includes it with the code coverage. Such a deal!
Before running your code coverage analysis, it's important to understand the four different types of code coverage that Devel::Cover tracks: subroutine, statement, branch, and condition. To illustrate, let's look at a simple piece of code and the tests for it.
Say I have a module, My::Math, with a subroutine, my_sqrt, that returns the square root of its parameter, but checks that it's not a negative number or undefined. If it is, it warns the user and returns 0:
1 =head2 my_sqrt( $n )
2
3 Returns the square root of I<$n>. If I<$n> is not defined,
4 or is negative, return 0.
5
6 =cut
7
8 sub my_sqrt {
9 my $n = shift;
10
11 if ( !defined($n) || ($n < 0) ) {
12 warn "my_sqrt() got an invalid value\n";
13 $n = 0;
14 }
15
16 return sqrt($n);
17 }
Then, somewhere in my test suite, I have a t/sqrt.t that tests that the module works as advertised:
use Test::More tests=>2; is( my_sqrt( 25 ), 5 ); is( int(my_sqrt(2) * 1000), 1414 );
Unfortunately, these two tests don't exercise as much of the code as they should. When I run the t/sqrt.t under Devel::Cover, the coverage percentages are pretty low:
File stmt branch cond sub -------------------------------- ----- ------ ------ ------ sqrt.pl 60.00 50.00 33.33 100.00
Let's look at each of these different coverage categories.
Subroutine coverage is simple. There's only one subroutine and the tests executed it at least once, so my subroutine coverage is 100 percent.
Statement coverage is the measure of how many of the statements in the code have been run at least once. There are five statements in the example: lines 9, 11, 12, 13, and 16.
9 my $n = shift;
11 if ( !defined($n) || ($n < 0) ) {
12 warn "my_sqrt() got an invalid value\n";
13 $n = 0;
16 return sqrt($n);
Since I never pass an undef or negative to the subroutine, lines 12 and 13 never get executed in the test. Therefore, the statement coverage is only 3/5, or 60 percent.
Branch coverage measures whether each possible branch has been taken. Any if statement has exactly two possible branches, regardless of how complex the expression being evaluated is. My subroutine only has one branch, and only one of its possible branches gets taken. Therefore, my branch coverage is 1/2, or 50 percent.
Conditional coverage is related to branch coverage, but looks inside the contents of the conditionals for possible combinations of values. This is usually represented as a truth table and takes into account short-circuit Boolean evaluation.
For the expression:
!defined($n) || ($n < 0)
there are three combinations of values:
!defined($n) $n < 0
------------ -------
0 0
0 1
1 1
Since my tests only pass positive, defined numbers, the last row in the truth table is the only one that's been exercised. Therefore, my conditional coverage is only 33 percent.
Improving My Coverage
Now that we understand the different types of coverage, how can I improve them? I'll start with passing undef:
is( my_sqrt(undef), 0 );
which improves my results:
File stmt branch cond sub ------------------------------- ----- ------ ----- ------ sqrt.pl 100.00 100.00 66.67 100.00
The main branch does get taken, which means that all the statements have now been executed, bringing statement and branch coverage up to 100 percent. However, I've only taken two of the three conditions in the truth table. To get to 100 percent conditional, I need to add a test for the negative number:
is( my_sqrt(-1), 0 );
Now, I have 100 percent coverage of all statements, all branches and all conditions:
File stmt branch cond sub ------------------------------ ----- ------ ------ ----- sqrt.pl 100.00 100.00 100.00 100.00
Devel::Cover in the Real World
Of course, when you use Devel::Cover on your code, the results won't be quite so clear. One thing that will help is to make a script to automate the running of your code and generating the resulting HTML pages. I call mine gocover, and it works on any module because everything is relative to the current directory:
cover -delete HARNESS_PERL_SWITCHES=-MDevel::Cover make test cover open ./cover_db/coverage.html
Note that this script is specifically for coverage testing on a standard module that uses make test to run its test suite. See the Devel::Cover documentation for examples of how to run other sorts of tests. The last line with the open command opens the main HTML file in my browser under Mac OS X. Adjust accordingly for your operating system.
When you run gocover, you'll notice things being significantly slower, taking about five to eight times as long to run because Devel::Cover is watching your program as it runs and collecting information along the way. Fortunately, the cover command runs quickly, generating a set of HTML files.
The first report file, coverage.html, is your thumbnail sketch of the files in your module, with coverage statistics for each of them. You'll notice a number of hyperlinks in the summary: Each filename and each percentage for branch, conditional, and subroutine coverage. Each of these links jumps to a specific subreport. All the reports are color coded—with green meaning something good and red meaning something bad—making it easy to skim the report for hotspots in your code.
The File Coverage page gives a full listing of a given file with metrics for each statement or subroutine as appropriate. The stmt column shows the number of times that the statement was executed, and the sub column shows how many times the subroutine was called. The branch and cond give percentages of coverage for the statement. The time column tells the number of milliseconds spent on each statement, which can help provide rudimentary profiling information.
Each of the Branch, Conditional, and Subroutine reports gives a summary of each branch, conditional, or subroutine, and summarizes the coverage for each. It's the same information as on the full File Coverage report, but condensed for easy skimming. You can also get to each of these summary reports from hyperlinks on the File Coverage page. Paul has made navigation between these pages very easy and helpful.
Improving your Coverage
So you've got a report on your module and you find that your coverage is weak. What do you do? First, remember that Devel::Cover is a tool, not an arbiter of correctness. It's up to you as the author to make the important decisions. That being said, it's a good idea to have all of your code exercised by your tests to make sure that everything works as you expect.
When I first ran coverage reports on WWW::Mechanize, I was disappointed to see how low my coverage numbers were. I'd been proud of WWW::Mechanize's testing suite, thinking it was fairly comprehensive, but Devel::Cover pointed out all the places I hadn't exercised. Most of the deficiencies fell into three areas.
The first, and the easiest to correct, was documentation coverage. I'd added a number of methods that I hadn't bothered to document, so fixing that was easy. I also found some methods that I considered as internal to my module, but that weren't prepended with underscores, so I renamed them.
Another source of the dreaded red boxes in my coverage reports were under-exercised default values. In a function that accepts default values, like a hypothetical text-writing function:
sub draw_string {
my $str = shift;
my $color = shift || "black";
...
}
the second statement has two different conditions that need to be tested: when a value for $color is passed in and when it isn't. I found that in all my tests, I'd been taking the default values, but had never exercised the code that overrides them. A few extra tests took care of covering them.
Finally, I never tested my warning conditions to see if they actually generated warnings. For example, Mech's agent_alias method lets you set your User-Agent string with an easy-to-read string like "Netscape 6," but if the alias isn't recognized, agent_alias emits a warning. Adding tests for this was simple with Test::Warn:
use Test::Warn;
my $m = WWW::Mechanize->new;
isa_ok( $m, 'WWW::Mechanize' );
warning_is {
$m->agent_alias( "Blongo" );
} 'Unknown agent alias "Blongo"', "Unknown alias squawks";
For checking behaviors where your code should die or throw exceptions, check out the Test::Exception module. Both should be in any module author's testing arsenal.
Pod::Coverage and Devel::Cover are both handy tools to help maintain your code, whether it's a module for distribution or just a simple web script. Even if you don't make your code run 100 percent under the tools, take the time to try it once. I guarantee you'll find at least one surprising thing in your code.
TPJ
