brian has been a Perl user since 1994. He is founder of the first Perl Users Group, NY.pm, and Perl Mongers, the Perl advocacy organization. He has been teaching Perl through Stonehenge Consulting for the past five years, and has been a featured speaker at The Perl Conference, Perl University, YAPC, COMDEX, and Builder.com.
I document all of my modules and scripts, and although I readily find out about program errors because people are quick to tell me their script either fails to run or does the wrong thing, almost nobody reports problems with the documentation.
Perl has a basic document format called "Plain Old Documentation," or POD. The Perl documentation reader, perldoc, follows the adage, "Be strict in what you output, and liberal in what you accept." It takes most of the bad POD formatting that I give it and turns it into something the user can read. If I make errors in my POD markup, perldoc often silently fixes them. I have the same problem with HTML. I can write bad HTML that the browser fixes so it can display it.
Perl comes with a POD validator, podchecker, based on the Pod::Checker module, which is part of the Perl Standard Library. Given a file with POD directives, podchecker tells me where I messed up. I tell podchecker which file (not which module, like perldoc) to check and it shows me the POD errors and questionable constructs; see Listing 1.
I am lazy, in the Perly sense of the word, so I want this to happen automatically. As I add new features to modules or move code around, I change the documentation. I may add more documentation, get rid of old explanations, or rearrange it. Anywhere in this process I might introduce a POD error. I do not want to use podchecker every time I make a change.
I could check the documentation just before I release a new version, but I usually forget to do that. I have automated most of the steps to release something to CPAN, so I should automate checking the documentation format as well.
Test::Pod
I wrote the Test::Pod module to automatically check my modules and scripts for POD errors. It does not check if I documented everything that I should have, like Pod::Coverage does. It simply wraps Pod::Checker, which only checks the POD markup, in a Test::Builder interface and calls it pod_ok().
Once I have my pod_ok function, which I show in gory detail later, I can use it in a test script. This snippet uses Test::More, the basis of many new test files. Test::More comes with the latest stable version of Perl, 5.8.0, which also comes with Test::Tutorial, which explains the basics of testing.
# t/pod.t use Test::More tests => 1; use Test::Pod; pod_ok( 'blib/lib/ISBN.pm' );
The test fails if Pod::Checker finds a POD error in the ISBN.pm file from my Business::ISBN distribution.
Most of my distributions have more than one module in it, though. Andy Lester showed me a cool hack to test all of the modules without remembering which modules I had. If I add a new module to the distribution, this test finds it automatically.
# t/pod.t
BEGIN {
use File::Find::Rule;
@files = File::Find::Rule->file()->name(
'*.pm' )->in( 'blib/lib' );
}
use Test::More tests => scalar @files;
use Test::Pod;
foreach my $file ( @files )
{
pod_ok( $file );
}
The File::Find::Rule module allows me to find all of the modules in the named directory and the build library ('blib/lib' in this case) very easily. Once I have all the filenames that I want to test in @files, I simply loop through @files to test each one.
I run this test like any other test file. Once I create the module Makefile from Makefile.PL, I run make test, which tells Test::Harness to do its magic. It runs all of the *.t files in the t directory, collects the results, and reports what it finds; see Listing 2.
If a test fails, the Test::Harness report is different. In this case, I edited the ISBN.pm file to remove an =over POD directive so the POD now has an error. Test::Pod reports that it found an error at line 400. Pod::Checker correctly identifies the problem as a missing =over. Test::Harness prints a summary of the failed tests at the end; see Listing 3.
Creating the Test Module
Test::Builder is the brainchild of Michael Schwern and chromatic, and in my opinion, it's the best thing to happen to Perl in years. This module allows other people to plug into Test::Harness with their own specialized modules. The Comprehensive Perl Archive Network (CPAN) has about 20 specialized Test modules so far.
These specialized modules have the same advantages as any other module—I can write a test function once and use it over and over again. The function standardizes the way I do things and moves more code out of the test files. I take any chance I can get to move code out of the test files. More code means more points of failure. I have to try really hard to mess up pod_ok().
The Test::Builder interface is very simple. In my specialized test module, I create a Test::Builder object, which is a singleton, so all of the specialized test scripts play well together.
my $Test = Test::Builder->new();
I create a test function, called pod_ok in Test::Pod, that tells Test::Builder if the test succeeded or failed, and optionally outputs some error information. Test::Builder's ok() method handles the result. If I think the test passed, I give ok() a true value, and a false value otherwise. The meat of pod_ok() is very simple—tell Test::Builder the test either passed or failed.
sub pod_ok
{
$pod_ok = _check_pod();
if( $pod_ok ) { $Test->ok(1) }
else { $Test->ok(0) }
}
Everything else in the function supports those simple statements in the pod_ok function. Test::Builder takes care of the rest.
To actually test the POD, I created an internal function, _check_pod, in Test::Pod; see Listing 4. I already defined the constants NO_FILE, NO_POD, ERRORS, WARNINGS, and OK, which described the conditions that Pod::Checker can report. The function returns an anonymous hash. The result is the value of the result key, error messages are the value for the output key, the number of errors is the value of the errors key, and the number of warnings is the value for the warnings key. The rest of the code puts the right things in the hash.
The first line in the subroutine takes a filename off of the argument stack, and the third line checks the file's existence. If the file does not exist, the function returns an anonymous hash with the result NO_FILE.
The next bit of code ties the variable $hash{output} to IO::Scalar. Pod::Checker can write messages to a file handle, and I want to intercept that output. It shows up in $hash{output} instead of the terminal.
I get a POD checker object from Pod::Checker, and then parse the specified file. Pod::Checker puts all output in $hash{output}. The do block puts the numbers of errors and warnings in the right keys, then returns the right constant for the condition that Pod::Checker reported.
Finally, _check_pod returns the anonymous hash that the pod_ok function can use to tell Test::Builder what happened. The pod_ok() function (see Listing 5) does the same thing it did before, although it has to do a little more work to decide if the test passed or not.
The first line takes a file name off of the argument list, and then passes that to _check_pod. The second argument specifies my expected result and default to OK. The third argument gives a name to the test that I can see in the verbose output of Test::Harness. It defaults to a message that includes the name of the file.
The rest of the function is a series of if-elsif statements, with one test for each possible condition. I can specify an expected result in the second argument. If Pod::Checker finds the condition I expected, then the test succeeds, and fails otherwise. If I do not specify an expected condition, pod_ok assumes I only want the test to pass if Pod::Checker finds neither error nor warnings.
Every branch of the if-elsif structure calls Test::Builder's ok() method. If that branch represents a success, pod_ok passes 1 to ok(), and 0 otherwise.
If pod_ok fails a test, it also uses Test::Builder's diag() method to give an error message.
Conclusion
Testing my work is simple if I use Test::Builder. I can create specialized test modules to check all sorts of things other than normal script execution. My modules have better-formatted documentation because Test::Pod automatically tells me about problems.
TPJ
Listing 1
podchecker /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** WARNING: (section) in 'perl(1)' deprecated at line 4926 in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** WARNING: (section) in 'perlmod(1)' deprecated at line 4926 in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** WARNING: (section) in 'perlbook(1)' deprecated at line 4926 in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** ERROR: unresolved internal link 'bind_column' at line 1819 in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** WARNING: multiple occurence of link target 'trace' at line - in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm *** WARNING: multiple occurence of link target 'Statement (string, read-only)' at line - in file /usr/local/lib/perl5/site_perl/darwin/DBI.pm /usr/local/lib/perl5/site_perl/darwin/DBI.pm has 1 pod syntax error.
Listing 2
localhost_brian[3150]$ make test cp ISBN.pm blib/lib/Business/ISBN.pm cp Data.pm blib/lib/Business/ISBN/Data.pm PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/load.t t/pod.t t/isbn.t t/load....ok t/pod.....ok t/isbn....ok 20/21 Checking ISBNs... (this may take a bit) t/isbn....ok All tests successful. Files=3, Tests=25, 358 wallclock secs (116.90 cusr + 2.96 csys = 119.86 CPU)
Listing 3
localhost_brian[3152]$ make test
cp ISBN.pm blib/lib/Business/ISBN.pm
Skip blib/lib/Business/ISBN/Data.pm (unchanged)
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e"
"test_harness(0, 'blib/lib', 'blib/arch')" t/load.t t/pod.t t/isbn.t
t/load....ok
t/pod.....NOK 1# Failed test (t/pod.t at line 12)
# Pod had errors in [blib/lib/Business/ISBN.pm]
# *** ERROR: =item without previous =over at line 400 in file
blib/lib/Business/ISBN.pm
# blib/lib/Business/ISBN.pm has 1 pod syntax error.
t/pod.....ok 2/2# Looks like you failed 1 tests of 2.
t/pod.....dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 1
Failed 1/2 tests, 50.00% okay
t/isbn....ok 20/21
Checking ISBNs... (this may take a bit)
t/isbn....ok
Failed Test Stat Wstat Total Fail Failed List of Failed
----------------------------------------------------------------------------
t/pod.t 1 256 2 1 50.00% 1
Failed 1/3 test scripts, 66.67% okay. 1/25 subtests failed, 96.00% okay.
make: *** [test_dynamic] Error 35
Listing 4
sub _check_pod
{
my $file = shift;
return { result => NO_FILE } unless -e $file;
my %hash = ();
my $output;
$hash{output} = \$output;
my $checker = Pod::Checker->new();
# i pass it a tied filehandle because i need to fool
# Pod::Checker into thinking it is sending the errors
# somewhere so it will count them for me.
tie( *OUTPUT, 'IO::Scalar', $hash{output} );
$checker->parse_from_file( $file, \*OUTPUT);
$hash{ result } = do {
$hash{errors} = $checker->num_errors;
$hash{warnings} = $checker->can('num_warnings') ?
$checker->num_warnings : 0;
if( $hash{errors} == -1 ) { NO_POD }
elsif( $hash{errors} > 0 ) { ERRORS }
elsif( $hash{warnings} > 0 ) { WARNINGS }
else { OK }
};
return \%hash;
}
Listing 5
sub pod_ok
{
my $file = shift;
my $expected = shift || OK;
my $name = shift || "POD test for $file";
my $hash = _check_pod( $file );
my $status = $hash->{result};
if( defined $expected and $expected eq $status )
{
$Test->ok( 1, $name );
}
elsif( $status == NO_FILE )
{
$Test->ok( 0, $name );
$Test->diag( "Did not find [$file]" );
}
elsif( $status == OK )
{
$Test->ok( 1, $name );
}
elsif( $status == ERRORS )
{
$Test->ok( 0, $name );
$Test->diag( "Pod had errors in [$file]\n",
${$hash->{output}} );
}
elsif( $status == WARNINGS and $expected == ERRORS )
{
$Test->ok( 1, $name );
}
elsif( $status == WARNINGS )
{
$Test->ok( 0, $name );
$Test->diag( "Pod had warnings in [$file]\n",
${$hash->{output}} );
}
elsif( $status == NO_POD )
{
$Test->ok( 0, $name );
$Test->diag( "Found no pod in [$file]" );
}
else
{
$Test->ok( 0, $name );
$Test->diag( "Mysterious failure for [$file]" );
}
}
