Andy manages programmers for Follett Library Resources in McHenry, IL. In his spare time, he works on his CPAN modules and does technical writing and editing. Andy is also the maintainer of Test::Harness, and can be contacted at [email protected].
Automated testing is a big part of Extreme Programming, but how do you do it in Perl? Actually, you've been using automated testing all along—when you install a new version of Perl or install a module. All the tools are there, in use, but you may have never noticed them before when you ran make test.
If you've installed a module from CPAN, and run make test, you've seen the testing modules at work, as in this snippet from the installation of WWW::Mechanize:
$ make test
perl "-MExtUtils::Command::MM" "-e" "test_harness...."
t/00.load.............ok
t/add_header..........ok
... 32 omitted lines ...
t/warn................ok
All tests successful.
Files=35, Tests=413, 46 wallclock secs
(36.10 cusr + 2.55 csys = 38.65 CPU)
After a few moments of flashing counters, the battery of tests is completed and you know that everything has run correctly on your specific machine. You can install the module with confidence.
Automated tests aren't just for module distributions. You can use Perl's tools to automate testing of your code even if it's not going to be uploaded to CPAN or even made into module. Think of each test that runs as both guardian angel and tireless assistant, doing the drudgery of checking your code every time you run the test suite, safeguarding against future breakage. My department has a battery of 11,000 tests that run once an hour, making sure all parts of our web site, from component to web interface, work and haven't been inadvertently broken. Testing can even help you write better code by forcing you to think early in the process about your API.
The Perl Testing Tools
There are two parts to the Perl testing tools: Test::Harness parses output created by Test::Simple in a simple format that's easy for humans to read. Each test program is a small Perl program with, by convention, a ".t" suffix for a filename. The test program prints out a plan that tells how many tests it plans to run, and then a series of test lines. Each test line starts with "ok" or "not ok," followed by a test number and an optional comment. For example, a program, prime.t, that tests a function that checks for prime numbers might create the following output:
$ perl prime.t 1..4 ok 1 - 2 is prime ok 2 - 3 is prime not ok 3 - 4 is not prime # Failed test (prime.t at line 9) ok 4 - 7 is prime # Looks like you failed 1 tests of 4.
We can see that the third test failed, and the comment tells us what was being tested, so we can easily find it in the source. However, this output is usually handled by the Test::Harness::runtests(), which runs our ".t" files, analyzes the output, and reports a summary.
$ perl -MTest::Harness -e'runtests(@ARGV)' prime.t
test....FAILED test 3
Failed 1/4 tests, 75.00% okay
Failed Test Stat Wstat Total Fail Failed List of Failed
———————————————————————————————————————-
prime.t 4 1 25.00% 3
Failed 1/1 test scripts, 0.00% okay. 1/4 subtests failed, 75.00% okay.
When we fix our tests in prime.t, and it runs clean as:
$ perl prime.t 1..4 ok 1 - 2 is prime ok 2 - 3 is prime ok 3 - 4 is not prime ok 4 - 7 is prime
then Test::Harness gives us the thumbs-up and gives timing statistics on what was run:
$ perl -MTest::Harness -e'runtests(@ARGV)' prime.t prime....ok All tests successful. Files=1, Tests=4, 0 wallclock secs ( 0.01 cusr + 0.00 csys = 0.01 CPU)
Writing Your First .t File
We've seen the test results, but what got tested and how? Our test.t is a regular Perl program, but with a ".t" extension. It exercises the code that we want to test, and uses Test::Simple's ok() function to do the test tracking, leaving the programmer free to think about what needs to be tested. Every test comes down to a pass/fail response: Did my code do what I expected it to do? The result of that question is a Boolean, and is passed to ok().
Let's take my is_prime() function, which should return True if the value passed is prime. Three is a prime number, so is_prime(3) should return True. If it does, our test passes. Here's a simple "prime.t":
#/usr/bin/perl use Test::Simple tests=>1; use Primes; # The module we're testing ok( is_prime(3) );
The Test::Simple line sets up the testing framework, based on the number of tests we specified, and makes the ok() function available for your use. Then, the ok() call means we're saying, "The test passes if is_prime(3) returns True." Assuming that my code is working correctly and the is_prime() call does return a True value, ok() takes the value and outputs
ok 1
which is what tells Test::Harness that the test actually passed. This is assuming that our call actually returns True. Conversely, is_prime(10) should return a False value, so we negate the return from it and pass that to ok(). We'll add a comment as well.
ok( !is_prime(10), "Ten is not prime" );
which prints
ok 2 - Ten is not prime
Test::Harness ignores the comment after "ok 2", but it helps the human to find failing tests. Get in the habit of giving each test a comment. It's not much fun to have to count tests in the course code because you know only that test 57 of 72 failed.
Note that ok() is keeping track of our test numbering for us. You may see old tests in the Perl core where the test program keeps track of its own test count, but Test::Simple frees you from that drudgery.
Any expression can be passed to ok() and will be validated in a Boolean context.
ok( $hostname eq "chimpy", "Valid hostname" ); ok( $parent =~ /^A(m|nd)y$/, "Correct parent name" ); ok( $statecount == 50, "Got all states, but not DC" );
The Importance of the Plan
Note how at the top of our test programs, we tell Test::Simple the number of tests to expect, which we call the plan. This lets Test::Harness validate that all tests have actually been run. If our test program exited after only running 10 tests, but we'd told Test::Simple to expect 15, then Test::Harness would know something was amiss and fail the test. In a way, validating the plan is itself a test, although the harness doesn't include it in the test count.
Sometimes you don't know how many tests you'll be running at compile time, when the use is executed. You can calculate the number of tests and then call the plan function directly, as long as it's before any ok() calls have been made. Say we want to check that all the HTML files in our directory only have lowercase letters with no digits or punctuation. The number of files might change over time, but not the rule.
use Test::Simple;
my @files = glob( "*.html" ); plan( tests => scalar @files ); for
my $filename ( @files ) {
ok( $filename =~ /^[a-z]+\.html$/, "$filename is lowercase" );
}
Finally, there may be times when it's just not possible to determine the number of tests before calling ok(). In this case, tell Test::Simple that you don't have a plan. Pass the string 'no_plan' in your use:
use Test::Simple 'no_plan';
Using prove
The prove utility comes with Test::Harness as of version 2.32, and lets you easily run tests on a set of files or one single file. It's basically a wrapper around the run_tests() function in Test::Harness, but it is designed to make it easy to get into a rhythm of test-code-test-code (more on this in the section entitled "Test-First Programming").
prove is run simply as
$ prove *.t digits...ok prime....ok All tests successful. Files=2, Tests=17, 0 wallclock secs ( 0.05 cusr + 0.03 csys = 0.08 CPU)
The -v option turns on the verbosity, so you can see each subtest, not just the summary, for a specific test:
$ prove -v prime.t prime....1..4 ok 1 - 2 is prime ok 2 - 3 is prime ok 3 - 4 is not prime ok 4 - 7 is prime ok All tests successful. Files=1, Tests=4, 0 wallclock secs ( 0.06 cusr + 0.01 csys = 0.07 CPU)
Any directories that prove sees will get expanded to mean all files in that directory, and the -r flag recurses into all the directories underneath that.
$ prove -r users/ users/admin/new....ok users/admin/privs..ok users/delete.......ok users/insert.......ok users/new..........ok All tests successful. Files=5, Tests=528, 8 wallclock secs ( 2.72 cusr + 1.63 csys = 4.35 CPU)
On Beyond ok()
With the knowledge of ok() to test your functions and prove to run the tests, you have all you need to create effective test suites. In fact, much of the Perl core uses functionality such as ok() as part of the tests that get run when you build Perl.
However, most of your tests will be doing the same sorts of things over and over, so you'll want to turn to Test::Simple's big brother, Test::More. These are wrapper functions around ok() that encapsulate common testing tasks and make debugging easier. Test::More is 100 percent compatible with Test::Simple, so you can start writing tests with Test::Simple and move to Test::More when you're comfortable.
Many times when you're testing, you'll want to see if a given value is what you expected. For instance, we may have a function allcaps() that should turn a string into all capital letters.
ok( allcaps( "Perl 5.8.1" ) eq "PERL 5.8.1" );
If the returned value is what we're expecting, the equality matches and the test passes. But what if it doesn't match? All we know is that the test failed, and on what line:
not ok 1 # Failed test (caps.t at line 3)
The is() function compares two values passed to it, and if they're not equal, shows the expected and actual results, making debugging easy.
not ok 1 # Failed test (caps.t at line 3) # got: 'PERL' # expected: 'PERL 5.8.1'
Now we can see that allcaps() seems to have dropped all nonalpha characters, and we know where to get started looking.
is() compares its values as strings, using the eq operator. If you want to use some other relational operator, especially if you want the values compared as numbers, you'll need to use the cmp_ok() function. The two following examples work the same, but just as with the is() function, cmp_ok() will show exactly what values caused the problem:
ok( $nfiles >= 3, "We have at least three files" ); cmp_ok( $nfiles, ">=", 3, "We have at least three files" );
is() has a counterpart, isnt(), which checks that one value is anything except for the second value, and like() has unlike() that works the same way with regexes. cmp_ok() doesn't need an opposite since each relational operator has its own opposite.
Throughout your test programs, you may find that the comments in your test calls aren't enough. The diag() function lets you embed diagnostics in your test output. Any strings passed to diag() are printed with a pound sign prepended, so that Test::Harness ignores them.
More Complex Checking
Test::More also has functions to help testing complex data structures. The isa_ok() checks that a given variable is defined and blessed into a specific package:
my $user = new My::User(); isa_ok( $user, 'My::User' );
If you do anything with objects, always remember to use isa_ok() throughout your tests—don't just use test constructors. Test anything that can return an object, like a factory method or a file-walking method that returns an object instance.
Hashes and arrays are also supported. Instead of writing code to walk arrays or hashes, use the is_deeply() function to make an element-by-element comparison of your structures. If the structures differ, is_deeply() reports on the first element that fails:
my @expected_stooges = qw( Larry Moe Curly Iggy ); my @test_stooges = get_stooges(); is_deeply( \@test_stooges, \@expected_stooges, 'stooge check' ); not ok 1 - stooge check # Failed test (stooges.t at line 9) # Structures begin differing at: # $got->[3] = 'Shemp' # $expected->[3] = 'Iggy'
Test-First Programming
You may find yourself saying, "All these tests look great, but how am I going to have enough time to write them?" Easy: Write the tests before you write the code. Test-first programming is a key principle of Extreme Programming, and even if you're not an XP fan, it's hard to dispute the value of test-first. A test defines your expectation of the code with simple and absolute clarity.
Say you're writing a function that will tell whether a number is prime. Before you even think about how you'll write the code, write some simple tests. Write tests for the first handful of integers:
use Test::More tests => 11; ok( is_prime( 2 ), "Two is" ); ok( is_prime( 3 ), "Three is" ); ok( !is_prime( 4 ), "Four is not" ); ok( is_prime( 5 ), "Five is" ); ok( !is_prime( 6 ), "Six is not" ); ok( is_prime( 2711 ), "Some big prime" );
Those are good tests for when someone passes in nice, well-behaved input, but what about weird cases? What about a negative number? Or a noninteger? Your function should handle all those as well, so write your test cases for them:
diag( "Check the weird cases" ); ok( !is_prime( -1 ), "Negatives are never prime" ); ok( !is_prime( 0 ), "Zero is not prime" ); ok( !is_prime( 1 ), "Neither is one" ); ok( !is_prime( 3.14159 ), "Fractions aren't" ); ok( !is_prime( "five" ), "Strings sure aren't" );
It doesn't matter what your code does, so long as you define it. Put it in the POD for the function and explicitly describe the behavior. (Maybe you want is_prime() to emit a warning: You can check that with Test::Warn, available on CPAN.)
Now that you've written tests for some common cases and a few edge cases, you can write your function. Start with the POD documentation for the function. The documentation explains the special cases that we came up with.
is_prime( $n ) returns True if <i>$n</i> is a prime number (is positive and has exactly two factors: 1 and itself). Nonintegers are never primes.
Finally, write the code:
sub is_prime {
my $n = shift;
return 0 if $n <= 0; # Negatives aren't prime
return 0 unless $n == int($n); # Non-integers aren't prime
my $nfactors = 0;
for my $i ( 1..$n ) {
++$nfactors if ($n % $i == 0);
}
return $nfactors == 2;
}
TODO & SKIP Blocks
Sometimes you'll write a set of tests, but not have the code completed, and you'll want to check your code back into CVS. You have the subroutine stub created, but it's time to go home for the day. You don't want to check-in code that fails a test, so you wrap the unfinished tests in a TODO block:
TODO: {
local $TODO = "Haven't written pi digitizer yet";
is( pi_digit(1), "3" );
is( pi_digit(2), "1" );
is( pi_digit(100), "7" );
}
The TODO block notifies Test::More that although the tests should be run, they should all fail. Any tests that succeed will be flagged as having unexpectedly succeeded. This will show as a warning at the end of your run of prove or make test, but the test run as a whole will still pass.
The counterpart to the TODO block is the SKIP block. SKIP blocks are for sections of code that might not run on all given machines, or under all given circumstances. You might have some code that shouldn't get run if a certain module isn't installed or if there's not an Internet connection. In these cases, we don't want to run the tests at all.
Say you have a test that uses the Test::HTML::Lint module to validate your HTML output and report on it. Unfortunately, not everyone using your module will have Test::HTML::Lint:
SKIP: {
eval "use Test::HTML::Lint";
skip "Test::HTML::Lint not installed", 2 if $@;
html_ok( $main_page, "Main page is valid" );
html_ok( $admin_page, "Admin page is valid" );
}
Here, if we're unable to use the module, the skip() function is called, which then jumps out of the SKIP block. The two html_ok() calls are never made. Note that we told skip() that there were two tests we were skipping; otherwise, our test numbering would be off and the plan would fail.
Wrapping Up
Automated testing has changed the way I think about programming. I'm confident in the new code I write, and refactoring is a joy, instead of nerve-wracking, because I'm no longer afraid of breaking existing code.
Keep an eye on the prove utility that comes with Test::Harness. It's a work in progress based on the past two years of testing work in my department. I welcome your comments and suggestions to make Perl's testing tools make your coding life easier.
TPJ
