Creating Module Distributions with Module::Install

The Perl Journal June 2003

By Brian Ingerson

Brian has been programming for 20 years, the last five of those in Perl. He lives in Portland, Oregon, and can be contacted at [email protected].

There is no doubt that Perl has the best code reuse infrastructure of all the open-source programming languages. This is due to the wonder known as the "Perl module." Almost every modern programming language worth a second look has a modular extension mechanism. But that's not what I'm getting at. The Perl module is different.

The reason has every bit as much to do with community and heroics as it does with technology and programming. Some very dedicated people have spent many years of their lives collectively nurturing the Perl module and, in the process, it has become something of a superstar of modern programming. The Perl module has a lot going for it:

• Good Documentation. Perl modules have an exceptionally high quality of documentation, and the ubiquitous POD format converts nicely into all sorts of other formats including UNIX manpages, LaTeX, and HTML.
• Object Orientation. Object-oriented interfaces are available for a large portion of modern modules.

• Consistency of Installation. 99.9 percent of all Perl modules install the same way: perl Makefile.PL && make install. This has made it possible to create sophisticated installation tools.

• Testability. Perl modules have a standard for running unit tests that allow them to be tested by users with a single make test.

• Huge Repository. CPAN has over 5500 modules. No other programming language can top that.

The Perl forefathers spent a lot of time, thought, and hard work developing good standards for Perl modules. Because of this, Perl was able to excel in this domain while other languages have struggled.

But the Perl module has been in a bit of a rut lately, due to weaknesses that have begun to appear in the module installation process, some of which I'll discuss in this article. Perl has had many accomplishments, but the next big development in Perl may hinge on this installation question. However, the Perl community is (justifiably) wary of large overhauls that may cause more turmoil than they are worth.

In this article, I'll describe a new module called Module::Install, coauthored by Autrijus Tang and myself. It looks at things from a whole new perspective and breaks down many common barriers to module authoring. It also honors the three guiding principles of Perl:

• There's More Than One Way To Do It (TMTOWTDI).
• Do What I Mean (DWIM).

• It Just Works!

There were other key ideas that affected the design of Module::Install:

1. Module::Install should reduce the effects of module dependencies. If an author of a module (let's call it module X) incorporates Module::Install into a module, Module::Install itself need not even be available on CPAN to the users of module X.

2. The average module user should never need to directly install Module::Install.

3. Even if Module::Install becomes widely used, a need for backward compatibility should never be a limitation.

4. Module::Install should get smarter over time. You should be able to teach it new tricks every time you use it.

MakeMaker

In the beginning, it was decided that the best way to build and manipulate Perl modules was with the UNIX "make" utility. This program is nearly ubiquitous and is a sharp tool for automating all sorts of processes, especially those having to do with building and configuring software. That means every Perl module (or more accurately, module distribution) needs a "makefile"—the file that contains all the build rules.

The problem is that you can't create a makefile that is portable enough to work on all systems. This has to do both with differences in the system command processors (shells) and differences in the various implementations of make itself. The forefathers got around this problem in a clever way: They wrote a Perl module called ExtUtils::MakeMaker that could generate a proper makefile on any system. All it needed was a set of simple directives from the module author.

The convention has always been to put these directives in a Perl program called Makefile.PL. This has the advantage that an author can write any Perl code they want in order to set the directives. Often, this involves inspecting the local system, or prompting the person installing the module. In its most basic form, a Makefile.PL looks like this:

use ExtUtils::MakeMaker;
WriteMakefile
    NAME => 'Acme::Pie',
    VERSION => '3.14';
When anybody runs the command:
perl Makefile.PL

ExtUtils::MakeMaker will generate a rather large makefile for their system. This Makefile is capable of all sorts of tasks relating to Perl module building. If you dare to study the MakeMaker sources, you'll find them to be an arcane, nonextensible behemoth of a code base. In short, enough spaghetti code to feed the Italian Army. The code basically works, but it ain't gonna get much better. This ugliness leaves us with the good and the bad.

The good things about ExtUtils::MakeMaker are:

• It can produce a makefile that will work on all Perl platforms from Windows to UNIX to VMS to Mac OS (the old one!).
• The makefile can do everything that a Perl module needs to do— run tests, build and link libraries, create documentation, and even generate a module distribution ready for CPAN.

• It has been shipped with Perl for a long time. That means a module author can expect it to be on virtually every installation of Perl that they would ship their module to.

The bad things about ExtUtils::MakeMaker are:

• It relies on make. Why would such a powerful language need such a crufty little old UNIX utility to do what amounts to little more than invoking other commands?
• A make utility isn't always available on all platforms by default. Win32 is a common problem spot as it requires "nmake" to be installed.

• It can't realistically be overhauled. There are literally millions of Perl installations in the world with all sorts of administrative policies. How would you get them all to upgrade? It's like rolling a snowball up a mountain.

• It doesn't do what it doesn't do. Authors of modules with a complex build process end up writing lots of extra custom code inside Makefile.PL to make (no pun intended) up for the missing features of ExtUtils::MakeMaker.

The Cure

So the Perl Module is kind of stuck in its legacy. Or is it?

Module::Install offers some fairly clever solutions to the problems listed above. Its modus operandi is "embrace and extend," but in a better, more Perlish sense of that phrase. Module::Install simply makes use of all the goodness offered by ExtUtils::MakeMaker, and adds any extras as needed. Let's look at it in action:

use inc::Module::Install;
name        ('Acme::Pie');
version     ('3.14');
check_nmake ();
&Makefile->write; 

This does the same thing as the previous example, but in a completely different way. Let's see how: The first thing you'll notice is that we didn't use Module::Install at all. We used inc::Module::Install—and therein lies the core of Module::Install's subtle trickery.

Authors and Users

There are three types of people in the world—well, the Perl world, anyway:

• Perl Module Authors.
• Perl Module Users.

• Perl Module Users who are also Authors.

Module::Install is different from most Perl modules because it is only installed by module authors. When the author runs perl Makefile.PL, the module called inc::Module::Install copies the module called Module::Install into the ./inc/ directory, if it doesn't already exist. Therefore, the new file has the path ./inc/Module/Install.pm.

Next, inc::Module::Install adds ./inc/ to @INC, Perl's module-path lookup variable. Finally inc::Module::Install does a require Module::Install. The module that gets loaded is the one that was just copied. And the Makefile.PL proceeds.

There are several (potentially dozens) of submodules that Module::Install might need to make use of. Whenever one of these modules is needed, it goes through the same process of copying to the local directory and then loading into memory. Understanding this is the first key to understanding Module::Install.

All of the modules that get copied into the local ./inc/ directory need to be added to the MANIFEST file so that they will be shipped to the user. This can be done with the make manifest command, and Module::Install will warn you if you try to do a make dist without doing this first.

When the user gets the module, all of Module::Install's parts will already be packaged right inside the module—and that's the whole point. When the user runs the Makefile.PL, it will load inc::Module::Install directly from the local directory. This happens because Perl always has "." in the @INC path, so no user ever has to install Module::Install directly.

If the user also happens to be an author with his own (and possibly incompatible) version of Module::Install, his copy of inc::Module::Install will load the local copy from ./inc/. This way, there can never be a conflict between different versions of Module::Install. The one that the module shipped with will always be the one used, so backward compatibility need not be maintained.

Features on Demand

Back to the example Makefile.PL. The next two lines are:

name        ('Acme::Pie');
version     ('3.14');

These do exactly what you would think. They set the NAME and VERSION parameters so that Module::Install can call ExtUtils::MakeMaker::WriteMakefile() internally for you. But don't assume they aren't special. They are.

name and version are not exported subroutines as you might think. They are simply two of myriad subroutines that exist in some unknown module. As long as that unknown module's name begins with Module::Install::, the subroutine will be found.

Module::Install uses Perl's AUTOLOAD facility to catch the unknown subroutines, locate them in some module, and make sure that this module is inside the local ./inc/ directory. Module::Install only adds the modules that you actually need for your Makefile.PL. Note that when a module gets bundled into ./inc/ it is "pod stripped." In other words, all the documentation is removed. Since the ./inc/ modules never get installed by the user, the pod is just a waste of space.

Next we have:

check_nmake ();

This only comes into play when a user is installing your module on an MSWin32 system. It will check to see if nmake is installed. If not, it will download it from the Microsoft web site. Of course, it will ask the user first, just to be polite. This solves the common problem of people not having nmake on Windows.

Finally we have:

&Makefile->write; 

Makefile is a subroutine that returns a Module::Install::Makefile object. This object holds all the info that is needed to create a Makefile by invoking ExtUtils:MakeMaker::WriteMakefile. The reason we don't just say write is because there are several modules that have write functions, as we'll see shortly. The name and version subroutines also happen to come from Module::Install::Makefile, so we could have written things like this:

use inc::Module::Install;
&Makefile->name    ('Acme::Pie');
&Makefile->version ('3.14');
&Makefile->write; 

Reducing Dependencies

Module::Install works independently. My recent CPAN module, only.pm, has been on CPAN for over two months, and it uses Module::Install. But as I am writing these words, Module::Install hasn't even been released. It simply doesn't need to be because all the parts of Module::Install that only.pm needs are shipped along with only.pm.

In fact, Autrijus and I have started moving all of our modules over to Module::Install. We're testing our own creation. The whole Module::Install idea stems from a module that I released last year called CPAN::MakeMaker. Autrijus figured out how to accomplish all my goals in a simpler, more extendable way. When Module::Install is released, it will completely replace the need for CPAN::MakeMaker, which I will likely remove from CPAN.

Module::Install should be released on CPAN by the time you read this. If not, keep an eye out for it.

Bundles of Joy

Module:Install has another keen feature: bundling. You can actually convince Module::Install to pull in entire modules that it needs for building things. A great example is Michael Schwern's Test::More. It is great for writing tests, but some people don't use it because it doesn't exist by default on older versions of Perl. It's a tradeoff of whether or not to burden the users for the sake of writing better tests that, in the end, the users probably don't care about.

Module::Install makes it a nonissue. It can simply pull in the author's version of Test::More right into the author's module. Here's an example:

include_deps    ('Test::More', 5.004);
build_requires  ('Test::More');

The first line instructs Module::Install to bundle all the parts of your local Test::More into ./inc/, as long as they support Perl 5.004 and higher. If they don't support 5.004, no bundling will take place.

The second line simply says that Test::More is required for building this module. By the way, the bundled Test::More modules are POD-stripped so the extra weight is minimal.

What About Module::Build?

Anybody who has been following the Perl module authors' scene in the past couple years should know about the Module::Build project. This is actually Ken Williams's attempt to do the impossible: replace MakeMaker. And he's doing a darn good job of it, too.

The main goal of Module::Build is to do everything important that MakeMaker does, but without using make. Ken has also done a fabulous job of keeping the code object oriented and clean as a whistle. That makes it easy for others to contribute to the project.

One of the hurdles Ken faces is getting people to use his module. Like Test::More, authors will have to make a choice between using the ubiquitous MakeMaker or the riskier new upstart, Module::Build.

Module::Install will be a great ally for Ken. That's because Module::Install can use the same Makefile.PL that was created for MakeMaker to use Module::Build instead. Here's how to convert our Makefile.PL to use Module::Build:

use inc::Module::Install;
name        ('Acme::Pie');
version     ('3.14');
check_nmake ();
&Build->write; 

All I did was change Makefile to Build. Everything else remains the same! If I really wanted to cover my bases, I could ship with both methods in place like this:

use inc::Module::Install;
name        ('Acme::Pie');
version     ('3.14');
check_nmake ();
&Makefile->write; 
&Build->write; 

If we also bundle in the Module::Build software, then there is almost no risk in using Module::Build. And then one day when Module::Build has replaced MakeMaker everywhere, authors can start dropping MakeMaker support.

No Big Thing

You may think that it's not a wise thing to add extra weight to all the CPAN modules. I personally wouldn't lose too much sleep over it. In the general case, you're only adding a few kilobytes once everything is gzip compressed. And much of this added weight is directly beneficial to your module.

Of course, it is possible to overdo it. I wouldn't suggest bundling the Tk module, for instance. But as long as you employ a modicum of wisdom, you (and the rest of the CPAN community) should be just fine.

Metadata Support

The future of a better CPAN lies in the ability of CPAN to easily obtain good metadata about each module. For instance, it would be immensely beneficial for CPAN to be able to report the license that each module is distributed under. Unfortunately, there is no good way to determine this unless the module author declares it somewhere.

Ken Williams realized this and created the META.yml standard file for holding module metadata. His Module::Build project encourages the use of META.yml. Michael Schwern has also added support for generating a META.yml file.

Module::Install has very easy-to-use metadata support. Just add as many metadata fields as you wish to the Makefile.PL and Module::Install will make sure they get into the META.yml file.

Here is the Makefile.PL file for only.pm:

use inc::Module::Install;
name           ('only');
version_from   ('lib/only.pm');
abstract       ('Load specific module versions; Install many');
author         ('Brian Ingerson <[email protected]>');
license        ('perl');
include_deps   ('Test::More', 5.004);
build_requires ('Test::More', 0);
clean_files(qw(lib/only/config.pm));
clean_files(qw(t/lib t/site t/distributions t/version t/alternate));
create_config_module();
check_nmake();
&Meta->write;
&Makefile->write;

As you can see, creating a META.yml is no harder than creating a makefile. Here's what the META.yml looks like:

name: only
version: 0.26
abstract: Load specific module versions; Install many
author: Brian Ingerson <[email protected]>
license: perl
distribution_type: module
build_requires:
  Test::More: 0
private:
  directory:
    - inc

CPAN for Scripts

Traditionally, CPAN has been for distributing Perl modules, but not really for distributing Perl scripts. Some modules such as LWP and YAML come with a few helper scripts, but finding a distribution that only installs a Perl script in your path is rare indeed.

This distinction is somewhat artificial, in my opinion. I think the real reason that people don't generally do this is that it isn't easy to figure out how to do it. Here's a recipe for doing it with Module::Install:

First, create a script called "Hello World" that looks like this:

#!/usr/bin/perl -w
use strict;
print ;"Hello, world\n";
__END__
=head1 NAME - Hello World Script
Don't forget your documentation!
=cut
Then write the Makefile.PL:
use inc::Module::Install;
name('Hello-World');
version('1.23');
install_script('hello-world');
&Makefile->write;
It's that easy. Run these commands:
perl Makefile.PL
make manifest

And upload the tarball to CPAN. Well, actually there's a bit more to it than that: You still need to create some tests and a README and a Changes file. And, yes, you need to write a more useful script with better documentation. But that's about all.

Extending Module::Install

The best part of Module::Install might not come from Autrijus or myself, but from you. You can easily create your own extensions to Module::Install. The trick is to put your functionality into a module called Module::Install::PRIVATE::FooBar. By using the private namespace, you'll avoid any possible collisions with future Mode::Install:: modules.

Say you had a Makefile.PL that looked like this:

use inc::Module::Install;
name('Acme::Pie');
    
print "Are you 18 years or older? ";
if (<> !~ /^Y/i) {
    die "Can't install this module for minors\n";
}

&Makefile->write;

You could hide the query in a module, Module::Install::PRIVATE::Assertions:

package Module::Install::PRIVATE::Assertions;
use base 'Module::Install::Base';

sub check_18 {
    print "Are you 18 years or older? ";
    if (<> !~ /^Y/i) {
        die "Can't install this module for minors\n";
    }
}

1;

Then your Makefile.PL would look like this:

use inc::Module::Install;
name('Acme::Pie');
check_18();
&Makefile->write;

That's so much cleaner. And you can use it over and over again for all your modules.

Conclusion

Module::Install is just coming onto the scene. We hope that its fresh outlook on building Perl modules will help to keep Perl's future bright.

Module::Install is just one of several Module::* modules that we hope will reinvent the way Perl authors do business. One of these will be Module::Test. Although it might not make its debut for a couple of months, its mission is clear—to do for Perl testing what Module::Install does for Perl modules.

TPJ