Jaap is currently a student in applied physics and philosophy. He can be contacted at [email protected].
Allegedly, a certain Mr. Wall has claimed that "It is easier to port a shell than a shell script." This, of course, is not true; shells are, although conceptually simple programs, in fact quite complex. Still, there are a few projects that try to improve the shell environment by adding Perl to the mix, most notably Psh and Zoidberg. In this article, I will show you some features of the Zoidberg Perl shell (see Figure 1).
There are two main ways in which a Perl shell can make your life easier: In the first place, you can use Perl syntax at the command prompt; and secondly, you can modify or extend your shell environment using Perl. UNIX users, in general, spend a lot of time working in a shell environment, so being able to tune this environment to your every wish can save you a lot of irritation. And of course, if you type Perl faster than shell script, it is nice to have a shell that understands you. Now, of course, the older shells allow for extensions: While bash has a notoriously obscure code base that is not easy to hack, zsh on the other hand even allows you some module interfaces. But these programs don't have Perl bindings and adding them could well be more work than starting from scratch.
Zoidberg (or zoid, for friends) was written from scratch and is a pure Perl implementation. Due to an object-oriented and modular design, Zoidberg can be regarded as a framework as well as an application. So far, it has not yet seen a stable release, so some parts of the code are still a bit rough, but we work very hard to fix that. You should also keep in mind that once you start hacking your shell, it is in general a good idea to have another shell available just in case; after all, you don't want to be locked out of your account.
To install zoid, try something like:
perl -MCPAN -e 'install q/Bundle::Zoidberg/'
Now run "zoid" to start the Zoidberg Perl shell.
Perl at the Command Prompt
Traditionally, shell languages have been ugly things. Their syntaxes are hacked together, resulting in an arcane language with lots of obscure exceptions ("for historic reasons" is an oft-used phrase in the manuals). The sole objective of these languages is to be as expressive as possible with the minimum number of characters. In some respects, shell languages are not unlike Perl.
So why use Perl instead of shell script? An important reason is that Perl is a language that has a much broader scope than shell script; it is more powerful. The UNIX philosophy dictates that everything is a file, and I think no language is so tuned for text manipulation as Perl; so a lot of your daily administration tasks can be done with simple Perl statements that are the equivalent of many lines of shell script. In a Perl shell you can, for example, use regular expressions to modify the filenames in a certain directory.
The downside of using Perl syntax is that it was intended for scripts and isn't really optimized for shell usage. Some people claim that the ultimate Perl shell is something like:
perl -e 'while (<STDIN>) { eval $_; warn if $@ }'
But after a while, it gets very tiresome to keep typing system "ls" instead of the plain ls that most shells will understand. Of course, you can define a subroutine for this or even use Shell.pm's AUTOLOAD function, but it won't be as intelligent as you want it to be. The next logical step is then to add a routine that tries to recognize which commands should be system'ed and which commands should be eval'ed; if you follow this way, you eventually end up with a program like zoid.
In Zoidberg you can use two different syntaxes. The first one is a partial implementation of the shell command language as specified by POSIX (full implementations include bash, zsh, etc.). The second syntax of course is Perl. This is just your normal Perl interpreter, but Perl code used at the command prompt undergoes a bit of source filtering to make it play better with the shell environment.
For example if you type:
zoid$ print map "\n" @PATH
then @PATH is imported from Env.pm before the evaluation of the command line. In this example, Zoidberg understands this is a Perl command because the word "print" is a reserved word.
Let's do another example: Say I want to change all filenames in a directory to their lowercase equivalent; this typically happens to me after using a brain-dead ftp client. Using Perl, this is a simple one-liner, while the shell equivalent would need some program, such as sed, awk, or perl to do the string manipulation.
zoid$ mv($_ => lc($_)) for grep /[A-Z]/, <*>
If you use zoid as your login shell, it will also will be used by other programs to execute commands. For example, from vim you can now use the following command to get a word count for the file you are editing.
:!cat % | {/^\S/}g | wc -
All lines starting with a whitespace are supposed to contain example code so we grep them out. The second part of this pipeline shows Zoidberg's notation to have a Perl grep in a pipeline. The curly braces force this command to be evaluated as Perl code instead of a filename and the g modifier at the end wraps the code in a loop.
Zoidberg also has support for multiline editing, which becomes very handy when you want, for example, to define a new subroutine on the command line. This is where Term::ReadLine::Zoidberg distinguishes itself. Other readline libraries allow you to edit only one line at a time—once you press return, you can't go back to the previous line. Not so in zoid, and what's more, you can even go to a "multiline mode" by pressing <ESC>-m.
Scripts
All scripts used by zoid are normal Perl scripts. This means that, for example, the rc files ("/etc/zoidrc" and "$HOME/.zoidrc") are just Perl scripts. To interface with the shell, these scripts use the Zoidberg::Shell module. This module exports an AUTOLOAD function that allows for shell commands to be called as Perl subroutines and a method called shell() that is the zoid equivalent to the Perl eval function. Below is an example zoidrc file that can be saved as "~/.zoidrc":
use Zoidberg::Shell;
$shell = Zoidberg::Shell->current();
cat($ENV{HOME}.'/TODO')
if $shell->{settings}{interactive};
unshift @INC, "$ENV{HOME}/lib/perl5/";
A Debugging Session
One of the things zoid can be used for is to test new modules interactively. It can be very useful to quickly test the output of some methods while you are coding a module. Take, for example, the following code and save it as "~/lib/perl5/My/Module.pm":
package My::Module;
sub new { bless {} } # simple constructor
sub test { return 'fooobar' }
1; # keep require happy
Now to load this module, we simply use it on the command prompt:
zoid$ use My::Module zoid$ $mm = My::Module->new() zoid$ p $mm->test()
Now you realize that there is a typo in the word "foobar," so you go back to your editor, open the module, and correct it.
Next, we need a method to load the changed version of the module. At the moment, zoid has an undocumented built-in called "reload," but it is broken, so let's define our own. The following code could be entered directly at the command prompt, but let's put it in your zoidrc file:
use Zoidberg::Shell;
my $shell = Zoidberg::Shell->current();
$shell->{commands}{reload} = sub {
# transform module name to file name
my $file = shift;
$file .= '.pm' unless $file =~ /\.\w+$/;
$file =~ s{::}{/}g;
# look up the filename in %INC
$file = $INC{$file} || $file;
# load the file
eval "do '$file'";
# forward errors
die if $@;
};
The hash $shell->{commands} is tie'ed to an object that helps to keep track of plug-ins, but for the moment, we can pretend that it is just a hash with anonymous subroutines. Now we can use this command to reload the module:
zoid$ reload My::Module zoid$ p $mm->test()
This works fine for object-oriented modules, but how about libraries? If your module exports some functions, you can use it from the command prompt and call the exported routines. Another option is to change the namespace in which Perl commands are evaluated—this is done simply by using the "package" keyword:
zoid$ package My::Module zoid$ p test()
To return to the default namespace, type package Zoidberg::Eval.
Extending Zoidberg
The parser already recognizes two different syntaxes, but how about adding another one? All you need for this is a plug-in that defines some rule to recognize the syntax, and a handler routine to execute commands given in this syntax. Let's teach Zoidberg SQL to see how this works.
To start with, we need to write a plug-in module that manages a DBI object. Save this module as "~/lib/perl5/SQLFish.pm" (remember that we added "~/lib/perl5" to @INC in the previously mentioned zoidrc file).
package SQLFish; use strict; use Zoidberg::Fish; use Zoidberg::Utils qw/error output/; use DBI; our @ISA = qw/Zoidberg::Fish/;
Zoidberg::Fish is the base class for Zoidberg's plug-ins. It gives you a constructor and routines needed for the plug-in framework. If you use Zoidberg::Utils, your plug-in will blend in nicely with the rest of zoid. The plug-in is an object that lives below the main Zoidberg object. Use $$plugin{shell} to access the parent object.
sub connect_db { # create DBI object
my $plugin = shift;
$$plugin{db}->disconnect if ref $$plugin{db};
$$plugin{db} = DBI->connect(@_);
# make DBI survive forks
$$plugin{db}{InactiveDestroy} = 1;
}
This routine will later be exported as a built-in command that is used to setup the database connection. Of course, you can call it from the zoidrc file if you want to be connected at all times.
Next, three methods follow that plug-in in several stages of the parser:
sub word_list { # claim commands
my $plugin = shift;
my ($meta, @words) = @{ shift() };
return grep /^$words[0]/,
qw/SELECT INSERT UPDATE DELETE/ if wantarray;
return 'SQL' if $words[0] =~ /^[A-Z]+$/;
return undef;
}
This method tells the parser that all commands written in all caps are SQL. The list context is used for tab completion of SQL commands.
sub parser { # perform selected expansions
my ($plugin, $block) = @_;
@$block =
$$plugin{shell}->$expand_param(@$block);
return $block;
}
This method overloads the default word expansions for the SQL syntax. We only expand for variables here, so we can use environment variables in our SQL statements. Path expansion, for example, is omitted here because we want to be able to use a "*" in the SQL command without escaping or quoting it.
And the last method:
sub handler { # execute SQL statements
my $plugin = shift;
my ($meta, @words) = @{ shift() };
# catch connect command
if ($words[0] =~ /^connect/i) {
shift @words;
return $plugin->connect_db(@words);
}
error "no db connection" unless ref $$plugin{db};
my $sth =
$$plugin{db}->prepare(join ' ', @words);
$sth->execute;
output [ map {join ', ', @$_}
@{$sth->fetchall_arrayref} ];
}
1;
__END__
This method actually gets to execute the SQL commands. We use the utility functions error and output here. error is like die but has another output format and helps us with stack traces. output is like print but uses Data::Dumper when passed references; the way it is used here, it outputs data in multiple columns when possible. And we make an exception in the evaluation for statements starting with connect—this will prove useful later on.
Next, we define a plug-in configuration script to make Zoidberg use this module. Save this script as "~/.zoid/plugins/SQL.pl".
{
module => 'SQLFish',
export => ['connect_db'],
parser => {
word_list => 'word_list',
parser => 'parser',
handler => 'handler',
}
}
This is just a Perl script containing only a hash defining the hooks that are in the module so that zoid can use them. We export the connect_db method as a built-in command and we list the parser hooks we defined. For more advanced plug-ins, one can also import events.
Now one might wonder what the advantage is of having SQL in your command shell instead of using a dedicated application for that. For one thing, you can combine SQL statements with general shell commands, so things like this will work:
zoid$ SELECT firstname, lastname FROM users
| {s/\b(\w)/uc($1)/eg}p
zoid$ SELECT * FROM users | wc -l
zoid$ sql{ select from users where clue > 0 }
> users.txt
In the last example we needed the parenthesis to protect the first ">"; this is a notation that works for all syntaxes in Zoidberg. Of course, once we tag the command as "sql" there is no need for caps to have it recognized as SQL.
You can also use the Zoidberg framework very easily to build a dedicated application, or at least an application that looks dedicated. This is demonstrated later.
Modes
Zoidberg has a built-in called mode, which is used to switch your default syntax. Say you want to type a series of SQL commands and you think typing commands in all caps is tiresome. Try this:
zoid$ mode SQL zoid$ connect DBI:mysql:test zoid$ select * from users
Here you see why we put that exception for connect in the handler routine of the plug-in. Once you are in a mode, you need the bang ("!") to "shell out." So in the SQL mode, you can still execute shell commands like this:
zoid$ !ls zoid$ select * from users | !grep root
To get back to Zoidberg's default mode, type:
zoid$ mode -
But wait, there is more! A mode can also be the name of a Perl module; in that case, you get to keep shell command syntax, but all commands are considered subroutines in the specified module.
zoid$ use CPAN zoid$ mode CPAN::Shell-> zoid$ i /MimeInfo/ ... zoid$ install File::MimeInfo ... zoid$ mode -
The "->" is there to tell zoid that CPAN::Shell always expects a class name as the first argument for a subroutine. Replace it with "::" for modules that don't need this argument. Note that this is purely an example—there is already a CPAN plug-in for Zoidberg that can be used after typing mode CPAN; this plug-in also supports things such as tab expansion.
Write Your Own Shell Application
Say you want to write a program with a shell-like interface quickly; using Zoidberg can spare you the hassle of writing a command parser and all that. We again use the SQL plug-in of the previous section and write a simple script to start our SQL shell:
#!/usr/bin/perl
use Zoidberg::Shell;
use lib $ENV{HOME}.'/lib/perl5/';
# You have Env::PS1 ?
$ENV{PS1} = '\C{blue,bold}DBI\C{reset}: ';
$ENV{PS2} = ' : ';
my $shell = Zoidberg::Shell->new(
settings => {
# non-default rcfile
rcfiles => [$ENV{HOME}.'/.sqlshell'],
mode => 'SQL',
}
);
$shell->main_loop; # run the shell
$shell->round_up; # clean up pending objects
The trick here is the "mode" setting; actually you just run Zoidberg in a predefined mode. Of course, you still are allowed to use Perl, so you now have your own SQL Perl shell (but remember that default aliases like 'p' and 'pp' are not defined here). Also, you can still access system commands using the bang ("!"). Now try this script with a module name as mode string.
Ongoing Development
As I stated in the introduction, Zoidberg has not yet seen a stable release. This means it is neither feature complete nor bug free. But I think that the development has reached a point where it has become clear how the stable release might look, and I hope you have caught a glimpse of that in this article.
One of the things I'm currently working on is forking more functionality from Zoidberg into separate CPAN packages. At the moment, Bundle::Zoidberg consists of the Zoidberg package itself and two packages that have already forked from Zoidberg's code base: Term::ReadLine::Zoid and Env::PS1. Especially forking Term::ReadLine::Zoid has been very good for the project. One of the subsystems I hope to release in a separate package is the parser and job-control code. Forking functionality makes the code more accessible for other applications and also forces us to have very clean interfaces between subsystems of the program and will thus allow for better test suites.
If you encounter any bugs while playing around with zoid, please report them through http://rt.cpan.org/ and feel free to ask questions on the mailing list.
TPJ