Simon is a freelance programmer and author, whose titles include Beginning Perl (Wrox Press, 2000) and Extending and Embedding Perl (Manning Publications, 2002). He's the creator of over 30 CPAN modules and a former Parrot pumpking. Simon can be reached at simon@ simon-cozens.org.
Last time we met, I talked about the very important work I'd been doing in allowing Perl hackers to waste a lot of time on Internet Poker servers, with the Online Poker Protocol and Games::Poker::OPP.
I also mentioned that when I wrote that module, I needed a protocol debugger to help me understand how the binary protocol works, which turned out to be an interesting little project in its own right.
The Problem
In the good old days, when people set out to create a communications protocol, they did so following the UNIX design philosophies, keeping the protocols simple, easy for humans to debug and understand, and, primarily, text-based. (Eric Raymond has a lot to say about this in "The Importance of Being Textual," Chapter 5 of The Art of Unix Programming; see http://www.catb.org/ ~esr/writings/taoup/html/ch05s01.html.)
For instance, if I have a problem with a mail server, I know that I can happily telnet to port 25 on the offending machine and type commands in SMTP to help understand the problem and the server's responses. (In fact, I've been putting together a guide to help people do just this at http://simon-cozens.org/programmer/ phrasebook.html) This is a major bonus if I'm writing clients for the protocol since I can telnet in and chat away with the server myself, discovering what responses to expect in edge cases.
Then the discipline of computer science progressed, and people began designing and implementing binary-based protocols, generally with fixed-width commands making them difficult to extend, and removing the ability to effectively debug them via telnet.
Unfortunately, the poker protocol I began implementing last month is one such protocol. I wanted to check that I was packing the data correctly and getting the responses I expected, but I didn't yet have a working client that I could use to confirm that! In short, the problem is that my keyboard doesn't have a NULL key, nor would I be able to read the output I get back even if it did.
So I decided that the nicest way around the problem was to create a version of telnet that could easily send null characters and the like, and could translate any such characters it got back into something visible.
The Overengineering Stage
Of course, there were many ways I could implement such a beast. One idea would be to write a wrapper around netcat, which would turn binary data into something readable and pack user input into binary data. But I started thinking that I'd like to have line-editing capabilities and other bits and pieces, and I could foresee this nice little program turning into a massive monster.
But then someone pointed me at the lovely Term::Visual module, which provides a Curses-based terminal front end to an application. Anyone who's used a text-mode IRC client will be familiar with the sort of interface provided by Term::Visual; see Figure 1.
There's a title line at the top, a couple of status lines near the bottom, a line for user input underneath that, and the main "conversation" goes on in the middle. Term::Visual does its magic by hooking into the POE module. POE is both a Perl module and a way of programming; it provides many of the features of a time-scheduling operating system, so before we get into looking at Term::Visual in any depth, let's take a look at how POE works.
POEtry in Motion
When Linus Torvalds announced the source of his new operating system to Usenet, apparently he was particularly excited to be able to start a process that wrote "aaaaa..." to standard output and another one that wrote "bbbbb..." and have the two processes run simultaneously:
abababababab...
Let's emulate that great moment with POE. The POE equivalent to a process is called a session.
use POE; POE::Session->create(); POE::Session->create();
At the moment, these sessions don't do very much. This is for two reasons. The first reason is that we haven't given them much in the way of code. Unlike operating-system processes, POE sessions don't do anything by themselves, but only respond to interrupts, which POE calls events. There are certain system-defined events, such as _start, which are posted to sessions by POE itself, but other events need to be defined and posted explicitly.
The second reason this won't do very much is that, just as with an operating system, having two processes around isn't all that useful unless the kernel is running. POE has a kernel as well, and we need to tell it to run when we're ready.
Let's address these two issues and come up with a slightly expanded version of the code:
use POE;
POE::Session->create(
inline_states => {
_start => sub { $_[KERNEL]->alias_set("a_process") }
});
POE::Session->create(
inline_states => {
_start => sub { $_[KERNEL]->alias_set("b_process") }
});
POE::Kernel->run;
Now we're getting further, but not much further. The kernel wakes up and runs the _start events of each session. The _start events are associated with subroutine references, and these subroutines tell the kernel the name it should give to each session. If you're wondering what $_[KERNEL] is, that's simply looking at the KERNELth element of the parameters passed to the subroutine, where KERNEL is a constant exported by POE, which specifies where in the argument list the POE::Kernel object will come.
However, once we've named each session, there's little else to do, so the kernel quits. Let's rectify that by creating and calling some user-defined events:
use POE;
POE::Session->create(
inline_states => {
_start => sub { $_[KERNEL]->alias_set("a_process") },
say_something => \&say_a
});
POE::Session->create(
inline_states => {
_start => sub { $_[KERNEL]->alias_set("b_process") },
say_something => \&say_b
});
$|++;
POE::Kernel->post("a_process", "say_something");
POE::Kernel->run;
sub say_a { print "a"; $_[KERNEL]->post("b_process", "say_something"); }
sub say_b { print "b"; $_[KERNEL]->post("a_process", "say_something"); }
Unlike Linux, POE's event loop isn't preemptively multitasking; instead, sessions need to cede time to each other. In this case, our a session responds to the say_something event by printing an "a" and then posting another say_something to the b session. Similarly, the b session prints its "b" and sends a say_something event to the a session. All we need is an initial say_something event to be posted to one of the sessions to kick the whole thing off, and:
ababababababab....
Hoorah. Our two sessions can talk to each other and process events. Now we can reveal the secret: Term::Visual works by providing a ready-made session that turns events it receives from POE into Curses graphics on the screen. But what can we use to generate these events?
Hymn of the Big Wheel
POE communicates with the outside world primarily through a mechanism called a wheel. A wheel is a Perl object that not only provides an interface to some kind of filehandle, but is also capable of posting events into POE. For instance, here's a POE session that emulates tail -f:
POE::Session->create(
inline_states => {
_start => sub {
my $log_watcher = POE::Wheel::FollowTail->new(
Filename => "my_log_file.txt",
InputEvent => "got_record",
);
$_[HEAP]->{watcher} = $log_watcher;
},
got_record => sub { print $_[ARG0],"\n"; }
}
);
When this session starts up, it creates a new POE::Wheel::FollowTail. This wheel opens a file and watches for new lines being added to it. When it sees a new line, it posts whatever event we've defined as being its InputEvent—in our case, that's got_record.
Now, we don't want this object to be immediately destroyed at the end of the _start block, so we need to stash it somewhere for safe keeping. POE provides another named parameter, the HEAP, which is a hash reference that sessions can use to keep stuff in. Accordingly, we file away our log watcher in the heap.
This time when our kernel runs, as well as spewing out a's and b's, it will detect when lines are put on the end of the log file, and call our session's got_record event. The event gets passed an argument—the line that has been added, which is passed as the named parameter ARG0, and so we print it out.
That's a simple wheel; there are more complex ones, as we'll see when we dissect the protodebug program. But for now, that's all the POE we need to know. Let's take a quick detour into how we're going to present the data flowing between the client and the server before diving back into Term::Visual.
Come On, Feel the Noise
There are two data presentation problems with our protocol debugger; the first is that we want to display binary data in a way that's not going to freak out the user. This happens in protodebug's to_visual subroutine:
sub to_visual {
my $out;
for (split //, shift) {
if (/\r/) { $out .= "\\r\n"; }
elsif (/\n/) { $out .= "\\n\n"; }
elsif (/[[:print:]]/) { $out .= $_; }
else { $out .= sprintf(" 0x%02x ", ord $_) }
}
return $out;
}
This caters to three types of data coming from the wire. The first type is end-of-line characters, in one form or another; these are made visible as "\n" or "\r", but additionally a "real" newline is added to the output. This means that an SMTP conversation would appear something like this:
220 alibi.simon-cozens.org ESMTP Exim 3.35 blah blah blah\r >>> HELO sailor\n 250 alibi.simon-cozens.org Hello sailor [195.188.xx.yy]\r
which is more or less what you'd expect; the newline stops the display from looking absolutely hideous, but the representation of the newline ("\r" or "\n") tells you what sort of newline it actually was.
The second class of data is data that is already visible, and so we don't need to do anything specific to it. The POSIX character class [:print:] describes this case neatly for us:
elsif (/[[:print:]]/) { $out .= $_; }
And the third class is obviously just generic binary spew, which we escape as a hex character:
else { $out .= sprintf(" 0x%02x ", ord $_) }
This means that a "logged in" reply from a Poker server would look like this:
0x00 0x00 0x00 0x15
Now we need to turn to the other way around, where we have to give the user a friendly way of specifying a binary stream. We try to make this as flexible as possible by providing multiple ways to describe the same binary byte. We use a "nibbling" tokenizer to shift tokens off the user's input and add them to the binary output stream; first, we handle the "0x.." case for specifying a hex byte:
sub to_binary {
my $output;
my $input = shift;
while ($input) {
$input =~ s/^0x([\da-fA-F]+)//
and do { $output .= chr(hex($1)); next};
We also want to handle backslash notation for characters like "\0", "\n", "\cJ", and so on:
$input =~ s/^\\(\w+|\s)//
and do { $output .= eval "\"\\$1\""; next; };
Then, of course, there are ordinary printable characters that can be reached from the keyboard:
$input =~ s/^([\x21-\x5b\x5d-\x7e]+)//
and do { $output .= $1; next };
But not space, it may surprise you to note! The reason for this is to allow the user to break up bytes and control sequences in a readable manner; you don't want to type, for instance:
0x3\0\0\0test\0\04\n
if you have the opportunity to type:
0x3 \0\0\0 test\0 \04 \n
To make this possible, a literal space has to be input using "\", but other whitespace is ignored:
$input =~ s/^\s// and next;
The nibbler technique, categorized by this s/$^regexp// and do { ...; next;} idiom, is a great way to make simple tokenizers; for instance, it's allowed us to give backslash-space priority over a space on its own.
Finally, if we now come across a character we haven't catered for, we should give an error:
$vt->print($window_id, "Couldn't understand ".to_visual($input)); return; }
Those are the two subroutines that marshal data from and to the user.
Sound and Vision
Now let's slip back into interfacing these routines, the TCP connection and Term::Visual. First, we'll set up a new Term::Visual object and specify some color preferences for it:
my $vt = Term::Visual->new( Alias => "interface" );
$vt->set_palette( ncolor => "white on black",
st_frames => "bright cyan on blue",
st_values => "bright white on blue",
stderr_bullet => "bright white on red",
stderr_text => "bright yellow on black",
err_input => "bright white on red",
);
The Alias parameter to the constructor is just like the alias_set method we used in our POE sessions—we give the terminal an alias so that we can post events to it during the course of the program.
The next stage is to set up our window, which will contain the two-line status bar. We need to tell Term::Visual what the status bar is going to look like. We do this with a set of printf-like format strings:
my $window_id = $vt->create_window(
Window_Name => $0,
Status => { 0 =>
{ format => " [%8.8s] ",
fields => [qw( time )],
},
1 => {
format => " %s to: %s:%s",
fields => [qw( status host port )]
},
},
Buffer_Size => 1000,
History_Size => 50,
Title => "Protocol debugger" );
This will set up one line to contain a little clock, of the form "[08:30 AM]", and another line to tell us about the connection status—whether "connecting to somehost:1234" or "connected to somehost:1234." It will also set the title line at the top of the screen, and set up buffers and history appropriately.
Now we're ready to go. We need to create the main POE session that is going to handle all the action, and then we can run the POE kernel. Here's what the main session looks like:
POE::Session->create
(inline_states =>
{ _start => \&start_guts,
update_time => \&update_time,
connect_success => \&switch_wheels,
connect_failure => sub {
$vt->print($window_id, "An error occured!: $_[ARG2]");
}
got_term_input => \&handle_term_input,
got_packet => sub { $vt->print($window_id, to_visual($_[ARG0])) },
}
);
There aren't that many events we need to cater for: First, when the kernel starts up, we need to start connecting to the remote server and kick off something to update the clock on the status bar. start_guts handles all this. Then there's the event that gets fired every minute to actually do the clock update.
After we've established the connection, we need to turn our initial TCP socket into a filehandle (or rather, a wheel) that we can send data to and get data back from; the connect_success and connect_failure states deal with what happens here.
And when we're up and running, if the user says something, we need to tell the server about it; similarly, if the server says something, we need to tell the user about it. This last state is so easy to handle, we do so on the spot: we run the data we've received from the server through the to_visual routine discussed above, then tell the terminal object to display it on our main window.
Let's start our analysis with the _start state. First, we need to tell Term::Visual how it should inform us about new input. We have to register a callback state so that it knows what to do with any input it receives. We do this by posting the name of a state to its send_me_input state. Our input handler state, as we've defined above, is called got_term_input, so that's what we'll send it:
sub start_guts {
my ($kernel, $heap) = @_[KERNEL, HEAP];
$kernel->post( interface => send_me_input => "got_term_input" );
We also call the update_time state; this will not only update the clock once, it will schedule itself to be called again every minute, keeping the clock updated:
$kernel->yield( "update_time" );
And finally, we connect to the remote server. We'll first update the status bar to say something like "Connecting to myserver:1234":
$vt->set_status_field( $window_id, host => $host,
port => $port, status => "Connecting" );
And we create a new wheel to connect to the host; as before, we store this on the heap. As well as telling it what host and port to go to, we need to tell it what to do on success and failure; as before, these are the states we've named in the previous session constructor:
$heap->{connector} = POE::Wheel::SocketFactory->new
( RemoteAddress => $host,
RemotePort => $port,
SuccessEvent => 'connect_success',
FailureEvent => 'connect_failure',
);
Updating the time is pretty easy, although we'll borrow some help from POSIX's strftime function to handle the formatting for us:
sub update_time {
use POSIX qw(strftime);
$vt->set_status_field( $window_id,
time => strftime("%I:%M %p", localtime) );
$_[KERNEL]->alarm( update_time => int(time() / 60) * 60 + 60 );
}
POE's alarm method will arrange for us to be called back at the beginning of the next minute.
When the connection happens, we need to create a new wheel that can be used to talk to the remote host. The SocketFactory wheel's success state (connect_success) will be called with the TCP socket, and so we set up another wheel that communicates with that socket:
sub switch_wheels {
my ($heap, $kernel, $connected_socket) = @_[HEAP, KERNEL, ARG0];
delete $heap->{connector};
$vt->set_status_field( $window_id, host => $host, port => $port,
status => "Connected" );
$heap->{socket_wheel} = POE::Wheel::ReadWrite->new
( Handle => $connected_socket,
Driver => POE::Driver::SysRW->new,
Filter => POE::Filter::Stream->new,
InputEvent => 'got_packet',
);
}
Again, we need to tell the new wheel what to do when it gets some data; specifying callback events is a big part of the POE I/O model. This is the got_packet event that was so simple to implement above.
The final event we need to handle is the input by the user. Term::Visual will call the callback event we registered, got_term_input, with any data received from the terminal. This is mildly complicated because the input can be in one of three forms. First, it can be an exception: the user hits "^C" or "^\" and expects to be dumped out of the application. Thankfully, Term::Visual tells us about such an exception directly:
sub handle_term_input {
my ($heap, $input, $exception) = @_[HEAP, ARG0, ARG1];
if (defined $exception) {
warn "got exception: $exception";
$vt->delete_window($window_id);
exit;
}
The second form is a command; we handle this just like an IRC client, with a command being defined as a forward slash at the start of the line, followed by a command word. At the moment, the only command we care about is QUIT, but it would be easy enough to extend the program to support a CONNECT command to connect to a different server:
if ($input =~ s{^/\s*(\S+)\s*}{}) {
my $cmd = uc($1);
if ($cmd eq 'QUIT') { $vt->delete_window($window_id); exit; }
warn "Unknown command: $cmd";
return;
}
Anything else we encode using the to_binary routine we previously detailed, and then send the result out to the server through the connection wheel. We also echo the output to the terminal, so the user can see what was said:
my $output = to_binary($input);
if ($output) {
$heap->{socket_wheel}->put($output);
$vt->print($window_id, ">>> ".to_visual($output)."\n");
}
And that's it! All our states are defined, all our bases are covered, and all we need to do is kick off the POE kernel and let the user talk to the server:
POE::Kernel->run;
The protocol debugger comes in at just over 150 lines of code, and swiftly became an invaluable tool in implementing the poker modules, as well as debugging several other network issues I came across. And, thanks to POE and Term::Visual, it has a friendly interface that makes it not too much of a burden to use, and was crafted up in little under an hour.
POE is a wonderfully flexible set of tools for developing event-based applications in Perl. There are a whole range of components like Term::Visual, which can take the pain out of programming servers and clients for all kinds of protocols. Take a look at the POE home page (http://poe.perl.org) to learn more.
TPJ
