Expect: The Firefighter’s
Friend
Cameron Laird
Most UNIX administrators are familiar with the Expect utility. They generally know it as a way to automate password entry in FTP clients or password-management system commands.
As crucial as its use can be in these applications, Expect has many other capabilities that may be less well known. Moreover, Expect is available for Windows. This article briefly surveys the range of Expect uses, then concentrates on ways Expect solves one-time emergencies. One major role that systems administrators play is to repair the damage — missing files, lost passwords, and so on — that arises in day-to-day information technology (IT) operation. These operations don’t have to be as tedious as they seem. In many cases, Expect can turn hours of drudgery into a short scripting session.
Tcl/Tk Plus More
The first essential to understand about Expect is that it’s a full-blown, general-purpose programming language. Yes, it’s a handy tool for automation of password management, but that’s only a showy side effect of its more general capabilities.
Expect is an “extension” of the Tcl/Tk high-level language. That means it does everything Tcl/Tk does, and knows a few commands more, in the broad categories of:
- Debugging
- Character-oriented dialogue management
- Pseudoterminal management
Pseudoterminals take responsibility for “funny” character handling, as with password entry made invisible by manipulating terminal characteristics. “Dialogue management” has to do with pattern matching the “conversation” between a user and an application: the user types something, the program responds, the user types again, and so on. Expect defines a convenient language for making calculations based on such dialogues. You can, for instance, use it to automate a dial-up connection:
spawn tip modem expect "connected" send "ATDT$number\r" expect "CONNECT" ...This is the beginning of a working Expect script that dials $number to establish a connection before continuing with its work. The send line feeds the ATDT sequence to the modem. The two expect commands shown here know how to wait until they receive the proper prompts (“connected” and “CONNECT”) before proceeding. Administrators who’ve worked with modems know how clumsy it can be to try to program such interactions without Expect. Completeness I’ll emphasize a few more times the point that Expect is a complete programming language, in much the way C or Java is. Many of the people who use it most refer to the one serious book published on the topic, Don Libes’ Exploring Expect. Libes is the computer scientist at the National Institute of Standards and Technology who originally wrote Expect, and continues to manage its maintenance and enhancement. His book is remarkably well constructed and accurate, and has been a valuable reference to a generation of administrators. The problem is that the book’s first edition is so comprehensive that it has never been updated. Since its publication in 1994, Tcl has acquired powerful capabilities to handle binary data, network connections, Unicode characters, concurrent programming, clock arithmetic, and much more. When someone tells you what Expect can do, he’s probably right; when he tells you that a task is beyond Expect, though, be careful he’s not talking about a version from a decade ago. The completeness of more recent releases makes Expect an excellent general-purpose upgrade from the shell or Perl scripting you might otherwise use for “fire fighting”. Here’s an example from my own work: I recently had to repair a busy server that had a corrupt C run-time library. At least, that’s what we tentatively concluded; we never made the time to diagnose the problem completely. What we knew was that the machine was in active networked use, continuously serving several thousand customers. However, some configuration files periodically became corrupt. Moreover, its development system was missing. There was no way to do compilations or even reliably copy over binary executables. Our most urgent need was to scan the configuration files periodically, detect corruptions, and launch processes to correct the damage. The only facilities at hand appeared to be the login shell and such UNIX “built-ins” as sed and awk. Yuck. While clever programmers have taught sed to invert numeric matrices and play checkers, I’m only expert enough for simple text transformations. I’ve never figured out how to use sed with Unicoded content. Worse, at least one of the remedies I needed to manage involved resetting passwords. That’s hard to do without Expect, and it didn’t look as though I could generate Expect on the server, or copy it there. Work Through Whatever Channel You Have That problem has a solution, though. A well-equipped development host could reach the server through the network, and that’s all Expect needs. While a conventional password automation with Expect might look like:
spawn passwd $user expect password: send $password\r expect password: send $password\r expect eofmy first step was to set up a “remote control” form for the same sequence:
spawn telnet $server
expect login:
send root\r
expect Password:
send $root_password\r
# Now we're logged-in to $server.
expect $shell_prompt
send "passwd $user\r"
expect password:
send $password\r
expect password:
send $password\r
expect eof
My actual script encapsulated this in a proc, Tcl’s “subroutine”.
Moreover, we ssh rather than telnet, and you should, too. I’ve simplified
the code in this article slightly from the real-world source, to highlight the
principles most likely to apply universally.
At this point, I was using Expect on a networked host to manage passwords on the server. I still had the challenge of running calculations over the contents of the configuration files, with few local tools.
Expect met that challenge, too, of course. I could see the corruption perfectly well by logging in to the server and editing the files. All that was left was to teach Expect to do the same. I quickly threw together a small framework for “remote reads”. With just a bit more “plumbing”, an ambitious programmer could wrap this up as a low-performance network file system — one that extends Tcl’s conventional input-output facilities to operate on any file, not just local ones. The heart of the program is this:
# Turn off screen echoes.
log_user 0
# Log in through telnet.
spawn telnet $host
expect login:
send $user\r
expect Password:
send $pass\r
expect $prompt
# Just echo the contents into Expect's "front end".
send “cat $filename\r”
# I could also use a timeout in place of detection of $prompt.
expect {
^$prompt {
}
-re "^(\[^\r]*)\r\n" {
puts "The line is '$expect_out(1,string)'."
exp_continue
}
}
By itself, this program simply connects to $host, then echoes the contents
of $filename to the screen. The value of this, though, is that all the data
are programmatically accessible. In the real application, I filtered $expect_out(1,string)
in various ways to detect corruptions. By bringing all the calculations into
Expect, I could conveniently process the suspect files one line at a time from
a remote host, while retaining the power to manipulate passwords and do other
sensitive operations.
Other languages and tools can do similar work. I don’t know of any other that make it so easy, though. If I were working with any other language, it probably wouldn’t have been worth my time to set up the automations I did. We likely would have reinstalled a development system, and worked directly on the host. As sensible as that sounds, it would have taken a bit longer just to install a working development system than the actual time to script and test everything I needed with Expect acting as a remote controller.
Not Just for Breakdowns
Expect is great for this kind of “fire fighting”. If a system has
enough functionality that you can do what you want “manually”, then
Expect can “glue” together the pieces to automate the operation.
It’s not just broken systems that need this help, however. Or, perhaps more precisely, Expect is also handy with systems designed with broken interfaces. All kinds of operations “off the desktop” are candidates for Expect programming:
- Modem control
- Real-time databases and physical control devices hosted on special-purpose industrial operating systems
- Network nodes with serial-line connections
- Testbeds for operating-system kernel experiments
proc set_switch {switch_name subnet_assignment} {
# ...
}
While you invoke set_switch from within Expect, you’ll be out
of the office next week, and you want to leave something in your co-workers’
hands that they can use safely. All you have to do is write:
package require Tk
package require Expect
proc make_labeled_entry {label variable} {
frame .$variable
label .$variable.label -text $label -width 18
entry .$variable.entry -textvariable $variable
pack .$variable -side top
pack .$variable.label .$variable.entry -side left
}
make_labeled_entry "Switch name" switch_name
make_labeled_entry "Subnet assignment" subnet_assignment
label .label
button .button -text "Update switch_name" -command {
set_switch $switch_name $subnet_assignment
.label configure -text "$subnet_assignment assigned to \
$switch_name."
}
pack .label .button -side top
When run, this script yields the small GUI control panel seen in Figure 1. Colleagues with point-and-click skills are likely to find this more comfortable than Expect prompting, and the only cost to you is fewer than 20 lines of source code.
Summary
When you have fires to fight — small, one-time jobs too tedious to welcome, but too important to neglect — let Expect help you. Expect has all the convenience of conventional shell scripting, but gives networking, pseudoterminal management, Unicode, and GUI capabilities for free. As Don Libes once observed, “Expect is, after all, a tool made for dealing with crappy interfaces.” Judge for yourself how much of your daily job fits that description.
Learn more about Expect, including tutorials, the latest sources, and so on, at:
http://starbase.neosoft.com/~claird/comp.lang.tcl/expect.html
My thanks to Andreas Kupries for his help in preparation of this article.
Cameron Laird is Vice President of Phaseit, Inc. (http://www.phaseit.net), where he frequently consults on systems administration issues. He stopped counting after the first hundred hours Expect saved from his own work days. Cameron welcomes correspondence to [email protected].