Systems Administration with Scsh
Evan Sarmiento
Shell scripting languages provide systems administrators with powerful tools to automate mundane tasks, saving hours of work. Most of these languages, however, do not provide a well-designed object system, nor are they optimized for recursion and abstraction. Scheme, a dialect of LISP, incorporates all the features of a good programming language as listed above. Because Scheme cannot interact with UNIX primitives alone, there is a Scheme software package called scsh (the scheme shell). Scsh provides a set of macros for the Scheme programming language, adding the ability to call on all standard UNIX system calls and even adds regular expressions.
In this article, I will briefly introduce and describe the important features of the Scheme language and present some of my own scripts written in scsh, which exhibit many of the listed features. This article presumes that you know how to program using Scheme. If you are new to Scheme, read the Notes section at the end of this article, which provides numerous learning resources.
You can download scsh from:
http://www.swiss.ai.mit.edu/ftpdir/scsh/
There are easy-to-follow installation instructions on the site. For FreeBSD users, scsh is present in the ports collection, /usr/ports/shells/scsh.
Scheme comes in many variants, such as MIT Scheme, Scheme 48, MzScheme, and others. Scsh is based on Scheme 48. However, I find MIT Scheme to be the most complete. You can download MIT Scheme from:
http://www.swiss.ai.mit.edu/projects/scheme/
Scheme tutorials and the book Structure and Interpretation of Programs (Second Edition by Harold Abelson and Gerald Jay Sussman, with Julie Sussman, The MIT Press) are provided online at:
You can also find out more about each variant by looking at:
Running Scsh
When you execute scsh, you’re running in interactive mode, which is similar to the interactive mode for Python. You can type in individual commands, and the interpreter will evaluate them line-by-line. To use scsh for batch processing, append this header to the top of your scsh script files:
#! /usr/local/bin/scsh -s !#Benefits of Scsh Programming in scsh provides numerous benefits. You will have access to both a scripting language and a systems-programming language when using scsh, because it is just a set of macros for Scheme. scsh uses ‘‘process notation’’, which allows easy access to the command line. Scsh also provides low-level access to the operating system. The current release of scsh provides full access to Posix, which includes: fork, exec, wait, sockets, write, open, close, seek, tell, chmod, chmod, chgrp, chown, locking, and more. Another benefit of using scsh is that it is very portable. It is based on Scheme and, like Java, Scheme code is executed through a byte-code interpreter. There are Scheme 48 virtual machines for numerous operating systems, including Windows, which allow you to run scsh in almost any environment. The Scheme Philosophy Before discussing the specifics of scsh, it is useful to understand the peculiar syntax and fundamental data structures of Scheme. Scheme was derived from Lisp, improving upon many of the Lisp features. The Lisp programming language was mainly used in two fields: math and artificial intelligence. Scheme is a dialect of Lisp that stresses elegance and simplicity. It is a stripped-down version of Common Lisp. In fact, the language specification is about 50 pages, while the Common Lisp’s specification is about 1,300 pages. Scheme is often used in computer science courses to teach abstraction and programming concepts. Even though it is used primarily as a teaching language, Scheme has numerous applications — the GIMP allows users to write GIMP-loadable modules in Guile, a GNU Scheme variant, and Scwm, a Window manager, allows users to write their own extensions in Scheme. The elegance and readability of Scheme is what prompted Olin Shivers to write scsh, a set of macros that provide access to low-level operating system functions and usual shell commands. Sh provides a terse, inflexible, and hard to read syntax, which does not provide access to lower level functions like sockets or signals, unlike scsh, which encompasses both a shell scripting language and a systems programming language. Prefix Notation Scheme may be hard to learn at first, but learning scheme will certainly increase your programming skills. What separates Scheme from other languages is its use of Prefix Notation. In Scheme, operations are placed before operands. For example, you would normally write this equation as such:
Regular: 5 / 3 * 10 Prefix Notation: (* (/ 5 3) 10)Even simple arithmetic is changed. For example, 1 + 1 + 1 + 1 becomes:
(+ 1 1 1 1)The concept of Prefix Notation is important to understand in order to use scsh. All operations in Scheme are done through Prefix Notation syntax, which may seem cumbersome at first, but simplifies coding. Lists Lists are an integral part of the Scheme programming languages. A list is just a data structure. The most obvious application of a List would be in a loop, where your program needs to take an element from the front of the list and perform some evaluations on it. When I program in scsh, I commonly use lists to parse output from programs. I can also specify that the program output be translated into a list of strings. Another benefit of Lists is that you can have functions return more than one value using a list. This is how the (pipe) system call works — it returns a list consisting of two new ports. Lists are manipulated through the functions: list, cons, car, cdr, and append. Scheme itself is based on the idea of a list. When the Scheme interpreter tries to execute the command (+ 1 1 1 1), the interpreter thinks of it as a list, where the first element of the list is the operator. You create a list by using the list function. The following examples are performed in interactive mode:
(define q (list 1 2 3 4 5))This creates a list assigned to the name q. You do not have to assign lists to names; you can pass them anonymously to functions. You can retrieve the item off the front of the list by using car:
(car q) 1You can remove one item from the list by using cdr. Using cdr on a list creates a temporary copy of the list in memory with the first item removed. The original list is still intact:
(cdr q) '(2 3 4 5)cons adds an item to the beginning of a list, while append adds a list to the end of a list:
(cons 5 q) '(5 1 2 3 4 5) (append q (list 5)) '(1 2 3 4 5 5)What is Scsh? As stated before, scsh is a set of Scheme macros that allow you to interact with the operating system, even providing access to system calls. Scsh is different from other shell scripting languages because it works with Scheme and uses a syntax that many people are unfamiliar with. However, Scheme does provide recursion, abstraction, object orientation, and readability features, which make it perfect to use for systems programming or shell scripting. Keep in mind that scsh is not a “command language”; it is a systems programming language, meaning that it encompasses both a way to interact with the shell, and a way to interact with low-level functions of the operating system. A programming language is just a notation for expressing computation. Scsh has its own way of expressing commands, called S-expressions, which will be described in the next section. Ports A port, which appears throughout all scsh documentation, is like a file descriptor, from which you can read and write data. In scsh, there are certain functions that either require ports as arguments, or return ports. You will probably be using functions that return ports. When I’m programming in scsh, I usually need to parse output from programs or text-files. There is a specific function in scsh that allows you to run a program and have all output returned on a certain port. You can then read all the data from the port and turn it into any sort of data structure. I usually turn output into a list. Ports are also used when opening files. When a user calls the (open-file) system call, a port is returned and you can perform subsequent operations, such as reading and writing, on that port. Extended Process Notation Interaction with the operating system is done through S-expressions. S-expressions themselves are just data structures for representing complex data. They can be either byte strings or lists of S-expressions. Usually, while coding in scsh, you will be dealing with lists of S-expressions. An S-expression could look like this:
(belong (all (your "base")) (to "us"))This S-expression has no meaning and is only an example of the syntax. When analyzing this S-expression, you can see that belong is a function that acts on the return value of (all (your "base")) and (to "us")). Scsh has a notation, taking the form of S-expressions, for controlling UNIX processes, which are called process forms. An extended process form is a specification of a UNIX process to run, in a particular environment:
(pf redir_1 redir_n)where pf is the process form and redir_i are arguments. Listed below is a set of process forms that do the normal set of commands, such as creating a pipe, running a process in the background, redirecting output into the calling process or out to a file:
| — fork/pipe. & — Run a process in background. begin — Runs code in a fork. < — Open a file for read. > — Open a file for crease/truncate.The list below provides scsh functions and examples that are commonly used in shell scripting. Note that “>” means that we are working at the interactive scsh prompt: • (run . EPF) — Runs a process; returns an exit code. For example:
(run (echo Hello)) Hello• (run/sexps . EPF) — Run a process and return its output as a list of strings. This is very useful as shown later. This example executes ps | tail -1, transforming the output to a list for further parsing:
> (define output (run/sexps (| (ps) (tail -1)))) > output '(61661 v0 S 0:00.45 aterm -tr -tn xterm-color -bg black -fg \ white -sh 30 -font nexus) > (car output) 61661• (argv n) -> string — Returns the Nth argument given to the program. • (begin . EPF) — Runs code within a fork. For example: (begin (run (ls))) This forks a new process to execute ls. • (with-cwd directory . EPF) — Changes the current directory to directory and executes the other arguments. For example:
(with-cwd "/usr/src" (run (ls)))• (& . EPF) — Runs a process in the background. For example, compare these two commands. The first is a command written in Sh, the second is written in scsh. It is a simple operation and both commands get the last ten lines of /var/log/smb.log and grep for the string File.
Sh: tail -10 /var/log/log.smb | grep File Scsh: (run (| (tail -10 /var/log/log.smb) (grep File)))System Calls As noted before, scsh provides access to low-level system calls, like dup, sockets, open, close, write, read, etc. All of these operations are done through the ports. Below is a table of system calls supported in scsh. I took a sampling from each section (not all of the system calls are listed), and you can find the rest in the documentation. UNIX I/O • (dup fd/port) -> fd/port — This function duplicates a file descriptor or a port, returning the newly created descriptor or port:
> (define port (open-file "/etc/inetd.conf" open/read)) > (define port-dup (dup port))• (open-file fname flags [perms]) -> port — This function opens the filename referenced by fname and returns a port. Perms is an optional argument, allowing you to set permissions on the file you are opening. The are numerous flags you can use, which are listed in the scsh reference or in cheat.txt, however, the most commonly used flags are:
open/read open/write open/append > (define port (open-file "/etc/inetd.conf" open/read)) (open-input-file fname [flags]) -> port (open-output-file fname [flag perms]) -> port (open-fdes fname flags [perms]) -> integerThis returns a true file descriptor; ports are more useful and do all the same things. • (set-fdes-flags fd/port flags) — Allows you to set flags on a file descriptor or a port. You can find the flags in the reference manual:
> (define port (open-file "/etc/inetd.conf" open/read)) > (set-fdes-flags port open/async)• open/async — Enables Asynchronous writing on the port. • (pipe) -> [rport wport] — This returns two unused ports, rport and a wport. All output on rport is sent to wport. • (read-line [fd/port retain-newline?]) -> string or eof-object — When a file descriptor or port is specified, it returns the first line waiting on the port:
> (define port (open-file "/etc/inetd.conf" open/read)) > (read-line port) "# $FreeBSD: src/etc/inetd.conf,v 1.44.2.1 2000/03/25 22:09:59 \ jhb Exp $" > (read-line port) "#" > (read-line port) "# Internet server configuration database"• (read-string nbytes [fd/port]) -> string or #f — Returns a string from a specified port:
> (read-string 30 port) "# $FreeBSD: src/etc/inetd.conf"• (write-string string [fd/port start end]) — Writes a string to a specified port:
> (define port (open-file "/home/evms/.zshrc" open/append)) > (write-string "echo Boo" port)Locking • (make-lock-region exclusive? start len [whence]) -> lock-region — Returns a lock-region record. Records are similar to a struct in C:
> (define q (make-lock-region 0 30 50))• (lock-region fdes lock) — Locks a file descriptor according to the lock region created with make-lock-region. File System • (create-directory fname [perms override?]) — Creates a directory fname. • (delete-directory fname) — Deletes a directory fname. • (delete-file fname) — Deletes a file fname. Processes • (exec prog arg1 ...) — Executes prog with args:
> (exec "/bin/ls" "-al")• (fork) -> proc — Forks a new process and returns a process object. Process State • (umask) -> fixnum — Returns the current umask:
> (umask) 18• (set-uid uid) — Sets the processes UID:
> (set-uid 0)Signals • (signal-process proc/pid sig) — Signals a process whose pid is pid with a signal specified by sig. You can use almost any signal, but the most common ones are:
signal/kill signal/stop signal/alrm signal/hup (signal-process (cadr (run/sexps (| (ps aux) (grep inetd) \ (tail -1)))) signal/hup)The (cadr ...) evaluates to the pid of the inetd process. As you recall, run/sexps turns the output into a list and cadr takes the second element from the list and returns it. These system calls are the ones I use most. There are socket system calls in scsh, but they are complicated and won’t be explained in this article. The scsh documentation, however, has great examples on sockets. OO in Scheme Object oriented programming is becoming a dominant trend for systems programming, and it can also be used for administration tasks. For example, you could create a computer object, which has the locally defined functions ping, restore, reboot, and local variables that indicate the IP address and hostname. When used in a network environment, this script would be modular and powerful. I would use Scheme to write this because it has the capacity for OO. OO in Scheme is mainly done through the use of state variables and lambda. All variables in Scheme maintain state. Here, I will use the example of a piggy bank object. You can deposit, withdraw, and display the amount of change within the piggy bank. Of course, you can modify this code to do something that is actually useful:
(define (create-piggybank) (let ((balance 0)) (define (deposit amount) (set! balance (+ balance amount))) (define (withdraw amount) (if (<= (- balance amount) 0) (display "Error: you have no balance!") (set! balance (- balance amount)))) (define (print_balance) (display balance)) (define (dispatch operation) (if (eq? operation 'deposit) deposit (if (eq? operation 'withdraw) withdraw (if (eq? operation 'print_balance) print_balance)))) dispatch))See Figure 1. System Automation Scsh is especially useful for system automation. Below are some of the shell scripts I’ve created using scsh, comparing them to a script written in sh that performs the same function. A jail is similar to chroot, but with more restrictions. Jail is implemented within FreeBSD. This script initializes all of the jails within /jail. It is used on bootup. Within the /jail directory, note that the jails are named by their IP addresses rather than their hostnames. Bourne Shell:
#! /bin/sh JAIL="/usr/sbin/jail" for dir in $(ls /export); do echo $dir $JAIL /export/$dir $dir $dir /bin/sh /etc/rc doneScsh:
#! /usr/local/bin/scsh -s !# (with-cwd "/jail" (define (jail dir host ip) (run (/usr/sbin/jail ,dir ,host ,ip /bin/sh /etc/rc))) (for-each (lambda (j) (jail j j j)) (directory-files)))Comparison The sh version of the script looks substantially shorter and easier to read, but, there are substantial benefits in the scsh version. One benefit is the ability to define functions within regions like (with-cwd), which allowed me to keep the (jail) function hidden from users of the script and I think it makes it easier to read. (directory-files) is also much shorter than writing for dir in $(ls /export); do.... (lambda) is used to create anonymous functions in Scheme. As shown, I am passing a function as an argument to (for-each) to be applied to every item of the list (directory-files). Lambda is very powerful and can be used for a variety of purposes. Using lambda, you can write a scsh script that returns functions or creates functions from a template. This second shell script, similar to one in ‘‘Useful Scripts for Overworked Administrators’’ by Mark Prager (Sys Admin, June 2001) pings a list of servers to test their availability. If the servers are down, the administrator is contacted. Bourne Shell:
#!/bin/sh hosts="next-station postfix ns router" for i in $hosts; do if [ "$(ping -c 1 $i | grep "100% packet loss")" != "" ]; then /home/evms/sms "$i is down." echo "$i is down" | /usr/bin/mail kaworu fi donescsh:
#! /usr/local/bin/scsh -s !# (for-each (lambda (host) (define (alive?) (run/strings (| (/sbin/ping -c 1 ,host) (grep "100% packet loss")))) (if (alive?) '() (begin (run (/home/evms/sms ", host is down.")) (run (| (echo ", host is down") (/usr/bin/mail kaworu)))))) (list "next-station" "postfix" "ns" "router" "teqnix" "next" "logsrv"))This example shows that I am also employing abstraction by defining a function within lambda. This example is a bit larger than the sh script, but it is certainly more readable. Through the vnconfig program in FreeBSD, you can create mountable partitions through disk images. This scsh shell script accepts two arguments: the directory that you want to back up, and the path to where the disk image should be stored:
#! /usr/local/bin/scsh -s !# (begin (if (eq? (length (command-line)) 1) (run (echo "Usage: ./createbackup <tree> <file>For example, ./createbackup /usr creates a self-mounting image <file>
containing all files within /usr")) (let ((totalsize (car (run/sexps (| (du -c ,(argv 1)) (grep \ "total")))))) (run (echo "It appears that" ,(argv 1) "is" ,totalsize "K. \ Starting the backup process..")) (begin (&& (dd if=/dev/zero of=,(argv 2) bs=1k count=,totalsize) (vnconfig -s labels -c vn0 ,(argv 2)) (disklabel -r -w vn0 auto) (newfs vn0c) (rmdir /backup_mount) (mkdir /backup_mount) (mount /dev/vn0c /backup_mount) (cp -R -f ,(argv 1) /backup_mount) (umount /backup_mount) (rmdir /backup_mount)))) (run (echo "Backup complete."))))
This example illustrates the advantage of Prefix Notation. Instead of having to include a && after each command, I only have to include it once, which makes the script easier to read. Also, let is a very important constructor that allows you to define variables within a region so that it can not be altered by the programmer. This is another example of abstraction.
Conclusion
There are many shells available for the various *NIX operating systems. Each one encompasses both an interactive prompt and a batch mode where more complicated commands can be written. Most shell scripting languages are archaic and have limited access to the operating system. Scsh merges shell scripting and systems programming to produce easy to read, portable, and fast code. When comparing the scsh shell scripts to their sh counterparts, you can see the difference scsh makes.
References
Shivers, Olin, “A Scheme Shell”. 1994. MIT Laboratory for Computer Science.
Shivers, Olin, and Brian D. Carlstrom. “Scsh Reference Manual”. 1994. SCSH 0.5.2 Documentation.
Kelsey, R. and J. Reeves and M. Sperber. “The Incomplete Scheme 48 Reference Manual for release 0.57”, Scheme 48 0.57 Documentation.
Abelson, H., and G. J. Sussman and J. Sussman. “Structure and Interpretation of Computer Programs 2nd edition”. 1996. MIT Press.
IEEE Working Group. “IEEE Standard for the Scheme Programming Language”. 1991.
Evan Sarmiento is a tenth-grade student at Boston University Academy. He enjoys FreeBSD kernel hacking and network administration. He can be contacted at: [email protected].