Java Q&A

These days, a great deal of software development relies on previously composed components that must be glued together and packaged into an application. In such a world, where components may be poorly documented black boxes, it seems particularly important to provide programmers with the means to flexibly experiment with components during the development process.

In this article, I'll examine Skij, an interactive scripting language for the Java environment. Skij takes the form of a Scheme interpreter written in Java. The core language is extended to let Scheme programs manipulate arbitrary Java objects. Skij forms a platform for a number of other interactive tools for Java, including a web-based interface and a variety of graphic object inspectors. All of these interfaces run from within a standard Java VM, and can run alongside existing Java applications, such as debugging and scripting tools.

Skij, which I created while working at IBM's T.J. Watson Lab, is available from IBM's alphaWorks web site (http://www .alphaworks.ibm.com/tech/Skij/) for noncommercial purposes. It has been used by several hundred programmers on applications ranging from database manipulation and web servers, to component glue, education, and debugging.

Interacting with Java

Java can be considered to be a semidynamic language. The environment consists of self-describing objects that are managed by garbage collection. But because Java is strongly typed and inherently batch compiled, it does not readily support a flexible, improvisational style of programming. Fortunately, even though the Java language is rigid, the Java environment has enough run-time flexibility to permit interactive tools to be built on top of it.

Useful tools (like a screwdriver, for example) combine a handle for users to grasp and a blade or other part that transmits force to the work object. Just so, an interactive scripting tool for Java consists of two main parts: a language interpreter or other human-controllable interface, and a means by which that language can reach down into Java and perform arbitrary run-time operations.

In the case of Skij, the interpreter or "handle" is for the Scheme language. The "blade" part of Skij is a package called Dynvoke (short for "Dynamic Invoke"), which allows arbitrary run-time operations on Java objects. This facility is based on the built-in Java Reflection API, but because that API is rather low level, Dynvoke encapsulates it in a way that makes it suitable for interactive use.

Interacting with Java through a Scheme listener is quite powerful, but limited by the low informational bandwidth of a text-based interface. Once the basic interaction facilities are built, it becomes possible to conceive of alternative interfaces, such as a visual object inspector. These can be built using the tools described here.

How to Embed Scheme in Java

Scheme is in widespread use as a scripting and extension language. It is a dialect of Lisp and was initially developed by Guy L. Steele and Gerald J. Sussman in the 1970s. Scheme was designed to have a minimal language core with simple, clean semantics. Its features include lexical scoping, a single namespace for variables and procedures, natural support for higher order procedures, and space-efficient tail calling. (More information and various Scheme implementations can be found at http://www-swiss .ai.mit.edu/projects/scheme/.)

The basic design principles that Skij uses to integrate Scheme and Java are:

• Implement Scheme objects as Java objects; use existing Java classes where possible.
• Any Java object can be a Scheme object (and vice versa).

• Provide Scheme procedures for reaching down to Java functionality.

All Scheme values are represented as Java objects. Scheme has a fixed set of built-in object types, including strings, pairs, symbols, and a variety of numeric types. Where possible, existing Java types are made to serve for the built-in Scheme types, even though this causes small divergences from the Scheme standard. For instance, all numbers are represented in wrapped form using the Java.lang.Integer and java.lang.Double classes, and Scheme strings are represented using java.lang.String. Scheme vectors are represented by Java arrays. Skij defines its own classes for pairs, symbols, various types of procedures, and so on.

Skij extends Scheme by allowing any Java object (usually including null) to be a Scheme value, used as an element of a list, and so on. Internally, Skij typically uses variables with the static type of Object, and casts their values to specific types when necessary.

Ways to Use Skij

The Skij interpreter can be run on its own, or it can be run in combination with existing Java applications or components. There are several different modes of use:

• Application. Skij can be invoked as a standalone application (that is, it provides the main method to the Java VM). In this mode, users can enter Skij commands, access arbitrary Java classes, and start up multiple applications or applets from within Skij. Generally, the system console is used as the primary Scheme listener, but users can choose to create other listeners that appear in Java windows.

• Applet. Skij can run as an applet, using the Skijlet class. In this mode, the Scheme listener appears in a Java window contained in the browser. This mode is useful for demonstrating Skij, as well as for performing dynamic interaction in the context of the browser's JVM.

• Scripting. When Skij is started, it can be given an initialization command. In a scripting app, this command can be used to perform some fixed task rather than starting up the Scheme listener window.

• Component. Skij can be included as a component of a larger Java application, and called upon to perform computations or internal scripting functions.

• Debugging. Skij can be included in an application as a debugging facility. In this mode, the application would typically supply the main method to the JVM. A menu item or other user interface can be used to bring up a Skij listener that runs alongside the application, and can be used to inspect or modify application objects. This requires only minimal additions to the application.

The Reflective Primitives

The interface between Scheme and Java is centered around a set of seven primitive Scheme functions. These access the Dynvoke package to perform reflective operations on arbitrary objects:

• (new class args^*) creates and returns a Java object of the specified class, passing the arguments along to the appropriate constructor. class can be a symbol, string, or class object; see Example 1(a).
• (invoke object method-name args^*) invokes the named method on the object, with the specified arguments. Returns the value returned by the method, if any; see Example 1(b).

• (invoke-static class method-name args^*) invokes the named static method of class, which may be a class object or the name of a class, with the specified arguments. Returns the value returned by the method, if any; see Example 1(c).

• (peek object field-name) returns the value of the named field; see Example 1(d).

• (peek-static class field-name) returns the value of the named field; see Example 1(e).

• (poke object field-name new-value) sets the value of the named static field.

• (poke-static class field-name new-value) sets the value of the named field.

Dynvoke: A Higher Level Interface to Reflection

Java's reflection API gives a program access to the individual metaobjects representing classes, methods, and fields. While it allows a method to be invoked reflectively, it does not implement any method resolution scheme. In other words, there is no convenient way to say: "send object x message y with arguments a*" and have something reasonable happen, which is the level of abstraction needed for interactive scripting.

Dynvoke is a Java class that provides a higher level reflective interface to the Java environment. Listing One presents a simplified version of Dynvoke called SimpleDynvoke.

A Note on Types

There are two ways the notion of type or class is used in programming languages. The terminology used for these separate but related ideas is inconsistent and often confusing. According to the Java specification, a variable has a type, while an object has a class; see The Java Language Specification (JLS), by James Gosling, Bill Joy, and Guy L. Steele, Jr. (Addison-Wesley, 1996), Section 4.5.5. The distinction between these is sometimes hard to grasp:

• An object has a single class, but may be assigned to variables of possibly many different types.

• A variable can have an interface type, but no object can have an interface as its class.

• The notion of types includes primitive types such as int, but there are no primitive classes (although there are, confusingly, java.lang.Class objects that represent primitive types).

In Dynvoke, I deliberately blur the boundary between these concepts. For instance, method lookup in Java normally is controlled by the class of the target object and the declared types of the arguments. In a reflective invoke call, there are no types available, and so the classes of the argument objects are used instead.

The type/class terminology, while mandated by the JLS, is not clear for our purposes, if only because the run-time manipulable objects that represent types are themselves of class java.lang.Class. To make the relationships between these concepts clearer, I generally use the term "static type" to refer to the type of a variable, and "dynamic type" to refer to the class of an object.

An API for High-Level Reflection

The first step is to figure out what the API for high-level reflection should look like. Unlike the built-in reflection API, which is based around the internal structure of Java classes, the high-level API will be based on operations of interest to users. These are:

• Creating an object of a given class, with arguments passed to the appropriate constructor.

• Invoking a method on an object, given the method name and a set of arguments.

• Reading and writing fields of an object.

There are other things you might like to do at run time, such as define new classes and methods. This is possible but considerably harder, since it would require generation of Java bytecodes. Here, I restrict myself to operations that can be accomplished via reflection.

In general, these operations take as arguments a target object (except in the case of new object creation), a name of a method or field (again, except in the case of creation), and possibly some arguments. All are implemented as static methods of SimpleDynvoke. The arguments of these methods (except for the name) will be of type Object, so that any Java reference object can be passed in as a target or argument.

I'll start with a simple operation -- reading the contents of a field. The caller supplies the object and a field name (as a string), and the value of the field is returned, assuming it exists and is accessible. This is handled by the peek method:

public static Object peek(Object object, String fieldName) throws IllegalAccessEx ception, NoSuchFieldException {

Note that the code that actually tries to access the field can throw exceptions, which peek passes upwards to its caller.

The peek method merely provides some simple plumbing that makes reflection slightly easier to use. Still, it illustrates my strategy for the more complex parts of Dynvoke: First, find the appropriate reflective object, then make use of it. In this case, both parts of the task are pretty trivial.

Method invocation is harder, because the task of finding the right method to call is complicated. Using the method involves a bit more work as well, because you need to handle exceptions in the called method.

The method Dynvoke uses to call methods is invoke. Its API looks like this:

public static Object invoke(Object obj, String methodName, Object[] args) throws Throwable { ...

Since the methods called might throw arbitrary exceptions, it declares that it may throw Throwable, a class that includes all Java errors and exceptions. The low-level reflection API wraps exceptions in an InvocationTargetException -- Dynvoke unwraps this and throw the more informative original exception.

Finding the Right Method Object

Finding the right method is the hard part of dynamic invoke. This task is handled by lookupMethod, which takes as arguments the classes of the target and arguments, along with the following method name:

static Method lookupMethod(Class targetClass, String name, Class[] argClasses) throws NoSuchMethodException {

Method lookup is done according to a procedure that is essentially the same as that specified in the JLS, Section 15.11. The differences arise from the fact that some parts of the standard Java method lookup procedure take place at compile time, and some at run time. In this case, there is no compile-time information to work with, so all processing must take place at run time.

The method lookup proceeds as follows:

1. Obtain a list of all accessible methods from the target class using the getMethods() call.

2. Examine each method to see if it is appropriate for this call by checking that all of the following hold: (a) its name matches; (b) it has the correct number of arguments; and (c) the static type of the parameter for each argument matches the dynamic type of the actual argument. (The concept of matching types will be described later.)

3. From the appropriate methods, select using these rules: (a) If exactly one was found, return it; (b) if there is more than one qualifying method, the most specific is selected using mostSpecific, which implements rules from JLS 15.11.2.2 (if a most specific method can't be found, throw an exception); (c) if there are no appropriate methods, throw a NoSuchMethod exception.

This procedure works only for reference (nonprimitive) types. Extra work is needed to handle methods that take primitives or null values as arguments.

Matching Types

Step 2(c) in the aforementioned procedure refers to the idea of matching the dynamic type of an actual argument with the static type of a method parameter. This operation is performed by the matchClasses and matchClass methods. matchClasses iterates over two parallel arrays of class objects, calling matchClass on each pair and returning True if all pairs match. Since SimpleDynvoke doesn't handle primitives, the implementation of matchClass is simple -- it's just a call to the Class.isAssignableFrom method. Dynvoke uses a more complex method.

Object Creation

The process of looking up a constructor is similar to looking up a method, but somewhat simpler because there is no target object, and you only need to look at a single class for suitable constructors. The constructors are found via getConstructors, and argument matching takes place in the same manner as type matching.

In fact, a good deal of code in Dynvoke is duplicated, or nearly duplicated, to deal with both methods and constructors. This suggests that the Java Reflection API should have based both these classes on a single abstract class or interface.

Complications

The basic process of method lookup is complicated by a number of factors, including static methods, primitives, security issues, and changes in JDK 1.2. SimpleDynvoke ignores these issues, but the full-strength version of Dynvoke handles them.

Primitives. Methods and constructors may take primitives as arguments. By their nature, reflective APIs can only accept reference objects for arguments. Java provides wrapper classes for each primitive that support passing primitives where a reference object is called for (for example, the reference class Integer corresponds to the primitive type int).

As it happens, the low-level reflection API does a good job of translating between primitives and their wrapped forms. For instance, a method like boolean isPrime(int n) can be called through reflection by passing in an object of class Integer. It will then return an object of class Boolean. The reflection API does all of the wrapping and unwrapping automatically.

However, the presence of primitives seriously complicates method lookup. The full-blown version of Dynvoke has an extended version of matchClass that knows how to match wrapped values in arguments with primitive types declared in methods. This is not a matter of simply matching up Integer and int, because methods that take an int arg may be legitimately called with a short- or byte-valued argument. Thus, the code for matchClass has to know about these relations, which are called "widening primitive conversions" and are described in JLS 5.1.2.

Null Arguments. Any method argument with a reference type may be passed the null value instead of an object. But SimpleDynvoke can't handle null values, since it tries to call getClass on each argument. Dynvoke handles nulls by checking for them when the class array is generated, and uses a special value in the class array (which could be anything, but happens to be the class object for java.lang.Void) to indicate them. matchClass is also extended so that Void will match any argument with reference type.

Static Methods. Static methods have no target object. However, in all other respects they are treated like normal methods. All we need for them is an alternate interface to invoke:

public static Object invoke(Class myClass, Object obj, String methodName, Object[] args) { ...

This method allows the caller to specify a target class, instead of relying on obj for that information. In the case of a static call, obj is ignored and would typically be null.

Security and JDK 1.2. In JDK 1.1, only public methods and fields of public classes may be accessed through reflection. This severely restricts the usefulness of reflective tools. The restriction might be reasonable for scripting-like applications, since a script should probably only be able to access the public interface of an object. But for debugging and rapid prototyping, this restriction is annoying. In cases where public methods return results that are in nonpublic classes, the restriction can be a fatal interference for a reflective scripting language (for example, the enumerators returned by the getElements method of java.util.Hashtable are nonpublic in some versions of the Sun JVM).

In JDK 1.2, Sun introduced extensions to the reflection API that allow nonpublic constructors, methods, and fields to be used via reflection. In this new scheme, each of the classes for these reflective objects is a subclass of java.lang.reflect.AccessibleObject. This class provides a flag that indicates whether the object can be used for reflective access, and a method (setAccessible) that can set the flag. When running under JDK 1.2, any time Dynvoke might return a possibly nonpublic reflective object, it will set it to be accessible first.

Given this change, we now need to look at all methods and constructors when doing a lookup, not just public ones. For methods, I have been using getMethods() to obtain a list of all possible methods. This method of java.lang.Class returns all (and only) public methods, including inherited ones. The other way to get a hold of methods from a class is with getDeclaredMethods(), which returns all methods, including nonpublic ones, but does not include inherited methods. So, in JDK 1.2, lookupMethod uses getDeclaredMethods and must do its own walk up the inheritance tree to find possibly inherited methods.

Access to Other Java Facilities

In addition to object manipulation facilities, Skij includes features that provide access to other features of the Java environment such as threads, events, synchronization, and exception handling. Because Scheme is capable of representing procedural objects, these features were implementable without alterations or extensions to the basic Scheme language (that is, no new syntax or special forms are required). Instead, they rely on combinations of new primitives, macros, and specialized Java classes.

For example, event handlers in Skij are handled by a class called GenericCallback that implements all of the AWT Listener interfaces. A GenericCallback object encapsulates a Skij procedure of one argument. When the callback object receives an event, the procedure is applied to the event; see Example 1(f).

Synchronization, in contrast, is handled by an additional primitive procedure that takes as arguments an object to lock and a Scheme procedure of no arguments (sometimes called a "thunk"):

(%synchronize object thunk)

The thunk procedure is then run within a context in which the object is locked. This is a good illustration of how Scheme's first-class procedures can reduce the need for special language syntax and features.

Other Interfaces

The Skij listener is a text-based interface to Dynvoke's reflection capabilities. It is possible to build other sorts of interfaces that allow users to take greater advantage of these capabilities. For instance, a tabular inspector can be used to view objects and navigate object structures; see Figure 1. This inspector gathers all available information about a given object, using reflection and taking advantage of Java's convention that zero-argument methods starting with get or is are accessors.

A typical use of the inspector is to traverse the object structure starting from a known object until the object of interest is found. Often you want to perform operations on this object. Skij includes a variable, inspected, that is always bound to the object contained in the topmost inspect window. This lets the user refer to an object obtained via the inspector from a Skij listener.

The tabular inspector shows all known information about a single object. Viewing patterns of relations between objects requires a different kind of graphic representation. Skij provides a graph inspector that can illustrate such relationships; see Figure 2.

Both of these visualization tools are written in Skij and are included in the distribution. There are ways to transfer object references between the inspectors and the Skij listeners, so that if an interesting object is found with the inspectors it may be assigned to a Scheme variable and controlled more closely.

Related Work

There are other scripting tools available for Java, such as Jacl (a Tcl interpreter; http:// www.scriptics.com/products/java/) and BeanShell (which interprets a Java-like language; http://www.ooi.com/beanshell/). There are also other Java-based Scheme interpreters, including SILK (http://www .norvig.com/SILK.html) and Kawa (http:// www.gnu.org/software/kawa/), but they lack many of Skij's Java integration features -- notably dynamic method invocation -- and are therefore less useful as interactive tools.

Conclusion

In the end, what interactive scripting tools such as Skij buy you is a better feeling for the behavior of program objects, which translates to lower cognitive overhead and faster development times. Once you are used to having these tools, you won't want to live without them.

DDJ

Listing One

import java.lang.reflect.*;
import java.util.Vector;
class SimpleDynvoke {
  // A simplified version of the real Invoke. Doesn't handle statics, 
  // constructors, primitives, java2, caching, null arguments
  public static Object invoke(Object obj, String methodName, 
                 Object[] args) throws NoSuchMethodException, Throwable {
    Class myClass = obj.getClass();
    Class[] signature = classArray(args);
    Method method = lookupMethod(myClass, methodName, signature);
    if (method == null)
      throw new NoSuchMethodException("no applicable method");
    else {
      try {
        // actually invoke the method
        return method.invoke(obj, args); }
      // pass exceptions upward.
      catch (InvocationTargetException e) {
        throw(e.getTargetException()); }
    }
  }
  static Method lookupMethod(Class target, String name, 
                      Class[] argClasses) throws NoSuchMethodException {
    // first try for exact match
    try {
      Method m = target.getMethod(name, argClasses);
      return m;
    }
    catch (NoSuchMethodException e) {
     if (argClasses.length == 0) { // if no args & no exact match, out of luck
        return null; }
    }
    // go the more complicated route
    Method[] methods = target.getMethods();
    Vector goodMethods = new Vector();
    for (int i = 0; i != methods.length; i++) {
      if (name.equals(methods[i].getName()) &&
          matchClasses(methods[i].getParameterTypes(), argClasses))
        goodMethods.addElement(methods[i]);
    }
    switch (goodMethods.size()) {
    case 0: {
      return null; }
    case 1: {
      return (Method)goodMethods.firstElement(); }
    default: {
      return mostSpecificMethod(goodMethods);
    } }
  }
  // 1st arg is from method, 2nd is actual parameters
  static boolean matchClasses(Class[] mclasses, Class[] pclasses) {
    if (mclasses.length == pclasses.length) {
      for (int i = 0; i != mclasses.length; i++) {
        if (!matchClass(mclasses[i], pclasses[i])) {
          return false; }
      }
      return true;
    }
    return false;
  }
  static boolean matchClass(Class mclass, Class pclass) {
    return mclass.isAssignableFrom(pclass);
  }
  static Method mostSpecificMethod(Vector methods) 
                                           throws NoSuchMethodException {
    for (int i = 0; i != methods.size(); i++) {
      for (int j = 0; j != methods.size(); j++) {
        if ((i != j) &&
            (moreSpecific((Method)methods.elementAt(i), (Method)methods.elementAt(j)))) {
          methods.removeElementAt(j);
          if (i > j) i--;
          j--;
        }
      }
    }
    if (methods.size() == 1)
      return (Method)methods.elementAt(0);
    else
      throw new NoSuchMethodException(">1 most specific method");
  }
  // true if c1 is more specific than c2
  static boolean moreSpecific(Method c1, Method c2) {
    Class[] p1 = c1.getParameterTypes();
    Class[] p2 = c2.getParameterTypes();
    int n = p1.length;
    for (int i = 0; i != n; i++) {
      if (!matchClass(p2[i], p1[i])) {
        return false;
      }
    }
    return true;
  }
  // given an array of objects, return an array of corresponding classes
  static Class[] classArray(Object[] args) {
    Class[] classes = new Class[args.length];
    for (int i = 0; i != args.length; i = i + 1)
      classes[i] =  args[i].getClass();
    return classes;
  }
}

Back to Article