An XML Document to JavaScript Object Converter

By Alexander Hildyard

One of the ironies of Web technology is that many of the most interesting advances, such as the Document Object Model (DOM), Dynamic HTML, and style sheets, cannot be fully taken advantage of by Web developers unless they decide to target a specific browser vendor and version. Extensible Markup Language (XML), which is fast gaining currency as the industry standard for Web-based data transmission, is the latest in the parade of tempting technologies that demand patience from Web developers interested in client-side deployment and require creative thinking to apply in a diverse population of browsers.

At the time of writing, only Microsoft had shipped an XML parser with its browser (Internet Explorer 4.x), thereby limiting prospective Web-based XML clients to a proprietary rather than an open solution, despite the fact that XML itself represents an open standard. An extensive infrastructure of SGML tools exists at the server level as middleware with which to prime, filter, annotate, and parse XML. However, to date, the bottleneck in exploiting XML's open and generic standard remains at the level of the Web-based client, which has only a limited interface with which to interpret XML-encoded documents, and a slow one at that.

I have come up with an approach that brings some of the benefits of XML-based documents to non-XML browsers. My workaround is a server-side conversion of XML documents to JavaScript code; this code gets interpreted by the browser and results in a data structure roughly equivalent to the parse tree that would have been produced by an XML-enabled browser. Transforming XML documents from tag-stream to DOM provides a similar benefit of increased accessibility for the data consumer that moving data from databases to XML data sources provides for data producers. With XML represented at the level of the DOM, Web-based consumers are freed from both the need for an XML parser and also from the need to have direct access to original XML data sources. Applets, scriptlets, ActiveX controls, and other client-side components have the same programmatic access to browser-based XML documents as they have to the rest of the browser's DOM.

As it turns out, my workaround offers significant advantages over a pure XML approach: It's a lot faster, and the code to manipulate XML-derived objects is cleaner and more concise.

Current IE 4 Limitations

The XML parser in IE 4 takes the form of a small embeddable applet (xmldso.class) that exposes XML documents as a tree-based set of collections, each collection containing child collections and/or elements as data/value pairs. By traversing the tree, it's possible to enumerate each collection in turn.

While this is great as an initial bridge between XML and the browser's own internal DOM, there are reasons why the Microsoft applet should only be seen as a first step towards accessing an XML document's underlying data. For practical considerations, it often will make sense to recast an XML object tree as a static set of JavaScript data objects.

XML obviously has the benefit of being a "visual" representation of data. Therefore, transforming it into the "programmatic" data represented by scripted data objects might seem a backward step. However, at least in the context of the browser, the justifications for doing so seem to outweigh the costs.

The first of these is that the Microsoft applet works only in IE 4, while a large subset of JavaScript will work equally well with Netscape 2.x and IE 3.x and later. Thus, JavaScript objects can provide both backward compatibility with the older generation of browsers, as well as offering a programmatic method for abstracting XML data and making it more accessible.

The second justification is simply the processing overhead of having to navigate such a structure as a tree, via the browser's JavaScript-to-Java requests for information and the applet's Java-to-JavaScript return of that information. Table 1 presents comparative timings for the Microsoft JScript parser applet to extract all the information from an XML document and for the same document to parse as a static JavaScript data set (using the XML/DOM converter). Even though the JavaScript document is more than twice as big as the XML data source (with the trial data in question), it renders around 700 times faster for a 30-entry data set, and about 3700 times more quickly for a 300-entry data set. With larger data sets, this figure will soon inflate exponentially to 10,000 or even 100,000. Does this mean that JavaScript is fast? Not at all. Like any interpreted language, it will never offer the performance of a compiled one. What it means is that the Microsoft applet is both slow and unscalable -- in fact, prohibitively slow for any data set containing more than 50 pieces of data. Fifty pieces of data is barely enough to hold ten book records containing first name, surname, place of publication, date, and title.

And thirdly, due to the structure of XML, without parsing an entire document, it is often difficult to form a clear view of exactly what data is made available and where we should search for it. To illustrate this, consider the XML document in Example 1.

In the absence of an explicit document type definition (DTD), XML has no obligation to ensure that any particular collection contains the same elements, or even that they occur in any particular order. You'll need to look at all the subcollections of a specific collection within a particular XML document to decide what potential information is or isn't available. In this example, we would guess from looking at the first AUTHOR collection that an AUTHOR contained three elements: FIRSTNAME, INITIAL1, and SURNAME. But we would be mistaken, since our second AUTHOR collection adds the element EMAIL and our third reveals the additional element INITIAL2 and the subcollection SPECIALFLAGS.

On the one hand, XML's rendering of this data is highly economical, since it only needs to specify such data as is relevant to a given collection; this, in turn, cuts down on the size of the resulting document and increases traversal speed. But on the other hand, it also means that, for example, it would be impossible to walk an XML document and prime a nonhierarchical HTML table with its data in a single pass, because you'd have no idea how many fields were present within each element collection to start with. What we really want is to know in advance everything that a collection will contain, without the obligation of putting that description into the XML document itself.

Converting XML to DOM

Accordingly, much of the XML converter's job involves trying to evaluate exactly what elements and subcollections hang off a particular collection, while making only a single pass of the XML document. It does this by constructing the complete path from the root of the document to each leaf-node (for example, an element that contains only data and has no children). Having done so, it checks whether each path is "unique," and, if it is, stores it for later evaluation. For the hypothetical document in Example 1, this would generate a set of entries of the form:


AUTHORS . AUTHOR . FIRSTNAME
AUTHORS . AUTHOR . INITIAL1
AUTHORS . AUTHOR . SURNAME
...and so on.


By recasting the AUTHORS collection into such a format, it becomes clear that each collection can be described as the set of data available under a given element tag descriptor at a particular depth within the XML document, and that this description will be exhaustive and complete. Thus, the AUTHORS collection contains no elements and two subcollections, AUTHOR and PSEUDOAUTHOR; the AUTHOR collection contains the elements FIRSTNAME, INITIAL1, INITIAL2, SURNAME, EMAIL, and SPECIALFLAGS; and the SPECIALFLAGS collection contains two elements, FLAGS1 and FLAGS4. With such a representation of its data fields, we're now in a position to encapsulate that document within a set of JavaScript objects, each of which contains a fixed set of properties, by iterating backwards from the leaf-nodes of each unique path of descent. This is done in order to construct the immediate container for each set of elements.

For example, FLAGS1 and FLAGS4 share the parent SPECIALFLAGS, so we can express this in JavaScript as an object called SPECIALFLAGS that exposes two objects called FLAGS1 and FLAGS4 respectively:


function SPECIALFLAGS
{
this.FLAGS1 = new Array();
this.FLAGS4 = new Array();
return this;
}


With every element in our XML document expressed either as an array or a new JavaScript object, we can start to build our original document into its DOM-based representation with code like:


document.AUTHORS = new AUTHORS();
document.AUTHORS.AUTHOR[ 0 ] = new AUTHOR();
document.AUTHORS.AUTHOR[ 0 ].FIRSTNAME = "Alexander";
...


Here, we make use of JavaScript's capacity to add arbitrary properties and methods to the DOM. The XML document is converted into a flattened tree or an array-of-arrays placed at top level within the DOM's active document. You can see a full rendering of this document as JavaScript objects in Example 4. [Ed. Note: Examples 2, 3, 4, and 5 are available online, as are Listings Two and Three; see "Source-Code Availability" on page 3. Listing One appears on pages 64 and 66.]

Parsing XML Using xmldso

Microsoft's XML data-source object exposes XML documents as a set of element collections and, with the help of a simple "stack" object, such documents can be parsed recursively with very little code at all. A bare-bones parser might be nothing more than Listing Two; its output is provided in Example 2. In this case, stack x is used to iterate recursively through the XML document, pushing each collection in turn in a breadth-first traversal of the document. Popping the top rather than last element of the stack each time ensures that we keep the document the "right way up" (for example, element collections are rendered in the order in which they appear in the XML document).

While it might seem more logical simply to pop the last element each time and change the loop that stacks the child elements so that they are stacked back to front, there seems to be a bug in the xmldso applet that causes elements to be omitted if you try this. Despite tests with various versions of IE 4, I haven't yet found a way around this.

The bare-bones parser in Listing One is fine to give us a picture of what's in the XML document, but unfortunately, since it's stateless, it gives us no clues as to the internal hierarchy of that document (what contains what). To achieve that, we need to know where we are within the XML document at each moment, and we can only know that if we can remember where we've already been.

The easiest and simplest way to do this is to set up a parallel stack that accumulates the nodes we've already visited. Listing Three does just this, and its output is provided in Example 3, both also available online. Stack x iterates through the document exactly as before, but while it does so, stack y, meanwhile, builds up an incremental description of that document in terms of its object hierarchy. Each node is checked to see if it exists uniquely at that level within the document; if not, an array of such nodes is generated, and the node in question is placed at its correct index position within that array. This is done to cope with the hypothetical situation:


<AUTHOR>
<SURNAME>Hildyard</>
<SURNAME>Emberton</>
<SURNAME>Twilite</>
</>


Failing to assign an array of nodes for SURNAME in this instance would simply cause the same single field, SURNAME, to be successively assigned the values Hildyard, Emberton, and Twilite within the relevant instance of its AUTHOR container, which is clearly not what we would want. Therefore, for each collection, numInstances holds the number of instances of each subelement, and thisInstance identifies the index of that subelement beneath its immediate parent -- the node we're processing.

We arrive at Listing One by putting Listing Three together with the relevant code necessary to identify unique paths of descent and, thereby, to generate JavaScript wrapper objects for the contents of our XML documents. Most of the work here takes place in the final section, which calculates and renders the JavaScript container objects. Basically, the depth of each "unique" node is used to decide which parent (if any) it belongs to, and sets of immediate child nodes are grouped under each parent. (See Example 5, available online, for a visual walk-through of this breadth-first traversal.)

Writing out the respective JavaScript object wrappers is straightforward, and takes place after parsing has been completed at each level: The first element of each stack in newTerms is the name for a JavaScript container, and the rest of the elements in that stack are the set of objects encapsulated by that container. The first term is popped and rendered as a function declaration, and then the rest of the stack is popped and rendered as properties.

Since newTerms is reassigned each time parsing moves on to another level, we escape the danger of some XML tagname "shadowing" the same tagname at a different level. Of course, this has the side effect that if a particular tagname occurs at multiple depths within our XML document, we will generate a fresh JavaScript wrapper for it at each level; but the worst this does is make our JavaScript structures a little larger than optimal -- a trade-off that seems worthwhile when faced with the needlessly complicated alternative of trying to work out every dependency in advance. As a result, the converter will cope correctly with collections that appear at multiple levels, for example SURNAME in the following fragment:


<AUTHOR>
<SURNAME>Emberton</>
<CITATIONS>
<SURNAME>Hildyard</>
<SURNAME>Betts</>
</>
</>


It also correctly renders recursively self-descriptive collections like AUTHOR in the fragment:




<AUTHOR>
<FIRSTNAME>Alexander</>
<CITATIONS>
<AUTHOR>
<SURNAME>Hildyard</>
</>
</>
</>


Finally, as mentioned earlier, if the same element (rather than element collection) appears multiple times within a given collection, the converter generates an array, giving each instance of the element a logical index. To check whether an element is an array or not, it's sufficient for end users simply to test the length property of the appropriate data field. For example, to determine whether SURNAME was expressed as an element or an array within a set of AUTHOR collections, you might write a script like:

<br>
var iterator = 
<br>
document.AUTHORS.AUTHOR[ 0 ]
<br>
.SURNAME;
<br>
if ( iterator.length != null )
<br>
{
<br>    
// Iterate array of 
<br>    
//surnames and display
<br>    
for ( i = 0; 
<br>

i < iterator.length; i ++ )
<br>          
alert( iterator[ i ]);
<br>
}
<br>
else
<br>
{
<br>    
// Not an array; 
<br>    
// display surname
<br>    
alert( iterator )
<br>
}
<br>

Note that length must be explicitly tested against "null" rather than against "zero" or "false." "Null" is specifically reserved by JavaScript for variables that haven't been defined (although they may have been declared).

Running Listing One on our now-familiar XML document yields Example 4. The document is broken up in exactly the same way as it was in Listing Three; but since the JavaScript containers have also been generated automatically, you can simply cut and paste the resulting output into an HTML or script file of your choice. You could then process that file with IE 3, Navigator, or, indeed, any browser that supports JavaScript (but doesn't support XML) in order to have access to the data that your original XML data source had contained.

While the code presented here is in the form of a stand-alone HTML page with a single JavaScript function, for practical purposes you'll probably want to abstract it as a .js script, and then include it in any page that needs to render XML at the document-object level. Note that after conversion you may need to rename any objects generated from XML tags that include arithmetic operators (such as a plus or minus) or that contain other escape sequences prohibited by JavaScript in object declarations.

Conclusion

XML is not ready for prime time on the browser population of the public Internet. But if you just want to get your foot on the XML bandwagon, want exclusively to target IE 4 clients, and have a tiny database of less than 50 items, you can use the Microsoft XML parser. If you also want to target Netscape Communicator clients, or want backward support for IE 3 and Netscape Navigator 3 browsers, use my converter. For anything else, it's probably best either to wait before committing yourself to XML, since Microsoft's Active Data Objects (ADO) and Java's JDBC are both compelling reasons for keeping ODBC-compliant databases in their present format, providing reliable and comparatively fast data exchange between server and client.

Alternatively, it may well be more forward-looking to forget about parsing XML yourself and skip straight to XML middleware: Take a look at Allaire's Web Distributed Data Exchange (WDDX), which uses XML-based "packets" to serialize database information at the server into WDDX modules, and deserialize that information at the client into JavaScript objects in a fashion very similar to our converter. On the plus side, the WDDX interface has already been implemented for JavaScript, Cold Fusion, Perl, and COM, and a Java version is forthcoming. Furthermore, WDDX is fast, effective, scalable, platform-neutral, requires no knowledge of XML, and it's free.

Another cross-platform solution is webMethods' Web Interface Definition Language (WIDL) (see "Lab Note," Web Techniques, November 1998). While WDDX provides what is essentially a transport riding on HTTP, WIDL in contrast offers more of a static interface to existing XML and HTML data, although it can also be distributed, and invoked via C/C++, Java, JavaScript, VB, and other languages. Finally, the Information and Content Exchange (ICE) standard (promulgated by an industry consortium that includes Vignette and Inso) offers a protocol geared specifically towards e-commerce, and can give you secure Web-based content syndication. In the meantime, there's little doubt that "pure" browser-based XML parsers will improve, but for the moment Microsoft's solution must remain more of a trial run and showpiece than a feasible means for getting XML from a server's database to a client's browser.

(Get the source code for this article here.)

---

Alexander is a freelance author, Web developer, and consultant. He can be contacted at [email protected].