XML-Binary Optimized Packaging

Andrey is a software developer in the Fixed Income division of Goldman Sachs & Company. He can be contacted at andreybutov@ yahoo.com.

For several years now, the development community has followed the emergence of XML-based solutions to common data-representation problems. As a metalanguage, XML is a success. A cursory search reveals several of the more popular custom languages utilizing XML as a metalanguage, including MathML for representing mathematical concepts [1], Scalable Vector Graphics (SVG) for representing two-dimensional vector and raster graphics [2], and Really Simple Syndication (RSS) [3] for distributing news headlines and other relatively frequently updated data such as weblog postings, for consumption by, among other things, RSS aggregators.

While most XML-derived languages are still used mostly for specialized representations of textual data, not all data domains are suitable for text-based representation. Consequently, there is a growing need to embed binary data into XML documents—in most cases, mixing binary data with plain-text XML tags for contextual description. While there is no standard for doing this as of yet, several approaches have had more success than others. In particular, SOAP [4] has long been used as a means of exchanging structured information, but there are numerous other proprietary implementations of technologies meant to overcome the binary-data dilemma.

In January 2005, the World Wide Web Consortium released a document presenting the latest version of its recommendation of a convention called "XML-binary Optimized Packaging" (XOP). The recommendation presents a way of achieving binary data inclusion within an XML document. There are several common arguments against pursuing the path laid out in the recommendation that are worthy of exploration; however, in its recommendation of XOP, the W3C presents a valid and interesting piece of technology worthy of examination.

The Current State of Things

The W3C's XOP proposal stems from a long-standing need to encode binary data in XML documents, which are, in nature, text-based animals. Before exploring the W3C proposal itself, take a look at some of the problems surrounding the inclusion of binary data in XML documents, and how these problems are currently being circumvented.

XML is usually presented as a way of describing text data within the context of a well-defined document containing metainformation (which is also text based) meant to bring some structure and form to that text data. There are, however, several domains that do not lend themselves nicely to being represented with textual data only. Most of these domains include the need to represent sound or video data in one form or another. In fact, the W3C itself has outlined several use cases that display a need to encode binary data in an XML document [5].

Assume that you want to somehow include Figure 1 in a transmission of an XML file describing a context in which the picture plays some role. You cannot simply place the binary data composing this picture in between two tags, as in Example 1(a), then transmit the XML file. The problem stems from the "text-centric" nature of XML. Binary data such as this can contain not only null characters (0x00 bytes), but can contain other string sequences, such as </ (0x3C, 0x2F), which can be detrimental to XML specifically. Both the null byte and the string sequence representing an XML closing tag would cause the XML parser on the other side of the transmission to improperly interpret the file. If you're lucky, the parser would catch the error because the XML file itself would no longer be properly formed; otherwise, the binary data itself could be improperly interpreted, causing a relatively easy-to-spot problem of improper syntax turn into an issue dealing with data integrity (such as in the case of transmitting binary financial data rather than photographs). By the way, the 2-byte sequences previously mentioned are only two culprits. They are enough to make the point, but there are others.

There are, in fact, several approaches to circumventing the binary inclusion problem. One popular (albeit naïve) approach is to place the binary segment into a <!CDATA]]> tag [6]. Developers new to XML tend to think of this tag as a panacea, when in reality, using CDATA only mitigates the problem caused by directly embedding binary data between two arbitrary tags. In this case, while it is true that the data inside the CDATA section will not be validated as XML, CDATA sections cannot be nested, and thus, instead of crossing your fingers and hoping your binary data does not contain a null byte, you cross your finders and hope that the binary data does not contain the sequence ]]>. (Early in my career, I was part of a group of developers who used this exact method to implement a piece of technology responsible for delivering binary financial data inside an XML document via a TCP connection to be displayed in a Flash applet through a web browser. At the risk of nullifying my point, I do not remember ever having an incident of the XML parser throwing a fit over an accidental ]]> sequence—but are you willing to take the chance with your application?)

Another approach is to encode the binary data into some string representation. In fact, the XML Schema [7] defines the base64Binary type that can be used for this purpose, as in Example 1(b). This approach is widely used, not only because of the fact that it effectively circumvents the "bad bytes" problem of directly embedded binary data, but also because of the simplicity of implementation. There are many free and shareware utilities that can be used to Base64-encode data. It does, however, have its drawbacks—one issue dealing with space, and one dealing with time (and all this time you thought sacrificing one would bring benefits from the other). Encoding binary data in base64Binary typically produces output that is about one-third larger than the original binary data [8], and at the same time, it takes time to both encode the data at the source, and decode the data at the destination, although the cost there would depend on a number of factors.

Yet another approach to the binary inclusion problem is to simply not include the binary data at all. Instead, a link (typically in the form of a URI reference) is presented instead of the binary data, while the data itself is kept at the referenced location. Example 1(c) presents an example of this. This approach has the advantage of not only bypassing all the issues of direct binary data embedding and encoding size/time performance hits, it also allows for the binary data itself to be modified at the referenced source without retransmitting the XML file containing the reference (although this could also be a bad thing, depending on context).

Indeed, there are various current technologies that implement one or, more typically, a combination of these approaches to enable inclusion of binary data in XML documents. Probably the most common technology that addresses this issue is Simple Object Access Protocol (SOAP) [4], or more accurately, SOAP with Attachments (SwA) [9], which utilizes a system of references in conjunction with MIME to deal with binary data in XML. An example of another similar technology is Microsoft's Direct Internet Message Encapsulation (DIME) [10]. Both of these technologies warrant pages (books?) of additional description, and fall outside the scope of this article.

W3C Recommendation

The first document in the chain of XOP-related publications was a Working Draft published in February 2004. Public comments on the first W3C Working Draft of XOP were requested at the time of publication [11] and a W3C public archive of the related messages is available for browsing at http://lists.w3.org/Archives/Public/ xml-dist-app/. Subsequent comments on the topic resulted in one more Working Draft [12], a Candidate Recommendation [13], a Proposed Recommendation [14], and finally, the W3C Recommendation serving as the topic of this article [15].

The recommended convention proposes a more efficient way to package XML documents for purposes of serialization and transmission. An XML document is placed inside an XOP package (see Figure 2). Any and all portions of the XML document that are Base64 encoded are extracted and optimized. For each of these chunks of extracted and optimized data, an xop::Include element is placed to reference its new location in the package. The recommendation makes a note that the encoding and eventual decoding of the data can be part of managing the XOP package—meaning that upon encoding, the internal links are automatically generated, and upon extraction, the links are automatically resolved, and the data can even be presented in its original intended format (sound, picture, and so on [15]).

At its core, the recommendation proposes a rather simple idea. Binary data can be included alongside plain-text XML data, and the XOP package processing would take care of the optimization (which promises to result in a much smaller dataset than the equivalent Base64-encoded data), the internal references would be resolved, and all levels of software above the XOP package management would not have to worry about managing the binary data either on the encoding or the decoding side.

An additional benefit, which I believe would drive this technology forward in comparison with DIME or SwA, is that existing XML technologies would continue to work with the XOP package data. This includes already popular XML-based technologies such as XQuery [16] for managing collections of data, XSLT [17] for content transformation, and even XML Encryption. In fact, XML Encryption [18] is referenced in the XOP recommendation itself as the solution concerning issues of security in XOP.

Community Feedback

The W3C recommendation (as well as its predecessors) raised some common arguments in the development community, including one stemming from a misunderstanding of the recommendation:

• What the W3C recommendation is NOT. The recommendation does not propose a binary format for XML documents. Although the W3C XML Binary Characterization Working Group does exist for the purpose of exploring the benefits of forming a binary protocol for XML, the recommendation of XOP focuses on a different issue entirely. XOP is merely a way of serializing XML in a way that would make it more efficient to include Base64-encoded data.
• What about the purity of XML? This argument is akin to people arguing that ANSI C ruined the purity of K&R C. A programming language (and in this case, metalanguage) remains alive partially due to its ability to adapt to the needs of its users. Any piece of technology that remains too rigid to evolve with those needs eventually dries up. It is precisely the flexibility of XML that has put it in the place it stands today.
• What about a simpler solution? Gzip anyone? Using gzip or some other compression scheme on XML data only has the potential of mitigating the size of the data. However, with this approach, the data is no longer human/parser readable, as would be the case with XOP, where only the Base64-encoded data is optimized and stored away, leaving the rest of the content of the XML well formed, self describing (in as much as XML is self describing), and in a way still able to be manipulated by various XML-focused tools. A compression solution would also not allow large chunks of binary data to be placed at the end of the file, leaving the parser with no choice but to deal with the entire data set upon decompression.
• Optimization of Base64 types only? This is actually a valid point. The recommendation does imply that XOP is currently enabled to only optimize Base64-encoded data. If the recommendation does evolve into some sort of a formal specification, and the development community picks it up as a practical solution, it would not be at all surprising if the demand grows for XOP to remove this limitation. This remains to be seen.

Conclusion

The existence of SwA, DIME, and countless proprietary technologies is evidence that there is a need to solve a relatively long-standing problem in the development community. The W3C Recommendation attempts to address this problem in a way that would put the burden of dealing with binary data inclusion into the specification of XOP rather than keep it at the application level, which in the long run results in various proprietary solutions, such as the case with SwA and DIME, and produces many forked roads. XML-binary Optimized Packaging is a sound, well thought-out idea that, in spite of common arguments, deserves consideration.

References

1. David Carlisle, Patrick Ion, Robert Miner, and Nico Poppelier, Editors. Mathematical Markup Language (MathML) Version 2.0 (Second Edition). World Wide Web Consortium, W3C Recommendation, October 21, 2003. http:// www.w3.org/TR/MathML2/.
2. Jon Ferraiolo, Fujisawa Jun, and Dean Jackson, Editors. Scalable Vector Graphics (SVG) 1.1 Specification. World Wide Web Consortium, W3C Recommendation, January 14, 2003. http://www.w3 .org/TR/SVG11/.
3. Gabe Beged-Dov, Dan Brickley, Rael Dornfest, Ian Davis, Leigh Dodds, Jonathan Eisenzopf, David Galbraith, R.V. Guha, Ken MacLeod, Eric Miller, Aaron Swartz, and Eric van der Vlist. RDF Site Summary (RSS) 1.0, 2001., http://web.resource.org/rss/1.0/spec.
4. Martin Gudgin, Marc Hadley, Noah Mendelsohn, Jean-Jacques Moreau, and Henrik Frystyk Nielsen, Editors. SOAP Version 1.2 Part 1: Messaging Framework. World Wide Web Consortium, W3C Recommendation, June 24, 2003. http://www.w3.org/TR/soap12.
5. Mike Cokus and Santiago Pericas-Geertsen, Editors. XML Binary Characterization Use Cases. World Wide Web Consortium, W3C Working Draft, February 24, 2005. http://www.w3.org/TR/ xbc-use-cases/.
6. Tim Bray, Jean Paoli, C.M. Sperberg-McQueen, Eve Maler, and Frangois Yergeau, Editors. Extensible Markup Language (XML) 1.0 (Third Edition). World Wide Web Consortium, W3C Recommendation, February 4, 2004. http:// www.w3.org/TR/REC-xml/.
7. Paul V. Biron, Kaiser Permanente, and Ashok Malhotra, Editors. XML Schema Part 2: Datatypes Second Edition. World Wide Web Consortium, W3C Recommendation, October 28, 2004. http:// www.w3.org/TR/2004/REC-xmlschema-220041028/datatypes.html#base64Binary.
8. Sameer Tyagi. "Patterns and Strategies for Building Document-Based Web Services." Sun Technical Articles, September 2004. http://java.sun.com/developer/ technicalArticles/xml/jaxrpcpatterns/ index.html.
9. John J. Barton, Satish Thatte, and Henrik Frystyk Nielsen, Editors. SOAP Messages with Attachments. World Wide Web Consortium, W3C Note, December 11, 2000. http://www.w3.org/TR/ SOAP-attachments.
10. Jeannine Hall Gailey. "Sending Files, Attachments, and SOAP Messages Via Direct Internet Message Encapsulation," MSDN Magazine, December 2002. http://msdn.microsoft.com/msdnmag/ issues/02/12/DIME/default.aspx.
11. Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Working Draft, February 9, 2004. http://www.w3.org/ TR/2004/WD-xop10-20040209/.
12. Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Working Draft, June 8, 2004. http://www.w3.org/TR/2004/ WD-xop10-20040608/.
13. Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Candidate Recommendation, August 26, 2004. http://www.w3.org/TR/ 2004/CR-xop10-20040826/.
14. Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-Binary Optimized Packaging. World Wide Web Consortium, W3C Proposed Recommendation, November 16, 2004. http://www.w3.org/TR/2004/PR-xop10-20041116/.
15. Martin Gudgin, Noah Mendelsohn, Mark Nottingham, and Hervi Ruellan, Editors. XML-binary Optimized Packaging. World Wide Web Consortium, W3C Recommendation, January 25, 2005. http://www.w3.org/TR/xop10/.
16. Scott Boag, Don Chamberlin, Mary F. Fernandez, Daniela Florescu, Jonathan Robie, and Jerome Simeon, Editors. XQuery 1.0: An XML Query Language. World Wide Web Consortium, W3C Working Draft, February 11, 2005. http:// www.w3.org/TR/xquery/.
17. James Clark. Editor. XSL Transformations (XSLT) Version 1.0. World Wide Web Consortium, W3C Recommendation, 16 November 1999. http://www .w3.org/TR/xslt.
18. Joseph Reagle. XML Encryption Requirements. World Wide Web Consortium. W3C Note, March 4, 2002. http:// www.w3.org/TR/xml-encryption-req.

DDJ