This article contains the following executables: XSCHEME.ARC CTEST.ARC
David is a technical editor for DDJ and the author of XLisp, XScheme, and the TelePath conferencing system. He can be reached at DDJ, 501 Galveston Drive, Redwood City, CA 94063.
Whenever the question of compiler benchmarks raises its ugly head, the axiom "lies, damned lies, and benchmarks" immediately jumps to mind. Nevertheless, benchmarks can be a valid means of comparison, especially when you have the opportunity to benchmark the actual application program you're developing. This leads to a second commonly spouted maxim about benchmarks: "The best benchmark is the program you are actually going to use." Unless you benchmark your actual application program, your experience of a particular compiler's performance will vary.
With this in mind, I recently measured the performance of a program I use all the time -- XScheme. This choice of a test program was not made because it happened to fit my personal benchmarking needs, but because it presents a general-purpose, nontrivial test of a compiler's functionality and performance. While XScheme is by no means an exhaustive testbed for all compiler features or application requirements, it did suit my immediate needs, and I'm sharing my findings with you.
Consequently, I examine in this article nine different C compilers for MS-DOS: Four protected-mode compilers -- Watcom C/386, Metaware High C 386/486, Intel's 386/486 C Code Builder, and Microway NDP C-386; and five real-mode compilers -- Microsoft C, Borland C++, JPI TopSpeed C, Zortech C++, and MIX Power C.
The Playing Field
My tests were performed on an Everex Step 386/20 with 4 Mbytes of memory, a 64K cache, an 80387 floating point coprocessor, and a Seagate ST4096 80-Mbyte MFM hard disk drive under MS-DOS, Version 5.0. I used DOS 5.0 partly because it supports disk partitions larger than 32 Mbytes, making it possible to install all of the compilers at the same time. In addition, this was an excellent opportunity to test these compilers under the new operating system.
All of the timings in this article were generated with FILES = 30 and BUFFERS = 10 in CONFIG.SYS. I mention this because the installation procedure for Intel's C Code Builder suggests that BUFFERS be set to no less than 30. The tests for each compiler should be on equal footing, so I used BUFFERS = 10 for C Code Builder as well.
Just for comparison purposes, I tried the XScheme compile with BUFFERS = 30. As you might expect, C Code Builder compiles somewhat faster. Of course, so does Borland C++, and I imagine the others do, too. In spite of the recommendation in the Code Builder installation procedure, I think the results presented here give an accurate representation of the relative performance of all the compilers, and that the performance of any of them could be improved by increasing the number of buffers.
Both the Metaware and Watcom compilers I tested generate code that runs in protected mode on the 386. This requires the use of a DOS extender. I used the Phar Lap 386 | DOS-Extender SDK, which provides an environment for running protected-mode programs under DOS and includes an assembler, linker, and debugger.
I've also taken a brief look at Rational System's Oxygen program. Oxygen is a product designed to be used with Microsoft C to allow the protected mode copies of the Microsoft compiler and linker to be run under DOS. This allows for compiling much larger programs, as all of extended memory can be used by both the compiler and linker. It also seems to considerably speed up compile and link times.
The Yardstick
There are two important aspects of a compiler's performance: generating compact and efficient code and compiling and linking efficiently. The latter is particularly important during program development when large portions of the program are recompiled often, as changes are made to global header files or large modules.
To test compile and link speeds, I chose a program I wrote called "XScheme," an interpreter for the Scheme dialect of Lisp. XScheme is a relative of XLisp, which I wrote before creating XScheme. (For more information on XScheme, see the accompanying text box "XScheme's Timely Role.") The entire program consists of about 10,000 lines of C code (including blank lines and comments). Lisp-like languages are not known for parsimonious use of a machine's instruction cycles; therefore, timing the speed of XScheme execution is a fair test of the binaries produced by a given compiler. Although there's far more source code to XScheme than could be printed here, it's available electronically for those of you who are interested in it as a language implementation, as well as those who wish to confirm the results presented in this article; see "Availability" on page 3.
Because all of the compilers tested here claim ANSI conformance, I spent some time adding ANSI prototypes to XScheme so that I could test the compilers with their ANSI compatiblity functions enabled. (Editor's note: Adding ANSI prototypes didn't uncover a single problem in the XScheme code!) I have measured the compilation speed both with and without optimization enabled. My assumption is that optimization is usually not necessary during development and that disabling it usually leads to faster compile times.
In addition to measuring the time required to compile XScheme, I have also measured the performance of the resulting executables. I've done this by timing the execution of an implementation of the Fibonacci function written in Scheme and run under each of the interpreters generated during the compile tests.
As a second performance measure, I've written a program called "CTest" that contains two separate tests. To test floating-point performance, I've used a modified version of a fractal tree program published in the article "Fractals in the Real World" by Dick Oliver (DDJ, April 1991). It does a large number of floating-point operations in the process of generating the fractal tree and seems like a better test than some of the do-nothing sequences of floating-point operations that have been used in the past. So that only floating-point performance is measured, I've written dummy versions of the graphics routines that actually draw the tree. CTest is also available electronically.
The second test in CTest is a C version of the Fibonacci test that I ran under XScheme. This is really a subroutine call test because it performs relatively few operations other than calls.
CTest reports its results in terms of iterations per second. In the case of the floating point test, this is the number of trees generated per second. To compute this, 100 trees were generated and the elapsed time was divided by 100. In the case of the Fibonacci test, this is the number of computations of fib(25) per second. To compute this, fib(25) was computed 100 times and the elapsed time was divided by 100. The only exception to this is in the case of the Power C compiler. Its performance was so slow that only 50 iterations of the tree test were performed and the elapsed time was divided by 50 to get the number of trees per second.
Switch Settings
For the XScheme test, I used the large memory model for all real-mode compilers and the flat memory model for the protected-mode compilers. I used software floating point (although the tests I ran didn't contain any floating-point code). For the optimized versions, I optimized for speed rather than size.
For the CTest test, I used the small memory model and tried to use the best optimization switches to get good floating-point performance. I used switches that allowed the compilers to generate inline floating point instructions where possible, and I forced the use of the 387 coprocesser chip. Table 1 provides descriptions of the various switches used.
Table 1. Description of switches used in compiling the test programs
Microsoft C
-AL Use the large memory model.
-AS Use the small memory model.
-FPi Generate inline floating-point instructions, with emulation if an
80x87 doesn't exist at runtime.
-FP87 Generate inline floating-point instructions; an 80x87 is required
at runtime.
-Gs Disable stack overflow checking.
-Za Enforce ANSI compatibility.
-Ox Maximize optimization.
JPI TopSpeed C
option(ansi=>on) Check for ANSI compatibility.
optimize(speed=>on) Optimize for speed over size.
optimize(cpu=>386) Generate code for the 80386.
optimize(copro=>387) Generate code for the 80387.
Zortech C
-ml Use the large memory model.
-ms Use the small memory model.
-f Generate inline floating point instructions.
-o Optimize.
-A Check for ANSI compatibility.
Power C
/ms Use the small memory model.
/f8 Use hardware-only floating-point library.
Watcon~ C
-fpi Generate inline floating-point instructions, with emulation if an
80387 doesn't exist at runtime.
-fpi87 Generate inline floating-point instructions; an 80387 is required
at runtime.
-za Disable non-ANSI extensions.
-s Disable stack overflow checking.
-oatx Optimize:
a Alias checking relaxed
t Speed favored over size
x Generate some library functions inline, enable loop
optimizations, no stack overflow checking.
Metaware High C
-Hansi Check for ANSI compatibility.
-O Turn on full optimization.
-f387 Generate code for the 80387 coprocessor.
Microway NDP C
-n2 Generate inline 80387 code.
-n3 Advanced 80387 stack utilization.
-ansi Check for ANSI compatibility.
-O Perform code hoisting and peephole optimizations.
-on Turn on all optimizations.
-OLM Additional memory optimizations, add speed optimizations related
to moving code out of loops and speeding up loops.
C Code Builder
-n Include 80387 emulation library (link switch).
-t Include information for link-time type checking.
-O0 Select the lowest level of optimization.
-O3 Select the highest level of optimization.
The Players
The benchmark results that appear in Tables 2 and 3 speak for themselves, and you can draw your own conclusions. But, in looking over the various packages, you may want to consider more than the raw data. Therefore, I'll briefly describe the features of these packages and note any problems I encountered during the tests.
Table 2: (a) XScheme compile and execution times (fib 25), and file sizes generated; (b) Switch settings used.
(a) Real-mode Compilers
Without With
optimization optimization
Compile Size Execute Compile Size Execute
------------------------------------------------------------------------
Microsoft C 1103 137638 94 1344 139190 93
w/Oxygen 664 802
MSC with -qc 160 183922 141
Borland C++ 144 136434 99 146 132130 89
Zortech C 151 132382 90 310 133470 88
TopSpeed C 338 130724 103 377 123156 99
Power C (wouldn't compile) (wouldn't compile)
Protected-mode Compilers
Watcom C 304 98132 132 334 96320 132
High C 666 104456 64 719 102104 62
C Code Builder 594 239841* 80 635 213597* 70
NDP C (wouldn't compile)
* The size includes a bound in DOS extender
(b)
Compiler Without optimization With optimization
----------------------------------------------------------------------
Microsoft C -AL -FPi -Gs -Za -AL -FPi -Gs -Za -Ox
MSC with -qc -AL -FPi -Gs -Za -qc
Borland C++ -ml -f -A -H=xs_bc.sym -ml -f -A -G -A -H=xs_bc.sym
Zortech C -ml -b -A -ml -b -A -o
TopSpeed C option(ansi=>on) option(ansi=>on)
pragma(speed=>on)
Power C (wouldn't compile) (wouldn't compile)
Watcom C -fpi -za -s -fpi -za -s -oatx
High C -Hansi -Hansi -O
C Code Builder -n -t -O0 -n -t -O3
NDP C (wouldn't compile)
Table 3: CTest benchmark results. Execution times are in iterations per second (a); Switches used in the CTest benchmark (b).
(a) Real-mode Compilers
fib(25) tree size
-------------------------------------------
Microsoft C 1.23457 0.308642 20838
Zortech C 1.21951 0.284091 16480
TopSpeed C 1.5625 0.320513 22562
Borland C++ 1.21951 0.306748 27470
Power C 1.190476 0.082372 21360
Protected-mode Compilers
Watcom C 0.98039 0.34364 15320
High C 1.23457 0.469484 23969
C Code Builder 1.49254 0.571429 87452*
NDP C 1.42857 0.425532 32129
* The size includes a bound in DOS extender
(b) Compiler Switches
Microsoft C -AS -FPi87 -Gs -Ox -Za
Zortech C -ms -f -o -A
TopSpeed C option(ansi=>on),
optimize(speed=>on,cpu=>386,
copro=>387)
Borland C++ -ms -f287 -G -O -A
Power C /f8 /ms
Watcom C -fpi87 -za -s -oatx
High C -f387 -Hansi -O
C Code Builder -n -t -O3
NDP C -n2 -n3 -ansi -O -on -OLM
Protected-Mode Compilers
Watcom C/386 8.0 Watcom C/386 Standard Edition comes with a compiler as well as a linker, a make utility, a librarian, an editor, and a number of other utilities. The Professional Edition, which I used for this article, adds a protected-mode version of the compiler, a profiler, and a source-level debugger. The Watcom compiler generates executable files that work with Phar Lap's 386|DOS-Extender.
One noteworthy problem is that the protected-mode compiler is not compatible with HIMEM.SYS that comes with DOS 5.0. Watcom says that the next version will support HIMEM, but until then you must remove HIMEM from your CONFIG.SYS file in order to use the protected-mode version of the compiler.
The only trouble I ran into while compiling XScheme under Watcom C was that I overlooked the section in the manual describing how continuation lines work in their make program. Traditional Unix make programs use the backslash character at the end of a line to indicate that the line is to be continued on the next line. Watcom decided to change this since the backslash (\) character is used by DOS as a separator in path specifications. Instead, they use the ampersand character (&) to indicate a continuation line. Once I modified my makefile, XScheme compiled without any further problems.
Metaware High C 2.32 Metaware High C 386 comes with a protected-mode compiler, an editor, a source-level debugger, and a profiler as well as a number of utility programs. Oddly enough, it does not include a make program, so I used the Borland make utility to build the test programs. It produces executables that work with the Phar Lap 386|DOS-Extender.
When it comes to problems, there's little to say -- Metaware High C compiled XScheme without a hitch.
Microway NDP C 3.1 Microway NDP C comes with two versions of the compiler -- one for their NDP Tool Kit and the other for the Phar Lap 386|DOS-Extender. For this article, I used the NDP Tools version of the compiler. In addition to the compiler, NDP C comes with a librarian, an editor, and a number of utilities.
NDP C doesn't come with a make facility, so I used the Borland make utility to build the test programs. Unfortunately, I was unable to get XScheme to compile under NDP C. It seems that NDP C does not allow the name of a macro to be passed as a parameter to another macro. Because this feature is used by XScheme to parse arguments to internal functions, I was unable to perform the tests that involved XScheme compiliation and execution. I was able to compile the CTest program and I've included those results.
Intel 386/486 C Code Builder Kit The C Code Builder Kit is one-stop shopping for 32-bit developers and includes a 32-bit C compiler, a DOS extender compatible with the 0.9 DPMI specification, a source-level debugger, a linker, a librarian, and a make utility.
I was particularly excited about C Code Builder because I've had many requests for protected-mode versions of programs I've developed and that I distribute free of charge to anyone who wants to use them noncommercially. I've been unable to comply due to the licensing fees required for the DOS extenders required to run them in protected mode. With C Code Builder, I'll finally be able to build protected-mode executables and distribute them free to anyone who is interested. This royalty-free policy should be of interest to commercial developers as well.
The C Code Builder Kit was a late arrival so I didn't have much time to work with it. Luckily, I didn't need much time to get both test programs compiled and running. I ran into only a minor problem in compiling XScheme which involved a redundant function declaration not required for an ANSI compiler. Once I got rid of the offending declaration, everything compiled without a hitch.
Real-Mode Compilers
Microsoft C 6.0 Microsoft C comes with both DOS and OS/2 versions of the compiler as well as a librarian, a linker, a debugger, a make program (nmake), and various utilities. It also includes an integrated development environment, but I used the command-line version for this article. In addition, Microsoft C supports developing applications for Windows 3.0 (using the Windows SDK). This feature was, of course, not addressed by our test suite.
Because I was running a number of compilers that didn't get along with HIMEM.SYS, I had it disabled. This didn't cause any problems until I tried to time the compilation of XScheme. To do this, I used a simple timer program that I wrote for this article. The program takes a DOS command as an argument and times how long it takes to execute that command. Unfortunately, Microsoft C ran out of memory when run from this timer program. To get XScheme to compile, I had to reenable HIMEM to free up some additional memory for Microsoft C. Under DOS 5.0, doing so enables the operating system to "load high," freeing up memory below 64OK for the compiler's use.
This points out a potential problem with using Microsoft C in real mode: It sometimes runs out of memory doing large compiles. Rational Systems' Oxygen solves this problem, as described above. Oxygen allows you to run under DOS the protected-mode version of Microsoft C normally used only under OS/2. For systems with extended memory, this makes much more memory available to Microsoft C and the Microsoft linker, and makes compiles faster (as you can see in Table 2).
Borland C++ 2.0 Borland C++ comes with the compiler itself, a librarian, a linker, a source-level debugger, a profiler, an assembler, a make program, and various utilities. It also includes an integrated development environment, but I used the command-line compiler for this article. Another nice feature of Borland C++ is that it includes the Whitewater Resource Toolkit to allow you to develop Windows 3.0 applications without any additional software. This can save you considerable money over buying both Microsoft C and the Windows 3.0 SDK. Note that even though this is a C++ compiler, I only tested its ANSI C features.
To speed up compilation under Borland C++ (although it was hardly slow to begin with), I used their precompiled header option. This creates a symbol table file the first time it encounters a header file. It then uses the information from that file instead of reparsing the header the next time it is encountered. This improved compile times by about six percent.
Borland provides a program called tkernel that allows their compiler to run in protected mode. This makes it possible to compile much larger source files without running out of memory on machines with extended memory. I tried using this for the compile time tests but it didn't speed up compilation, so the results reported here are for the real-mode compiler.
Products Mentioned
Watcom C/386, Version 8.0 Watcom 415 Philip Street Waterloo, Ontario Canada N2L 3X2 519-886-3700 $1295
Metaware High C, Version 2.32 MetaWare Inc. 2161 Delaware Ave. Santa Cruz, CA 95060-5706 408-429-META $995
Microway NDP C, Version 3.1.0 MicroWay P.O. Box 79 Kingston, MA 02364 508-746-7341 $895 (for 386), $1195 (for 486)
Intel 386/486 C Code Builder Kit Intel 5200 NE Elam Young Parkway Hillsboro, OR 07124-5961 503-696-8080 $695
Microsoft C, Version 6.00A Microsoft Corp. One Microsoft Way Redmond, WA 98052-6399 206-882-8080 $495
Borland C++, Version 2.0 Borland International 1800 Green Hills Road P.O. Box 660001 Scotts Valley, CA 95066-0001 408-438-8400 $495
Zortech C++, Version 2.1 Zortech, Inc. 4-C Gill Street Woburn, MA 01801 617-937-0696 $450
JPI TopSpeed C, Version 3.00 Jensen & Partners International 1101 San Antonio Road Mountain View, CA 94043 415-967-3200 $396
MIX Power C, Version 2.0.1 MIX Software 1132 Commerce Drive Richardson, TX 75081 800-333-0330 $19.95
Phar Lap 386|DOS-Extender SDK Version 3.0 Phar Lap Software 60 Aberdeen Avenue Cambridge, MA 02138 617-661-1510 $495
Rational Systems Oxygen, Version 1.1a Rational Systems 220 North Main Street Natick, MA 01760 508-653-6006 $199
JPI TopSpeed C 3.0 JPI TopSpeed is a multilanguage development environment that supports C, C++, Modula-2, and Pascal. Any or all of these languages can be used from a single environment to develop applications. For this article, however, I used the command-line version of their C compiler. The TopSpeed package also includes a source-level debugger, a linker, and a profiler.
TopSpeed doesn't include a standard make facility but does support a project facility for building programs with multiple modules. Actually, the TopSpeed project system is much more powerful than Unix-style make facilities. It includes a macro language with conditionals and a very flexible method for adding language processors. It also keeps track of the options used to compile source files and will recompile a file when the options that affect it change.
TopSpeed C also compiled XScheme without a problem.
Zortech C++ 2.1 Zortech C++ comes with a compiler, a linker, a librarian, and a source-level debugger, as well as various other utilities. It also includes an integrated development environment. One advantage of Zortech C++ is its availability for a large number of platforms. In addition to the DOS compiler, Zortech produces OS/2, Unix, and Macintosh compilers. This could make cross-platform development much easier. Zortech C++ also supports developing applications for Windows 3.0 using the Microsoft Windows SDK.
Zortech C++ provides a protected-mode version of its compiler that allows the compilation of very large source files. I tried using this version in the compilation tests, but because it didn't improve compilation times, I've reported the real-mode compiler results here.
Finally, I should mention that even though this is a C++ compiler, I only tested its ANSI C features.
MIX Power C 2.0.1 MIX Power C includes a compiler and a linker. A source-level debugger and a profiler are available as a separate package. It doesn't include a standard make program but does support a simple project facility for building programs with multiple modules. Unfortunately, I couldn't compile XScheme because Power C can't handle nested calls to the same macro. For instance, the following pair of macro definitions won't work:
#define car(x) ((x)->n_car) #define foo(x) car(car(x))
I was able to compile the CTest program and I've included those results. I should note that Power C is a very inexpensive package and perhaps shouldn't be compared directly with these higher-priced compilers.
Conclusion
Be mindful when comparing the execution performance of the protected-mode compilers with the real-mode compilers. The environments in which they run are considerably different. Also, their integer performance will vary because the protected-mode compilers use 32-bit integers and the real-mode compilers use 16-bit integers. Finally, an obvious advantage to running in protected mode is having access to much larger amounts of memory. These tests do not show that difference but you should keep it in mind when deciding whether to use a protected-mode compiler.
XScheme's Timely Role
A few years ago, I received a call from a graduate student at MIT who was looking for a way for computer science students to use the Macintosh version of XLisp for homework assignments. The problem was that XLisp is based on Common Lisp and the course materials assumed Scheme, a dialect of Lisp invented at MIT by Gerald Sussman and Guy Steele.
One of my original goals in developing XLisp was to build a small, yet powerful subset of Lisp for small computers. Originally, I based XLisp on MacLisp. Later, I converted it to be more or less compatible with Common Lisp when that standard came out, but I've always been uncomfortable with that shift. Common Lisp is a huge language and implementing even a subset of it resulted in a language that was much larger than what I had originally planned for XLisp. Scheme seemed to be a way to solve this problem: It's an elegant, powerful language without a lot of extra baggage.
So with a few minor edits, XLisp became XScheme. At first, XScheme was a straight interpreter like XLisp. However, when I added the proper handling of tail recursion (a feature required of every implementation of Scheme), I converted it to its present form, a bytecode compiler/interpreter. The version I used for the C compiler article is 0.28. I've assigned it a version number of less than 1.0 because I want to add support for debugging before considering XScheme complete.
One notable change to previous versions of XScheme is the addition of prototypes to test the ANSI C features of the compilers in the accompanying article. The use of prototypes is controlled by the__STDC__pre-processor symbol, so it is still possible to compile XScheme with non-ANSI compilers.
I've written a bench function in XScheme to test the executables generated by a given compiler and have included it in Listing One (page 93) as an example of XScheme syntax. bench times the evaluation of an arbitrary expression. To implement this function, I've added two functions to XScheme. The time function returns the current time in seconds. The diff-time function takes two times returned by time and computes the difference in seconds. bench also reports the number of calls to the garbage collector that occur during the evaluation of the expression.
Finally, the fib function in Listing One computes Fibonacci numbers. There is a call to the bench function to time the execution of a call to fib to compute the 25th Fibonacci number. Keep in mind, however, that XScheme is not a benchmark in the traditional sense, but provides the opportunity to test a compiler's performance under real world conditions.
-- D.B.
Listing One
(define (bench expr)) (let ((gc-start) (gc) (start (time))) (eval expr) (let ((end (time)) (gc-end (gc))) (write expr) (display ", elapsed time: ") (write difftime end start)) (display ", gc calls: ") (write (- (car gc-end) (car gc-start) 1)) (display ", memory: ") (write (list-ref gc-end 5)) (newline)))) (define (fib n) (cond ((= n 0) 0) ((= 1) 1) (else (+ (fib (- n 1)) (fib (- n 2)))))) (bench '(fib 25)) SEE EXECUTABLES: XSCHEME.ARC CTEST.ARC
Copyright © 1991, Dr. Dobb's Journal
