C++ Conformance Roundup
compiled and edited by Herb Sutter
There is no C++ compiler or library today that implements the Standard perfectly, but some are getting close. Here, finally, is a first attempt to quantify just how close.
1. Introduction and Overview
The official ISO C++ Standard was published in 1998. It has four main parts:
- ISO C++98 (1998), Clauses 1-16. C++ Language (e.g., classes, templates, exceptions, namespaces, overloading)
- ISO C++98 (1998), Clauses 17-27. C++ Standard Library (e.g., vector, iostreams, algorithms)
- ISO C90 (1990). Incorporated into C++ Standard by reference
- ISO C90, Amendment 1. Incorporated into C++ Standard by reference
This roundup is designed to report which compiler and library implementations support the Standard well. It is not meant to be a buyer’s guide. There are many important measures of a good C++ product that are not included here — things like compile speed, executable size, and GUI tool ease of use. What this roundup is specifically designed to do is promote products’ conformance to the C++ Standard. But what’s the best way to do that? Well, our goals were to attempt to measure which compiler and standard library implementations support the Standard well, and so to send a clear message to implementers that the C++ community really cares about adherence to the standard. I think the latter has been achieved: the message has certainly been sent, and some benefits have already been seen. Still, we’ve also learned along the way that measuring conformance can be tricky, and that it’s hard in not-so-obvious ways, many of which are made clearer in the comments as we go along.
The people involved in creating this roundup represent all major parts of the global C++ community: all major conformance test suite vendors, nearly all major compiler and standard library implementers, and many industry experts (consultants, trainers, standards committee members, expert working programmers, and the inventor of the C++ language) have participated and contributed comments that you can read here.
Why Conformance Is Important: A Visit To the Land of Ad Hoc
Imagine that you’re trying to take a train trip across a country — let’s call it Ad Hocland — where every town uses a different track gauge, or distance between the rails. Traveling inside each town is easy enough, but making the cross-country trip is difficult. There are really only two main ways to solve the problem:
1) Change trains at each town border. That would be inconvenient for both schedulers and passengers, and vendors of train engines and cars would have to build different models for different markets, which in turn would be more expensive.
2) Build engines and cars with telescoping axles to fit different gauge sizes on the fly. That would be still more expensive, and the more complex machinery would require that much more development time and maintenance, and would be more likely to fail. In both cases, the additional expense is of course passed right along to you, the hapless passenger.
Now imagine that you’re trying to write code in a language — let’s call it AdHoc — with no official standard. The dialects may be similar to each other, but writing AdHoc code that works correctly under more than one AdHoc compiler or interpreter or runtime environment becomes difficult. We have the same two options, and both lead to inconvenience and expense for everyone — but especially for the poor developer who just wants to get his code done and have some degree of reassurance that it’s actually going to work on whatever platforms he might need to target tomorrow.
ISO conformance is important for C++ implementations because it enables both code portability and wider consistent use of the C++ language and library. Even if you never need to port your code to another compiler, conformance helps you because you can read a C++ book and have some assurance that the code you’re reading will have the same effect on your platform as on the author’s.
What’s In This Roundup
This roundup evaluates C++ compilers and standard library implementations only for conformance to the ISO/ANSI C++ Standard. We’re not evaluating compile speed, executable size, debugging and GUI tools, or other features that can also be important when deciding which tool to use for a commercial development project. To supplement the reviewers’ own test plans, in the summer of 2000 we conducted a public poll on the Internet newsgroup comp.lang.c++.moderated to ask real-world developers what areas of conformance mattered to them. In particular, this list includes areas that had caused these folks code portability headaches in the past. The resulting areas were given specific attention in this roundup. These results “real-world” conformance are published as Tables 3, 4, and 5 in the web version of this article. See www.cuj.com/roundup/.
For the C++ language itself, the main areas were:
- Dependent names should be used correctly. This is a relatively new feature added late in the standardization process.
- for loop variable scope rule should be implemented correctly.
- Partial ordering of function templates should be implemented correctly.
- Template friend functions should work correctly.
- Template partial specializations should work correctly.
- using declarations should work correctly.
For the C++ Standard library, the areas were:
- auto_ptr should conform to the standard. (In the past there have been varying implementations of auto_ptr because its specification was changed several times during standardization.)
- Header <limits> should be implemented correctly. This header includes platform-specific characteristics of numeric and other values. Important for portability.
- Header <sstream> should be implemented correctly. This header provides the modern stringstreams, which replace the pre-standard strstreams.
- Headers named <cname> should exist and put the C Standard library items into namespace std.
- Locales, which provide internationalization support, should be fully implemented.
- Math functions, such as abs, should correctly provide not only the versions taking int and/or double arguments as required by the C Standard, but also the additional overloads taking other numeric types as required by the C++ Standard.
- New-style headers should work correctly (e.g., <iostream> instead of <iostream.h>, where the issue is particularly important because the two versions of the headers provide different functionality).
The review has three main parts:
1. Independent Reviews.
You will see three independent reviews, one each by Dinkumware Ltd., Perennial Inc., and Plum Hall Inc. — the three major suppliers of C++ compiler and standard library conformance test suites. Using multiple review teams lets us give you the opportunity to compare independent views of the state of the art. Each review reports how well the given products conform to the C++ Standard, in the opinion of that reviewer based on the reviewer’s own test suite [1]. Also, the web version of this article contains the results of tests for real-world conformance issues. See www.cuj.com/roundup/.
2. Reviewees’ Comments.
Nearly all major suppliers of C++ compilers and standard libraries have actively participated in this review, and have provided their own comments on the process and results. Many comments include information about their company’s conformance plans for upcoming releases.
3. Comments From the Independent Expert Panel.
Many C++ standards committee members and top authors have monitored the review process and have contributed their own comments about this roundup’s results, and related pearls of wisdom.
The Future
One year from now, we hope to repeat this roundup along with a comparative report of which products progressed the most in that time. This will help to encourage and reward C++ products’ conformance today, as well as their continued work over the next year. It will also help test suite vendors to continue updating and refining their products, for example to test conformance to the recently-approved Technical Corrigendum #1 to the C++ Standard.
2. Reviews
Each conformance suite vendor tested the products using its own suite, and has written an introduction to the reviews to summarize what they think is important for you to know. The scores presented here are summaries of the overall review results; the detailed results are available online (see www.cuj.com/roundup/).
Introduction From Dinkumware, Ltd.
Dinkumware, Ltd. is in the library business, period. So unlike the other two reviewers, we test only the library portion of a Standard C++ implementation. The Dinkum C++ Proofer is our flagship testing product. It derives from the same code base, written by P.J. Plauger in 1995, as the Plum-Hall LibSuite++ library test suite. That code base consists of nearly 2,000 separate tests, each of which derives directly from a requirement stated in the library portion of the C++ Standard. Dinkumware has evolved the Proofer considerably since 1995, however, as we have applied changes in the C++ Standard to both our library and our testing products in tandem. Indeed, the Proofer is our principal quality-assurance tool for our library products (and conversely).
The Dinkum C++ Proofer has a number of options for testing subsets of a fully conforming library. We can adapt to missing language features, such as template partial specialization. We can also configure the suite to test just for Embedded C++ features, just for the Standard Template Library, or just for a combination of these two useful components (as in the Dinkum Abridged Library). For the tests presented here, however, we test all libraries for conformance to the full C++ Standard.
The only concessions we made are revealed under “testing problems,” where a single compiler or library fault would cause numerous unrelated library tests to fail. As an extreme example, consider our testing of the Silicon Graphics SGI-STL with Comeau’s C++ for PC Linux. The front-end parser (from Edison Design Group) chokes on the expression for the macro LDBL_EPSILON in the underlying gcc C library. (It uses a C99 language extension.) Since the common Proofer test harness uses this macro, every single test fails. Rather than report a zero success rate for this combination of libraries and compiler, we patched our Proofer and obtained a more revealing result.
We also factor out several other aspects of conformance that can have a global impact, in the interest of providing more revealing test results. For example, the C++ Standard imposes a number of additional requirements on the underlying Standard C library. Only two Standard C libraries tested here (from Dinkumware and from IBM) fully conform, but we try not to heavily penalize the Standard C++ libraries under test for C library shortcomings. Similarly, the C++ Standard imposes tight requirements on what names are declared in the global and std namespaces, and on what names cannot be declared in these namespaces. We identify the inevitable real-world compromises without lowering the figure of merit for each Standard C++ library under test.
It should come as no surprise that our library scores high with our own tests. But we hasten to point out that the Proofer tests any library as a black box. Some of the testing problems that we mention for other libraries revealed “parochial” tests, where we inadvertently overspecified behavior that happens to match our implementation. In each such case, we corrected our Proofer baseline to be more fair to all comers. The results reported here reflect the corrections we made to the Proofer.
What may be surprising is that our library scores 100 per cent even on compilers that do not fully conform to the C++ Standard. Both versions of gcc V2.95.2 that we tested with the Dinkum C++ Library have missing language features. But these are not features needed by the library, at least the way we implement it. We also list a number of “compiler limitations” for both of these compilers, some of which do cause a few tests to fail. But we round all test results to the nearest per cent, so a success rate greater than 99.5 per cent is reported here as 100 per cent.
In summary, we have tried hard to characterize each Standard C++ library we tested in ways that are both revealing and fair. We provide only a single figure of merit, the raw score of C++ Proofer tests passed, expressed as a percentage. The scores vary from 75 to 100 per cent. We feel that this is a broad enough range to indicate relative degrees of conformance with some accuracy, but not broad enough to warrant presenting scores for subcomponents of the library.
Introduction From Perennial Inc.
All implementations were tested using the Perennial C++VS Validation Suite Version 5.4 July 2000. The versions of the implementations and the command line options used and the platforms the implementations were tested on are listed in the individual test results for each implementation (available on the CUJ website). The tests exercised were only those tests contained in C++VS that measure conformance to the ISO/IEC 14882:1998, C++ International Standard. The testing comprised of running 35,993 tests for those implementations that were language only, or 38,626 tests for library only implementations, and the combined total of 74,619 tests for implementations containing both language and libraries.
The test suite was run in the ‘conformance mode’ for every implementation, with no allowances for features that may not have been implemented. C++VS can be run in modes that allow for non-conforming features, such as C library functions not being defined in namespace std, lack of C++ style headers for C library includes (e.g., <stdio.h> vs. <cstdio>), old-style exception handling, and others. None of the options that allow for non-conformance were used for this Conformance Roundup.
The test results are organized according to the hierarchy of the ISO standard from which they were derived. C++ shares features and conformance requirements with ISO C90, and ISO C90 Amendment 1. As such, the appropriate tests are derived from the ISO C standards, and all remaining tests are derived from requirements in the ISO C++ Standard. The tests are listed by the clause containing the applicable requirements. The listing indicates the clause number and title, the number of tests run, the number of tests passed, and the percentage of tests passed.
An “OverallSummary” report (see www.cuj.com/roundup/) makes a comparison of the test results obtained for each implementation in three areas:
1) Language and Library tests — where full implementations with language and library support were tested;
2) Language Tests — to compare the results of full implementations language tests with those containing only language support; and
3) Library tests — to compare the library results of full implementations with those that are library only.
For each clause of the C++ standard, we report the number of test cases run, the number that passed, and the percentage that passed.
Perennial’s C++VS is composed of thousands of small test cases, each of which reports a distinct pass or fail. The tests are all “assertion based,” with each assertion traceable to a specific conformance requirement in the ISO C++ Standard. All of the Perennial test cases test for conformance to the ISO C++ Standard, based only on requirements contained in that standard. ISO C++ requirements that are shared with ISO C90, and ISO C90 Amendment 1, are identified as such. We did not test for anything that falls outside the scope of being “required” for conformance to the ISO C++ Standard.
[Some readers may wonder why Perennial’s ratings of the standard library products were consistently lower than the other reviewers’. To explain that, and to give you a taste for how conformance testing approaches can differ, I’ve asked Perennial to comment on the difference. I think you’ll find this interesting and illuminating. — hps]
For library implementations, our results had generally lower pass percentages that those of the other reviewers. Our analysis shows this is due to three main differences:
1) Strict Conformance. We have various “non-conforming” modes available to make allowances for common limitations, but we ran our tests in strictly-conforming mode. For example, we required that the C libraries be defined in namespace std (affects over 2,000 tests), and we required C++ style headers such as <cstdio> for the C library (over 1,000 tests).
2) ISO C, Amendment 1. Our ISO C and Amendment 1 library tests make up a large portion of the testing we did for the C++ libraries, and are included in the suite. We tested them because they are required for conformance to the C++ Standard.
3) Larger Numbers of Test Cases in Specific Clauses. For three particular clauses in the Standard, we have many more test cases than the other reviewers. Test failures in these clauses where we have lots of tests had a greater negative impact on the overall results.
When we remove failures due to those three causes, our results closely track the other reviewers’. Because we tested for strict conformance, and for Amendment 1, we feel our results are an accurate reflection of products’ conformance to the entire Standard.
Introduction From Plum Hall Inc.
Each test case in a Plum Hall test suite is intended to provide compilable and executable tests for one specific sentence from the Standard. Sometimes that test is a minimal example for just that one sentence, more often we attempt to test the interaction between that one sentence and one or more other features from the Standard. In the compiler tests (but not in the library scores), we also have negative-tests, which contain syntax that violates some requirement in the Standard, and we provide a harness for reading the compiler’s output to verify that some implementation-defined diagnostic message was produced. The Plum Hall results present only the compiler tests for C++ clauses 2 through 16 (i.e. no C tests, no C Amendment 1 tests, certainly no C99 tests). The Plum Hall library results present only the C++ clauses 17 thru 27 and Annex D, and exclude the tests that cover clause 7 of the C Standard.
In our review results, we have summarized the scores according to the “clause dot subclause” of the C++ Standard. For each subclause, we computed a score from 1 to 10 based on a simple “percentage correct,” rounding to the nearest integer — any score of 96% or higher is a “10” for that subclause (leaving some slack for errors in the testing, or controversies in the Standard, etc). But if there were fewer than 10 test cases in a subclause, and the compiler didn’t receive a perfect score for that subclause, then we reported just a “dash” (or “hyphen”), meaning “refer to the actual numbers.”
For example, here are a few lines of one report:
17.3_Method_of_description_ - 0/1 10 1/1 - 0/1 - 0/1 - 0/1 17.4_Library-wide_requirements 9 21/24 8 19/24 7 16/24 8 19/24 8 19/24 18.1_Types 10 1/1 10 1/1 10 1/1 10 1/1 10 1/1
For each of the about 120 “clause-dot-subclause” categories, we report a pair of columns for each reviewed product: the first number is a zero-thru-ten score, followed by the actual raw scores (# passed /# tests). For example, the score “9 21/24” means that the subchapter had 24 tests, the product succeeded on 21 out of those 24, and that the fraction 21/24 rounds off to “9 out of 10.”
Towards the bottom you can find the overall totals for each product. These are percentages plus the raw “N out of 2256” scores. The number of test cases, 2256, is roughly equal to the number of sentences in the Standard.
Review Summary: Compilers with Standard Libraries as Shipped
The first set of reviews focuses on compilers “as shipped,” with the included standard library. (In Comeau’s case, Comeau does not include a standard library but recommends using a slightly modified version of the free SGI standard library; that was deemed to be Comeau’s “as shipped” library.)
For each compiler product, Table 1 shows two scores from each reviewer: a compiler conformance score showing how well the product implements the core C++ language (e.g., templates, namespaces, exceptions), and a library conformance score showing how well the product implements the C++ Standard library facilities (e.g., auto_ptr, vector, streams).
Review Summary: Third-Party Standard Library Products, on Representative Compilers
Some companies produce implementations of the C++ Standard library that you can use instead of the one shipped with your compiler. Because some compilers are stronger than others, the library’s conformance can vary depending on the compiler with which it’s being tested. Therefore, each library supplier selected a “preferred” compiler on which they wanted their library to be tested to show it in its best light. In addition, because of the popularity of Dinkumware Ltd.’s library as a replacement for the older Dinkumware library shipped with Microsoft’s compiler, we additionally tested Dinkumware’s library on Microsoft Visual C++.
Table 2 shows the results for this category.
Review Summary: Public Poll Results, Compiler Issues
Plum Hall additionally tested the reviewed products for their conformance with the issues identified by real-world developers via the public poll. The results have been divided into compiler/language feature issues (e.g., templates, namespaces), summarized in Table 3, and library feature issues (e.g., auto_ptr, library headers), summarized in Table 4 and Table 5.
Each result is shown as a score out of 10. In the detailed Plum Hall results, you will see a few more rows for specific features that had been suggested by the poll, like these:
AutoPtrThatConforms 9 42/45 8 34/45 8 38/45 9 40/45 9 40/45 Header<limits> 10 71/72 10 70/72 9 67/72 9 64/72 10 69/72
Plum Hall states: “Our approach to the ‘specific feature’ tests was to search (‘grep’) the body of test cases to locate tests which embody the specific feature. For the two examples above, ‘AutoPtrThatConforms’ designates the set of tests containing the characters ‘auto_ptr’, and ‘Header<limits>’ designates the set of tests containing ‘<limits>’. If there were fewer than ten test cases in a subclause, and the compiler didn’t receive a perfect score for that subclause, then we reported just a ‘dash’ (or ‘hyphen’), meaning ‘refer to the actual numbers.’”
Of the features that mattered to the real-world developers who contributed to the poll, the compiler/language-related ones that were most poorly implemented across the board were dependent names and using declarations, and the library-related ones that were most poorly implemented across the board were locales and the library’s templated member functions.
Recognition for Special Achievement: Edison Design Group (EDG)
Edison Design Group (www.edg.com) does not appear by name in this roundup because it does not sell compilers to end users; what it does sell is one of the most compliant and advanced C++ language implementations available today. EDG’s product is a compiler front-end, so it implements the C++ language only (e.g., templates, exceptions, namespaces, classes) — not the C++ Standard library (e.g., streams, containers, algorithms), and not even the compiler back-end or optimizer. Just the language. It’s no coincidence that several well-rated compiler products in this roundup, such as Comeau and KAI, use EDG’s front-end.
Although EDG’s is not an end-user commercial product, it has contributed greatly to the industry’s overall level of conformance to the C++ language standard by: a) contributing to the existence of many well-conforming products as evidenced in this review, since without EDG’s technology we would have fewer honorees; and b) providing quick and early implementations of new features in the draft standard that the standards committee could work with, which helped to make the committee’s work that much faster and safer. EDG deserves this special recognition for that contribution throughout the standardization effort to date [2].
3. Reviewees’ Comments
Borland/Inprise
We’re proud of the levels of conformance to ANSI standards we’ve shown in all third-party reviews. We will continue to push forward in our goal of providing the best tools in the industry.
Comeau Computing
Numerous additions were made to Comeau C++ throughout 2000, additions which bring Comeau C++ to near complete language compliance to Standard C++. This is effectively confirmed by the many high scores in Comeau C++’s review. The scores in this review also serve to support subjective postings often made by posters on the Internet that Comeau C++ is the most compliant and robust multi-platform compiler.
Take note that CUJ reviewed a beta version of Comeau C++ 4.2.44, as well as a beta version of the standard library we provided. By the time this article is printed, they should both be out of beta. They should also both have even more Standard C++ support.
To check on our upcoming conformance plans, visit http://www.comeaucomputing.com/features.html. There, we will post our final features lists, giving Comeau C++ near full language support (for instance, we hope to have the first implementation of export). Furthermore, by the time this roundup is published we expect to have released Comeau C++ to about seven new platforms. This is in addition to our existing numerous ports, available both generally and as custom ports. See http://www.comeaucomputing.com/faqs/genfaq.html#history for details.
In October 2000, the C++ committee voted on officially releasing its first group of defect reports to Standard C++. Some are already incorporated into Comeau C++, and this process will continue through 2001. As an aside, Comeau C++ also tracks Standard C in its C mode, and already supports various C99’isms.
With respect to the review process used and the review results produced for this article, Comeau Computing is still trying to understand the testing process, and the results presented by the various testing vendors despite many attempts for clarification. As one example, testing vendors reported that we did not provide “Cname” style headers, when we did, even after pointing this out. Obviously then, corrections to things such as this would make our scores even higher.
Dinkumware
The Dinkum C++ Library has tracked the evolving C++ Standard closely since early 1993. Versions of this library are shipped with Microsoft VC++, IBM Visual Age C++, and a number of other compilers for both desktop (Windows, Unix, Linux) and embedded development. The Dinkum C++ Library has thus been configured to work well atop a number of different C libraries, with a number of different compilers. Dinkumware directly licenses an upgrade library for VC++. We also offer the Dinkum C Library, which has tracked the C Standard since 1990. (A C99 version is imminent.) We directly license our combined C and C++ libraries prepackaged for some versions of gcc.
We believe that the Perennial and Plum Hall tests do not show our libraries in the best possible light. Compiler limitations cause many tests to fail for reasons unrelated to the library features under test. For example, Pete Becker made a few global changes to our copy of the Perennial test suite and retested our upgrade library for VC++. By avoiding one or two compiler shortcomings, he raised the success rate from 64 per cent to 88 per cent. (Sadly, Perennial was unable to test our gcc package.)
For another example, we know that the PC Linux version of gcc has a nasty bug. It botches some conversions of unsigned char array names to pointers. We made numerous changes to both our libraries and to the Dinkum C++ Proofer to avoid this bug. Evidently, the Plum Hall tests have not been overhauled the same way. Thus, we believe that our own Proofer tests give the truest picture. The Dinkum libraries are highly compliant, and well ahead of the competition in this regard.
Hewlett-Packard
The Perennial Test Suite was run on a machine installed with HP-UX 11.0.
HP-UX versions prior to 11i lack wide-character support in the C library. Because over 15,000 of the 38,626 total Perennial library tests fall into this category, the 50.9% library conformance rating is expected. This figure substantially understates overall library conformance.
In section 19, many tests rely upon indirect inclusion of standard header files. In Rogue Wave’s Standard Library implementation, which we use, types are forward declared where possible. For example, in some implementations, the header file <stdexcept> will include <string>. Rogue Wave’s library does not. This is legal, as the C++ Standard does not specify header file dependencies. We suggest to our users that they explicitly include the standard header files that they need.
In section 20, many tests use explicit instantiation as a method to test conformance. aC++ fails over 1,000 tests in this section because it is unable to explicitly instantiate adaptors for pointers to members, not because of inadequate or incorrect library coverage. In addition, some explicit instantiations do not specify required throw specifications. aC++ fails 400 tests because of incompatible declarations.
In aC++ versions 3.26 and later the runtime library provides support for namespace std. The Dinkumware Proofer was run using a prior version of the compiler. We expect this is the reason for the 79% conformance ranking.
aC++ does not accept template template parameters, some forms of explicit member template specializations, and exported templates. However, work is ongoing to increase our conformance in these areas.
IBM
VisualAge C++ includes both a conventional, “one-file-at-a-time” or “batch-mode” C++ compiler and an incremental compiler. The batch-mode C++ compiler was used in all of the tests that were conducted for this review. The incremental and batch-mode compilers are based on the same technology, and we expect that the results reported using the batch compiler are identical to the results that would have been reported had the incremental compiler been used.
IBM VisualAge C++ version 5.0.0.1 has recently become available, but it was not available at the beginning of the Conformance Roundup review period, so version 5.0.0 was reviewed instead.
Future versions of IBM VisualAge C++ will address the non-conformances demonstrated in this review, and will implement the library and core language changes that will be published in the forthcoming TC (Technical Corrigendum) to the C++ International Standard.
The VisualAge C++ compiler team wishes to thank the reviewers for the time and effort that they devoted to this Conformance Roundup.
Intel
Intel C++ Compiler 5.0 is the first commercially available, fully supported C++ compiler for the Intel® Pentium® 4 processor, and will soon support the Intel Itanium processor. For more information go to http://developer.intel.com/software/products.
Intel C++ plugs into the Microsoft Visual C++ environment as it runs on either IA-32 or Intel® Itanium™ architecture computers. It produces executables that are source and object code compatible with the 32-bit and 64-bit Microsoft Visual C++. This gives developers flexibility. They can use the compiler to develop IA-32 applications using IA-32 systems or — still using IA-32 systems — use the pre-release compiler to begin development of Itanium architecture based applications.
KAI Software (formerly Kuck & Associates, Inc.)
The KAI C++ compiler, KCC, attempts to provide high level of conformance with the ISO C++ Standard across multiple platforms. KCC provides C++ headers and C++ libraries, but relies on the native C compiler, standard system headers, and the native C library for conformance with the C Standard. There is a temptation to “redo” C implementation details for a given platform, but KAI has chosen to concentrate on C++ issues.
Where the native C library does not supply long double precision math functions, KCC does not attempt to supply them. Where system headers define simple C functions, such as isalpha, as macro definitions, the KCC headers cannot import such function names into namespace std. The current KCC configuration only recognizes digraphs alternate tokens and trigraphs sequences when the “—strict” option is invoked.
Metrowerks
Metrowerks strives to provide its customers with the best C++ compiler/library product available. Conformance to the C++ Standard is a high priority, and we have earned a reputation for quality in this area.
CodeWarrior allows programmers to build applications from one unique IDE (Integrated Development Environment) and is available on Windows, Macintosh, Linux, Solaris, and Palm OS, as well as the most popular microprocessors. Its cross-platform compilers support C, C++, Java, and assembly programming languages. This broad support for multiple platforms and languages is one reason Metrowerks’ CodeWarrior products are used by more than 250,000 registered users worldwide.
Microsoft
At Microsoft, we take the C++ language and the ISO Standard very seriously. Visual C++ is used daily by over 1.2 million developers worldwide, and is the most extensively used development language within the company.
Our new C++ compiler is substantially closer to compliance with the ISO Standard than the Visual C++ 6.0 one. Unfortunately, the terms of the review precluded consideration of the latest version, and was based on a compiler released three years ago. [This review was limited to shipping, or at most late-beta, products. We’re looking forward to including the new .NET version in next year’s roundup! — hps.]
The latest Visual C++ compiler in Visual Studio.NET includes
- covariant return types
- additional template support
- in-class initialization of static integral data members
- correct for-loop scoping
- the C Standard library in the std namespace, and
- function-try blocks.
These new features are available today. Please see http://msdn.microsoft.com/vstudio/nextgen/beta.asp for more details of the beta-1 release of Visual Studio.NET.
Our commitment to ISO compliance is underscored by our partnership with Dinkumware, Ltd. (http://www.dinkumware.com). This will ensure that the version of the C++ Standard Template Library in Visual Studio.NET will be an extremely compliant and robust implementation.
It is a stated goal of the Visual C++ team to produce a fully ISO-compliant compiler, and this goal continues to guide all current and future development.
RogueWave Software
The Standard C++ Library by Rogue Wave Software implements and conforms to the C++ International Standard, sections 17 through 27 and Annex 4, as specified by ISO/IEC 14882:1998. Rogue Wave actively participates in the ANSI C++ committee, closely tracks its resolved issues, and addresses them in our implementation.
The Rogue Wave implementation is portable to 13 operating systems and 12 compilers. We achieve this exceptional portability by implementing our library using the C and POSIX libraries available on each platform. Although some bugs from these libraries impact the functionality and conformance of our implementation, Rogue Wave tries to detect these problems and correct the behavior of the underlying library.
Rogue Wave’s implementation of the library is periodically tested with most major library conformance test suites. While there is some minor nonconformance, most problems are caused by the combination of the library and the tests exercising a particular language feature, like explicit template instantiation, in a particular way. Often, invalid assumptions or incorrect interpretation of the Standard by the test suite are the cause of false failures as well.
The Rogue Wave library provides and correctly implements all the features highlighted in the Roundup as especially important to C++ library users. Specifically, however, the <cname> kind of C++ C library wrapper headers and the C++ overloads of the math functions are not provided for all platforms. Correctly implementing this functionality requires the cooperation of the C library provider. Since many compiler vendors, including HP and Sun, ship the Rogue Wave implementation of the C++ library, we rely on them to integrate the two libraries.
Our library provides all required member templates. Since we support a broad range of compilers, some of which do not yet correctly implement all language features, the library also provides alternative mechanisms wherever possible to accomplish the same effect.
Sun Microsystems
Here at Sun Forte tools we are grateful for the recognition of our commitment to standards conformance. Another area that Sun takes very seriously is binary compatibility from release to release. The Sun Forte C++ compiler is actually more compliant than the tests in this report show. For example, in the areas of member templates and partial specialization, the compiler is fully compliant, but the Rogue Wave 2.0.1 Standard C++ Library is missing the support. Since Rogue Wave does not maintain binary compatibility from one version of SCL to the next, Sun continues to use this relatively old version of SCL to maintain binary compatibility. Sun customers and ISVs have historically required this version-to-version binary compatibility as a higher priority than rigid standards adherence. Customers who want or need such standards conformance can replace the Rogue Wave SCL Sun ship with other SCLs such as later versions from Rogue Wave, the Dinkumware SCL, or versions based on STLport.
In January 2001, Sun released Forte C++ 6 update 1, an update release to the Forte C++ 6 release of July. New in this release was tuning for UltraSPARC III, support for Interval Arithmetic in C++, and probably most important for C++ users, significant improvement in compile-time performance. Forte C++ 6 update 1 maintains full binary compatibility with previous releases of Sun C++ compiler products while continuing to support compliance to the ISO standard.
STLport
This C++ Library Roundup is a move towards better C++ environments. This review of Standard C++ library implementations is going to help in better planning for multiplatform C++ projects.
Some conformance scores presented in this article may be surprisingly low, mostly because test suites have extra dependencies on C library and header inclusion assumptions, as has been pointed out by other vendors and reviewers. Even though such scores may not be immediately useful for application developers, they are certainly of great value for library and test vendors.
Hopefully we are going to see more roundups. A performance roundup would definitely be of great interest for developers. Currently there are no benchmarks for C++ Standard libraries available to the public, so it would be valuable to come up with some charts prepared by independent reviewers, like we did for conformance roundup.
What readers may expect in the future from STLport: standards conformance has always been of the highest priority for STLport, and it’s going to stay that way. CVS access is coming up to facilitate contributions and provide regular development snapshots. There are projects to improve installation and to provide STLport as a drop-in C++ library replacement package on some platforms.
4. Expert Panel Comments
This roundup’s organization and execution were monitored by an expert panel consisting mainly of C++ standards committee members and distinguished authors. Here are comments from three members of the expert panel — Bjarne Stroustrup, Andrew Koenig, and Kevlin Henney — on this year’s process and results.
Bjarne Stroustrup
Bjarne Stroustrup is the creator of the C++ programming language and the author of the books The C++ Programming Language, The Design and Evolution of C++, and The Annotated C++ Reference Manual. He remains active on the ISO/ANSI standards committee.
Comparing compilers is hard. There are many criteria that we, as practising programmers, must apply. This roundup addresses one important such criterion: standards conformance. I consider conformance one of the pillars of portability. And for a user, portability is the foundation of choice and freedom. For starters, my ideal compiler will implement every feature of ISO Standard C++, implement it correctly, and not quietly provide me with extensions (that could lock me in).
No individual doing a full time job building applications can do a thorough job at verifying this. No compiler or standard library builder can unaided verify this. So, we have conformance test suites. However, such suites are expensive to build, (therefore) expensive to use, and their output is hard to interpret. This means that conformance testing is something done by experts for experts. Conformance is therefore often underestimated by programmers and placed low on a product manager’s priority list of requirements.
Thus, I think this “roundup” has already done significant good. It has been noted that part of the community cares about conformance (some of us passionately so). Compiler, library, and test suite vendors have had a chance to discuss conformance issues and apparent anomalies in a forum that also included users. The implementations and test suites are already better for the exercise. Thanks Herb, for making this real! And thanks to the many test suite, compiler, and library writers for putting in significant effort to take part and to learn from the experience. I hope to see a follow up — with significantly better scores — next year.
You can tell a lot about an implementation from reading the test results, but you need to spend significant of time studying the details. Evaluating test scores is not for novices. A compiler scoring 92% will not necessarily accept more real-world programs than one scoring 85%. In my opinion, what matters most is how well a compiler supports the major programming paradigms. However, a C++ validation suite cannot weigh its scores with a measure of importance of each language rule. A compiler failing 200 obscure tests can be better than one failing one crucial one. I suspect this is at least part of the reason that the scores were lower than I had expected based on my experience with many compilers. I’m quite aggressive in my use of “new” features, yet I rarely use workarounds. (I guess my style is colloquial and that I have simply “internalized” key workarounds.)
Looking at the numbers, I felt a strong urge to ask questions about specific features. For example, which compilers (correctly) require template<> to precede a specialization? I can get a pretty good idea if I can “dig” into the data to the subsection level (14.7) and I would be even happier if I could dig to the paragraph level (14.7[3]). It would be wonderful if vendors would publish their detailed scores [Perennial, at least, has in fact done this. See the Perennial's test results. — mb]. A collection of links to such vendor scores would be an extremely useful tool for C++ programmers with sufficient knowledge to use the Standard. What would it take to make test suite output a practical tool for expert C++ programmers in addition to serving implementers?
Naturally, the test suites place a lot of emphasis on showing which features are present and correctly implemented. In addition, I’d like to see an increased emphasis on detecting extensions. I’d like to know what extensions are present in an implementation as default installed, and also which ones might still be present after throwing an “ISO standard” switch. For example, which compilers accept specializations without a template<> prefix?
This roundup is a good sign that the C++ community is maturing. We have passed the stage where people could naively believe that stability is ensured by sticking to a single vendor (just to be burned by version skew). I look forward to the (near) future where we can take conformance scores in the high 90s for granted. Then, we will be able to move on to issues that makes sense only when standards conformance is high, such as performance benchmarks and cross-platform consistency of library performance.
Andrew Koenig
Andrew Koenig is the Project Editor of the ISO/ANSI C++ standards committee and author of the books C Traps and Pitfalls, Ruminations on C++, and Accelerated C++.
This conformance testing effort leaves me with two impressions. One is that we have started doing something important. Programmers should know what kinds of programs they should write in order to be confident that those programs will run robustly everywhere. Moreover, they need to know which features a particular implementation fails to implement correctly if they intend to use that implementation. The other impression is that perhaps the long run is longer than it looks at first. Numbers, by themselves, give scant insight, and the requisite insights are not always quick or easy to obtain.
I once had a high-school physics teacher who gave exams in which every question but the first depended on the answer to the preceding question. Once you got a question wrong, you were almost certain to get all the following questions wrong also. Invariably, more students would score zero on his exams than any other grade. Did those grades really measure students’ knowledge of physics?
Similarly, suppose a test suite makes an assumption that is true of most — but not all — C++ implementations. Then some implementations may fail a bunch of tests for the wrong reason: even though they implement correctly the feature being tested, the test makes an inappropriate assumption for those implementations.
Let’s make this discussion more concrete: the <string> standard header is required to declare (among other things) an operator<< that takes an ostream as its left argument (the actual types are more complicated than that, but the idea is right), so that programmers can write strings onto output streams. Most implementations meet that requirement by having <string> include <iostream>. Implementations are permitted to work that way, but they are not required to do so.
Now consider the following program:
// include <string> but not <iostream>
#include <string>
int main()
{
std::cout << "Hello, world!" << std::endl;
return 0;
}
This program will compile and run on any implementation in which <string> includes <iostream>, but will fail on an implementation that does not do so, even if that implementation conforms to the Standard in every respect. If the author of the program has available only implementations in which <string> includes <iostream>, how is the author to know that the program makes such an assumption? Similarly if a conformance test contains such an assumption, it may report failures on correct implementations, and the author of the test may be able to discover these spurious failures only by gaining access to an implementation in which <string> does not include <iostream>.
In other words, this first large-scale test of implementations’ conformance is also a test of the test suites themselves, and the analysis needed to sort out the reason for the failures is only beginning.
There is another reason that the long run is longer than it might appear, and that is that programmers do not know how important each individual test is to pass. There are some failures that programmers will never notice in practice, and others that are overwhelmingly important. Still others will be relevant only to particular programmers and not to others. Therefore, it is not automatically clear that an implementation that passes 90% of all tests is better than one that passes only 70%. It might be that the first implementation fails in ways that will inconvenience every programmer who uses it, and the second one fails in such obscure ways as to be invisible in practice. We simply don’t know until we have analyzed the reason for the failures.
In other words, I think these tests as the first step on a path that is longer than it may appear at first. The next step should be to analyze the results in detail, with an eye toward gaining insight into what these results mean to programmers in practice, and how important particular tests are and in what contexts.
Kevlin Henney
Kevlin Henney is an independent consultant and trainer based in the U.K. He is a columnist for Java Report and C/C++ Users Journal (C++ Experts Forum), and a regular speaker at software conferences.
Having your code not compile when you move it from one compiler to another is perhaps the most visible consequence of portability — or rather its absence. Standard conformance also shows up in the less tangible area of skills: what knowledge and techniques are portable?
For anyone trying to teach and communicate C++ concepts and idioms — whether through articles or courses — non-conformance is as much of a pain as the compiler messages that greet the supposedly portable code of those trying to learn. “Ah, yes, well the code/recommendation/technique works if your compiler and library conform to the ISO standard, but if you are using compiler X you have to tweak it like so, for compiler Y you have to mod this and this, oh, and you simply can’t do this with compiler Z,” is unfortunately a common enough wrapping to what may otherwise be a killer explanation or technique. It becomes a learning experience for all, but often for the wrong reason.
A couple of years on from the Standard and the roundup seems to show a reasonable, but still less than perfect picture of the state of conformance. There are still gaps that developers, consultants, and educators must bridge en route to their more important destinations. Advanced use of the language and libraries are likely to have a rough journey with some compilers and libraries, especially in the areas of template specialization and overloading, namespace lookup, and locales.
However, the good news is that the gaps are shrinking, and there are some smooth and continuous paths. There are fewer chasms than there were in the past, and many of the pitfalls can be worked around with a few small, conforming changes. For the common cases ISO C++ provides a more stable platform to teach to and learn from than C++ in the past. The moving target of compiler conformance seems to be moving towards this platform, although it is still far enough away as to not be transparent.
The three ages of C++ correspond roughly to the three editions of The C++ Programming Language: the first was pre-ARM; the second was ARM and pre-STL; the third is STL and ISO. Each generation brings with it not just a different language, but a different family of styles and techniques. It has become clear that there is less excuse to teach, learn, and develop C++ in the style of the older generations, when support for ISO C++ simplifies all these activities, and when its use is now more practical than theoretical.
Notes
[1] Dinkumware is both a reviewer and a vendor of one of the reviewed C++ Standard library products. They have endeavored to remain critically unbiased, yet it’s only natural that their own library scores best with their own conformance test suite because they regularly test the two with each other.
[2] Incidentally, there’s an important lesson to be learned here: Neither a large development team nor a centralized organization are necessarily required to make world-class products. Both EDG (recognized here for special achievement) and Dinkumware (this year’s best scorer for library products) are three-person companies producing world-class software, and the six people live and work in five different locations.
Download Test Results
CUJ Editor's View
Herb Sutter ([email protected]) is chief technology officer of PeerDirect Inc. and the architect of their heterogeneous database replication products. He is secretary of the ISO/ANSI C++ standards committees, a member of the ANSI SQL committee, and author of the book Exceptional C++ (Addison-Wesley).
