The Proof Is Out There

Surviving rigorous testing is one of the last major hurdles any serious software project must overcome to deliver a stable, trustworthy product. Methodologies may vary depending on an application's scope and intention, but many developers are unaware of the plethora of tools available to assist in this process. Fortunately, vendors and the open source community have created a range of solutions that operate within many development environments.

Software testing comprises several subclasses, commonly defined as unit, functional, performance, load, smoke and integration testing—but there are many others. For instance, Mock Client is a testing methodology for exploring the unpredictable ways customers might interact with the application. (See the interview with Ship It! authors Jared Richardson and William Gwaltney Jr. for more details.)

Most testing tool vendors focus on unit, functional, performance and load testing, as they're the easiest to automate into a build/test/debug cycle, with the first three frequently combined into a single offering. Testing can also be decoupled from the coding process via a centralized organization exclusively focused on software testing—a trend gaining popular acceptance in large enterprises with numerous ongoing application development projects, but one that runs contrary to the agile focus on developer testing via test-first programming.

Unit and Functional Testing
For the single developer responsible for the integrity of the source code he writes, unit tests are a way to ensure quality prior to checking work into a project repository. Future versions of popular IDEs may someday dynamically generate and execute unit tests on-the-fly with each new line of code written, but for now, most developers still need education and practice in writing effective unit test cases. Many commercial offerings are based on the open source JUnit for Java projects and NUnit for the .NET framework. Tools such as Parasoft's Jtest and .TEST (the company's .NET equivalent to its immensely popular Java-based testing tool) are especially helpful and usually integrate well with popular IDEs.

Functional testing determines whether or not the application behaves as intended. Do the GUI elements position themselves where they're supposed to? Does the application logic execute exactly the way the designer's UML sequence diagrams dictate it should? Tools such as Infragistics' TestAdvantage 2005 (designed to work with Mercury Interactive's WinRunner) can settle these and other GUI issues, and IBM/Rational's Functional Tester can answer questions about application logic flow. Often, these unit and functional testing tools extend their product's abilities to perform code analysis and formatting, link checking, regression testing, static analysis and test coverage.

Performance and Load Testing
Performance and load testing tools take the functionally approved application and iterate through its execution to identify bottlenecks in the code. These can be caused by inefficient algorithms, excessive external resource polling, slow network access and a host of other reasons. Rather than spending weeks hunting down the troublesome needle in the virtual haystack, performance testing tools can analyze any portion of an application's execution and locate the problem areas in a matter of hours, minutes or even seconds. The cost for such tools is inconsequential compared to the time saved by locating and abolishing the trouble spots. Load-testing tools complement performance analysis by simulating multiple application instances vying for resources such as database connections and queries, Web application server availability and transaction speed. Vendors such as IBM/Rational have distinct product offerings for functional versus performance testing, while others such as AutomatedQA's TestComplete combine these intentions into a single package.

Integrated Testing Suites

The growing practice of centralized testing within large IT enterprises depends on test experts who apply rigorous, repeatable methods to a wide array of software projects. Enterprise-class testing suites that integrate testing priorities, schedules and results provide stakeholders a view of the health of the development lifecycle. Integrated testing suites, such as Compuware's impressive QA Center and Segue's prioritization-ranking Silk Central Test Manager, aggregate and summarize test results into the grand production picture and offer report dashboards that can predict project failure or success. While their primary function is to roll up and report on the overall testing status of a project, most of these suites can execute the particular test in question by loading up the test parameters, targeting a selection of resources, running the test and reporting on the results. All that's missing is a human being to acknowledge the test results and decide on the level of perfection that qualifies as success.

Evaluation Criteria
Hundreds of testing tools are available, ranging from open source one-offs to polished commercial suites. The products in our guide are a diverse sample of the expansive selection of commercial testing tools available. Some are well-established, best-of-breed choices. Others offer innovative approaches to lifecycle management problems as old as development itself. An essential qualifier for inclusion is availability of the featured product's general release at press time. Hence, the testing features of Microsoft's Visual Studio 2005 Team System aren't included in this article.

Note that open source alternatives popularized by the success of JUnit and NUnit unit testing tools are detailed in a separate sidebar.

Why We Chose 'Em
The products weren't reviewed singly, but rather evaluated by vendor claims, market buzz and installation cost. The testing tools grid covers three classes of testing tools: unit and functional testing, performance and load testing, and integrated testing suites. The columns are the same for each class and contain the following categories:

Features and Claims focuses on the product's unique aspects and how it stands apart from similar solutions.

Market Buzz is a subjective analysis of the product's features and claims as they fare in the real world. This category is especially critical because of the wide range of choices available in the testing tool realm.

Cost is always a challenging category to assess, as most of the featured vendors don't advertise their products' prices on their websites. Whether due to total cost of ownership analysis or flux in the marketplace, costs listed (especially for the more expensive products) can often be negotiated based on volume and usage.

Software Development Podcasts
Starting with this guide, we're offering the Software Development Podcast, available for subscription in any RSS Reader or podcast-catching client, which features audio interviews with vendors, developers and authors who are subject-matter experts in the range of topics covered by the magazine. To subscribe, point your favorite RSS Reader or podcast-catching client to www.sdmagazine.com/podcasts/.

Mike Riley is a Naperville, Illinois-based advanced computing professional specializing in emerging technologies and new development trends. Contact him at [email protected].