Numega's Devpartner Studio

NuMega's DevPartner Studio will shorten and improve your testing process.

Five stars. NuMega’s DevPartner Studio impressed me so much that I want to share that with you up front. If you develop Windows software in Visual Basic or Visual C++, this suite of tools will help you develop better software. If you develop in Java, you will probably find DevPartner Studio beneficial as well. While Java tools are Windows-hosted, the suite supports both the Microsoft and Sun Java Virtual Machines. The suite includes CodeReview (source code analysis for Visual Basic), SmartCheck (a debugging extension for Visual Basic), BoundsChecker (a debugging extension for Visual C++), JCheck (a debugging extension for Java), SoftICE (a system-level debugger), TrueTime (a profiler for Visual Basic, Visual C++, and Java), TrueCoverage (a code coverage tool for Visual Basic, Visual C++, and Java), and FailSafe (an error recovery utility for Visual Basic).

DevPartner Studio uses several approaches throughout the development process to help you create solid code. It examines code for common logic problems, poor constructs, known bugs, Year 2000 problems, version compatibility issues, and standards compliance (these are all limited to Visual Basic in this release). It provides extensive debugging support to help isolate errors that are difficult to detect using the standard debuggers. It helps you measure your applications’ performance to find problem areas. It helps you determine how well you are testing your code by pointing out code paths that are never run. And it can help you recover from errors at run time in the field.

The studio package supports three languages: Visual C++, Visual Basic, and Java. It is important to note that DevPartner Studio’s applications are available separately, so if one functional area in one environment is all you need, you don’t have to purchase the suite. Because the suite includes SoftICE (an extremely powerful system-level debugger), it also supports Assembler.

Installation
The setup for DevPartner is one of the few things I found lacking in the suite. You must install each product individually. The main setup screen lets you select each installation. But the installation of the selected product runs as if it were a stand-alone install. The main screen does not indicate which products you’ve installed. I installed some products twice out of confusion. For this review, I installed more products than an average developer, but many developers will install six or more. Each product installation ran without incident. But the lack of a unified install (and uninstall) utility is a drawback for a product that is billed as a suite. Despite this confusion, however, I had the entire suite installed in well under half an hour.

Each product takes about 10MB of disk space to install, less if you pare off some options. Processor and RAM requirements that provide enough power to put the suite to good use are in the medium to high range. I don’t recommend installing on less than a 200MHz Pentium with 64MB of RAM. This is a suite of serious development tools, so those numbers are in line with the other development tools you are already using. The box lists lower requirements and it will run with less, but performance will suffer dramatically. I used a 233MHz Pentium with 128MB of RAM running Windows NT 4.0 for this review.

DevPartner Studio installs program groups and shortcuts, but you will rarely, if ever, use them. Its integration with Visual Studio environments is almost seamless. You launch the tools and view their results from within your current development environment.

CodeReview
CodeReview checks for potential problems in your Visual Basic source code. You can launch it from within Visual Basic. It offers you several options for checking the current project, as shown in Figure 1. You can select a subset of the project files on another tab. I accepted the defaults and ran it on a project our shop developed. It churned through the code for a few minutes, providing adequate feedback to let me know it was working, then spit out more than 1,000 possible errors of high severity. One of the pitfalls of automated source code analysis is that you can quickly go from knowing too little to far too much about the shape of your code.

Fortunately, I could use the suppress feature to eliminate a number of errors quickly. Variable naming conventions were not used in the project. I was able to suppress all errors flagged by that rule. Form controls were not locked. Some string functions and var functions were being used. I turned off the rules that either did not apply or were not of great concern. When you suppress, you can choose to suppress the particular instance of the errors, all errors of the type for a particular session, or all errors of the type for all testing. Within five minutes, I had winnowed my list down to 67 items. That’s still a fair number of potential errors to examine, but it wasn’t overwhelming and was worth the effort.

The CodeReview window is a docking and floating window within the Visual Basic environment. As a floating window, I could see more within it, but when docked and shrunk, it was more useful for examining code. If an error applies to a specific line of code, CodeReview will quickly display it for you.

CodeReview found a few problems that were not specific to a single line of code. It noted instances where object variables were created or destroyed an odd number of times. This is not necessarily an error, since flow of control may provide a logical explanation. But it certainly deserves examination.

It is important to note that CodeReview is not an automated expert beyond reproach; it plays devil’s advocate and points out possible problem areas. It ships with a rules database that covers many common potential problems, and uses the database to determine trouble spots. You can add, change, or delete rules from the database.

Each rule has a severity level, a category, a description, and optional links to a Visual Basic help topic or a knowledge base article in addition to the information about the error detection. The rules database is in Microsoft Access format.

CodeReview also includes an add-in to help you properly name controls as you create them in forms, and a highly configurable metrics application that uses a number of popular algorithms to provide extensive additional information about your code.

SmartCheck
If you’ve ever programmed in Visual Basic, you’ve probably seen an “Error 5: Invalid procedure call” message once, twice, or maybe a thousand times. This is the error you get on any function or sub call if one or more of the parameters is out of range, invalid, or can’t be converted to the type required. If you are using the return of one function as a parameter to another, it gets even more confusing. It basically means that something is wrong at the current line, and Visual Basic has stopped so you can figure out what it is. If you run under SmartCheck, a debugging extension for Visual Basic, it will tell you what is wrong by identifying the specific call and parameter in error.

SmartCheck debugs compiled Visual Basic EXEs and DLLs. This is very helpful. I have sometimes encountered problems that only occur in my compiled Visual Basic code and not inside the environment. If you know what you’re doing, you can debug Visual Basic with the Visual C++ debugger, but it is not trivial.

Most intrinsic Visual Basic functions and subroutines raise errors if invalid parameters are passed or some other exception is found. But there are a few, particularly regarding string manipulation, that simply return an empty string or other “harmless” value or silently fail to do what you intended. Error handlers do not catch these and the Visual Basic debugger does not stop for them. SmartCheck will report these.

If you use the Windows API, you usually get one of three outcomes on a call—it either does what you want, fails horribly (often crashing your program), or seems to fail harmlessly. The built-in Visual Basic debugger doesn’t have any real knowledge of the Windows API in terms of number of parameters, type of parameters, or acceptable values for those parameters. Nor does it know about return values and their meanings. SmartCheck understands the API and reports errant calls and bad return values.

SmartCheck lets you check your project settings to see if they are set to get the most benefit. To do this, you must compile to native code, with no optimization and with debug information being generated for best results. (In my test, enabling optimization had little impact.) Adding debug information results in an EXE that is only 1K larger, along with a companion Program Data Base (PDB) file that has the actual debug information in it. These two last points mean that you can make your final form EXE-enabled for SmartCheck, which I highly recommend.

SmartCheck also lets you tweak the level of information it provides. I recommend accepting the defaults, running your application, and browsing the results a few times before you attempt to customize. After you run under SmartCheck, it presents a display that is basically a log of everything significant (as defined by the options selected for the run) that occurred during execution. You can choose different filtering to limit the amount of information it displays. It lists all the events that occurred in an expandable, Explorer-like view. You can see all calls made, the parameter values, and returns. This is extremely useful. From the point of an error, you can quickly locate related calls to see how a variable was set to a given value, or see the flow of events.

BoundsChecker
BoundsChecker, a debugging extension for Visual C++, has been a favorite of mine for several years. I make sure applications pass BoundsChecker’s quality checkpoint before going into the field. Much like SmartCheck, it tracks events within your application and reports them. However, BoundsChecker can integrate more tightly with the Visual C++ debugger, since they both use compiled code.

BoundsChecker is extensively configurable. You can use it to look for a wide array of problems. Some of its greatest strengths are detecting leaks, overwrites, and Windows API failures. For maximum error checking, BoundsChecker instruments your code. You get better results this way, but compilation and execution both take longer. While I recommend you instrument your code from time to time, I use BoundsChecker mostly on code that has been compiled for standard debugging within the IDE. BoundsChecker detects almost all problems this way, and performance of the compiled code is much better.

To run BoundsChecker, you simply enable it in the BoundsChecker menu and then start a debug session (you can also use it without symbols, but the information isn’t as helpful). You can debug normally, or simply operate the application. Depending on settings, BoundsChecker may pop up during execution to alert you to a problem and let you break into the debugger, acknowledge the error, suppress further reports, or stop. When you complete execution, the BoundsChecker window pops up to show you the results of the session, as shown in Figure 2.

Its tight integration with the Visual C++ environment lets you easily navigate to source code from a reported problem. If BoundsChecker reports a problem that you’re sure is fine, you can add the message to a file of items to suppress for your application and it won’t be reported in future sessions. BoundsChecker ships with standard suppression files for MFC, the Inprise libraries, and the C and Visual Basic run times, so you aren’t subjected to reports of potential problems in the libraries.

JCheck
JCheck, a debugging extension for Java, is similar in many respects to SmartCheck. JCheck’s Event Debugging feature has a user interface that bears a strong resemblance to SmartCheck, and it provides roughly the same functionality. It shows the sequence of events within threads and calls within events in a tree-based browser. Extensive details are given for each logged item, and the source for each is displayed as you select and move through.

JCheck also has a Thread Inspector feature. This is a completely different interface that graphically maps out your application’s execution in a tree diagram showing threads, objects, and calls. It lets you see the big picture of your application’s operation and drill down into components for greater detail.

SoftICE
SoftICE is possibly the most powerful low-level PC debugger available. If you are developing a VxD device driver or an application that is concerned with process internals, SoftICE is almost without equal. I also found it useful when a bug tripped up the integrated C++ debugger. SoftICE will let you browse within the operating system and across process space with ease. If you have never needed this kind of capability, you’re not alone. Only a few developers need this kind of power, and the somewhat primitive interface can be confusing and intimidating to use for normal debugging. However, it’s nice to have it available if you need it.

TrueTime
TrueTime is a multilingual profiling tool. It includes editions for Visual Basic, Visual C++, and Java. These supported languages differ in their degree of integration to instrument a build and collect the execution data. Once you collect the data, the languages share the same interface for viewing the data, which is what sets TrueTime apart from other profiling tools.

You can quickly drill down into your application to see where time is being spent, browsing by function or looking within the source code. You can bounce back and forth between the two views to isolate problem areas within a function at the source line level. At the function call level, you can bring up a window of details to get profile information about all the functions called from within the function and information about where the function was called from. You can also add additional columns to the display to see the maximum and minimum times spent in functions and other attributes. You can save “session files” to review later and export function lists as Comma Separated Value (CSV) data to use in a spreadsheet or database.

TrueCoverage
How good is your testing? If you’re like most developers, your answer is probably vague: “pretty good,” “bad,” or “testing, what testing?” It’s hard to know how much of your code is actually being exercised. TrueCoverage helps you put some more finite numbers together. You must do a special build to use TrueCoverage. It instruments your code (like TrueTime, there are editions for Java, Visual Basic, and Visual C++) and creates a special version that tracks code usage. You then run this version and perform your tests. This works best with an automated testing tool for a couple of reasons—the special version runs considerably slower and it’s best to test scenarios that you can repeat exactly.

After you run a special instrumented version of your application, the TrueCoverage interface pops up with a summary of information about the execution session, as shown in Figure 3. You get numbers at the summary level—how many lines and functions you have and how many of each were actually run, along with percentages. You can then see detailed information about what functions were run, and even get a count of times executed on each source line. You can filter the view to only show functions of a certain size, or those that have been tested in a given range of percentages, to help you determine what additional tests are needed.

You can merge results, so you don’t have to come up with a single test to cover all code. Your goal is that your merged file of all tests should have no untested code, except perhaps the handling of errors so severe that you can’t run tests reliably (out of disk or resource memory, for example).

In Visual C++, you can use TrueCoverage and BoundsChecker in tandem to produce an excellent measure of how solid your application is. If you run through all your code and BoundsChecker detects no errors, you have a very clean program.

FailSafe
FailSafe adds error handling to your Visual Basic source code. It tracks the code carefully so you can easily remove it if you change the amount of error checking. When you instrument your code for FailSafe, you’re offered choices between several templates that provide different levels of error detection, reporting, and recovery.

After instrumenting code at the maximum level, I purposefully inserted a bug and ran to it so that I would see the form with detailed information about the error. I had options to view the call stack, add information about how the error occurred, or view information about my system and any other applications that were running.

This level of detail will let your beta testers provide you with information about any errors they encounter. You can adjust the level of FailSafe instrumentation as you roll out to wider audiences, and since a lot of the implementation is in Visual Basic code, you can modify it to create return e-mail messages and error logs to help you further, without requiring your users to understand and report to you the technical details of the errors they encounter.

Wrap It Up, I’ll Take It!
I started this review by telling you that I was awarding five stars to DevPartner Studio, so that fact should come as no surprise now. DevPartner Studio adds significant value to your development suite and doesn’t force you to change the way you do things now. It helps you avoid many problems by finding bugs and bottlenecks while evaluating your testing process. It helps improve quality while simultaneously shortening the length of time required to complete a project. You might get more bang for your buck with it if you develop in Visual Basic, but that’s because C++ and Java have more complex constructs for error handling than Visual Basic does. With DevPartner Studio, you can simply add another step your process—a small step with a big pay off.