Herb Isenberg

Rapid application development (RAD) environments give programmers the ability to quickly create client/server and other complex applications. In doing so, however, RAD environments introduce a new set of problems, especially when it comes to software testing.

Traditionally, testing has adhered to a rigid paradigm implemented in terms of:

• Embedded test-script code, characterized by the mixing, or embedding, of test-case logic, navigation sequences, and data within the same test script.
• Exact verification, which is based on the notion that expected results are either true or false.
• Informal test-case format, the narrative style in which test cases are often written. The style is informal in that it lacks a well-defined set of test criteria for organizing test-case components.
• Low-tech testing tools, which are limited to "capture/playback" due to their inability to easily perform field-level (or object-level) verification.

Such a testing system is even more important for designing and building automated testing systems in environments where the interfaces (screens and windows) continually change and data is constantly revised and modified. In this article, I'll present an automated testing paradigm I've coined "flexible testing system" (FTS) that represents a shift in how we test application software. What makes this methodology possible is an emerging class of intelligent, object-oriented testing tools. Specifically, I'll describe how to build an FTS that can automate regression testing of business functionality, while providing valuable information during user-acceptance testing cycles. This FTS is application independent and facilitates data and environments that are in constant transition. The FTS I'll describe is built around AutoTester, an automated testing and verification tool that runs on DOS, Windows, and OS/2 Presentation Manager.

AutoTester records user events and generates a documented test script, rather than simply capturing keystrokes and mouse events. An editor lets you create or enhance scripts, including those for toolbars and standard features such as cut and paste. The testing tool sports a "learn window" tool that automatically "learns" a window and all controls contained within it and enables testing of host-based applications through terminal-emulation packages. AutoTester also provides script synchronization with application execution, and supports testing of applications built with KnowledgeWare/Sterling's ObjectView, Powersoft's PowerBuilder, and Gupta's SQLWindows.

I work in the Information Service Department (ISD) of Charles Schwab & Co. The group I work with (the Business Systems Development Testing Group) provides testing support for application developers and business users. The development environment is quickly evolving from a centralized, legacy CICS mainframe configuration, to a Windows-based, multilayered, distributed architecture, where CASE tools are employed to rapidly prototype and develop new applications. The automated testing systems designed for the old configuration couldn't keep pace in our RAD environment. For instance, our order-entry system was being designed and constructed in the new development environment, and test cases were failing because they couldn't locate fields correctly when their screen positions changed. It was obvious that the testing system was not working and that we had to come up with a better method.

Our solution was to create intelligent field-navigation and location routines that utilized "relative context awareness." This allows the test case to vary its execution based on the context at playback time instead of development time. Eliminating the field-position problem enabled us to run hundreds of test cases per hour, daily. This was the first big step toward creating an FTS.

Our FTS methodology grew quickly. A bad-data issue turned into the dynamic, multilevel-verification solution, which expanded our entire perspective on automating testing.

FTS is based on structured test-script code, multilevel verification, and a dynamic test-case format. Structured test-script code separates test-case logic, screen navigation, and data, as follows:

A screen program is written to service all the screen/windows in the system. Screen programs exist independently of test cases and have three major functions: defining the screen itself and its title or label, creating variables for all the objects/fields on the screen, and navigation. The screen program has the intelligence to verify when its screen is up and in focus, as well as to locate and determine the status of any object on the screen. Screen programs are CALLed by test cases whenever they need to service a screen (for example, when entering or validating data).

The navigation component of a screen program is central to FTS methodology. It is the ability to find screen objects that have unexpectedly changed position that makes a testing system flexible. For example, assume that in Version 1.0 of some software, the customer-name field is the fourth field on the screen and can be located by pressing the Tab key three times. In Version 2.0, the customer-name field is the sixth field on the screen, and no one has communicated this change to the testers. The question is, how do you design a testing system so that this unexpected change does not interrupt the flow of testing?

FTS methodology handles this by first locating the cursor in a "home" position on the screen. This is usually the first field/object, located at the top-left corner of the screen. FTS then captures the cursor's row/column coordinates at the home position and stores them in temporary variables (the testing tool should be able to automatically capture row/column coordinates). FTS then moves the cursor to the next field/object (using the Tab or other navigation key) and checks if it is on the correct field; if not, it checks the current row/column position against the home position. It continues navigating the screen until the cursor is on the correct field or back to its home position. If the cursor returns to home position without locating the field, then the field probably has either been removed from the screen or had a drastic name change. At this point, FTS cancels the current test case and begins the next one.

Another benefit of building screen programs is easy maintenance. When an application screen changes, only the single corresponding screen program must be updated. (Remember, test cases are independent of screen programs, so they are not affected.) For example, if 1000 test cases have been built that access the customer-entry screen and that screen changes, only the screen program must be updated.

Data for individual or logically grouped sets of test cases is stored in separate ASCII text files. Each test case (which defines its own data requirements) will READ an ASCII file containing data to either input or verify.

The major benefits of storing data in separate ASCII text files are maintenance and ownership. Again, single-pointed maintenance can be achieved by storing data for a large group of test cases in a single ASCII text file. Additionally, data can quickly be added or modified to accommodate rapid changes in the system being tested. No technical knowledge of the testing tool is required to make these changes; the user need only know the application and how to use a word processor or text editor. Users can maintain their own testing system and even add new test cases in some instances. This takes the burden off developers and gives the users more control and ownership. User ownership greatly increases the productivity of the testing system over time, and has far-reaching, beneficial effects on the overall organization.

Given the independence of test data and screen navigation, what remains in an individual test-case script? A READ statement (loop) to pick up its data requirements; CALLs to screen programs and logs in the order specified by the test case; and information or logic specific to the individual test case (flags that point to fields/objects required to execute particular business functions for this test case).

Multilevel verification is the ability of the testing system to perform dynamic data verification at multiple levels, and provide the information (system or edit messages) necessary for evaluating and/or verifying the data. (Verification refers to correctness of data, while evaluation refers to system information such as system messages or edits.) The more levels of data verification/evaluation, the more flexible the testing system.

Dynamic data verification is the process whereby the automated testing tool, in real time, gets data from a field, compares it to an expected value, and writes the result to a log file. It also expresses the ability of the testing tool to make branching decisions based on the outcome of the comparison. Ideally, the testing tool should combine the GET and COMPARE into one LOOK function, to simplify dynamic-verification coding.

To support the development of a flexible automated-test system, your test tool must have four basic characteristics:

• Fixed and relative context awareness.
• Variables.
• Variable indirection.
• Logic and branching.

The concept of relative context awareness means that the context of one object or control can be determined with reference to the position or location of another. In a GUI environment, where windows can be sized and repositioned, relative context means both the relative position of an item within a window and its aspect ratio, or relationship to the size of the window. For text-based applications, this means being aware of the location of a text string, for example, relative to the position of the cursor. This relativity of context awareness is critical for reducing or minimizing the effect of otherwise cosmetic changes, such as moving a window or a field.

Variables are named elements or objects in the test script whose content can be varied during execution. An obvious type of variable is an edit control in a GUI application or an input field in a text system; a perhaps less-obvious type is a window or screen. By assigning a variable to contain the actual string of input characters, the contents of the control or field can be varied from one test case or run to another without duplicating the entire test script for each iteration. The potential values for a variable can then be supplied either from an external file, from the user at the keyboard, as a result of calculations, or from another window or screen in the application.

For example, a series of transactions may be created in a spreadsheet or extracted from a database into an exchange file format, such as ASCII. Then, by defining a loop in the test tool that reads each record consecutively, substituting the file values for each defined variable in the test script, a test sequence that processes a single transaction can be transformed into one that processes an unlimited number of transactions. These external values can supply not only inputs but also expected outputs, including responses such as error messages or state transitions.

Variable indirection is another dimension of variability where a variable is used to store the name of the target variable: The term "indirection" refers to the fact that the first variable points to the second, so that it is accessed indirectly. This level of abstraction greatly compresses the number and size of test scripts by requiring only one script per common element or control within a system. (Variable-indirection code by Marin Data Systems; CompuServe 72172,3661.)

For example, a test script could be created to verify standard requirements of a certain type of control, such as a check box: setting the focus with the keyboard, mouse, or a mnemonic; verifying the status of the check box; and setting the status to either checked or unchecked. By using one variable to name each control and its parameters, then another to indicate which control is active, a single set of tasks can be used to verify the behavior of any check box in the system by simply passing the name of the control and its parameters to the test script.

As an example, imagine a window named "Update Employee" with an Edit control named "Last Name." To verify that the field contains the value, "Smith," the text-file-based test script has the commands Verify Panel, Update Employee and Verify Field, Last Name, "Smith." The window variables to support this verification are UPDATE_EMPLOYEE.MAIN (the panel itself) and UPDATE_EMPLOYEE.LAST_NAME (the edit-control field). The code for the test script is:

1. Verify Panel, UPDATE_EMPLOYEE
2. Verify Field, LAST_NAME, "Smith"

The variable-indirection feature allows the automated-test designer to write a small number of highly capable control programs that, along with the careful use of English-like variable-naming standards, provides a self-documenting, easy-to-read testing system.

"Logic" refers to the ability of the tool to make decisions based on conditions at the time of playback and vary the sequence of test execution as a result. The most-basic level includes If/Then/Else decisions that execute or skip instructions within a single test script based on the outcome. For example, the test script might first verify whether a particular control or field has focus before input is attempted; if the focus were not correct, the test script could set the focus before proceeding.

Another level of logic and branching includes the capability to branch from one test script to another, then return to the original one and continue. This is known as "nesting," because one or more test scripts can be nested within another. For example, a test script that encounters an error during playback might branch to another script whose function is to log the error, then return to the original script and continue execution. The error-logging script might also verify whether the error is a problem with context; if so, it is required to restore the expected context before playback continues. To accomplish this, it might also branch to a test script whose function is to recover the context, or restore the state of the application to a known place. The error-logging script is nested within the test script, and the context-recovery script is nested within the error-logging script.

The advantage of logic and branching is that it supports modularity and flexibility. "Modularity" means that common tasks, such as logging errors or recovering context, can be developed once and shared across all other scripts. This saves development time and permits single-point maintenance when a change is needed. If all scripts use the same error-logging routine, a decision to capture new system-status information as part of error documentation would need to be made in only one place (instead of in every script that could potentially encounter an error).

Logic and branching also maximizes flexibility, as it allows a single test script to modify its behavior based on results at playback. Otherwise, a separate test script would be required for each possible condition or pathway, or the exact state of the system would have to be known beforehand. For example, if the test script needed to create a new file of a certain name, it could first verify that there was no existing file of the same name; if one was found, the steps to delete it could be executed first. This flexibility means the state of test data and files would not have to be as rigidly controlled in order to maximize the probability that the test would execute successfully.

FTS is an evolving technology aimed at increasing longevity and decreasing maintenance. Longevity reflects the automated testing system's ability to intelligently adjust and respond to unexpected changes to the application under test.

Reduced maintenance applies not only to time but to necessity as well. One-point maintenance saves time, while features such as heuristic navigation reduce maintenance necessity.

FTS is beginning to provide solutions to testing challenges arising from new development technologies and environments. It is important to remember that flexible testing systems execute test cases, but do not define or create them. Therefore, FTS is only a piece of the automation puzzle, but its principles may one day be applied to a model for automating the definition and creation of test cases. This moves us closer to realizing the goal of a system that automatically tests itself.

AutoTester Inc.

8150 N. Central Expressway, #1300

Dallas, TX 75206

214-368-1196

Description                          AutoTester Code    

Save cursor position.                LocateCursor host $FOCUSWND W.COLSAVE  
                                      W.ROWSAVE
Loop through all cursor positions.   Label"LOOKLOOP"
 Look for field label"Name"          Look text"Name" host"$FOCUSWND" @ -9, 0, -4, ,0
(cursor-relative LOOK function).                                         cursor area
Found label? No?                     If No
Go to next line.                     >tab
Only one field per row.
Are we back where we started?        >LocateCursor host $FOCUSWND W.COL W.ROW
                                     >Compare W.ROW W.ROWSAVE
                                     >If Not equal
Not done yet, keep looking.          >>Goto"LOOKLOOP"
NOT Find Field Label
Display screen message.              >Message"Name FIELD NOT FOUND" wait 1>
Write out log file.                  Log $CURLOG"Name FIELD NOT FOUND"
Set Field flag to No.                > Assign FLAG.FIELD ="NO"
Call outline to print the screen.    >Call"PRT_SCR"
Return to test-case outline          
 called from.                        Resume

(a)
Level 1   Exact field-level verification, (1--1).
Level 2   Line LOOK, inclusive or exclusive.
Level 3   Area LOOKs, inclusive or exclusive.
Level 4   Cursor-relative LOOK.
(b)
Level 5   Exact message with capture only on error.
Level 6   Partial message match with capture.
Level 7   Capture message only.

Test Case ID:          CUST.01.
Function:              Add a new Customer.
Data Assumptions:      Customer database has been restored.
General Description:   Add a new customer via the Customer
                       Add screen, and validate that the
                       new Customer was displayed correctly
                       on the All Customer screen.