Establishing a Supporting Process
When you arrive at work each morning, review and respond to any test failures reported for your code. In doing so, you will either address the functional defects in your code or update the test to reflect the correct behavior of the code.
These are the recommended responses to test-case failures :
- If the previous behavior is still the correct behavior, fix the code that was broken.
- If the functionality was modified intentionally, verify the new behavior by updating the test-case assertions.
- If the assertion is checking something that you don't want to check or something that changes over time (if it is checking the day of the month), comment out the assertion.
- If the test case doesn't check something you want to assert or if the test case is no longer valid, delete the test case. For instance, this might be appropriate if someone adds validation checks to some methods, and now the inputs that were originally generated are no longer valid.
- Additionally, you might want to further enhance the tests with additional logic, assertions, or new test cases.
The end result of this repeated daily process is a regression suite that evolves with the application, which is more robust and more intelligent than it was the day before. By spending a fraction of each day to keep the test suite in sync with the application and by continually enhancing it by building more intelligence into the tests, development teams can increase the lifespan and value of the test cases they already created, and expose most code-regression errors as soon as they are introduced—which is when they are fastest and easiest to diagnose and fix.
Extending the Test Suite
Automated test-case generation is what makes Behavioral Regression Testing practical for teams that haven't been building functional test cases for every piece of code as developed. Automated test cases are invaluable for quickly generating a baseline test suite to help you identify whether code modifications change code behavior. However, they may need to be extended to verify the continued operation of use cases, check scenarios that require a complicated setup, or simply improve test coverage. This requires injecting human intelligence into the automatically generated test suite.
One way to extend the test suite is to enhance the automatically generated test cases (that is, by adding more function calls, using more realistic inputs, ensuring that objects are properly initialized, and so on). Another way is to write your own functional tests using whatever test frameworks are applicable for your application (JUnit, Cactus, CppUnit, NUnit, and the like). If the team has any legacy test cases (for example, unit test cases that developers wrote for one-time verification), those should also be integrated into the test suite. Additionally, leverage any other technologies you have at your disposal to generate test cases that cover vital functionality and/or provide increased coverage of your application. These tests should all be executed in concert with the automatically generated test cases, letting you see the cumulative coverage of the complete test suite.
For example, if you are working on a web application and/or SOA, tests that exercise the application from those perspectives should also be added. Ideally, such tests should be converted to source-code level tests (such as JUnit or HTTPUnit). This correlates front-end and message-layer behavior to the back-end code, which helps you:
- Isolate the code responsible for any issues discovered in the Web or SOA test scenarios.
- Determine how much of your code is exercised by your complete test suite.
Pattern-matching and data-flow static analysis are also valuable extensions to the regression test suite. Traditionally, people do not consider static analysis as part of the regression test suite, but it can be a very important tool in detecting regressions. Tools that can run pattern-based and data-flow static analysis are good at identifying mechanical errors that would take humans a long time to find. Think of it this way: This morning, no violations were reported for your code. You change the code today, then when you arrive at work tomorrow morning, you see a violation reported for that code, alerting you to a coding problem that—if not corrected—causes resource leaks. This is essentially a regression failure—it indicates that the recent code modifications introduced problems into the code. If static analysis was not performed as part of the regression testing, this problem might have gone unnoticed until load testing later in the development process, when it would be considerably more difficult to diagnose and repair.
