Understanding the Key Phases
There are two key phases involved in creating and using Behavioral Regression Test Suites:
- Phase 1. Automated one-time test generation. A one-time process that involves generating a regression test suite that can be run on a regular basis and will not report any errors unless the captured behavior changes (Figure 1).
- Phase 2. Automated test generation for new code + Automated daily execution + extend tests. A daily process that involves automated generation of regression tests for new code, as well as automated execution of the existing test suite. After the execution completes, you review and respond to any functional changes that the test suite detected. The response process involves injecting human intelligence into the test suite.
Once per project, an automated unit test generation tool scans the project code base (accessed from the source control system), then automatically generates and executes unit test cases with assertions that capture the project code's current functionality for a wide range of automatically generated test scenarios. Test execution results should be stored on a central server, where they can be shared team-wide, and the generated tests should be added to source control. At this point, you might want to fine-tune the test suite to minimize noise (by ignoring any assertions that fail on a second run against the exact same code base, ignoring time-sensitive and date-sensitive assertions, and so forth). However, such fine-tuning is completely optional at this phase.
Because your goal is to establish a behavioral baseline rather than verify the application's current functionality, there is no need to actually review—or even glance at—the generated tests or the assertions/observations/outcomes attained. Assuming that your application is currently functioning as expected, just blindly verify all the reported assertions. Or, even better, configure the tool to automatically set the actual outcomes as the expected ones.
No test failures should be reported for the initial run of this test suite. To minimize the amount of noise presented, failures should be reported only when code behavior changes as a result of code-base modifications (Figure 2).
This daily process involves running the evolving code base against the behavioral regression test suite on a regular basis. Running in batch mode, the unit-testing tool scans the modified project code base (accessed from the source control system), automatically executes the existing regression test suite, and also automatically generates regression tests for new code. Test failures are reported only if the current code behavior does not match the behavior captured in the baseline test suite.
This alerts you whenever the previous day's changes caused the tests to fail. If this occurs, you need to review and address the reported failures. At minimum, you need to respond to failures daily to keep the test suite in sync with the app. As resources permit, you may also extend the test suite as needed to cover any critical application functionality not yet represented in the test suite.
Ideally, the results from this automated test execution not only reveal which test case assertions failed as a result of the previous day's code modifications, but also indicate exactly which developer modified the code that broke each assertion. For instant feedback on whether their code changes broke the existing functionality, developers can import information into their IDE about the regression failures caused by their modifications. Because the regression failures are directed to the developers responsible for them, the overall process of fixing them is much more streamlined than it would be if all developers were looking at the same list of regression failures. And since the results are available in their IDEs, resolution is faster and easier.
