Regression Testing

A while ago a new developer approached me to discuss some changes he wanted to deploy to the test environment. He had re-factored parts of the application workflow management and redesigned a number of reports. We talked about it and he stressed that these where major changes and a lot of work to test. While we continued discussing I commented “that’s okay it will take some time but we have a regression test set just for this that we can use to work with and that will help us…”

He looked at me with total bewilderment and uttered: “Didn’t you get anything of what I said. These are major changes. No regression test will cover that! They will all fail!!”

I looked at him and realized…

He was right

His understanding is based on a common perception where regression tests mean something like running the same step by step tests as before to see that they have the same step by step results as before. And if that were the case he would obviously be right. There was no way I could execute the tests the same way as before simply because the workflow method and GUI functionality had changed dramatically. Nor could the results be exactly the same given the design changes he had made.

But I was right too

His reaction made me realize that over the years my understanding of a regression test set has changed dramatically. I don’t see test cases as something that provides a step by step description. I see test cases as a doable and executable follow-up of a test idea. A test case to me consists of the following elements:

• A test idea; what do you want to learn, verify/validate or investigate?
• A feature; which part of the test object do you investigate?
• Test activities; a way to exercise your test idea and see how the software behaves
• Trigger data; data that should trigger the behavior you want to see
• An oracle; something that tells you how to value the behaviors information/data

With that understanding I looked at the changes and had concluded that the existing test cases were still useful. Even if the workflow management was re-factored I could still apply the test ideas we had used before. In essence the same features were affected and my oracles should still be valid. To me adjusting the test activities and trigger data, in this case to match the workflow changes, is something I tend to do anyway. With these adjustments I try to make re-running test cases more effective as variation adds new chances of finding new or different bugs.

With the reports the differentiation was slightly different. Test ideas, features, test activities and trigger data could stay more or less the same but my oracles, the current template reports, had changed. The reports now had a new layout but their content had changed only minimally.

Reference test

To avoid future confusion I came up with a new name for my tests which, as far as I am aware, has not been used in software testing before. I will call these tests Reference Tests.

A Reference Test then is a test where under similar circumstances similar input results in a similar outcome of the test if evaluated against an identical oracle

I am aware that ‘similar’ and ‘identical’ in this definition are relative concepts and I have chosen them on purpose. It expresses that each time that such a test is used the tester needs to be aware which information it tries to show or uncover, how it is doing that and to what purpose. This discourages mindless repetition or pass/fail blindness. It encourages thoughtful selection and execution of tests and deliberate evaluation of test results.