Regression Testing

As a follow up in the testing definition series it was my intention to continue with covering Test Types. Initial investigation showed what I had already feared. Such a post would become a Herculean task and probable my longest post ever. So I will continue with that particular endeavor sometime later tackling it one step at a time. This post for starters covers one of the most common but also one of the most peculiar types of testing

“Regression Testing”

Regression Testing is so common as a testing type that the majority of books about software testing, and agile for that matter, that I know, mention regression testing. Almost as common however is that most of them either or both do not tell what regression testing is or do not tell how one should actually go about and do regression testing. To be fair an exception to the latter is that quite a few, particularly the ones with an agile demeanor, tell that regression testing is done by having automated tests but that is hardly anymore informative is it.

Before I go further into regression testing as being peculiar first inline with the previous posts a list of regression testing definitions:

  • Checking that what has been corrected still works. (Bertrand Meyer; Seven Principles of Software Testing 2008)
  • Regression testing involves reuse of the same tests, so you can retest (with these) after change. (Cem Kaner, James Bach, Bret Pettichord; Lessons learned in Software Testing 2002)
  • Regression testing is done to make sure that a fix does what it’s supposed to do (Cem Kaner, Jack Falk, Hung Quoc Nguyen; Testing Computer Software 2006)
  • Regression testing is the probably selective retesting of an application or system that has been modified to insure that no previously working components, functions, or features fail as a result of the repairs. (John E. Bentley; Software Testing Fundamentals Concepts, Roles, and Terminology 2005)
  • Retesting to detect faults introduced by modification (ISO/IEC/IEEE 24765:2010)
  • Saving test cases and running them again after changes to other components of the program (Glenford J. Myers; The art of software testing 2nd Edition 2004)
  • Selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements (ISO/IEC/IEEE 24765:2010)
  • Testing following modifications to a test item or to its operational environment, to identify whether regression failures occur (ISO/IEC/IEEE 29119-1:2013)
  • Testing if what was tested before still works (Egbert Bouman; SmarTEST 2008)
  • Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. (Standard glossary of terms used in Software Testing Version 2.2, 2012)
  • Testing required to determine that a change to a system component has not adversely affected functionality, reliability or performance and has not introduced additional defects (ISO/IEC 90003:2014)
  • Tests to make sure that the change didn’t disturb anything else. Test the overall integrity of the program. (Cem Kaner, Jack Falk, Hung Quoc Nguyen; Testing Computer Software 2006)

Looking at the above definitions the general idea about regression testing seems to be:

“To ensure that except for the parts of the areas* that were intentionally changed no other parts of these areas or other areas of the software are impacted by those changes and that these still function and behave as before”.
(*Area is used here as a general expression for function, feature, component, or any other dimensional divisions of the subject under test that is used)

The peculiar thing now is that however useful and logical such a definition is it only provides the intention of this type, or should I say activity, of testing. Regression testing could still encompass any other testing type in practice.

To know what to do you first need to establish which areas are knowingly affected by the changes and then which areas have the most likelihood of being unknowingly affected by the change. Next to that there probably are areas in your software where you do not want to take the risk of them being affected by the changes. In his presentation at EuroSTAR in 2005 Peter Zimmerer addresses the consequences of this in his test design poster by pointing out that the wider you throw out your net for regression effects the larger the effort will be:

  • Parts which have been changed – 1
  • Parts which are influenced by the change – 2
  • Risky, high priority, critical parts – 3
  • Parts which are often used – 4
  • All – 5

Once you have identified the areas you want to regression test you still need to figure out how to test those areas for the potential impact of the change. The general idea to solve this, in theory at least, seems to be to rerun previous tests that cover these areas. As this might mean running numerous tests for lengthy periods of time many books and articles propose to run automated tests. This will however only work if there are automated tests to use for testing these areas to begin with. And even if there are you still need to evaluate the results of any failed test and there is no clear indication of how long that may take.

How do you know that these existing tests do test for the impact of the change? After all they were not designed to do so. For all you know they might or might not fail due to changes to the area that is tested by them. Either result could therefore be right or wrong in light of the changes. The test itself could be influenced by an impact of the change on the test (positive or negative) that was not considered or identified yet.

All in all regression testing is easily considered to be necessary, not so easy to determine, difficult to evaluate on success and considerably more work then many people think. Even so next to writing new tests it probably is the best to solution to check if changes bring about unwanted functionality or behavior in your software. My suggestion to you is to at least change the test data so that these existing tests have a better chance of finding new bugs.

Advertisements

Seven questions – What questions do I have?

The previous two questions helped you to find why testing is necessary, what information you need to answer the first question (business value) and which test ideas help you deliver meaningful and relevant information. This post now extends this to areas that help you identify the circumstances in which you will have to do your work. It ends with a little advice that you should not take things for granted especially if you do not understand them.

DID-A-TEST

Originally called Jean-Paul’s test this mnemonic represents a set of surveying questions that helps you identify working conditions. Once you have the answers to these questions you should check if and if so how this influences your ability to test and the ability to give more or less rich information to your stakeholders. You can use these questions to identify  boundaries and constraints to your testing possibilities and address them or at least be and make others aware of them. These questions are by no means exhaustive, but in my opinion they form a good starting point in exploring your test context.

Are the Developers available?

Developers are physically close of far from you. They are more or less available in time or more or less organizationally accessible to testers. The ability or inability to work together with development can influence your risk assessments, your insight into risk areas, your knowledge about development solutions and what is or is not covered by development testing activities. Additionally when addressing developers it is good to know the preferences and willingness of each developer with regard to working with testers.

How soon do you have access to Information?

Of course you can use the FEW HICCUPPS mnemonic (James Bach, Michael Bolton) to improve and expand your test ideas, but gathering information about the intended product or solution is a main starting point and important reference to work with. So getting access to the sources of information or even better being involved in the information gathering should start as soon as possible.

Do you control the test Data?

My interpretation of test data here is wide in the sense that I do not only mean the ability to enter different types of inputs, in different variations and quantities. I also mean the ability to set up and load data sets creating test scenarios. And the ability to set or remove states in the software. Being able to control the data is beneficial in speeding up test execution, creating typical test situations and helps to quickly repeat the test case if necessary.

Having control of the test data is only one side of the story. The other side of the story is that you need to find the right ‘Trigger Data‘ to use. Trigger Data is any data item, set of data or data state specifically created and used to invoke, enable or execute your test case (scenario).

Are the Analysts available?

Like the developers the availability, both physical and in time, of the (business) analysts has an impact on the way you can interact with them. And like the developers analysts will have preferences and are more or less willing to work with testers. The impact of this might however be larger as analysts are often the first source of information about the products intended functionality and its means of satisfying the stakeholders needs and wants. They are often also a sort of gate(keepers) in communicating to business stakeholders. In that sense they can make a testers live more or less easy. Especially if testers are not expected to go outside of the projects boundaries.

Are the (other) Testers available?

In my experience working as the only tester on a project has an impact both on the way you work and to some extend to the quality of your work. Being able to pair, share thoughts or just have a chat with another tester can help you reconsider your work and develop new or different test ideas. The tester doesn’t necessarily have to be in your team to have this effect. Having other testers in your team brings both the benefit (and sometimes burden) of being able to divide work, get fast feedback on test ideas or test results and the possibility to focus or divert away from your strengths and weaknesses as a tester.

Do you have a quiet work Environment?

This question addresses two different aspects. The first aspect is the infrastructure. Do you know what it’s components are? Do you have a separated test environment? And if so are you its only user? Do you know how to get access to it? Are you allowed to change it yourself or do you need others to do it for your? Is your test environment similar to the real production environment?

Secondly it addresses the circumstances of your workplace. Do you work in isolation, in cubicles, or in a large office garden? Is your work uninterrupted or are you (in)voluntarily involved into other work processes and activities? Does that influence your performance and well-being? What the influences are obviously depends on you as person and the real circumstances. But it is wise to take note and consider possible consequences. There are many studies into this field. Here are few articles that might trigger your interest: “Designing the work environment for worker health and productivity” by Jacqueline C. Vischer;  “Interrupt Mood” by Brian Tarbox or “Where does all that time go” by Michael Bolton.

Are the Stakeholders (that matter) available?

Stakeholders come in many forms and shapes, but they have one thing in common. They are in someway involved in the creation and/or use of the software solution. That not only means they need to be informed about the product that also means that they have expectations and opinions about the product itself, what it is used for, and what the products needs to able to do to make it valuable to them. As a tester you should identify these expectations and opinions and tailor your information about the product so that it is meaningful to them.

In theory the effort you put into gathering, tailoring and presenting that information is based on how much the stakeholders matters to the product, the project and to some extend to you the tester. I say in theory because to do so in practice the stakeholders need to be available and accessible. If they are not or if it is difficult you should take the extra time and effort into account of your testing and test reporting.

Is there (mandatory) Tooling?

There are many types of tools available in the market to capture requirements, store test cases, log test execution or manage bugs. And likewise there are many tools available to use during testing. As a tester you need to find out which tools there are, which tools you are allowed to use, and which tools are mandatory to use. You will might not know all the tools you are faced with or are unable to use a tool that you already know and like. In that case you will have to get used to the ‘new’ tooling and learn to use it. Additionally many tools have inbuilt workflows and processes that take away time from actual testing. As a tester you should be aware of this and take this into account when testing.

Poutsma principle

Whenever I start on a new test assignment or pick up a new work item I need to search and find its purpose, its meaning and I need to understand how the chosen requirements offer a solution to the problem that is solved. Sometimes that is really easy.

Say you visit the 36th International Carrot Conference before going to CAST 2013.  You come home and decide to sell carrots for hungry rabbits online and you want to vary the amount of carrots or differentiate the type of carrot for different breeds of rabbits. You will need something like drop down list or input field to identify the different rabbit breeds.  And except for the sudden urge to sell carrots this is fairly easy to understand and test.

If however you are asked to test the software implementation of calculating results for a new Credit Risk Model used by an international bank you will have a lot more to understand. If so I remind myself of the Poutsma Principle:

If something is too complex to understand, it must be wrong.

I use this principle to remind myself to keep asking questions until I either understand it or except the argumentation of it as proof. In either case it helps me to break down requirements to a level that makes me confident enough to start testing and daring enough  so that I can also use my personal addition to the principle

And it is your job (as a tester) to proof it wrong.

If you want to know more about the Poutsma Principle you can follow this link.

Questioning Testing

This years EuroSTAR 2013 theme – Questioning Testing

I was a speaker at last years EuroSTAR and I am still enthusiastic and proud to have been there. Being a speaker adds very much to the experience and since I believe I have still more to share with the community I plan to enter for a talk in this years event. Something I  can recommend to everybody willing to learn, share and invest the necessary time.

Sending in a good abstract is however not so easy and needs next to having a good idea also the ability to write a good proposal. Last December, to share how we managed to write proposals that allowed us to go to many of conferences as a speaker  Huib Schoots, Derk-Jan de Grood and I held a short workshop on proposal writing.  Derk-Jan wrote a small blog post about it. I am continuing some of that effort in this post.

Last week Anne-Marie Charrett was so kind to review one my proposals and give me some good tips. During that session I also showed her a mind map that I had made in while preparing. At some point during our session she pointed out to me that perhaps it was a good idea to share the mind map with the community. I hadn’t really thought of it myself but it immediately struck me as a good idea. So to help you on your way, and even at the risk of bringing in competition, I would like to share with you the mind map that I made while preparing my proposal. It summarizes the information that Michael Bolton and Allan Richardson shared on writing an abstract.

EuroSTAR Call for papers 2013

So good luck and maybe see you there!

Seven Questions – What will I test?

In the previous post I addressed why you test. In this post I will address what you are going to test. In the sections below I will hand you a number of ideas. Some of them are well-known in main stream testing and other may be a bit more out of the box to some of you.

Requirements

I think I am not exaggerating if for most testers and stakeholders covering the written requirements with test cases still is what you as a tester should do. This line of thought is more generally known under the title of Requirements Based Testing. As a definition this could be described as:

Requirements-based testing (RBT) process addresses two major issues: first,
validating that the requirements are correct, complete, unambiguous, and logically
consistent; and second, designing a necessary and sufficient (from a black box
perspective) set of test cases from those requirements to make sure that the design and code fully meet those requirements

Followers of this approach do admit that requirements, or rather documents, are hardly ever complete and that testers almost never have the possibility to design and execute all possible tests. This last insight is the foundation of an approach called Risked Based Testing (or sometimes called risk and requirements based testing). You can find more information on it here, here, and slightly more hidden in here. But personally I like this   interpretation of risk based testing, by J. Bach, best.

A major advantage of risk based testing, and in my opinion essential in finding out what to test, is that it is based not only on documents as a source, but that it is based on people as a source of information. People are a major source of requirements and especially of requirements that matter to them. Documents can tell you a lot, but people can tell you what’s missing in them, how to give meaning to them and therefore paint you a more complete picture.

A good source of information, besides risks, that people can also give is how they current software or the manual operation that they use to solve the problem now. Often this will be the basis for the acceptance criteria that they formulated, or that you will formulate together with them for the new software. Acceptance criteria are useful to give direction and focus to your testing and are mandatory elements in the structure of your testing report.

You the tester

Finding out what to test is not only about bringing the subject under test and its information to you the tester. It is also about what you bring to the subject. What knowledge of the domain do you already have? What test ideas or mnemonics do you know that are useful in the context of your subject?

Sources for Test Ideas and mnemonics

The list below has some online sources you can use and fit into your situation. Use them, adjust them and come up with your own ideas.

Test Cases

There are many definitions for a test case. My personal one is: “A test case is the execution-able version of a test idea. A test case aims at gathering information about the test ideas relation to the subject under test and the value of that information to the stakeholders.” If and how much of a test case is (pre-)scripted is based upon personal preferences, the test idea itself and the context in which it is executed. Attention should in any case not go towards the quality of scripting of a test case, but towards the relevance and value of the information that is delivered by the test case.

Designing test cases, scripted or otherwise, takes skill and experience. I therefore believe that as a start a good tester should have a base of test design skills and a testing mindset. As a start I would like to recommend the following literature:

  • TMap Next (Chapter 14) – Tim Koomen, Leo van der Aalst, Bart Broekman, Michiel Vroon
  • Lessons Learned in Software Testing – Cem Kaner, James Bach, Bret Pettichord
  • Essential Software Test Design – Torbjörn Ryber
  • What is good test case? – Cem Kaner

When working on test cases there are a few principles that I like to keep in mind. The DRY principle. DRY stands for “Do not Repeat Yourself”. If test cases draw their value based on the information they offer it is not useful to use variations of tests that do not deliver new information. This principle does have a catch as you do not always know in advance if a variation does or does not deliver new information or not.

Separation of concerns is another principle that lets you focus your attention upon some (single) aspect. This does not mean that the other aspects are ignored. It means that you address the subject under test with a singular point of view, deeming the other’s irrelevant at that point in time while not forgetting that relationships might exist.

Finally I am always interested in mentions like “You Ain’t Going to Need It” (YAGNI) or “No user would do that” popping up in any design discussion between other team members. The areas discussed are an interesting area to start investigating.

This concludes this third post of the Seven Questions series.

Seven Questions – Why do I test?

Reasons for testing

The question why something is tested has kind of a schizophrenic nature to it. Its answer is either so obvious that the question itself is ignored or it is so cumbersome that testers rather avoid to answer it. The latter is mostly the case if testers have to defend why testing is done in the first place. I cannot provide you with ready made answers for this, because the answer depends too much on the circumstances and context of your situation. What I can tell you is that it is worth while for you to figure out why your specific subject under test needs testing. If you can answer this for your specific situation, then the contextually acceptable general answer should be able to be derived from it.

What to take into consideration?

One of the more common ideas on why software testing is conducted is that it’s done to find bugs. The idea is that the fact that you do or do not find more then a certain amount of bugs in time is a measurement for release readiness. Obviously such a amount of bugs doesn’t say anything about how serious the remaining bugs are nor does it say anything about the bugs you did not find. Already more than 40 years ago Edsger W. Dijkstra (1969 p.16 and 1970) discovered that software testing can show the presence of bugs, but never their absence .

Another reason to do software testing is that there is some internal or external reason to do it. A standard, law or regulation exists that either states that you have to test or whose interpretation makes management believe that you should do tests, often in a certain predefined way, to meet the rules. This is not a bad thing as an external reason for testing.

A better, more internal, reason for testing is that the software is tested to provide information. Better still Information that is meaningful with regard to the product itself, its intended use, its real use, its potential (miss)use and related to the value this has to which stakeholders.

Your Challenge

It is your challenge as a tester to find out what the information is that the stakeholders value. Then extend this with the information they should value, even if for reasons thus far unknown to them. And finally to find a way how to provide that information so that its relevance and value is delivered to them in a meaningful way.

With some well-directed extra effort the value of testing can grow. Both to the tester and to the stakeholders.

Finding the right reason

Way back in 1998, with the introduction of the Euro to the financial markets I came in contact with software testing for the first time. As a business acceptance tester I was responsible of judging whether the new programs actually had the desired functionality and if we could work with them. Especially that latter part had the focus of my attention. Being one of the users myself and being involved in the requirements design of the product I found it easy to understand why this had to be tested and what value to look for.

More often than not software testers are not so familiar with the everyday practical needs and demands of the product they are working on. In this case I have two approaches that I prefer to use and that have served me well in the past. The first is to approach testing heuristically, with for instance the Heuristic Test Strategy Model,  and explore the product with helpful mnemonics like FEW HICCUPPS . The second approach is to converse and to keep on conversing with the stakeholders and ask them all they need to know.

Who else would know better what matters to them than the people who matter (to the product and/or the project) themselves.

Why do I test, summarized.

Seven Questions

This is the opening blog of a small series of posts in which I elaborate on a test approach heuristic using 7 questions that I have developed over the years.

Thinking about testing

As a tester I have seen many approaches to software testing pass me by. A few of them, like TMap (Next) and ISTQB were picked up by the Rabobank and I have had the mixed pleasure of working with them. But regardless of how different the approaches voice out to be from each other they all seem to have a number of things in common:

  • They are mostly oriented on management (of testing)
  • They focus on processes and deliverables
  • They do not teach you how to actually test something in practice
  • They hardly make any connection to software development in general
  • They are supposedly mastered after certification

I admit that both TMap and ISTQB (initially) helped to give testing a positive foothold in many organisations and have underlined that testing should get its place in software development. Even so the five elements I have described above should also show that there are fundamental flaws in how these approaches apply testing. Following them does not guarantee you to get fully involved into software development nor does it teach you how to test in practice. Usually as a compensation for these flaws testers go to boot camp like courses to teach them more practical testing skills, like determining test coverage and applying test design techniques. Even so for many testers the start-up of their professional life is focused on getting the certification and maybe some introduction to actual testing. And then….well…..for of them most it ends here and they go out to work and follow the processes and deliver a bunch of documents. If your new to software testing this will probably keep you busy for a while, but eventually you (should) start to ask yourself questions like: Is there no better way to test this? Do I really have to write these elaborate test plans / test scripts / test cases that nobody seems to really care about? Why don’t the developers agree with me on my defects? Why is my work not valued?

I have asked myself these and similar questions and over the years I have come up with a set of alternative questions whose answers guide me through a development / test cycle. These questions demand creativity, knowledge and skilled experience to answer them. And any answer you can come up with this time will differ the next time you ask yourself that same question again.

The seven questions I use are:

I have done a talk on these seven questions at EuroSTAR 2012 and will do the at Belgium Testing Days 2013. Contact me there if you want to meet and talk, or sent me a tweet @arborosa .

Here are some twitter reactions that I got from talking at EuroSTAR 2012