Software Testing

Seven Questions – Why do I test?

/ Arborosa / 1 Comment

Reasons for testing

The question why something is tested has kind of a schizophrenic nature to it. Its answer is either so obvious that the question itself is ignored or it is so cumbersome that testers rather avoid to answer it. The latter is mostly the case if testers have to defend why testing is done in the first place. I cannot provide you with ready made answers for this, because the answer depends too much on the circumstances and context of your situation. What I can tell you is that it is worth while for you to figure out why your specific subject under test needs testing. If you can answer this for your specific situation, then the contextually acceptable general answer should be able to be derived from it.

What to take into consideration?

One of the more common ideas on why software testing is conducted is that it’s done to find bugs. The idea is that the fact that you do or do not find more then a certain amount of bugs in time is a measurement for release readiness. Obviously such a amount of bugs doesn’t say anything about how serious the remaining bugs are nor does it say anything about the bugs you did not find. Already more than 40 years ago Edsger W. Dijkstra (1969 p.16 and 1970) discovered that software testing can show the presence of bugs, but never their absence .

Another reason to do software testing is that there is some internal or external reason to do it. A standard, law or regulation exists that either states that you have to test or whose interpretation makes management believe that you should do tests, often in a certain predefined way, to meet the rules. This is not a bad thing as an external reason for testing.

A better, more internal, reason for testing is that the software is tested to provide information. Better still Information that is meaningful with regard to the product itself, its intended use, its real use, its potential (miss)use and related to the value this has to which stakeholders.

Your Challenge

It is your challenge as a tester to find out what the information is that the stakeholders value. Then extend this with the information they should value, even if for reasons thus far unknown to them. And finally to find a way how to provide that information so that its relevance and value is delivered to them in a meaningful way.

With some well-directed extra effort the value of testing can grow. Both to the tester and to the stakeholders.

Finding the right reason

Way back in 1998, with the introduction of the Euro to the financial markets I came in contact with software testing for the first time. As a business acceptance tester I was responsible of judging whether the new programs actually had the desired functionality and if we could work with them. Especially that latter part had the focus of my attention. Being one of the users myself and being involved in the requirements design of the product I found it easy to understand why this had to be tested and what value to look for.

More often than not software testers are not so familiar with the everyday practical needs and demands of the product they are working on. In this case I have two approaches that I prefer to use and that have served me well in the past. The first is to approach testing heuristically, with for instance the Heuristic Test Strategy Model,  and explore the product with helpful mnemonics like FEW HICCUPPS . The second approach is to converse and to keep on conversing with the stakeholders and ask them all they need to know.

Who else would know better what matters to them than the people who matter (to the product and/or the project) themselves.

Why do I test, summarized.

Seven Questions

/ Arborosa / 5 Comments

This is the opening blog of a small series of posts in which I elaborate on a test approach heuristic using 7 questions that I have developed over the years.

Thinking about testing

As a tester I have seen many approaches to software testing pass me by. A few of them, like TMap (Next) and ISTQB were picked up by the Rabobank and I have had the mixed pleasure of working with them. But regardless of how different the approaches voice out to be from each other they all seem to have a number of things in common:

  • They are mostly oriented on management (of testing)
  • They focus on processes and deliverables
  • They do not teach you how to actually test something in practice
  • They hardly make any connection to software development in general
  • They are supposedly mastered after certification

I admit that both TMap and ISTQB (initially) helped to give testing a positive foothold in many organisations and have underlined that testing should get its place in software development. Even so the five elements I have described above should also show that there are fundamental flaws in how these approaches apply testing. Following them does not guarantee you to get fully involved into software development nor does it teach you how to test in practice. Usually as a compensation for these flaws testers go to boot camp like courses to teach them more practical testing skills, like determining test coverage and applying test design techniques. Even so for many testers the start-up of their professional life is focused on getting the certification and maybe some introduction to actual testing. And then….well…..for of them most it ends here and they go out to work and follow the processes and deliver a bunch of documents. If your new to software testing this will probably keep you busy for a while, but eventually you (should) start to ask yourself questions like: Is there no better way to test this? Do I really have to write these elaborate test plans / test scripts / test cases that nobody seems to really care about? Why don’t the developers agree with me on my defects? Why is my work not valued?

I have asked myself these and similar questions and over the years I have come up with a set of alternative questions whose answers guide me through a development / test cycle. These questions demand creativity, knowledge and skilled experience to answer them. And any answer you can come up with this time will differ the next time you ask yourself that same question again.

The seven questions I use are:

I have done a talk on these seven questions at EuroSTAR 2012 and will do the at Belgium Testing Days 2013. Contact me there if you want to meet and talk, or sent me a tweet @arborosa .

Here are some twitter reactions that I got from talking at EuroSTAR 2012

DEWT 2 – Drinks, dinner, more drinks and lots and lots of talk (a.k.a. day 1)

/ Arborosa / 1 Comment

I was very pleased to be part of the 2nd DEWT workshop. This Peer conference took place on October 5 and 6 at Hotel Bergse Bossen in Driebergen, the Netherlands. The Dutch Exploratory Workshop on Testing (DEWT) is a workshop that falls into the series of peer workshops on testing like LAWST, LEWT, SWET and GATE. The main theme of this workshop was:

 Experience Reports: Implementing Context-Driven Testing.

Friday evening started with gathering in the grand cafe, dinner and a lightning talk by myself in the conference room. I had planned and prepared the lightning talk to be about the summarized content of my EuroSTAR conference talk “Seven Questions to Help You on the Path of Testing”.

A talk that I have also entered for the next Dutch Testing Conference. The Dutch Testing Conference has a process of sending review comments on your abstract and one of the remarks in the review, a couple of days before, made me change my mind about the lightning talk. That particular remark reminded me that the context driven approach to testing is far away from being common knowledge in the software testing world.

I wondered how we communicate our view on testing and why it is that we are not really effective in bringing the message of context driven testing across?

The section below describes what my thoughts were about the content of my lightning talk mixed and enhanced with parts of the discussions arose after the discussion and later that evening when we were having drinks and the next day during the conference.

Since I started to see myself as a follower of the context driven testing (CDT) approach I have seen many a tweet and blog pass by proclaiming that if you follow context driven testing results will be much better than if you are using the old school / traditional / factory / waterfall / IEEE/ ISTQB process approaches to testing. And roughly parallel with my start of being context driven I also started to work in an agile environment. And like CDT many agilists proclaim superiority on traditional or waterfall approaches of software development. And sure enough if applied consciously and thoughtfully both approaches often yield better results than the more process oriented ‘traditional’ approaches. But even so I believe both followers of CDT and agile are making a few fundamental mistakes in their communication.

First of all the agile manifesto and the context driven testing principles have been around for over ten years now and at that time it probably was a good idea to react against the then main stream ideas of software testing. However a lot of time has passed since then and most people who had (or still have) the same feeling towards the ‘old’ approaches have changed to following the ideas of CDT and / or agile by now. So this rhetoric will have no effect on them anymore. Well maybe it could create some kind of group identity, but that can hardly be the purpose in my opinion.

Second I wonder how many people actually work in a true traditional / waterfall assignment. Personally I admit to having worked in an environment based on so-called waterfall process principles, but even then I saw myself as being in a true waterfall assignment. And in that particular case hardly anybody fully committed to doing waterfall (or to be more specific TMap) by the book. And obviously this underlines the flaw of such process and deliverable oriented approaches, but it also shows that a lot of people in such an environment do not commit to the approach as such. So trying to sway them to follow your approach because their approach is ‘bad’ might look appealing, but kind of misses the point as a lot of people in that environment will feel you are not talking about them. And even if they feel addressed by your comments how will this make them change. For the better part they already agreed with you so what your telling is either no news or not adding helpful information.

Finally, and this is for me personally the main conclusion of the discussions. Being context driven is not about being against something. It is about being a (self) educated, thoughtful, skillful, open to your context testing craftsman. A craftsman that is likely to engage in to an open discourse with his peers and team mates, who’s drive it is to learn what is necessary to do the job and then learn some more just for the fun of it. As Joris Meerts aptly pointed out “Yes that does make us elitist.” For unfortunately there are not that may software testers that invest in maintaining or even extending their software testing (and general) skills, let alone seeking to collaborate with others while they are doing that.

Even after one and a half day of conferring the question on how to bring the context driven testing message and mindset across is still not answered. So let me know if you come up with useful ideas and I will collect them and come back to the subject on our next DEWT peer conference (or any time sooner when we meet).

Standing left to right:
Ray Oei, Jean-Paul Varwijk, Adrian Canlon, Markus Gartner (Germany), Ruud Cox, Joris Meerts, Pascal Dufour, Philip Hoeben, Gerard Drijfhout, Bryan Bakker, Derk Jan de Grood, Joep Schuurkes, Lilian Nijboer, Philip-Jan Bosch, Jeroen Rosink, Jeanne Hofmans
Kneeling left to right:
Tony Bruce (UK), Zeger van Hese (Belgium), Ilari Henrik Aegerter (Switserland), Huib Schoots, Peter Simon Schrijver