At the time of this blog post my family and me are on holiday in Iceland. Since we are not that often in Iceland we, amongst visiting relatives and friends, use the time to look into administrative and regulatory stuff that is easier to do in Iceland than from abroad.


One of the things necessary is to renew my wifes passport. For which you actually need to physically go to the Civil Registry, or in icelandic ´Syslumen´. The process of renewal is (boringly) straightforward. At the office you get a number, wait, identify yourself and pay for your renewal, get a form, wait, identify yourself again, hand over the form and update your data (including new digital photo and fingerprint), sign and wait for a couple of weeks to pick up your passport at the Civil Registry.


Since we´re only on holiday in Iceland a couple of weeks of waiting is not a real option. So to amend this my wife investigated and proposed the solution to sent the new passport to the consulate in our country. An option, once validated by the team lead, that was acceptable to the civil clerk. And thus the proper check box was looked for and found.

Into the process

After filling in the personal details instead of the offices address the address of the consulate was needed. The page itself did not offer any listing. The help page wasn´t really helpful either as it only pointed towards a government listing at another department. After some searching the consulate in Amsterdam and its address was found and the data could be entered. So everything was entered and the OK could be clicked. Nothing happened. Looking over the page the clerk found:

Færðu inn lögboðnar reit (Please enter mandatory data) next to a field asking for Póstnúmer (Zip code) that had been left empty as it had also been empty on the government listing. So what to do? My wife and the clerks colleague suggested to google it. And so she did. The zip code was entered and again the OK was clicked. The intranet page jumped back to the entry page and everything looked okay. But the clerk rightfully noted that the usual confirmation message was not shown and checked my wifes file. To her, and my wifes surprise no data was added, meaning the whole 20 minute worth of data was absent.

The process repeated itself a few times and eventually another colleague noted that the zip code contained letters. Something not used in Iceland itself. Why not leave those out of the field and move them somewhere else, say in front of Amsterdam. Now when clicking OK the confirmation appeared and a check showed that the file now contained all the data.


Even with only a couple of thousand Icelanders living abroad chances that they live in one of the eight countries (e.g. Canada, Great-Britain) using alpha numeric characters is realistic especially since many more countries use the country abbreviation in front of their zip code. So when my wife returned to tell about her plight she commented: Clearly neither the developer nor tester thought this field was important. But it really bugged me today. Further more she noted The same software company maintained the government listing and had all the zip codes removed, leaving empty spaces in the listing. That´s even more stupid.

Clearly someone must have convinced the developers and testers “You Ain´t Gonna Need It” (YAGNI).

Seven Questions – Why do I test?

Reasons for testing

The question why something is tested has kind of a schizophrenic nature to it. Its answer is either so obvious that the question itself is ignored or it is so cumbersome that testers rather avoid to answer it. The latter is mostly the case if testers have to defend why testing is done in the first place. I cannot provide you with ready made answers for this, because the answer depends too much on the circumstances and context of your situation. What I can tell you is that it is worth while for you to figure out why your specific subject under test needs testing. If you can answer this for your specific situation, then the contextually acceptable general answer should be able to be derived from it.

What to take into consideration?

One of the more common ideas on why software testing is conducted is that it’s done to find bugs. The idea is that the fact that you do or do not find more then a certain amount of bugs in time is a measurement for release readiness. Obviously such a amount of bugs doesn’t say anything about how serious the remaining bugs are nor does it say anything about the bugs you did not find. Already more than 40 years ago Edsger W. Dijkstra (1969 p.16 and 1970) discovered that software testing can show the presence of bugs, but never their absence .

Another reason to do software testing is that there is some internal or external reason to do it. A standard, law or regulation exists that either states that you have to test or whose interpretation makes management believe that you should do tests, often in a certain predefined way, to meet the rules. This is not a bad thing as an external reason for testing.

A better, more internal, reason for testing is that the software is tested to provide information. Better still Information that is meaningful with regard to the product itself, its intended use, its real use, its potential (miss)use and related to the value this has to which stakeholders.

Your Challenge

It is your challenge as a tester to find out what the information is that the stakeholders value. Then extend this with the information they should value, even if for reasons thus far unknown to them. And finally to find a way how to provide that information so that its relevance and value is delivered to them in a meaningful way.

With some well-directed extra effort the value of testing can grow. Both to the tester and to the stakeholders.

Finding the right reason

Way back in 1998, with the introduction of the Euro to the financial markets I came in contact with software testing for the first time. As a business acceptance tester I was responsible of judging whether the new programs actually had the desired functionality and if we could work with them. Especially that latter part had the focus of my attention. Being one of the users myself and being involved in the requirements design of the product I found it easy to understand why this had to be tested and what value to look for.

More often than not software testers are not so familiar with the everyday practical needs and demands of the product they are working on. In this case I have two approaches that I prefer to use and that have served me well in the past. The first is to approach testing heuristically, with for instance the Heuristic Test Strategy Model,  and explore the product with helpful mnemonics like FEW HICCUPPS . The second approach is to converse and to keep on conversing with the stakeholders and ask them all they need to know.

Who else would know better what matters to them than the people who matter (to the product and/or the project) themselves.

Why do I test, summarized.

Seven Questions

This is the opening blog of a small series of posts in which I elaborate on a test approach heuristic using 7 questions that I have developed over the years.

Thinking about testing

As a tester I have seen many approaches to software testing pass me by. A few of them, like TMap (Next) and ISTQB were picked up by the Rabobank and I have had the mixed pleasure of working with them. But regardless of how different the approaches voice out to be from each other they all seem to have a number of things in common:

  • They are mostly oriented on management (of testing)
  • They focus on processes and deliverables
  • They do not teach you how to actually test something in practice
  • They hardly make any connection to software development in general
  • They are supposedly mastered after certification

I admit that both TMap and ISTQB (initially) helped to give testing a positive foothold in many organisations and have underlined that testing should get its place in software development. Even so the five elements I have described above should also show that there are fundamental flaws in how these approaches apply testing. Following them does not guarantee you to get fully involved into software development nor does it teach you how to test in practice. Usually as a compensation for these flaws testers go to boot camp like courses to teach them more practical testing skills, like determining test coverage and applying test design techniques. Even so for many testers the start-up of their professional life is focused on getting the certification and maybe some introduction to actual testing. And then….well…..for of them most it ends here and they go out to work and follow the processes and deliver a bunch of documents. If your new to software testing this will probably keep you busy for a while, but eventually you (should) start to ask yourself questions like: Is there no better way to test this? Do I really have to write these elaborate test plans / test scripts / test cases that nobody seems to really care about? Why don’t the developers agree with me on my defects? Why is my work not valued?

I have asked myself these and similar questions and over the years I have come up with a set of alternative questions whose answers guide me through a development / test cycle. These questions demand creativity, knowledge and skilled experience to answer them. And any answer you can come up with this time will differ the next time you ask yourself that same question again.

The seven questions I use are:

I have done a talk on these seven questions at EuroSTAR 2012 and will do the at Belgium Testing Days 2013. Contact me there if you want to meet and talk, or sent me a tweet @arborosa .

Here are some twitter reactions that I got from talking at EuroSTAR 2012

250 hours of practice – January

As said in my post a couple of weeks ago, this year I would try to spend 250 hours on practicing and enhancing my testing skills. This post is a report on how I fared in January 2012. (Leaving my personal favourite untill the end…)

I started enthusiastically on January 2nd by following up on a post about the “Follow the link exercise” by Jeff Lucas. In short the exercise is to choose a blog post of your liking. You start reading it critically and then follow every link mentioned in the post. You then pursue this with every post that you read in a one hour session.

In my session, that actually lasted two hours, among others I followed up on a link to Alan Page’s blog “Tooth of the Weasel”. This post contained an overview of posts Alan wrote in 2011 so there were enough links in there to follow-up:
My job as a Tester
What is Testing?
Test Design for Automation
Numberz Challenge
Beyond Regression Tests
Judgment in Testing
Lost in the weeds

Although I had heard about Alan Page I was not yet familiar with his work. It pleasantly surprised me with some useful ideas and even some advice for my personal goals for this year. Let me give you some quotes I found interesting:

“What you do or don’t define as testing may differ per context.”

Automated testing “starts the same as always. Design your test first then automate where eligible. Coded tests do not replace, but enhance human tests.”

“Do not only use automated testing for regression. Vary the data, the sequence, randomize, to find new information” data driven testing

“Are testers’ second class citizens? NO. Are they whiners? Yes; Figure out how to get and earn respect!”

My second (larger) series of practice session(s) started with watching the 2011 GTAC keynote by Alberto Savoia with the ominous title “Test is dead”. You can read more about this on the blog post I wrote “Is testing dead?”

My third endeavour entailed reading the hardcopy of the book “Essential Software Testdesign” by Torbjörn Ryber. The E-book is free to download, but I liked the content enough to want to own it. Some warning is in order however. Even the hardcopy has a somewhat annoying number of typos, illogical sentences and even faults. Nevertheless the concepts Ryber discusses are helpful for many a tester.

Early in January the DEWT’s met up again. This time to discuss and prepare the TestNet event about context-driven testing. On January 18, some 150 testers visited the event to watch James M. Bach and Michael Bolton do a one hour introduction on context-driven testing using Go to meet  (which btw. worked brilliantly). After the break the DEWT Zeger van Hese, Ruud Cox, Ray Oei and myself gave a number of lightning talks followed by Q&A. Themes of the talks were “On being context-driven”; “Spin-Off”; “Context-Driven expert”; “Test Plan”.

All in all these activities got me some 20 hours of practice bringing me well en route for the 250 hours of testing practice. But to be honest I am even more of a test nerd. I have spent another 10-15 hours on following Twitter feeds with a peak while participating in a #Testchat lead by Lisa Crispin asking the following questions:
Q1: Have you worked on a “test automation project” that succeeded? What helped it succeed
Q2: What do you think upper management should know about testing? (not limited to automation)
Q3: related some to Q2: How do you keep your testing transparent to others on your team and in the organization?
Q4: Are testers on your team treated with the same respect as programmers?
Q5: sometimes the tester is undone by the process. Documentation outdated leading to looking like lack of knowledge

The last practice activity however was, for me personally, the most engaging, emotional and gratifying one.

In December I contacted Markus Gärtner to ask him for a challenge to see if I was worthy enough to enter the realms of the Miago-Do software school of testing. This actually the first step of the challenge having found a member. Markus offered my “The light saber” challenge. Several times during the challenge I would sent Markus my test investigation results and as many times Markus answered. I used several heuristic approaches, tried to inform the customer based on his needs and eventually offered a solution using personas. Somewhat to my despair Markus’s answers were getting shorter and repetitive and I asked Markus to debrief me.

We organized a one hour Skype session and went on with the challenge discussing results, progress en feelings during the challenge. Eventually we came to the point where Markus would reveal if I was allowed to enter Miagi-Do. The result got me stunned, silent and humbled for a moment… Not only was I a new member I was one of the members to, fully endorsed by other instructors, become a Black-belt.

I can only say again. Thanks guys, I am honoured.

What do you put in your test plan?

Theme event

This evening, January 18, I will be on stage during the TestNet theme event about Context-Driven Testing. The evening will start with a duo presentation by James Bach and Michael Bolton followed by a series of short presentations, lightning talks and discussion. In one of those lightning talks I will share a personal experience report. It describes how I changed the way I make test plans. Since most of you will not be able to be there (or might have never heard about TestNet*) I am sharing my experience with you in this post.


Even if a lightning talk last only for five minutes it still requires some preparation. So to extra prepare myself I placed my experience into perspective and placed the following message on Twitter:

” @Arborosa: Question to my tweeps: What items do you put in a test plan?  I’ll put the results on my Blog. (please retweet)”

This resulted in the following responses:

Rob van Steenbergen (@rvansteenbergen)
Scope of testproject, context of product (with mindmap), product risks and qlty attributes and risk approach, planning, who tests, stakeholders, testing tools, explanations abt testing for orgs that are still learning. TP is also promotion material for testing.

Stephan Kämper (@S_2K)
Well, what to put in a plan? A (current) goal of what you’re planning. The major way you’ll follow to reach said goal. A ‘Plan B’. (Known) risks – What’s the risk of following the plan? …the risk of *not* following it? Tools & Techniques? Not sure about these.

Nitin Hatekar (@nhatekar)
Entry and exit criteria for each test phase and specific test approach for each phase. Scope of testing and the estimates for completion of in-scope test efforts. A section for assumptions, risks & blockers as well.

Rik Marselis (@rikmarselis)
For your testplans take IEEE829 (1998) as a starting point. And see tmap.net for templates ;-) (And after a reprimand by Huib Schoots to be more serious) Don’t start with making the testplan. First make the outline of the testreport. That’s your deliverable! The testreport outline must be discussed with stakeholders. Then you have startingpoint for your testplan.
Jesper L. Ottosen (@jlottoosen)
Generally answers and descriptions to “how” – to the level required of the context. ie #itdepends ;-)
Jan Jaap Cannegieter (@jjcannegieter)
Write in your testplan the info your stakeholders need. So ask your stakeholders what kind of info they need. Write it for them!
Generally I see two trains of thought here. On the one side there is the idea of having more or less fixed items in a test plan. Things like scope, approach and (product) risks. On the other side the idea to not start with fixed items or a template, but to ask the stakeholders what information they need to have in the test plan. As you will see this kind of follows the change I made.
From old to new
Historically my organization has approached software testing by following a standardized test approach based on TMap. Similarly test planning is, or rather was, based on an extended TMap style “Master Test Plan” template. The raw template itself counts 24 pages when empty, but includes some examples and explanation. The idea is to fully fill in all items in the template, see list below, and get it signed off by the principal stakeholders.
In short the template was as follows (ChaptersParagraphs and Sub-paragraphs):
Colophon Strategy
Management summary Test levels
Goals Entry – exit criteria
Preconditions Test objects
Budget and milestones Scope
Assignment Dependencies
Introduction Project risks
Assignment Communication & Procedures
Client Reporting
Assignee Meetings
Scope of the assignment Procedures
Test basis Test product / Deliverables
Objective Project documentation
Preconditions Testware
Starting points Test Infrastructure
Release from assignment Workplace
Test strategy Test environment
Product Risk Analysis Budget, planning & organization
Test goals Budget
Compenent per characteristic Planning
Test goal vs component matrix Team composition

I have to admit that all items in themselves are in some way relevant to testing software. But one can argue the usefulness of some of these items and more so of having these items together in one document.

The latter is best illustrated by a remark my mentor made when after three months, of being a professional tester, I was writing my first Master Test Plan. He said: “Don’t waste time. Take one of my plans. Ctrl-H the project name, change the stakeholders and check if there is mention of specifics not relevant to your project and change them. All else you can leave the same. So, even if I resisted the idea, like my colleagues I learned to do the drill; fill the template in a copy and paste style. Only occasionally I had a stakeholder question what was in it and disturb from actual testing.

Some five years ago I changed departments and found myself in a place in which I not only was free to use only those elements that I felt were useful, but I could start changing the template and the use of it entirely. But there was resistance both from the testers and the stakeholders. The testers, I think, because some of them now had to think and communicate more and the stakeholders because this broke with the standard process and they too would have to get involved and think more. To break the deadlock I started with an experiment. I filled in the template not only complete but to the letter of the “law”. I ended up with a 36 page document which I immediately send out to all stakeholders with an invitation to meet next week, meantime thoroughly check it and be ready to sign off the document during the meeting.

At the meeting the stakeholders were sitting silently, sighing at the thought of having to go through all the 36 pages. I didn’t do that. Instead I asked how many of them had read the document. With 6 out of 8 I was actually impressed. I then asked  how many of them had reviewed it. Still 4 out of the 6. I then asked who found the document a pleasure to read, who fully understood its content and thought it was of value to the project. As I hoped for all the attendees broke out in commenting the document, its length, its irrelevance, the difficulty of the content etc….

I decided to then pull out the rabbit and said “I agree with you all. I too think it’s basically of no use. There is no point in reviewing it. But we still need to write a test plan. So why don’t you tell me what you actually do want to know about testing your product.”

We spend an hour or so discussion what they wanted to know about testing. Agreed that since we are a financial institution we still have to follow certain rules, regulations and guidelines and that I would deliver a new document the same week.

I ended up writing a document that was still 24 pages long. But now it not only adhered to the documentation standards but of those 24 pages 11 were purely related to testing as way to mitigate risks and provide information about the product for this project and another 4 on testing and test heuristics in general. The original document had no  explanation and only 8 pages related to actual testing.


Approach writing a test plan as you would approach any test activity. Figure out what information your stakeholders need, if there are other things to consider like rules, regulations or standards. Use your personal experience and other references you think useful and then write a plan that suits your model of the context and verify and confirm it with your stakeholders.

250 hours of testing practice

The promise

On January 3, 2011 Phil Kirkham posted a question on the Software testing club:

“so if you were to set a target of doing 2 hours practice a week every week this year, how would you spend your 100 hours ?”

Having missed the post initially I read the post as early as the week before Christmas. So I really had not enough time left to get to 100 hours in 2011. After reading the post and comments I felt however that Phil was making a valid point. One should spent time and effort on practicing and in my comment on his post I made the following promise:

“As 2012 is on the brink of starting I will try to put this into practice. As two hours seems a bit low I will spent 5 hours per week on practicing and some extra time on logging and writing short (monthly) posts about it.”


So today is January 1st and I am starting to live up to my promise. Every week of this year, except for the summer holidays, I will try to practice for at least 5 hours and log the things I do. At the end of every month I will write a post sharing my activities, providing short reviews and formulate my insights.

I have started practicing earlier today by reading “Essential Software Test Design” by Torbjörn Ryber. A book that I had downloaded as a PDF before and of which I found after a number of pages and comments from fellow DEWT’s that I wanted to have the hard copy. Later today I will make time to listen to TWiST # 76. I am not yet sure what I will do for the rest of the month, but as said before I will keep you posted.

For now I wish all of you a wonderful, succesful and entertaining 2012 and I hope to meet lots of you in person this year!

Testing is some value to someone who matters


I have a concern. We online testers have one thing in common: we care enough about our craft to take the time and read these blogs. That’s all very fine. However most testers, and this is based on my perception not research, most testers do not read blogs, or articles, or books or go to conferences, to workshops or follow a course. Well some of those testers do, but only when they think their (future) employer wants them to. And when they do they go out for a certificate that proofs they did so.

Becoming a tester

Regardless of where we start our carreer, be it in software engineering, on the business side or somewhere else, most testers start out with some kind of introductory test training. In the Netherlands, where I live, most of the time that means you’re getting a TMap, or sometimes an ISTQB training. And my presumption is that you get a similar message on what testing is everywhere:

  • Establishing or updating a test plan
  • Writing test cases (design; procedures; scripts; situation-action-expected result)
  • Define test conditions (functions; features; quality attributes; elements)
  • Define exit criteria (generic and specific conditions, agreed with stakeholders, to know when to stop testing)
  • Test execution (running a test to produce actual result; test log; defect administration)

But there are likely to be exceptions. For instance at the Florida Institute of Technology where Cem Kaner teaches testing.

Granted neither TMap nor ISTQB limit testing solely to this. For instance TMap starts of by defining testing as: “Activities to determine one or more attributes of a product, process or service” and up to here all goes well, but then they add “according to a specified procedure”. And there is where things start to go wrong. In essence the TMaps of the world hold the key to start you testing software seriously.  But instead of handing you down the knowledge and guide you to gather your own experiences they supply you with fixed roadmaps, procedures, process steps and artifacts. All of which are obviously easy to use for reproduction in a certification exam. And even this still could still be, as these methods so often promote, a good starting point to move on and develop your skills. Unfortunately for most newbies all support and coaching stops once they passed the exam. Sometimes even facing discouragement to look beyond what they have learned.


To make matters worse the continuing stance to make testing a serious profession has brought line managers, project managers, other team roles and customers the message that these procedures, processes and artifacts not only matter, but they are what testing is about. These items (supposedly) show the progress, result and quality of testing being undertaken. Line and process managers find it easy to accept these procedures, processes and artifacts as measurement items as they are similar to what they use themselves according to their standards. So if the measurements show that progress is being made and that all artifacts have been delivered they are pleased with the knowledge that testing is completed. Customers or end users go along in this but face limits in their belief of these measurements as they actually experience the end product. Like testers they are more aware that testing is never really ended and about the actual information about the product and not about testing artifacts.


New methods and approaches such as agile testing have brought the development roles closer together and have created a better understanding for the need and content of testing to both the team and the stakeholders. Other approaches, like context driven testing, focus more on enhancing the intellectually based craftsmanship of testing, emphasizing the need for effective and efficient communication, the influence of context on all of this and that software development is aimed at solving something. And thus the aim of testing is shifting from checking and finding defects to supplying information about the product. Regardless of this however and inspite of how much I adhere to these approaches I think they have a similar flaw as the traditional approaches. Like TMap or ISTQB neither of them go outside of their testing container enough to change the management and customer perception. They still let management measure testing by the same old standards of counting artifacts and looking at the calendar.


I think we as a profession should seek ways of changing the value of testing to our stakeholders. To make them understand that testing is not about the process or procedure by which it is executed nor about its artifacts, but about the information and value to achieve business goals it supplies.

I myself cannot give you a pret-a-porter solution so I challenge you, my peers, to discuss with me if you agree on this vision and if you do to form a new approach for this together. I will gladly facilitate such a discussion and deliver its (intermediate) results to a workshop or conference at some point.