Software Testing

No user would do that

/ Arborosa / 3 Comments

Still on Iceland

Being in a foreign country gives you a chance to visit shops that you haven´t been before. And doing so has heightened my attention to curious software behavior. Today we went out for some groceries at the Bonus supermarket. Untill we got to the check out nothing exciting happened. While waiting in line I noted some commotion by a customer as he argued with the cashier. Even if my Icelandic is not that well I could make out that the man had bought groceries for 30.213 ISK (approx. 200 USD) and had tried to pay with his credit card. Unfortunately for him the cash register signalled that his credit was insufficient to match the amount to pay. He however disagreed and demanded that the cashier tried again.

This system had been tested

Probably against her better knowledge she tried again so she could convince the client. As she tried again I noted a change in the layout of the cash registers touch screen. I couldn´t help myself and tried to see what  had changed. I noted that a red button had moved to the side of the screen (later I looked up its meaning and it said ‘Cancel Payment’) and a new grey button had appeared on the screen. The text on the button caught my eye as it was not in Icelandic but in English and said:

*TEST* Use another card *TEST*

To my surprise and probable to hers aswell she pushed where the red button had been and hit the new button. As far as I could tell the screen seemed to have returned to its former state. However the cashier caught the difference. The line displaying the  amount paid now had a value saying 15.107 ISK with the line below it saying the amount to pay was 15.106 ISK. The cash register had accepted half of the amount to pay from the previously overdrawn credit card, but still had an open amount. The cashier was puzzled. The customer less so and readily offered his girlfriends credit card to pay the rest. To no avail. Nothing happened, the cash registers screen locked and the cashier, her colleague and eventually her manager could not unlock the screen let alone solve the problem. The check out line closed, we paid at another line, and as we were leaving the shop I could just hear the manager calling a service desk…

YAGNI

/ Arborosa / Leave a comment

Context

At the time of this blog post my family and me are on holiday in Iceland. Since we are not that often in Iceland we, amongst visiting relatives and friends, use the time to look into administrative and regulatory stuff that is easier to do in Iceland than from abroad.

Syslumen

One of the things necessary is to renew my wifes passport. For which you actually need to physically go to the Civil Registry, or in icelandic ´Syslumen´. The process of renewal is (boringly) straightforward. At the office you get a number, wait, identify yourself and pay for your renewal, get a form, wait, identify yourself again, hand over the form and update your data (including new digital photo and fingerprint), sign and wait for a couple of weeks to pick up your passport at the Civil Registry.

Except

Since we´re only on holiday in Iceland a couple of weeks of waiting is not a real option. So to amend this my wife investigated and proposed the solution to sent the new passport to the consulate in our country. An option, once validated by the team lead, that was acceptable to the civil clerk. And thus the proper check box was looked for and found.

Into the process

After filling in the personal details instead of the offices address the address of the consulate was needed. The page itself did not offer any listing. The help page wasn´t really helpful either as it only pointed towards a government listing at another department. After some searching the consulate in Amsterdam and its address was found and the data could be entered. So everything was entered and the OK could be clicked. Nothing happened. Looking over the page the clerk found:

Færðu inn lögboðnar reit (Please enter mandatory data) next to a field asking for Póstnúmer (Zip code) that had been left empty as it had also been empty on the government listing. So what to do? My wife and the clerks colleague suggested to google it. And so she did. The zip code was entered and again the OK was clicked. The intranet page jumped back to the entry page and everything looked okay. But the clerk rightfully noted that the usual confirmation message was not shown and checked my wifes file. To her, and my wifes surprise no data was added, meaning the whole 20 minute worth of data was absent.

The process repeated itself a few times and eventually another colleague noted that the zip code contained letters. Something not used in Iceland itself. Why not leave those out of the field and move them somewhere else, say in front of Amsterdam. Now when clicking OK the confirmation appeared and a check showed that the file now contained all the data.

YAGNI

Even with only a couple of thousand Icelanders living abroad chances that they live in one of the eight countries (e.g. Canada, Great-Britain) using alpha numeric characters is realistic especially since many more countries use the country abbreviation in front of their zip code. So when my wife returned to tell about her plight she commented: Clearly neither the developer nor tester thought this field was important. But it really bugged me today. Further more she noted The same software company maintained the government listing and had all the zip codes removed, leaving empty spaces in the listing. That´s even more stupid.

Clearly someone must have convinced the developers and testers “You Ain´t Gonna Need It” (YAGNI).

Seven Questions – What will I test?

/ Arborosa / 1 Comment

In the previous post I addressed why you test. In this post I will address what you are going to test. In the sections below I will hand you a number of ideas. Some of them are well-known in main stream testing and other may be a bit more out of the box to some of you.

Requirements

I think I am not exaggerating if for most testers and stakeholders covering the written requirements with test cases still is what you as a tester should do. This line of thought is more generally known under the title of Requirements Based Testing. As a definition this could be described as:

Requirements-based testing (RBT) process addresses two major issues: first,
validating that the requirements are correct, complete, unambiguous, and logically
consistent; and second, designing a necessary and sufficient (from a black box
perspective) set of test cases from those requirements to make sure that the design and code fully meet those requirements

Followers of this approach do admit that requirements, or rather documents, are hardly ever complete and that testers almost never have the possibility to design and execute all possible tests. This last insight is the foundation of an approach called Risked Based Testing (or sometimes called risk and requirements based testing). You can find more information on it here, here, and slightly more hidden in here. But personally I like this   interpretation of risk based testing, by J. Bach, best.

A major advantage of risk based testing, and in my opinion essential in finding out what to test, is that it is based not only on documents as a source, but that it is based on people as a source of information. People are a major source of requirements and especially of requirements that matter to them. Documents can tell you a lot, but people can tell you what’s missing in them, how to give meaning to them and therefore paint you a more complete picture.

A good source of information, besides risks, that people can also give is how they current software or the manual operation that they use to solve the problem now. Often this will be the basis for the acceptance criteria that they formulated, or that you will formulate together with them for the new software. Acceptance criteria are useful to give direction and focus to your testing and are mandatory elements in the structure of your testing report.

You the tester

Finding out what to test is not only about bringing the subject under test and its information to you the tester. It is also about what you bring to the subject. What knowledge of the domain do you already have? What test ideas or mnemonics do you know that are useful in the context of your subject?

Sources for Test Ideas and mnemonics

The list below has some online sources you can use and fit into your situation. Use them, adjust them and come up with your own ideas.

Test Cases

There are many definitions for a test case. My personal one is: “A test case is the execution-able version of a test idea. A test case aims at gathering information about the test ideas relation to the subject under test and the value of that information to the stakeholders.” If and how much of a test case is (pre-)scripted is based upon personal preferences, the test idea itself and the context in which it is executed. Attention should in any case not go towards the quality of scripting of a test case, but towards the relevance and value of the information that is delivered by the test case.

Designing test cases, scripted or otherwise, takes skill and experience. I therefore believe that as a start a good tester should have a base of test design skills and a testing mindset. As a start I would like to recommend the following literature:

  • TMap Next (Chapter 14) – Tim Koomen, Leo van der Aalst, Bart Broekman, Michiel Vroon
  • Lessons Learned in Software Testing – Cem Kaner, James Bach, Bret Pettichord
  • Essential Software Test Design – Torbjörn Ryber
  • What is good test case? – Cem Kaner

When working on test cases there are a few principles that I like to keep in mind. The DRY principle. DRY stands for “Do not Repeat Yourself”. If test cases draw their value based on the information they offer it is not useful to use variations of tests that do not deliver new information. This principle does have a catch as you do not always know in advance if a variation does or does not deliver new information or not.

Separation of concerns is another principle that lets you focus your attention upon some (single) aspect. This does not mean that the other aspects are ignored. It means that you address the subject under test with a singular point of view, deeming the other’s irrelevant at that point in time while not forgetting that relationships might exist.

Finally I am always interested in mentions like “You Ain’t Going to Need It” (YAGNI) or “No user would do that” popping up in any design discussion between other team members. The areas discussed are an interesting area to start investigating.

This concludes this third post of the Seven Questions series.