Being challenged by James M. Bach

Posted on by
8

On March 31st I joined Huib Schoots and Pascal Dufour in picking up James Bach from Schiphol Airport and driving him to his hotel in Dordrecht. Having met Cem Kaner (briefly) and Michael Bolton (several times) I was looking forward to meeting James in person and I was not disappointed. Conversation en route was vivid and fruitfull  and continued during dinner at Villa Augustus and later on April 3rd after an Agile Meetup with presentation by James at Codecentric. During dinner James had offered Pascal a challenge and later that evening he offered me the following challenge on twitter:

Explain dendogram-based testing

My first thought was “What’s a dendogram?” So I googled it and saw something I had long forgotten from when I studied Social Sciences and certainly had never used in testing.

So for those of you who requested me to blog on testing and the social sciences this is my first post on the subject.

For all the others with this post I want to share my answer to the challenge with you and some of the reactions I got from James on it.

What is dendogram-based testing?

To answer this question I will start with a short description of what I understand a dendogram is. Followed with a description of how I think this could be used in testing.

Dendogram

A dendogram is a visual representation of data that shows hierarchical clusters of, potentially before unknown, groups of data in a cluster tree diagram. A picture of such a diagram is shown below.

Cluster analysis, for a dendogram, starts with a data matrix, where objects are rows and observations are columns. From this beginning, a table is constructed where objects are both rows and columns and the numbers in the table are measures of similarity or differences between the two observations.

  X1 X2 X3
O1      
O2      
O3      

 From measurements to relative observation results 

  O1 O2 O3
O1      
O2      
O3      

 You can then place the results in a cloud of n points or in a matrix and put each point in a class of its own (having n classes, each containing a single point). Then, look for the closest two classes (for a given distance, for instance the relative distance between measurement of a feature– but other distances will give other results, perhaps more appropriate for some data sets) and join them into a new class. You now have n-1 classes, all but one containing a single element. Then iterate this activity until you reach a single set containing al the classes. The result is presented as a dendogram: the leaves are the initial 1-element classes and the various “levels”) of the dendogram are various clustering’s of the data, into a decreasing number of classes.

It is also possible to calculate the distances and their relationships with the use of Euclidian algebra. This uses a number of formulas to calculate hierarchical distance between the data (clusters). But besides the relative complexity of the calculation I prefer to use the more visual and intuitive approach.

Dendogram-Based Testing

The concept of clustering  data in a hierarchical form and thus grouping it based on some measured characteristic opens a number of possibilities for testing. All of these possibilities have in common that both the activity of clustering of data and the end result, be it in numbers or a visual representation, form an additional source of information about the subject under test. The kind of information will largely differ based on the chosen measurement. With regard to testing I see the following, extendable list of areas for observation and measurement:

  • The relationship between functionality and code

Where code is differentiated into several elements such as (that are more or less commonly used):

  • Type of input
  • Used variables
  • Classes
  • Functions
  • Methods
  • Etc.
  • The relationship between test cases and code
  • The relationship between functionality and GUI elements
  • The relationship between defects and code
  • The relationship between defects and their cause of failure

The clusters of information that you get out of it provide the following information and activities for testing:

  • The different clusters of information can suggest different testing techniques
  • The different clusters of information can suggest different testing missions
  • The different clusters of information can suggest different elements of reporting

In essence the information derived from a dendogram is dependent from the kind of observation and measurement you choose, the skill you have in identifying the clusters and patterns and the skill to adapt your testing activities. In any case it forms a new oracle to be used in software testing.

The above description was made before I read the following document, that I had avoided on purpose:  http://melindaminch.com/docs/thesis.pdf

In my, opinion the thesis is a nice academic exercise that adds insight to testing, but lacks sufficient practicality for use in every day testing. But I will study it more carefully at sometime in the future and I will see if there is an opportunity to use HCE 3.0 that I have downloaded.

For now I prefer the more general intuitive approach as described above and instead of the more elaborate HCE program I would use a reverse mind map.

Giving me the below, originally not in my answer included, graphical steps:

Which translates into the following Dendogram

James response to my answer

Below I have pasted the, unedited, response James gave my to my answer.

Jean-Paul, I am impressed. This is an excellent answer. I concur with your reasoning. I’d like to try using dendograms in testing, when the opportunity arises.

The reason I gave you this challenge is that we who aspire to excellence must be comfortable describing test techniques that don’t yet exist– in other words, we must be confident in our ability to invent test techniques as needed. This is a challenge that cannot be solved well using Google, since almost nobody has applied dendograms to testing, and the one reference you cite does a poor job of it.

By researching this and replying as you did, you have marked yourself as a leader in the Context-Driven community. Thank you.

– James

Needless to say that I am proud to receive such a response. And I feel strengthened in my drive to grow and learn as tester and as a member of the context-driven community.

Belgium Testing Conference 2012

Posted on by
Reply

QA vs.Testing: Antagonism or Symbiosis?

The header above was the theme of the last Belgium Testing Days in Brussels on March 12-14. During the call for papers for this conference, several months ago, I was in the middle of having SOx compliance established for one of my projects. The theme caught my attention since it represented one of my feelings on the compliance process at the time.

I work at an internationally operating bank which has a few consequences for the context in which I work. The most obvious consequence is that a bank uses either financial software or software that enables financial processes. As a result the (in-house) developed software gets extra special attention with regard to accessibility (or perhaps rather inaccessibility), integrity and confidentiality (AIC). Every piece of software that is built or bought by the bank gets a so-called AIC assessment and depending on the result of the assessment and certain amount of checks, controls and measures are mandatory.

The AIC assessment itself is essentially internal to the bank. But being a bank, and especially being an international bank, this means that on top of the internal regulations all kinds of external government and financial market regulations are imposed on it. The bank QA department translates these regulations or standards into internal processes and rules. For most of the high level business processes such a translation seems fairly straight forward. These processes are often both described and measured at the same way as the regulations. It gets more difficult if you drill down into the organization and start taking all the contributing activities and tasks into account such as in my case the development of software or more particularly the testing of software.

Organizational response

One of the financial organizations common responses is to apply and design standards and procedures together with any number of deliverables.

These standards are prescriptive of nature. They tell you in general terms what their idea is. But, depending on your QA department it gets more specific. They tell you what you must do, how you should do it, in what order you should do it and how you should call it.

The so designed procedures and processes describe the steps you are supposed to do given some standard situation. And since seeing is believing they also formulate how you should proof that you followed the process. In many cases such proof is the delivery of a number of deliverables. Deliverables can be a lot of things, but typically they are in the form of documents, test ware libraries or reports.  Given a certain standard they follow a fixed format both in terms of content, that is what should be described, and in terms of lay-out, that is how it should be described.

Quality Assurance

To my experience for the most part the somebody defining the standards to be used, the procedures to be followed and the deliverables to be created are not the developers or testers nor the customer, but is a typical staff department:

The Quality Assurance department

I see quality assurance not as a singular activity. In my opinion it is a group of activities. Activities that have a difference in focus.One part of quality assurance is focussed on making the chosen framework useable and applicable within the organization. QA as the designer. The second part of quality assurance is closely related to this as it acts as the controller of what was previously designed. To this end it translates the designs into to points of measurement and puts values to the measurement results. QA as the controller. These two might go by different names in some organizations, like quality management, but in my opinion this still is part of quality assurance. The third part of quality assurance is the part in which the actual software development related activity is taking place. It is the part that also executes the previously designed steps and  reports back on them. QA as executor.

At this point QA starts to be seen as testing which is captured in the following definition that is often used for both:

The process of validating and verifying that a software program/application/product meets: The requirements that guided its design and development;  works as expected; and can be implemented with the same characteristics

This definition has a certain appeal. It is understandable; it is similar to other process oriented methodologies; it aligns with the QA concept. But in my opinion it limits testing to checking.

Testing

The previous definition is not the definition I would use for testing. In my opinion the kind of information testing provides depends on the what the stakeholder values as important for the softwares quality. Therefore my definition is:

Software testing is a means to provide information to someone who matters about the product or service under test in a certain context at a certain time.

If I apply this definition to my AIC, SOx compliant context I find that the current solution QA has offered me does not meet this definition. Do not get me wrong I understand and agree that the change process and software development should be both traceable and accountable. I do however not believe that the solution is to enforce processes, procedures and deliverables that are judged by their presence and adherence to layout. What matters is the content that they should be presenting.

Not the process story but the testing story should be told

An example

This example describes in summary which measures the SOx team and I agreed on that are required by software development projects, and specifically testing, necessary to comply to Sarbanes-Oxley Act (SOx) regulation.

A SOx regulation review is targeted at the state of software development and testing at the moment of release to production. The intermediate states or progress steps in establishing traceability and documentation compliance are out of scope for the investigation.

Manual Testing

Testware management for manual testing follows the guidelines as summarized in the following steps:

  • A test plan or a similar structure identifies functional test objects that, minimally, covers the functionality as specified in the requirement or change documents.
  • For each of these test objects specific test activities are logged
  • The test structure identifies the release, individual RFC’s and relates them to the test activities.
  • A  full and complete overview of test objects, test activities and the test results should be established prior to release of the changes to production. This can be either in a testware management system, like preferably HP QC, or in another reviewable form
  • All tests should either have passed or if not passed have a logged defect and or     stakeholder decision attached that indicates that this is acceptable to go into production at this point in time

Automatic Testing

In essence the testware management for automated testing follows the same guidelines as manual testing. Main difference is that the way of documentation is adapted to the structure of automated testing:

  • A test automation tool or input data sheet used by the test automation tool shows the automated tests with a reference to the test object, functionality or RFC that they test
  • A log file (preferred), or checklist, shows whether the tests have been executed and if they have passed or failed
  • A full and complete overview of test objects, automated tests and the final      execution of tests with test results should be established prior to release of the changes to production
  • All tests should either have passed or if not passed have a logged defect and or     stakeholder decision attached that indicates that this is acceptable to go into production
  • If for automatic testing a self designed tool or framework is used the      functionality of the tool and the execution of the test cases should be validated by peer review. Results of the review should additionally be captured in the test report

The above steps should result in the following deliverables:

-          Basic testware describing the test design, test execution, test results and defects
-          Test Report, containing an advice for release

Test report

You might have noted that there is no mention of describing tests in advance, of following test scripts, nor of following standard or using templates. The SOx compliancy team’s focus is not on how the testing is executed during the project. Its focus is to see if the implemented changes are tested and that the executed tests and test results can be matched to them. Their aim is to establish that the changes do not have an unexpected or undesirable effect on the companies annual balance or regulatory capital. To that end they check if the changes are implemented as intended.

The SOx compliance essentially asks for what James Bach describes as three stories that describe the testing story:

  1. A story about the status of the product
  2. A story about how you tested it
  3. A story about the value of the testing

The only thing now left for me is to convince the QA department that even when I have not followed their standard and procedures and not used their templates I have thought about the reason behind their questions and am still able to supply the information they need. Only a bit faster and more naturally.

250 hours of practice – February/March

Posted on by
1

It has been a while since my last post. It was not that I have been procrastinating. No I actually was too busy to write a post. There were many reasons for this and most of them are covered in this post. So lets start with the biggest time consumer.

BBST Foundation

For those who do not know what this is I can almost hear you thinking.

“A foundation course? Doesn’t this guy have enough experience already and now he does Black Box Software Testing Foundation course. ”

Well let me put it this way. There are foundation courses and foundation courses. The Black Box Software Testing (BBST) Foundation course is as it says a foundation for further education and it is a good place to start your testing career. It’s just that it has a distinctive approach to it. First of all it is a four-week online course. Which is longer then almost any other course on software testing. Second it focusses on getting the participants to become critical and thinking software testers. It does this not by handing you any predefined recipe. It gives you the ingredients and the possibility to work with them. Let me, as an example, give you a short overview of the content:

Basic definitions                                              Programming fundamentals and Coverage
What is a computer program                               Numbering
Types of testing                                                   Storage
.                                                                           Representation
Strategy                                                               Data and Control structures
What is testing                                                     Coverage
Testing missions
Testing strategy                                              The impossibility of complete testing
.                                                                           The basic combination rule
Oracles                                                                Paths and sub paths
System under Test                                              Data flows
Heuristics                                                             Sequences
Consistency oracles
More types of oracles                                       Measurement

All captured in :

  • 20+ articles and several books to read
  • 306 slides
  • Over 3 hours of video lecture
  • 5 quizzes
  • multiple individual and group assignments
  • Exam and exam grading
  • Over 70 hours of thinking, preparing and answering either individually or with team members from multiple time zones

Given that I also have worked these four weeks and maintained a family life I have made long days in February. But it was all worth it. Even if you have, like me, some experience in software testing it is a splendid (re)sharpener of your critical thinking skills with regard to software testing. Not only with the content of the course itself, but also with the interaction you have with the twenty or so other participants. Especially the group assignments and the assignments that need reviewing each others work give you a good insight in cultural and thinking differences people have. And from this and from the comments you can get a lot of more wisdom.

So some seventy of so hours spent on BBST Foundation. Well on my way towards the end goal of two hundred and fifty. But you might have noticed that this covers only February and this post is written half way March. So what more have I done.

Proposals

The end of February and beginning of March was also the time in which several deadline for calls for papers came in to view. I do not know if this really is considered practice, but it gets you to think and write about software testing anyway. Within one week I entered five proposals for a track and one for a tutorial for EuroSTAR, TestNet spring and autumn event and the Agile Testing Days. And I already am more than happy to tell you that the tutorial on the use of mind maps in testing, that I thought of together with testing buddy Huib Schoots, already got accepted.

But not only did I spent time on my own proposals. Zeger van Hese, this years program chair, invited me to help review some of the many proposals that EuroSTAR has gotten this year. And even if the amount of, anonimized, text is not so much I did want to do a serious review and evaluation of the proposals. (Like I would like others do with mine.) Some of them were good, some were bad and for some I was indecisive.

TestNet book

And there was more time spent reviewing. A couple of months ago I had committed to reviewing the TestNet jubilee book on the future of software testing. Obviously at the time I had not imagined to be this busy. But at the cost of some sleep I managed to finish the review prior to my next challenge.

The book itself is a great reference to over think where testing is going to and what choices you as tester need to make to make yourself both comfortable and future proof the coming five years.

Conferences

Although I have given talks and workshops before I had never been a speaker at a major international test conference. March 14 I made my debut on the international stage at the Belgium Testing Days. As I will be spending a separate post on the content of my talk I will limit for now with the remark that it was great fun to do and that it was great to meet both familiar and new  faces.

Meanwhile during this period Markus Gärtner published an interview with me on his blog as prequel to Europe’s first context-driven testing conference “Let’s Test“. The only downside for me seems to be that I am actually not able to attend, due to a lack of funds.

To complete the conference experience for the coming period I am invited to be a test expert at the Dutch Testing Conference (agile, context-driven testing and exploratory testing) and I invite participants to ask me questions during the conference.

250 hours of practice – January

Posted on by
2

As said in my post a couple of weeks ago, this year I would try to spend 250 hours on practicing and enhancing my testing skills. This post is a report on how I fared in January 2012. (Leaving my personal favourite untill the end…)

I started enthusiastically on January 2nd by following up on a post about the “Follow the link exercise” by Jeff Lucas. In short the exercise is to choose a blog post of your liking. You start reading it critically and then follow every link mentioned in the post. You then pursue this with every post that you read in a one hour session.

In my session, that actually lasted two hours, among others I followed up on a link to Alan Page’s blog “Tooth of the Weasel”. This post contained an overview of posts Alan wrote in 2011 so there were enough links in there to follow-up:
• My job as a Tester
• What is Testing?
• Test Design for Automation
• Numberz Challenge
• Beyond Regression Tests
• R-E-S-P-E-C-T
• Judgment in Testing
• Lost in the weeds

Although I had heard about Alan Page I was not yet familiar with his work. It pleasantly surprised me with some useful ideas and even some advice for my personal goals for this year. Let me give you some quotes I found interesting:

“What you do or don’t define as testing may differ per context.”

Automated testing “starts the same as always. Design your test first then automate where eligible. Coded tests do not replace, but enhance human tests.”

“Do not only use automated testing for regression. Vary the data, the sequence, randomize, to find new information” data driven testing

“Are testers’ second class citizens? NO. Are they whiners? Yes; Figure out how to get and earn respect!”

My second (larger) series of practice session(s) started with watching the 2011 GTAC keynote by Alberto Savoia with the ominous title “Test is dead”. You can read more about this on the blog post I wrote “Is testing dead?”

My third endeavour entailed reading the hardcopy of the book “Essential Software Testdesign” by Torbjörn Ryber. The E-book is free to download, but I liked the content enough to want to own it. Some warning is in order however. Even the hardcopy has a somewhat annoying number of typos, illogical sentences and even faults. Nevertheless the concepts Ryber discusses are helpful for many a tester.

Early in January the DEWT’s met up again. This time to discuss and prepare the TestNet event about context-driven testing. On January 18, some 150 testers visited the event to watch James M. Bach and Michael Bolton do a one hour introduction on context-driven testing using Go to meet  (which btw. worked brilliantly). After the break the DEWT Zeger van Hese, Ruud Cox, Ray Oei and myself gave a number of lightning talks followed by Q&A. Themes of the talks were “On being context-driven”; “Spin-Off”; “Context-Driven expert”; “Test Plan”.

All in all these activities got me some 20 hours of practice bringing me well en route for the 250 hours of testing practice. But to be honest I am even more of a test nerd. I have spent another 10-15 hours on following Twitter feeds with a peak while participating in a #Testchat lead by Lisa Crispin asking the following questions:
Q1: Have you worked on a “test automation project” that succeeded? What helped it succeed
Q2: What do you think upper management should know about testing? (not limited to automation)
Q3: related some to Q2: How do you keep your testing transparent to others on your team and in the organization?
Q4: Are testers on your team treated with the same respect as programmers?
Q5: sometimes the tester is undone by the process. Documentation outdated leading to looking like lack of knowledge

The last practice activity however was, for me personally, the most engaging, emotional and gratifying one.

In December I contacted Markus Gärtner to ask him for a challenge to see if I was worthy enough to enter the realms of the Miago-Do software school of testing. This actually the first step of the challenge having found a member. Markus offered my “The light saber” challenge. Several times during the challenge I would sent Markus my test investigation results and as many times Markus answered. I used several heuristic approaches, tried to inform the customer based on his needs and eventually offered a solution using personas. Somewhat to my despair Markus’s answers were getting shorter and repetitive and I asked Markus to debrief me.

We organized a one hour Skype session and went on with the challenge discussing results, progress en feelings during the challenge. Eventually we came to the point where Markus would reveal if I was allowed to enter Miagi-Do. The result got me stunned, silent and humbled for a moment… Not only was I a new member I was one of the members to, fully endorsed by other instructors, become a Black-belt.

I can only say again. Thanks guys, I am honoured.

What do you put in your test plan?

Posted on by
1

Theme event

This evening, January 18, I will be on stage during the TestNet theme event about Context-Driven Testing. The evening will start with a duo presentation by James Bach and Michael Bolton followed by a series of short presentations, lightning talks and discussion. In one of those lightning talks I will share a personal experience report. It describes how I changed the way I make test plans. Since most of you will not be able to be there (or might have never heard about TestNet*) I am sharing my experience with you in this post.

Twitter

Even if a lightning talk last only for five minutes it still requires some preparation. So to extra prepare myself I placed my experience into perspective and placed the following message on Twitter:

“ @Arborosa: Question to my tweeps: What items do you put in a test plan?  I’ll put the results on my Blog. (please retweet)”

This resulted in the following responses:

Rob van Steenbergen (@rvansteenbergen)
Scope of testproject, context of product (with mindmap), product risks and qlty attributes and risk approach, planning, who tests, stakeholders, testing tools, explanations abt testing for orgs that are still learning. TP is also promotion material for testing.

Stephan Kämper (@S_2K)
Well, what to put in a plan? A (current) goal of what you’re planning. The major way you’ll follow to reach said goal. A ‘Plan B’. (Known) risks – What’s the risk of following the plan? …the risk of *not* following it? Tools & Techniques? Not sure about these.

Nitin Hatekar (@nhatekar)
Entry and exit criteria for each test phase and specific test approach for each phase. Scope of testing and the estimates for completion of in-scope test efforts. A section for assumptions, risks & blockers as well.

Rik Marselis (@rikmarselis)
For your testplans take IEEE829 (1998) as a starting point. And see tmap.net for templates ;-) (And after a reprimand by Huib Schoots to be more serious) Don’t start with making the testplan. First make the outline of the testreport. That’s your deliverable! The testreport outline must be discussed with stakeholders. Then you have startingpoint for your testplan.
Jesper L. Ottosen (@jlottoosen)
Generally answers and descriptions to “how” – to the level required of the context. ie #itdepends ;-)
Jan Jaap Cannegieter (@jjcannegieter)
Write in your testplan the info your stakeholders need. So ask your stakeholders what kind of info they need. Write it for them!
Generally I see two trains of thought here. On the one side there is the idea of having more or less fixed items in a test plan. Things like scope, approach and (product) risks. On the other side the idea to not start with fixed items or a template, but to ask the stakeholders what information they need to have in the test plan. As you will see this kind of follows the change I made.
From old to new
Historically my organization has approached software testing by following a standardized test approach based on TMap. Similarly test planning is, or rather was, based on an extended TMap style “Master Test Plan” template. The raw template itself counts 24 pages when empty, but includes some examples and explanation. The idea is to fully fill in all items in the template, see list below, and get it signed off by the principal stakeholders.
In short the template was as follows (Chapters, Paragraphs and Sub-paragraphs):
 
Colophon Strategy
Management summary Test levels
Goals Entry – exit criteria
Preconditions Test objects
Budget and milestones Scope
Assignment Dependencies
Introduction Project risks
Assignment Communication & Procedures
Client Reporting
Assignee Meetings
Scope of the assignment Procedures
Test basis Test product / Deliverables
Objective Project documentation
Preconditions Testware
Starting points Test Infrastructure
Release from assignment Workplace
Test strategy Test environment
Product Risk Analysis Budget, planning & organization
Test goals Budget
Compenent per characteristic Planning
Test goal vs component matrix Team composition

I have to admit that all items in themselves are in some way relevant to testing software. But one can argue the usefulness of some of these items and more so of having these items together in one document.

The latter is best illustrated by a remark my mentor made when after three months, of being a professional tester, I was writing my first Master Test Plan. He said: ”Don’t waste time. Take one of my plans. Ctrl-H the project name, change the stakeholders and check if there is mention of specifics not relevant to your project and change them. All else you can leave the same. So, even if I resisted the idea, like my colleagues I learned to do the drill; fill the template in a copy and paste style. Only occasionally I had a stakeholder question what was in it and disturb from actual testing.

Some five years ago I changed departments and found myself in a place in which I not only was free to use only those elements that I felt were useful, but I could start changing the template and the use of it entirely. But there was resistance both from the testers and the stakeholders. The testers, I think, because some of them now had to think and communicate more and the stakeholders because this broke with the standard process and they too would have to get involved and think more. To break the deadlock I started with an experiment. I filled in the template not only complete but to the letter of the “law”. I ended up with a 36 page document which I immediately send out to all stakeholders with an invitation to meet next week, meantime thoroughly check it and be ready to sign off the document during the meeting.

At the meeting the stakeholders were sitting silently, sighing at the thought of having to go through all the 36 pages. I didn’t do that. Instead I asked how many of them had read the document. With 6 out of 8 I was actually impressed. I then asked  how many of them had reviewed it. Still 4 out of the 6. I then asked who found the document a pleasure to read, who fully understood its content and thought it was of value to the project. As I hoped for all the attendees broke out in commenting the document, its length, its irrelevance, the difficulty of the content etc….

I decided to then pull out the rabbit and said “I agree with you all. I too think it’s basically of no use. There is no point in reviewing it. But we still need to write a test plan. So why don’t you tell me what you actually do want to know about testing your product.”

We spend an hour or so discussion what they wanted to know about testing. Agreed that since we are a financial institution we still have to follow certain rules, regulations and guidelines and that I would deliver a new document the same week.

I ended up writing a document that was still 24 pages long. But now it not only adhered to the documentation standards but of those 24 pages 11 were purely related to testing as way to mitigate risks and provide information about the product for this project and another 4 on testing and test heuristics in general. The original document had no  explanation and only 8 pages related to actual testing.

Conclusion

Approach writing a test plan as you would approach any test activity. Figure out what information your stakeholders need, if there are other things to consider like rules, regulations or standards. Use your personal experience and other references you think useful and then write a plan that suits your model of the context and verify and confirm it with your stakeholders.

Is testing dead?

Posted on by
3

“Test is dead”

Last year, 2011, Alberto Savoia, presented the opening keynote at GTAC with the title “Testing is dead”. Alberto started with an all to recognizable Old Testmentality:

  • Top down
    • Thou shalt follow the spec
  • Rigid
    • Thou shalt not deviate from the plan
  • Distinct roles and responsibilities
    • Developers shalt develop
    • Testers shalt test
    • And never the twain shalt meet
  • Do not release until ready
    • Thou shalt not sell wine before it’s time

He concludes this with declaring that in essence the focus is on building it right.

Alberto’s then takes an elaborate detour to arrive at the conclusion that in the New Testmentality the focus is on building the right it. While doing this he comes to the conclusion that success, in the Post Agile era, does not depend on testing or on quality but on building the right it at the highest speed (to get the best realistic marketing edge). And so you do not actually need to test at all. Well…. at least at the start, in the right environment…. I could go on but I think Mark Tomlinson’s blog post says just the right it.

The reactions

Alberto’s keynote and later presentations like James Whittaker’s at EuroSTAR have spawned a load of critical reactions from the testing community. (At least the part that I am following.) Most of them were even downright negative and dismissive. As an example a quote of one of my fellow DEWT’s: ”Ik vind het  ‘Test is Dead’-paradigma in ieder geval tijdloze flauwekul” (in English this translates to “In my opinion the ‘Test is Dead’ paradigm is to be considered timeless nonsense”). I can understand these reactions and based on the stories and without critically thinking over what the paradigm was saying I had a similar mindset.

Requital

In short the ‘Test is dead’ paradigm argues that test will disseminate into two directions. It will either move down to the developers or it will move up to the users. The arguments for the movement down to development are that software in general has gotten better; that there are more possibilities to quickly fix in production, that software is able to self repair; that there are more standards,better software languages and that software is no longer localized but available in the, more stable, cloud. The arguments for moving testing up to the users are that the current testing slows down the development process and that it imitates user behaviour. Since speed has more value than quality and users can do a better job at being users then testers this kind of testing is no longer necessary.

I agree with the observation. It is true that software nowadays is different then the software that was produced when the first major test approaches were established. The shift to web-based software (delivery) and the growing knowledge and acceptance by the public of software updates has changed the playing field. I think a lot of what testers (still) do nowadays can be done by developers as well. Particularly the stuff I heard a fellow tester sigh about during a TestNet event “Come on do I still have to test input fields and buttons on correctness. Why don’t those lazy programmers write unit tests as they are supposed to.”. Oh and yes there are loads of testers that go and sit behind their computer and punch keys as if they were users. But are they really testers…..

I do not agree with the conclusion. When it comes to testing business logic, calculations, multi state or multi integrated programs, security, or usability more and bigger unit tests really do not cut it. Yes they take out the more or less obvious bugs, but still leave the less obvious and unimagined ones mostly untouched. If you want to catch those you need sapient testers who are able to use their investigative skills. Who are able to cooperate with and understand developers, business analysts and users. Testers who adjust their skills and the use of those skills to the context in which they work. Only these kind of tester can smoke out bugs that otherwise would have gotten away and provide information that allows others to make the right decisions. Even Alberto Savoia himself limits his arguments when he says during his keynote that eventually you have to build it right also and that security sensitive, risky or regulated software still needs a testing process.

So is testing dead?

Yes if you mean factory style mindlessly following standards and pre-scripted testing.

No if you mean sapient critical skillful and context-driven testing.

or in other words

Testing is dead, long live testing!

250 hours of testing practice

Posted on by
4

The promise

On January 3, 2011 Phil Kirkham posted a question on the Software testing club:

“so if you were to set a target of doing 2 hours practice a week every week this year, how would you spend your 100 hours ?”

Having missed the post initially I read the post as early as the week before Christmas. So I really had not enough time left to get to 100 hours in 2011. After reading the post and comments I felt however that Phil was making a valid point. One should spent time and effort on practicing and in my comment on his post I made the following promise:

“As 2012 is on the brink of starting I will try to put this into practice. As two hours seems a bit low I will spent 5 hours per week on practicing and some extra time on logging and writing short (monthly) posts about it.”

2012

So today is January 1st and I am starting to live up to my promise. Every week of this year, except for the summer holidays, I will try to practice for at least 5 hours and log the things I do. At the end of every month I will write a post sharing my activities, providing short reviews and formulate my insights.

I have started practicing earlier today by reading “Essential Software Test Design” by Torbjörn Ryber. A book that I had downloaded as a PDF before and of which I found after a number of pages and comments from fellow DEWT’s that I wanted to have the hard copy. Later today I will make time to listen to TWiST # 76. I am not yet sure what I will do for the rest of the month, but as said before I will keep you posted.

For now I wish all of you a wonderful, succesful and entertaining 2012 and I hope to meet lots of you in person this year!