Is testing dead?

“Test is dead”

Last year, 2011, Alberto Savoia, presented the opening keynote at GTAC with the title “Testing is dead”. Alberto started with an all to recognizable Old Testmentality:

  • Top down
    • Thou shalt follow the spec
  • Rigid
    • Thou shalt not deviate from the plan
  • Distinct roles and responsibilities
    • Developers shalt develop
    • Testers shalt test
    • And never the twain shalt meet
  • Do not release until ready
    • Thou shalt not sell wine before it’s time

He concludes this with declaring that in essence the focus is on building it right.

Alberto’s then takes an elaborate detour to arrive at the conclusion that in the New Testmentality the focus is on building the right it. While doing this he comes to the conclusion that success, in the Post Agile era, does not depend on testing or on quality but on building the right it at the highest speed (to get the best realistic marketing edge). And so you do not actually need to test at all. Well…. at least at the start, in the right environment…. I could go on but I think Mark Tomlinson’s blog post says just the right it.

The reactions

Alberto’s keynote and later presentations like James Whittaker’s at EuroSTAR have spawned a load of critical reactions from the testing community. (At least the part that I am following.) Most of them were even downright negative and dismissive. As an example a quote of one of my fellow DEWT’s: “Ik vind het  ‘Test is Dead’-paradigma in ieder geval tijdloze flauwekul” (in English this translates to “In my opinion the ‘Test is Dead’ paradigm is to be considered timeless nonsense”). I can understand these reactions and based on the stories and without critically thinking over what the paradigm was saying I had a similar mindset.

Requital

In short the ‘Test is dead’ paradigm argues that test will disseminate into two directions. It will either move down to the developers or it will move up to the users. The arguments for the movement down to development are that software in general has gotten better; that there are more possibilities to quickly fix in production, that software is able to self repair; that there are more standards,better software languages and that software is no longer localized but available in the, more stable, cloud. The arguments for moving testing up to the users are that the current testing slows down the development process and that it imitates user behaviour. Since speed has more value than quality and users can do a better job at being users then testers this kind of testing is no longer necessary.

I agree with the observation. It is true that software nowadays is different then the software that was produced when the first major test approaches were established. The shift to web-based software (delivery) and the growing knowledge and acceptance by the public of software updates has changed the playing field. I think a lot of what testers (still) do nowadays can be done by developers as well. Particularly the stuff I heard a fellow tester sigh about during a TestNet event “Come on do I still have to test input fields and buttons on correctness. Why don’t those lazy programmers write unit tests as they are supposed to.”. Oh and yes there are loads of testers that go and sit behind their computer and punch keys as if they were users. But are they really testers…..

I do not agree with the conclusion. When it comes to testing business logic, calculations, multi state or multi integrated programs, security, or usability more and bigger unit tests really do not cut it. Yes they take out the more or less obvious bugs, but still leave the less obvious and unimagined ones mostly untouched. If you want to catch those you need sapient testers who are able to use their investigative skills. Who are able to cooperate with and understand developers, business analysts and users. Testers who adjust their skills and the use of those skills to the context in which they work. Only these kind of tester can smoke out bugs that otherwise would have gotten away and provide information that allows others to make the right decisions. Even Alberto Savoia himself limits his arguments when he says during his keynote that eventually you have to build it right also and that security sensitive, risky or regulated software still needs a testing process.

So is testing dead?

Yes if you mean factory style mindlessly following standards and pre-scripted testing.

No if you mean sapient critical skillful and context-driven testing.

or in other words

Testing is dead, long live testing!

Advertisements

7 thoughts on “Is testing dead?

  1. I’m a Senior QA Analyst at a fairly large Digital Agency. Chances are very good you have been to a site we created or used a mobile app we have developed or designed. We work on large very publicly facing projects for big name clients.

    I’ve been reading on the whole “Test is Dead” thing on multiple blogs and watched the keynote that spurned it. While there are some good points presented (such as the decrease in rigid test case relevance) I do have to argue with the idea of users becoming testers. While this is a nice idea for a small start up or a Google product that is in beta for years it does not make sense for our clients. They are not paying us thousands or even millions of dollars to re-brand their site and have their users have a bad or buggy experience. Their brand is important and they want it presented in the way they paid for.

    What do you think if you hit a major brand site for information on their products or services and find a bug? I am sure you aren’t thinking about how modern their development process is. You now think about that brand in terms of having a shoddy website or mobile application. Not something our clients are looking for. It’s not possible to have a bug free launch but we need to be as close as possible.

    I suppose I should address the possibilities of moving the onus back onto the role of developer and while that could work there is still a level of comfort in having an individual or group of individuals double check their work before going to production. Agency work is fast paced and we are always trying to be innovative and cutting edge with what we produce and still do that work within a short time frame. This is not a recipe in which a developer is able to ensure a quality end result at all times.

    Companies who pay millions of dollars for software development work do not think testing is dead. In fact they really can’t get enough testing.

    To sum up there is room for innovation in our industry and there is a lot more we can do to move it forward but removing the role of tester is not the answer in all cases or even most of cases.

    Like

    1. Hi Kate,

      Thanks for the comment.
      Since I work at a rather large bank I understand what you are saying. Banks, insurance companies, webshops, and other brand sensitive organizations get a lot of value out of the public ‘good will’. Most of them are very aware of the potential impact that production bugs can have. As a result they do not have any intention to release software virtually un-tested to their customers. So I do not expect the role of testers to dissapear any time soon. But we should be aware that the environment in which we work is changing (agile, lean, etc.) and that we as testers will have to change with it.
      For a long time now Testing and QA seem to change only at a slow pace. And if you aim to be a stable factor in a otherwise turbulent environment you should probably not follow every whim that shows its face. But that doesn’t mean you ought not develop yourselves. It only means that testing as a craft has to be critical and thoughtful towards new developments and value them both for their merrits and their flaws. Good developments should however not be ignored.
      As far as I am concerned the “test is dead” paradigm is not one of those good developments.
      At least not for my context.

      Cheers,

      Jean-Paul.

      PS.:
      Fairly large digital agency is a nice cryptic description of a huge incorporated.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s