Test Levels! Really?!

Next in the series of software terminology lists is “Test Levels”. But there is something strange with test levels. Up until now almost every tester that I have worked with is familiar of the concept of software test levels. But I wonder if they are. What some call a test level, say Unit Testing, I would call a test type. However with a level like Component Testing I am not so sure. It seems only one level up from Unit Testing but now I am inclined to see it more as a test level. In my experience I am not alone in this confusion.

Sogeti’s brand TMap was one of the main contributors in establishing the concept of test levels (or at least so in the Netherlands). But since last year Sogeti acknowledges the confusion in their article “Test Levels? Test Types? Test Varieties!” and propose to rename it Test Varieties. Even ISTQB or ISO do not mention test levels (or test phases) if you like explicitly.

But test levels are a term with some historic relevance and as such they are part of my series of software testing lists. Even if nowadays I never use them anymore.

Acceptance Testing

  • Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. (ISTQB – Standard Glossary of Terms Used in Software Testing Version 3.01)
  • A formal test conducted to determine whether or not a system satisfies its acceptance criteria and to enable the customer to determine whether or not to accept the system. (Cunningham & Cunningham, Inc.; http://c2.com/cgi/wiki?AcceptanceTest)
  • Acceptance testing is the process of comparing the program to its initial requirements and the current needs of its end users. (G. Meyers, The art of software testing (2nd edition) [2004])

Chain Test

  • A chain test tests the interaction of the system with the interfacing systems. (Derk-Jan de Grood; Test Goal, 2008)

Claims Testing

  • The product should behave the way some document, artifact, or person says it should. The claim might be made in a specification, a Help file, an advertisement, an email message, or a hallway conversation, and the person or agency making the claim has to carry some degree of authority to make the claim stick. (Michael Bolton; Testing without a map, 2005)
  • The object of a claim test is to evaluate whether a product lives up to its advertising claims. (Derk-Jan de Grood; Test Goal, 2008)

Component Testing

  • The testing of individual software components. (ISTQB – Standard Glossary of Terms Used in Software Testing Version 3.01

Function Testing

  • Function testing is a process of attempting to find discrepancies between the program and the external specification. An external specification is a precise description of the program’s behavior from the point of view of the end user. (G. Meyers, The art of software testing (2nd edition) [2004])

Functional Acceptance Test

  • The functional acceptance test is carried out by the accepter to demonstrate that the delivered system meets the required functionality. The functional acceptance test tests the functionality against the system requirements and the functional design. (Derk-Jan de Grood; Test Goal, 2008)
  • The functional acceptance test is a test carried out by the future user(s) in an optimally simulated production environment, with the aim of demonstrating that the developed system meets the functional requirements. (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)

Hardware-software Integration Testing

  • Testing performed to expose defects in the interfaces and interaction between hardware and software components. (ISTQB – Standard Glossary of Terms Used in Software Testing Version 3.01)

Integration Testing

  • Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. (ISTQB – Standard Glossary of Terms Used in Software Testing Version 3.01)

Module Test

  • Module tests focus on the elementary building blocks in the code. They demonstrate that the modules meet the technical design. (Derk-Jan de Grood; Test Goal, 2008)
  • Module testing (or unit testing) is a process of testing the individual subprograms, subroutines, or procedures in a program. Module testing (or unit testing) is a process of testing the individual subprograms, subroutines, or procedures in a program. (G. Meyers, The art of software testing (2nd edition) [2004])

Module Integration Test

  • Module integration tests focus on the integration of two or more modules. (Derk-Jan de Grood; Test Goal, 2008)

Pilot

  • The pilot simulates live operations in a safe environment so that the live environment is not disrupted if the pilot fails.

Production Acceptance Test

  • The system owner uses the PAT to determine that the system is ready to go live and can go into maintenance. (Derk-Jan de Grood; Test Goal, 2008)
  • The production acceptance test is a test carried out by the future administrator(s) in an optimally simulated production environment, with the aim of demonstrating that the developed system meets the requirements set by system management. (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)

System Test / System Testing

  • Testing an integrated system to verify that it meets specified requirements. (ISTQB – Standard Glossary of Terms Used in Software Testing Version 3.01)
  • The system test demonstrates that the system works according to the functional design. (Derk-Jan de Grood; Test Goal, 2008)
  • System testing is not limited to systems. If the product is a program, system testing is the process of attempting to demonstrate how the program, as a whole, does not meet its objectives. (G. Meyers, The art of software testing (2nd edition) [2004])
  • System testing, by definition, is impossible if there is no set of written, measurable objectives for the product. (G. Meyers, The art of software testing (2nd edition) [2004])
  • A system test is a test carried out by the supplier in a (manageable) laboratory environment, with the aim of demonstrating that the developed system, or parts of it, meet with the functional and non-functional specifications and the technical design. (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)

System Integration Test

  • A system integration test is a test carried out by the future user(s) in an optimally simulated production environment, with the aim of demonstrating that (sub)system interface agreements have been met, correctly interpreted and correctly implemented. (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006) 

Unit Test

  • A unit test is a test carried out in the development environment by the developer, with the aim of demonstrating that a unit meets the requirements defined in the technical specifications (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)

Unit Integration Test

  • A unit integration test is a test carried out by the developer in the development environment, with the aim of demonstrating that a logical group of units meets the requirements defined in the technical specifications (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)

User Acceptance Test

  • The user acceptance test is primarily a validation test to ensure the system is “fit for purpose”. The test checks whether the users can use the system, how usable the system is and how the system integrates with the workflow and processes. (Derk-Jan de Grood; Test Goal, 2008)
  • The user acceptance test is a test carried out by the future user(s) in an optimally simulated production environment, with the aim of demonstrating that the developed system meets the requirements of the users. (TMap NEXT; Michiel Vroon, Tim Koomen, Leo van der Aalst, Bart Broekman, 2006)
Advertisements

TMap Day 2014

On October 28, 2014 I visited Sogeti’s TMap Day.
This year’s focus was on their new book “Neil’s Quest for Quality”,
subtitled “A TMap© HD Story”.

This blog post describes my first impressions of that day.  When I have read the book I will either return to this post and adjust it or write a separate one on the books content.

Note: I have read the book and have adjusted the post.

The book is written as a novel. It contains “the TMap Human Driven story, consisting of a business novel, building blocks, Mr. Mikkel’s musings and contributions from the innovations board in testing”.

A quality-driven approach

The new TMap presents itself as a quality-driven approach that is captured in the TMap© Suite which consists of the following three parts:

  1. TMap Next
  2. TMap© HD
  3. Building Block described in the TMap HD book and gathered, maintained and extended on http://www.tmap.net

Inspired on Lean, Agile and DevOps

Both authors explained their contribution to the book and expressed that the elements (more detail later) are mostly inspired on the market move towards Agile and DevOps and the whole is based on a Lean approach to software development.

Aldert Boersma, one of the authors, positioned quality-driven as follows.

Position_TMapHD

The approach described in TMap HD© distinguishes five basic elements:

TMapHD_Elements

The (short) TMap HD descriptions for these concepts are:
People
Only people realize moving from “Testing according to TMap” to “Testing with TMap”.People with a broad knowledge of quality and testing that is.

Simplify
Make things as simple as possible – but not more simple than that.
Start small and work from there. A complicated process will only lose focus on the result.

Integrate
Integration with respect to testing denotes to a shared way of working, with a shared responsibility for quality. Testing is not a stand alone process.

Industrialize
Industrialization is important in improving testing and optimizing quality. Test tools are used to test more, more often, and faster.

Confidence
The goal of TMap HD© is providing confidence in IT solutions through a quality-driven approach. Confidence is the fifth element over and above all others.

Building Blocks

The first four elements should help you choose and apply building blocks that give (build) confidence. There is no prescribed set of blocks to use and main blocks themselves are seen as larger blocks of which smaller parts are chosen.

The now available building blocks are:

  • Test manager
    In the Test manager presentation the remark was made that fewer people will be TM but the activities will remain
  • Test manager in traditional
  • Assignment
  • Test organization
  • Test plan
  • Product risk analysis
    Oddly during the test management presentation and in the book this was called
    Product Risk & Benefit Analysis but that is not part of the website (yet)
  • Test strategy
  • Performance testing
  • Test approaches
  • Crowd-testing
  • Test varieties
    This replaces the current Test Levels and Test Types.
    They are divided in Experience based and Coverage based
  • Test manager in agile
  • Permanent test organisation
  • Model-based testing
  • Quality policy
  • Using test tools
  • Quality-driven characteristics
  • Integrated test organization
  • Reviewing requirements

As a kind of closing motto for that morning the following phrase was handed as a summary for test managers “Do not report trouble but offer choices for the client”.

So…

The TMap Day and the book have left me with rather distinct, and slightly contradicting, impressions.

A move in the right direction

Sogeti has embraced the fact that software development and with it software testing has changed over the last decades. The rise of Agile and Lean on the one side and the decline of Waterfall hasn’t gone unnoticed and the new brand certainly addresses these developments. There is also an influence that Sogeti has carefully tried to avoid in mentioning but that I believe is clearly present. That influence is Context-Driven Testing. In spite of naming it environment, circumstance or situation TMap HD shares the principle that based on the context software testing is and should be different and use that what best suites the context.

Criticism

Ever since TMap, and TMap Next and particularly since their training and certification program appeared there has been a lot of criticism. This criticism especially focusses on the rigid factory school view on software and the limited value of TMap certification. While Sogeti itself did not react to this much many of the authors, most of them no longer working for Sogeti, did. The common denominator in their response was that the content was misunderstood and it was never meant to be followed by the letter.

Next to a wider interest for and influence of new software development approaches the emergence of building blocks shows that parts of the criticism is taken to heart and that TMap should and can now be used more flexible.

Superficial

In the Netherlands we have a saying “Oude wijn in nieuwe zakken” (litterally Old wine in new wineskins) expressing that although it looks new it’s still the same old stuff. I believe this applies also TMap HD. Even with influence of Agile and Lean and the introduction of Building Blocks in the book I am still left the feeling that beneath the surface the nature of the solutions is still the same as before. This feeling is enhanced by the fact that TMap Next is still declared to be the core of the testing approach and that all existing training courses and certificates remain. Especially that last part has led to the rigid and limited testing approach that many Dutch testers employ.

So while in theory there is hope for positive change I fear that in reality nothing much will change for the better.

Seven Questions

This is the opening blog of a small series of posts in which I elaborate on a test approach heuristic using 7 questions that I have developed over the years.

Thinking about testing

As a tester I have seen many approaches to software testing pass me by. A few of them, like TMap (Next) and ISTQB were picked up by the Rabobank and I have had the mixed pleasure of working with them. But regardless of how different the approaches voice out to be from each other they all seem to have a number of things in common:

  • They are mostly oriented on management (of testing)
  • They focus on processes and deliverables
  • They do not teach you how to actually test something in practice
  • They hardly make any connection to software development in general
  • They are supposedly mastered after certification

I admit that both TMap and ISTQB (initially) helped to give testing a positive foothold in many organisations and have underlined that testing should get its place in software development. Even so the five elements I have described above should also show that there are fundamental flaws in how these approaches apply testing. Following them does not guarantee you to get fully involved into software development nor does it teach you how to test in practice. Usually as a compensation for these flaws testers go to boot camp like courses to teach them more practical testing skills, like determining test coverage and applying test design techniques. Even so for many testers the start-up of their professional life is focused on getting the certification and maybe some introduction to actual testing. And then….well…..for of them most it ends here and they go out to work and follow the processes and deliver a bunch of documents. If your new to software testing this will probably keep you busy for a while, but eventually you (should) start to ask yourself questions like: Is there no better way to test this? Do I really have to write these elaborate test plans / test scripts / test cases that nobody seems to really care about? Why don’t the developers agree with me on my defects? Why is my work not valued?

I have asked myself these and similar questions and over the years I have come up with a set of alternative questions whose answers guide me through a development / test cycle. These questions demand creativity, knowledge and skilled experience to answer them. And any answer you can come up with this time will differ the next time you ask yourself that same question again.

The seven questions I use are:

I have done a talk on these seven questions at EuroSTAR 2012 and will do the at Belgium Testing Days 2013. Contact me there if you want to meet and talk, or sent me a tweet @arborosa .

Here are some twitter reactions that I got from talking at EuroSTAR 2012