What is software testing?

Over time the definitions of software testing and the ideas behind it have changed.
This post brings together an expanding collection of testing definitions in chronological order and thus provide an overview on changing views. Views that are based on the changes in computer and software development one would expect to be outdated but often still are in use today.

  • “How would we know a program exhibits intelligence?
    Alan Turing [1950]
  • “Make sure the program runs” and “Make sure the program solves the problem”
    Dan McCracken; Digital Computer Programming [1957]
  • “Errors plague us, but hidden errors make our job impossible. One of our recurring problems lies not in finding errors but in not finding errors. We must be alert for them; we can never be complacent about our results.” Herbert D. Leeds and Gerald M. Weinberg; Computer Programming Fundamentals [1961]
  • Testing is “any activity that increases our assurance or belief that a program performs ‘as it should’ .” William C. Hetzel; Program Test Methods [1973]
  • “Testing is the process of executing a program (or part of a program) with the intention or goal of finding errors.” Glenford J. Meyers; Software Reliability [1976]
  • “Testing is the process of executing a program with the intent of finding errors.”
    Glenford J. Meyers; The art of software testing [1979]
  • “Test means execution, or set of executions, of the program for the purpose of measuring its performance. That a program was executed with no evidence of error is no proof that it contains no errors; program errors are sensitive to the specifics of the data being processed.” “The goal of testing ought to be the uncovering of defects within the program.”
    Robert Dunn and Richard Ullman, Quality Assurance for Computer Software, [1982]
  • “Testing is any activity aimed at evaluating an attribute of a program or system. Testing is the measurement of software quality.”
    Bill Hetzel, The Complete Guide to Software Testing [1983]
  • “Testing is the act of executing tests. Tests are designed and then executed to demonstrate correspondence between an element and its specification. There can be no testing without specifications of intentions.”
    Boris Beizer, Software System Testing and Quality Assurance [1984]
  • “The Testing stage involves full-scale use of the program in a live environment. It is here that the software and hardware are shaken down, anomalies of behavior are eliminated, and the documentation is updated to reflect final behavior. The testing must be as thorough as possible. The use of adversary roles at this stage is an extremely valuable tool because it ensures that the system works in as many circumstances as possible.”
    Henry Legard, Software Engineering Concepts: Volume 1 [1987]
  • “The purpose of testing a program is to find problems in it”
    Cem Kaner, Jack Falk, Hung Quoc Nguyen; Testing Computer Software [1988]
  • “The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.”
    The 1990 IEEE Standard Glossary of Software Engineering Terminology [1990]
  • “Testing is the execution of software to determine where if functions incorrectly.”
    Robert L. Glass, Building Quality Software [1992]
  • “Testing helps detect defects that have escaped detection in the preceding phases of development. Here again, the key byproduct of testing is useful information about the number of and types of defects found in testing. Armed with this data, teams can begin to identify the root causes of these defects and eliminate them from the earlier phases of the software life cycle.”
    Lowell Jay Arthur, Improving Software Quality: An Insider’s Guide to TQM [1993]
  • “[Unit] Testing is a hard activity for most developers to swallow for several reasons: Testing’s goal runs counter to the goals of other development activities. The goal is to find errors. A successful test is one that breaks the software. … Testing can never prove the absence of errors — only their presence. … Testing by itself does not improve software quality. … Testing requires you to assume that you’ll find errors in your code. …”
    Steve McConnell, Code Complete: A Practical Handbook of Software Construction [1993]
  • “Testing is an unnecessary and unproductive activity if its sole purpose is to validate that the specifications were implemented as written. … testing as performed in most organizations is a process designed to compensate for an ineffective software development process. It is unrealistic to develop software and not test it. The perfect development process does not exist …”
    William Perry, Effective Methods for Software Testing [1995]
  • “A tester is given a false statement (‘the system works’) and has the job of selecting, from an infinite number of possibilities, an input that contradicts the statement. … [You want to find] the right counterexample with a minimum of wasted effort.”
    Brian Marick, The Craft of Software Testing: Subsystem Testing [1995]
  • “Testing is obviously concerned with errors, faults, failures, and incidents. A test is the act of exercising software with test cases. There are two distinct goals of a test: either to find failures, or to demonstrate correct execution.”
    Paul C. Jorgensen, Software Testing: A Craftsman’s Approach [1995]
  • “The penultimate objective of testing is to gather management information.”
    Boris Beizer, Black Box Software Testing: Techniques for Functional Testing of Software and Systems [1995]
  • “The most common quality-assurance practice is undoubtedly execution testing, finding errors by executing a program and seeing what it does.”
    Steve McConnell, Rapid Development: Taming Wild Software Schedules [1996]
  • “Software testing is the action of carrying out one or more tests, where a test is a technical operation that determines one or more characteristics of a given software element or system, according to a specified procedure. The means of software testing is the hardware and/or software and the procedures for its use, including the executable test suite used to carry out the testing.”
    NIST [1997]
  • “Software must be tested to have confidence that it will work as it should in its intended environment. Software testing needs to be effective at finding any defects which are there, but it should also be efficient, performing the tests as quickly and cheaply as possible.”
    Mark Fewster and Dorothy Graham, Software Test Automation: Effective use of test execution tools [1999]
  • “Testing is the process by which we explore and understand the status of the benefits and the risk associated with release of a software system.”
    James Bach, James Bach on Risk-Based Testing, STQE Magazine [1999]
  • “Testing is a process of planning, preparing, executing and evaluating, with the purpose to determine the attributes of an information system and to show the differences between its actual and demanded state” Pol, Teunissen, van Veenendaal, Testen volgens TMap 2e druk [2000]
  • “Software testing is a difficult endeavor that requires education, skill, practice, and experience. Building good testing strategies requires merging many different disciplines and techniques.”
    James A. Whittaker, IEEE Software (Vol 17, No 1) [2000]
  • “Software testing is the process of applying metrics to determine product quality. Software testing is the dynamic execution of software and the comparison of the results of that execution against a set of pre-determined criteria.”
    NIST The Economic Impacts of Inadequate Infrastructure for Software Testing [2002]
  • “Testing is done to find information. Critical decisions about the project or the product are made on the basis of that information.”
    Cem Kaner, James Bach, Bret Pettichord,
    Lessons Learned In Software Testing: A Context-Driven Approach [2002]
  • “Testing is a concurrent lifecycle process of engineering, using and maintaining testware in order to measure and improve the quality of the software being tested.”
    – Rick Craig and Stefan Jaskiel, Systematic Software Testing [2002]
  • “A software tester’s job is to test software, find bugs, and report them so that they can be fixed. An effective software tester focuses on the software product itself and gathers empirical information regarding what it does and doesn’t do. This is a big job all by itself. The challenge is to provide accurate, comprehensive, and timely information, so managers can make informed decisions.”
    Brett Pettichord, Don’t Become the Quality Police, StickyMinds.com [2002]
  • “Software testing is a process of analyzing or operating software for the purpose of finding bugs.”
    – Robert Culbertson, Chris Brown, and Gary Cobb, Rapid Testing [2002]
  • “How do you test your software? Write an automated test.
    Test is a verb meaning ‘to evaluate’. No software engineers release even the tiniest change without testing, except the very confident and the very sloppy. … Although you may test your changes, testing changes is not the same as having tests. Test is also a noun, ‘a procedure leading to acceptance or rejection’. Why does test the noun, a procedure that runs automatically, feel different from test the verb, such as poking a few buttons and looking at answers on the screen?”
    – Kent Beck, Test-Driven Development By Example [2003]
  • “My conclusion is that testing is an intellectual endeavor and not part of arts and crafts. So testing is not something anyone masters. Once you stop learning, your knowledge becomes obsolete very fast. Thus, to realize your testing potential, you must commit to continuous learning.”
    James A. Whittaker, How to Break Software: A Practical Guide to Testing [2003]
  • “All software has bugs. It’s a fact of life. So the goal of finding and removing all defects in a software product is a losing proposition and a dangerous objective for a test team, because such a goal can divert the test team’s attention from what is really important. … [The goal of a test team] is to ensure that among the defects found are all of those that will disrupt real production environments; in other words, to find the defects that matter.”
    Scott Loveland, Geoffrey Miller, Richard Prewitt, and Michael Shannon,
    Software Testing Techniques: Finding the Defects that Matter [2004]
  • “Software Testing [is] The act of confirming that software design specifications have been effectively fulfilled and attempting to find software faults during execution of the software.”
    Thomas H. Faris, Safe And Sound Software: Creating an Efficient and Effective Quality System [2006]
  • “Checking is the inspection of intermediate products and development processes, or verification. These all the activities directed to answering the question: Is the building done rightly?
    Testing is the inspection of end products, or validation. Is the product valid against its demand? Is the right thing build?” (translation) Koomen, Pol Test Process Improvement [2006]
  • “Software testing is a process where we check a behavior we observe against a specified behavior the business expects. … in software testing, the tester should know what behavior to expect as defined by the business requirements. We agree on this defined, expected behavior and any user can observe this behavior.”
    Andreas Golze, Charlie Li, Shel Prince, Optimize Quality for Business Outcomes: A Practical Approach to Software Testing [2006]
  • “Testing is a process of gathering information by making observations and comparing them to expectations.” “A test is an experiment designed to reveal information, or answer a specific question, about the software or system.”
    Dale Emery and Elizabeth Hendrickson [2007]
  • “software testing as a process of technical investigation, an empirical study of the product under test with the goal of exposing quality-relating information about it. (Validation and verification both fit within this definition.)”
    LAWST [2007]
  • “[Testing is] An empirical, technical investigation conducted to provide stakeholders with information about the quality of the product under test. A process focused on empirically questioning a product or service about its quality.”
    Cem Kaner [2007]
  • “An empirical, technical investigation conducted to provide stakeholders with information about the quality of the product OR SERVICE under test.”
    Scott Barber [2007]
  • “Testing is… an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous”
    Jon Bach, Telling Your Testing Story, SEASPIN [2008]
  • “Test:(verb) to critically evaluate; to determine the presence, quality, or truth of something; to conduct a trial.” – used by Paul Gerrard, Test axioms-definitions, http://testaxioms.com/index.php?q=node/3 [2008]
  • “Testing is strategic tool for risk control”
    Egbert Bouman, SmarTEST [2008]
  • “The test process has to produce information about errors found and the degree to which the quality has been improved by solving them. In turn, the tests have to indicate whether the modified system responds to the reason that led to the change, and if it contributes to the business requirements.” de Grood, TestGoal [2008]
  • “Testing is a process that provides insight into, and advices about, quality and its related risks” Van der Aalst, Baarda, Roodenrijs, Vink and Visser, TMap NEXT Business Driven Test Management [2008]
  • “The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. ”
    Jeanne Hofmans and Erwin Pasmans, Quality Level Management [2012]
  • ”Software testing is a process, or a series of processes, designed to make sure computer code does what it was designed to do and, conversely, that it does not do anything unintended. Software should be predictable and consistent, presenting no surprises to users.”
    The art of software testing / Glenford J. Myers, Corey Sandler, Tom Badgett. – 3rd ed. [2012]
  • “interact with the software or system, observe its actual behavior, and compare that to your expectations”
    Elisabeth Hendrickson; ‘Explore It! [2013]
  • “Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc.”
    James Bach, Michael Bolton [2013] http://www.satisfice.com/blog/archives/856
  • “Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. The information is compared against specifications, business requirements, competitive products, past versions of this product, user expectations, industry standards, applicable laws and other criteria.”
    Undisclosed company policy [2015]
  • “Professional, sophisticated software testing is an approach that lets people get the information that they need at the time they need it, in order to make timely decisions.” Trish Khoo, Rock solid software without hiring an army, [June 2016]
  • “Testing is the pursuit of actual or potential incongruity”
    James Thomas [2016] http://qahiccupps.blogspot.nl/2016/11/the-anatomy-of-definition-of-testing.html
  • “Testing consists of verification, validation and exploration activities that provide information about the quality and the related risks, to establish the level of confidence that a test object will be able to deliver the pursued business value”

Undated:

  • Context-driven testers choose their testing objectives, techniques, and deliverables (including test documentation) by looking first to the details of the specific situation, including the desires of the stakeholders who commissioned the testing. The essence of context-driven testing is project-appropriate application of skill and judgment. The Context-Driven School of testing places this approach to testing within a humanistic social and ethical framework.
    Ultimately, context-driven testing is about doing the best we can with what we get. Rather than trying to apply “best practices,” we accept that very different practices (even different definitions of common testing terms) will work best under different circumstances.
    Cem Kaner, James Bach
  • “Software testing is a process of executing a program or application with the intent of finding the software bugs” and “the process of validating and verifying that a software program or application or product: Meets the business and technical requirements that guided it’s design and development; Works as expected; Can be implemented with the same characteristic”
    ISTQB Foundation Exam
  • The process or method of finding error/s in a software application or program so that the application functions according to the end user’s requirement is called software testing.” Economic Times

Unknown origin:

  • “Testing is finding out how well something works”
  • “Testing is the process of evaluating a system or its component(s) with the intent to find out that whether it satisfies the specified requirements or not”
  • “Software testing is the process of evaluating a software item to detect differences between given input and expected output”
  • “Software testing is a process that should be done during the development process”

Why is software development (not) like building a house?

While this blog mostly addresses software testing that is not the only thing I do.
I am also Scrum Master and provide agile coaching. This post is about a topic I have run across particularly in agile environments but is also used in discussions about other development practices.

Why isn’t software development like building a house!?

Building a house starts at the foundation not at the roof!

Houses, bridges and building in general is predictable, why isn’t software building?!

These remarks and similar ones express the discontent some have with software development in general and agile software development in particular. A discontent that I believe arises as a side effect of two agile development basic elements: “transparency” and “prioritization”. Transparency has the effect that a lot of things that usually take place behind the scenes or that are handled implicitly become visible to a wider audience. Prioritization in agile regularly breaks with the seeming simplicity and logical sequentially of traditional software development as it focusses on delivering valuable but partial solutions first rather than complete solutions later. And it does so visibly.

The simplicity of building…

So is building a house really that simple?
Populistic concepts edict that building follows these, or similar, steps:

  • Pouring a foundation
  • Erecting the walls
  • Flooring
  • Adding a roof
  • Closing up with windows and doorsBuilding-Construction

Occasionally an architect, electricity and plumbing is added but not much more. In any case building is considered fairly simple and straightforward and essentially the question is raised why software development isn’t?

The answer to that question is actually simple.
If software development were only compiling the final build and deploying that to production it would be as simple. But software development is not only creating a build an deploying it and neither is building a house if you look further.

A more realistic view on building a house

Building a house doesn’t start with pouring a foundation, it doesn’t even start with the blueprint the architect makes. It started earlier with the design and production of all the materials the architect and the builder chose for construction of the house. Concrete, bricks, wooden beams, pipes, cables, lighting, etc. did not appear out of thin air and ready for use. All of these materials have undergone their own designing, prototyping, testing, and fabrication process before they could be picked by an architect and used by a builder on site. Let’s take a closer look at bricks to illustrate this.

In the Netherlands and Belgium alone there are over 45 different sizes of bricks, there are many mixtures of ingredients (clay, sand, minerals, water,etc.), colors, and textures. While some of the differences are merely decorative many of them are functional. And that is just the brick itself. Consider also all the possible ways of stacking them and all the kinds of cement that can be used to keep them together that took years and years of practical (live) tests.Wall_bonds

So before any first brick is laid not only the architect and builder have chosen which brick to use and how to use it. The manufacturer of the brick has also implemented design specifications, chose a mixture of ingredients, chose a way to manufacture it and had it tested for durability, fire proofing, insulation and structural integrity.

This is so for for most if not all materials used in building. Some of them may be more or less generic but all of them eventually have a specific usage in building a specific house.

Comparing building a house to software development

So how does building house then compare to software development. Well building a house basically can be seen as the sum of all individual material developments integrated into constructing a house. When looking at it in that way it is not very unlike software development of an application. In software development the individual components of the application are developed and eventually integrated into a full application.

Yet in practice building a house is not conceived as similar to software development. And that is not surprising. While in software development components are often developed as part of the same project in which the application is delivered. While in building the individual materials are most likely developed and manufactured separated both in place and time from the actual construction of the building.

Another similarity between building a house and developing software is that, contrary to the simplistic view, both are hardly ever right first time. In fact errors are so common in building that professions and consumer societies exist that are largely based on its shortcomings. Like software development, where there are software testers, auditors etc., building also has its inspectors, overseers etc..

A difference between software development and building a house is the visible duration. While building (and deploying) software can take a few minutes to up to a few days. Building a house takes a few days up to several weeks. On the other of side of the medal preparation of building a house is largely unseen. Which is quite the opposite to developing software. Getting up to a deployable build can last a few days to several months and sometimes up to several years.

Conclusion

I think the continued use of technical references and terminology in software testing spurs the comparison between software development and building. It seems to me that the remarks with which the post started find their basis in a incomplete vision. A vision that is based on contextual experiences. Working in an IT environment lets you see what a struggle software development is. And when your vision of house building comes from an owners perspective you are likely only to take the final construction as reference with all its supposed simplicity.

Even when the development processes leading up to construction is taken into account there still are significant differences in time, space and execution that outweigh the similarities.