Following the news – Code inspection is 80 percent faster than testing

During the CAST 2014 conference in New York I participated in a workshop by Laurent Bossavit and Michael Bolton called “Thinking critically about numbers – Defense against the dark arts”. Inspired by this workshop I took a look at one of the Dutch sites addressing news about software testing www. This is the second post to come out my curiosity.

On May 23, 2014 Testnieuws hosted an article “Code inspectie is 80 procent sneller dan testen” (translated Code inspection is 80 percent faster than testing). The article itself provides little more substantiation for the claim than a reference to research by IfSQ. Both the claim and usage of this as header seems to only serve to grab the readers attention. The article ends with an invitation to read more about it and this leads to what I think is the actual article “Status ICT-projecten vaak compleet onduidelijk” (translated “Status of ICT projects often completely unclear). This article describes that, especially government, projects need to have more objective information and they need to get it earlier. This way it is possible to determine the status of a project. Andres Ramirez, Managing Partner of the OSQR Group states that “better software leads to better projects” and “better source code leads to better software” and “the quality of source code can be objectively assessed by the guidelines from the Institute for Software Quality”. The last quote explains the IfSQ abbreviation used earlier.

A little further in the article the claim is used and even extended “Code inspection is 80 percent faster than testing, and finding and repairing code is much cheaper than testing”. Ramirez also adds “Research by IfSQ shows that regular code inspection during the production process ensures that software can be changed more easily. Inspected software is 90 percent cheaper to maintain.”. I am choosing to ignore these last claims for now and proceed to IfSQ the look into their research.

The IfSQ – Research Findings Relevant to the IfSQ Standards hosts about 50 or so reference to articles and research results divided into sections “Why should you inspect software?”, “When should you inspect software?” and “What should you look for?”. Noticeably the focus is strongly on code quality and, to my opinion, therefore not really on software quality as such. Also there seems to be need to position code inspection opposite to testing as suggested by titles like:

The second title points to a page with the title “Inspection is 80% faster than testing” which indicates I am on the right track. The page however only repeats “Code reading detected about 80% more faults per hour than testing.” and provides two, non IfSQ, sources for it without further argumentation. The sources are:

So, at least in this case, so called research findings by IfSQ do not point to research executed by IfSQ themselves, nor were they involved as both sources are quite old and IfSQ was established much later in 2005. Next step to identify which of the two sources holds the quote.

The first article was easily found. In summary the article describes a scientific study that applies an experimentation methodology to compare three (then) state-of-the-practice testing techniques: a) code reading by stepwise abstraction b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using 100 percent statement coverage. It compares three aspects of software testing: fault detection effectiveness, fault detection costs and classes of faults detected. It focussed on unit testing code using a limited set of specific programs, known errors and a mix of academics and professional developers.

Although it found difference between the three test techniques with in some instances an identifiable hierarchy of code reading, functional testing and structural testing non of the results came anywhere near the claim of being 80% faster. So my conclusion is that this article cannot be a valid source for this claim.

I could only find the second at IEEE and as a result the article could only be read by buying it. Setting aside my initial dislike of paying for information, especially if it so old, I tried to buy it. Unfortunately the cash module did not like my dutch creditcard. As a result I stuck to a number (4) abstracts and a course summary of where the article was used.

The second article came closer to the IfSQ description of code inspection in describing how its done, what is needed for it and what it can measure. Still none of the abstracts said anything about being faster than testing. They did mention percentages around 80% for defects found by software inspections. This to me is a different claim. Sounds to me that a ‘leaky’ inference was made and worse another attempt to gain credibility by bringing testing in disrepute.




2 thoughts on “Following the news – Code inspection is 80 percent faster than testing

  1. I think you’d like a conversation with Graham Bolton, founder of ifsq, I know him to be especially focussed in code quality (primarily meant for developers writing code) so that software testers can test the product without being hindred by bugs that could have been prevented by just a simple code review, f.e. Empty statements or magic numbers that might cause a problem. Ifsq work isn’t about disreputing software testing, it’s about improving code quality (quality as measured points in the levels) so that software testers can focus on the functionality (ao) of the product. The code inspections certainly don’t obsolete good testing! code inspections (can) help developers look and think more thouroughly about their code(hygiene). The article doesn’t state a negative about software testing as I interpret, but I might understand why it’s perceived that way. The article might not be have been the most favourable for the topic, but that doesn’t mean code inspection isn’t good or usable IMO. Wanted to provide some context from my experience with ifsq and code inspections. Hope this doesn’t offend anybody , as seems whatever I write/think does, it’s not meant as insult or as offensive.


    1. Hi Nathalie,

      Thanks for the response.
      In my opinion I haven’t said that code inspection isn’t good or not usable. Nor have I said anything other about IfSQ than that their focus particularly lies with code inspection rather than the broader field of software testing as a whole. And that in case of this particular instance the research they did consisted of referencing two rather old articles. Both of them, by themselves or combined, do as far as I can tell do not warrant the claim that code inspection is 80% faster than testing.
      I see a follow up article emerging in discussing the merits of using/merging/concurrently use a wide variety of quality oriented approaches in software development. But thats another matter.
      To address the last part of your comment. It is true we both come from different communities and follow different approaches to software testing. And sometimes our approaches differ so much that either of us would respectfully decline to work in such a way. Even so I believe we are both software testing professionals who take our job serious and have contributed to the software testing community as a whole. I may go into fierce discussion with you about something but I aim to do that on topic and not on person. Which is something I believe most context-driven testers do even when occasionally crossing the line.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s