Being challenged by James M. Bach

On March 31st I joined Huib Schoots and Pascal Dufour in picking up James Bach from Schiphol Airport and driving him to his hotel in Dordrecht. Having met Cem Kaner (briefly) and Michael Bolton (several times) I was looking forward to meeting James in person and I was not disappointed. Conversation en route was vivid and fruitfull  and continued during dinner at Villa Augustus and later on April 3rd after an Agile Meetup with presentation by James at Codecentric. During dinner James had offered Pascal a challenge and later that evening he offered me the following challenge on twitter:

Explain dendogram-based testing

My first thought was “What’s a dendogram?” So I googled it and saw something I had long forgotten from when I studied Social Sciences and certainly had never used in testing.

So for those of you who requested me to blog on testing and the social sciences this is my first post on the subject.

For all the others with this post I want to share my answer to the challenge with you and some of the reactions I got from James on it.

What is dendogram-based testing?

To answer this question I will start with a short description of what I understand a dendogram is. Followed with a description of how I think this could be used in testing.

Dendogram

A dendogram is a visual representation of data that shows hierarchical clusters of, potentially before unknown, groups of data in a cluster tree diagram. A picture of such a diagram is shown below.

Cluster analysis, for a dendogram, starts with a data matrix, where objects are rows and observations are columns. From this beginning, a table is constructed where objects are both rows and columns and the numbers in the table are measures of similarity or differences between the two observations.

  X1 X2 X3
O1      
O2      
O3      

 From measurements to relative observation results 

  O1 O2 O3
O1      
O2      
O3      

 You can then place the results in a cloud of n points or in a matrix and put each point in a class of its own (having n classes, each containing a single point). Then, look for the closest two classes (for a given distance, for instance the relative distance between measurement of a feature– but other distances will give other results, perhaps more appropriate for some data sets) and join them into a new class. You now have n-1 classes, all but one containing a single element. Then iterate this activity until you reach a single set containing al the classes. The result is presented as a dendogram: the leaves are the initial 1-element classes and the various “levels”) of the dendogram are various clustering’s of the data, into a decreasing number of classes.

It is also possible to calculate the distances and their relationships with the use of Euclidian algebra. This uses a number of formulas to calculate hierarchical distance between the data (clusters). But besides the relative complexity of the calculation I prefer to use the more visual and intuitive approach.

Dendogram-Based Testing

The concept of clustering  data in a hierarchical form and thus grouping it based on some measured characteristic opens a number of possibilities for testing. All of these possibilities have in common that both the activity of clustering of data and the end result, be it in numbers or a visual representation, form an additional source of information about the subject under test. The kind of information will largely differ based on the chosen measurement. With regard to testing I see the following, extendable list of areas for observation and measurement:

  • The relationship between functionality and code

Where code is differentiated into several elements such as (that are more or less commonly used):

  • Type of input
  • Used variables
  • Classes
  • Functions
  • Methods
  • Etc.
  • The relationship between test cases and code
  • The relationship between functionality and GUI elements
  • The relationship between defects and code
  • The relationship between defects and their cause of failure

The clusters of information that you get out of it provide the following information and activities for testing:

  • The different clusters of information can suggest different testing techniques
  • The different clusters of information can suggest different testing missions
  • The different clusters of information can suggest different elements of reporting

In essence the information derived from a dendogram is dependent from the kind of observation and measurement you choose, the skill you have in identifying the clusters and patterns and the skill to adapt your testing activities. In any case it forms a new oracle to be used in software testing.

The above description was made before I read the following document, that I had avoided on purpose:  http://melindaminch.com/docs/thesis.pdf

In my, opinion the thesis is a nice academic exercise that adds insight to testing, but lacks sufficient practicality for use in every day testing. But I will study it more carefully at sometime in the future and I will see if there is an opportunity to use HCE 3.0 that I have downloaded.

For now I prefer the more general intuitive approach as described above and instead of the more elaborate HCE program I would use a reverse mind map.

Giving me the below, originally not in my answer included, graphical steps:

Which translates into the following Dendogram

James response to my answer

Below I have pasted the, unedited, response James gave my to my answer.

Jean-Paul, I am impressed. This is an excellent answer. I concur with your reasoning. I’d like to try using dendograms in testing, when the opportunity arises.

The reason I gave you this challenge is that we who aspire to excellence must be comfortable describing test techniques that don’t yet exist– in other words, we must be confident in our ability to invent test techniques as needed. This is a challenge that cannot be solved well using Google, since almost nobody has applied dendograms to testing, and the one reference you cite does a poor job of it.

By researching this and replying as you did, you have marked yourself as a leader in the Context-Driven community. Thank you.

— James

Needless to say that I am proud to receive such a response. And I feel strengthened in my drive to grow and learn as tester and as a member of the context-driven community.

Advertisements

Belgium Testing Conference 2012

QA vs.Testing: Antagonism or Symbiosis?

The header above was the theme of the last Belgium Testing Days in Brussels on March 12-14. During the call for papers for this conference, several months ago, I was in the middle of having SOx compliance established for one of my projects. The theme caught my attention since it represented one of my feelings on the compliance process at the time.

I work at an internationally operating bank which has a few consequences for the context in which I work. The most obvious consequence is that a bank uses either financial software or software that enables financial processes. As a result the (in-house) developed software gets extra special attention with regard to accessibility (or perhaps rather inaccessibility), integrity and confidentiality (AIC). Every piece of software that is built or bought by the bank gets a so-called AIC assessment and depending on the result of the assessment and certain amount of checks, controls and measures are mandatory.

The AIC assessment itself is essentially internal to the bank. But being a bank, and especially being an international bank, this means that on top of the internal regulations all kinds of external government and financial market regulations are imposed on it. The bank QA department translates these regulations or standards into internal processes and rules. For most of the high level business processes such a translation seems fairly straight forward. These processes are often both described and measured at the same way as the regulations. It gets more difficult if you drill down into the organization and start taking all the contributing activities and tasks into account such as in my case the development of software or more particularly the testing of software.

Organizational response

One of the financial organizations common responses is to apply and design standards and procedures together with any number of deliverables.

These standards are prescriptive of nature. They tell you in general terms what their idea is. But, depending on your QA department it gets more specific. They tell you what you must do, how you should do it, in what order you should do it and how you should call it.

The so designed procedures and processes describe the steps you are supposed to do given some standard situation. And since seeing is believing they also formulate how you should proof that you followed the process. In many cases such proof is the delivery of a number of deliverables. Deliverables can be a lot of things, but typically they are in the form of documents, test ware libraries or reports.  Given a certain standard they follow a fixed format both in terms of content, that is what should be described, and in terms of lay-out, that is how it should be described.

Quality Assurance

To my experience for the most part the somebody defining the standards to be used, the procedures to be followed and the deliverables to be created are not the developers or testers nor the customer, but is a typical staff department:

The Quality Assurance department

I see quality assurance not as a singular activity. In my opinion it is a group of activities. Activities that have a difference in focus.One part of quality assurance is focussed on making the chosen framework useable and applicable within the organization. QA as the designer. The second part of quality assurance is closely related to this as it acts as the controller of what was previously designed. To this end it translates the designs into to points of measurement and puts values to the measurement results. QA as the controller. These two might go by different names in some organizations, like quality management, but in my opinion this still is part of quality assurance. The third part of quality assurance is the part in which the actual software development related activity is taking place. It is the part that also executes the previously designed steps and  reports back on them. QA as executor.

At this point QA starts to be seen as testing which is captured in the following definition that is often used for both:

The process of validating and verifying that a software program/application/product meets: The requirements that guided its design and development;  works as expected; and can be implemented with the same characteristics

This definition has a certain appeal. It is understandable; it is similar to other process oriented methodologies; it aligns with the QA concept. But in my opinion it limits testing to checking.

Testing

The previous definition is not the definition I would use for testing. In my opinion the kind of information testing provides depends on the what the stakeholder values as important for the softwares quality. Therefore my definition is:

Software testing is a means to provide information to someone who matters about the product or service under test in a certain context at a certain time.

If I apply this definition to my AIC, SOx compliant context I find that the current solution QA has offered me does not meet this definition. Do not get me wrong I understand and agree that the change process and software development should be both traceable and accountable. I do however not believe that the solution is to enforce processes, procedures and deliverables that are judged by their presence and adherence to layout. What matters is the content that they should be presenting.

Not the process story but the testing story should be told

An example

This example describes in summary which measures the SOx team and I agreed on that are required by software development projects, and specifically testing, necessary to comply to Sarbanes-Oxley Act (SOx) regulation.

A SOx regulation review is targeted at the state of software development and testing at the moment of release to production. The intermediate states or progress steps in establishing traceability and documentation compliance are out of scope for the investigation.

Manual Testing

Testware management for manual testing follows the guidelines as summarized in the following steps:

  • A test plan or a similar structure identifies functional test objects that, minimally, covers the functionality as specified in the requirement or change documents.
  • For each of these test objects specific test activities are logged
  • The test structure identifies the release, individual RFC’s and relates them to the test activities.
  • A  full and complete overview of test objects, test activities and the test results should be established prior to release of the changes to production. This can be either in a testware management system, like preferably HP QC, or in another reviewable form
  • All tests should either have passed or if not passed have a logged defect and or     stakeholder decision attached that indicates that this is acceptable to go into production at this point in time

Automatic Testing

In essence the testware management for automated testing follows the same guidelines as manual testing. Main difference is that the way of documentation is adapted to the structure of automated testing:

  • A test automation tool or input data sheet used by the test automation tool shows the automated tests with a reference to the test object, functionality or RFC that they test
  • A log file (preferred), or checklist, shows whether the tests have been executed and if they have passed or failed
  • A full and complete overview of test objects, automated tests and the final      execution of tests with test results should be established prior to release of the changes to production
  • All tests should either have passed or if not passed have a logged defect and or     stakeholder decision attached that indicates that this is acceptable to go into production
  • If for automatic testing a self designed tool or framework is used the      functionality of the tool and the execution of the test cases should be validated by peer review. Results of the review should additionally be captured in the test report

The above steps should result in the following deliverables:

–          Basic testware describing the test design, test execution, test results and defects
–          Test Report, containing an advice for release

Test report

You might have noted that there is no mention of describing tests in advance, of following test scripts, nor of following standard or using templates. The SOx compliancy team’s focus is not on how the testing is executed during the project. Its focus is to see if the implemented changes are tested and that the executed tests and test results can be matched to them. Their aim is to establish that the changes do not have an unexpected or undesirable effect on the companies annual balance or regulatory capital. To that end they check if the changes are implemented as intended.

The SOx compliance essentially asks for what James Bach describes as three stories that describe the testing story:

  1. A story about the status of the product
  2. A story about how you tested it
  3. A story about the value of the testing

The only thing now left for me is to convince the QA department that even when I have not followed their standard and procedures and not used their templates I have thought about the reason behind their questions and am still able to supply the information they need. Only a bit faster and more naturally.