Exploratory Testing Research

Good research on testing is hard to find. Why? One reason is that testing does not belong to the field of Computer Science. I mean, sure, some of it does. There is some value to describing and testing an algorithm to efficiently cover a directed graph. But covering directed graphs is not my problem, most of the time. Most of the time, my problem is how to work with other people to simplify a complex world. Most of the time, the testing problem is an exploration and modeling problem within a socially distributed cognitive system. Whew! Whatever that is, it ain’t Computer Science.

Therefore, I am delighted to present two excellent examples of rigorous scientific research into exploratory testing– both of them coming from the field of Cognitive Science.

1. Jerry Weinberg’s 1965 Doctoral ThesisHere, Jerry runs an experiment to determine strategies people use when trying to comprehend a pattern of behavior in a system. In this case, the system is a set of symbols that keep changing, and the task is to predict the symbols that will come next. By observing the pattern of prediction made by his test subjects, Jerry is able to draw inferences about the evolution of their mental models of the system.The upshot is this: tom some extent it is possible to see how testers think while they are thinking. I use this principle to evaluate testers and coach them to think better.
2. Collaborative Discovery in a Scientific DomainThis paper by Takeshi Okada and Herbert Simon is fantastic!They study how pairs of scientists, working together, design and conduct experiments to discover a scientific principle. This is EXACTLY the same thought process used by testers to investigate the behavior of systems.Notice how Okada and Simon collect information about the thought processes of their subjects. It’s very much like Weinberg’s approach, and shows again that it is possible to draw solid inferences and make interesting distinctions about the thought processes of testers.This is important stuff, because we need to make the case that exploratory testing is a rich activity that can be observed, evaluated, and also systematically taught and improved. These two papers deal with the observation and evaluation part, but I think they suggest ways to teach and improve.