When Cem Kaner introduced the term “exploratory testing” in the mid-80’s, everyone ignored it. When I picked up the term and ran with it, I was mostly ignored. But slowly, it spread through the little community that would become the Context-Driven School. I began talking about it in 1990, and created the first ET class in 1996. It wasn’t until 1999 that Cem and I looked around and noticed that people who were not part of our school had begun to speak and write about it, too.
When we looked at what some of those people were saying, yikes! There was a lot of misunderstanding out there. So, we just kept plugging along and running our peer conferences and hoping that the good would outweigh the bad. I still think that will happen in the long run.
But sometimes it’s hard to stomach how the idea gets twisted. Case in point: James Whittaker, an academic who has not been part of the ET leadership group, and also has little or no experience on an industrial software project as a tester or test manager, has published a book called Exploratory Software Testing.
Whatever Whittaker means when he talks about exploratory testing is NOT what those of us mean who’ve been working on nurturing and developing ET for the last 20 years. As far as I can tell, he has not made more than a shallow study of it. I will probably not write a detailed review (though his publisher asked me to look at it before it was published), because I get too angry when I talk about it, and I would rather not be angry. But Adam Goucher has published his review here.
Another guy who shows up at the conferences, B.J. Rollison, also gets ET wrong. He’s done what he calls “empirical research” into ET, at Microsoft. Since he, again, has not engaged the community that first developed the concept and practices of ET, it’s not altogether surprising that his “research” is based on a poor understanding of ET (for instance, he insists that it’s a technique, not an approach. This is similar to confusing the institution of democracy with the mechanics of voting), and apparently were carried out with untrained subjects, since Rollison himself is not trained in what I recognize as exploratory testing skills.
Experimental research into ET can be done, but of course any such work is in the realm of social science, not computer science, because ET is a social and psychological phenomenon. (see the book Exploring Science, for an example of what such research looks like).
Now even within the group of us who’ve been sharing notes, debating, and discovering the roots of professional exploratory thinking in the fields of epistemology and cognitive psychology and the philosophy and study of scientific practice, there are strong differences of opinion. There are people I disagree with (or who just dislike me) whom I still recognize as thoughtful leaders in the realm of exploratory testing (James Lyndsay and Elisabeth Hendrickson are two examples). Perhaps Whittaker and Rollison will become rivals who make interesting discoveries and contributions, at some point. Time will tell. Right now, in my opinion, they are just skating on the surface of this subject.
Hi James,
before I posted this I tried to read the commenting policy out of respect but found that the link at https://www.satisfice.com/blog/archives/57 isn’t working.
[James’ Reply: Thanks for reporting that problem. The link now works.]
After looking around for a bit I found the policy in question, however I’m not sure if this is the latest version/the one that you actually wanted to link to.
[James’ Reply: Yes, it’s correct.]
With regard to ET Skaters, I only today commented on an article from Rob Lambert (http://post.ly/5ogr) about the importance of integrity in our profession.
Disclaimer: I haven’t read the book yet and probably won’t now after reading Adam’s review.
From Adam’s review it sounds to me as if the integrity went down the drain here.
If a book about ET is being published, the people who did earlier work on it need to be quoted.
It should cover the area in a way that brings value to the reader. I’d argue that judging from the number of pages only and the poor review that there is a mismatch between the value to the reader and the price of the book.
Furthermore the release of a book like this based on the name of a well-known figure in the testing field could be called an exploitation of the people buying it as especially in our field tester names often are a synonym for quality.
Coming back to my original point, for the sake of making money integrity is taking a second place. In the end it only damages James W so I think in the long run this might backfire which I find more sad than anything else.
I had the chance to hear BJ Rollins keynote speak at SIGIST a couple of months (years?) back and also attended his workshop. While I can clearly see that he has a lot of experience I was a bit bewildered by his stance to ET. Basically he was saying that all testing is exploratory so there’s nothing special about it.
[James’ Reply: I don’t have the impression that Rollison has much experience as an actual tester or test manager. If he ever saw me for a job interview, I would have to go over that carefully.
Anyway, this is an example of Rollison’s strangely shallow view of the subject. Everyone who comes to race car driving school ALREADY drives a car. So, by Rollison’s logic, the ability to drive a car makes race car driving school uninteresting and irrelevant, even for people who want to drive very fast and very well. Does that make sense to you?
Of course all testing is exploratory to some degree. But not many people are good at ET. My quest is to develop methods of training people in the skills of ET so they can operate on a high level. I teach a sort of racing school for testers. Rollison has publicly declared the that things I do are a kind of empty snake oil, designed to extract money from gullible clients. Hmmm. Well. Does he actually know what I do? I haven’t seen him contribute at the peer conferences. I’m not aware of a single new method or idea that Rollison has contributed to the field of testing, though he does like to sneer at people who have contributed.]
Sort of saying everyone is special so really nobody is. He then put his MS spin on his testing approach. I don’t mind opinionated people but that was a bit too much for me. I’m not sure if MS breeds this kind of politically minded people spinning facts until they serve their needs. The sad thing is that this is not necessary at all and that more honest discussions would get everyone further. But maybe not as much money in the long run. I’m with Rob L in that one, passion and integrity over money.
I don’t expect to get this published but wanted to share some of my thoughts and impressions.
Thomas
I’m wondering if we could figure out someone who introduced the “testing” term. Imagine what James The First Tester could feel observing the difference between “his testing” and present-day testing. I might be wrong but the same thing is about exploratory testing. It doesn’t matter what James Whittaker does or doesn’t. He does what he does. It matters what you do. People are not stupid. They are able to get the best part from all approaches they are aware of.
Anyway, thanks for all your articles and books.
[James’ Reply: The first serious tester I know of is Socrates, but he didn’t use that term, or anything much like it.
I know one of the guys from the very first project in the computing industry that ever had a dedicated test team: Jerry Weinberg. Jerry wrote about testing in his 1961 book on computer programming. How he wrote about testing is consistent with what I write about it.
The way I speak of testing is also consistent with how the word is used in the philosophy of science. In any case, that’s my goal.]
If you want a copyright or trademark, apply for it. Don’t get jealous of other people making money or using words that everyone learns how to spell in elementary school.
[James’ Reply: Copyright is not the applicable law. Trademark is. I don’t trademark the words I use because I don’t believe anyone should own dictionary words.
I don’t own exploratory testing. As I said in my post, there are people I disagree with whom I would still describe as leading thinkers. I wish they would just go along with me, but I think it’s fair that they don’t.
The reason I’ve spoken up on this is not that I think other people have no right to their own ideas. It’s simply that I think these particular people are doing bad work. Since I know I’m going to be asked what I think of what they are doing, I’m going on the record about it.]
I wonder why you guys have not gathered all you’ve done about ET in a book after these 20 years…?
When I as a beginner searched for materials about exploratory software testing, all I got was some articles and presentation or comments in blogs and suchlike.
When I googled and found the book (James Whittaker’s) about ET I said: FINALLY! and I ordered the book immediatelly. ( I actually ordered your book as well at the same time: Leasson learned in software testing…)
[James’ Reply: I wrote a chapter on ET for two different books, and Lessons Learned discusses it.
The reason my community has not written a book about it is mainly that we are still developing our ideas about it and trying them out on projects, in classes, etc. These ideas go deep. However, another reason is that we’re all busy with our work. It’s hard to write a good book. Whittaker didn’t write much, actually, if you check out what he actually says.
I spent six months at home writing Secrets of a Buccaneer-Scholar. If I could take another four months off I’d finish my ET book.]
I have to wonder how personally attacking JW helps make your point about what you disagree about in his book.
[James’ Reply: The point of my post was not to review his book. I had a different goal in mind: to contradict a specific claim he made, and to publicly put him on notice, since he appears to ignore what I say in private conversations.]
Granted I haven’t read it, but pointing fingers and calling him an “academic who has not been part of the ET leadership group, and also has little or no experience on an industrial software project as a tester or test manager” doesn’t help make your point.
[James’ Reply: Really? I think it does. I think it’s relevant to report my opinion. If you don’t respect my opinion, that’s okay. I don’t know who you are, either.]
The ideas of exploratory testing may have been taken by others and turned into something you disagree with – but that’s the nature of these things. Take the opportunity to refine your explanations of ET in response to their work instead of belittling their work. Be proud that you helped start something. You don’t own the concept, and someone doesn’t need to be part of your good boy’s club (ET leadership group) to contribute to the conversation of what ET can be.
[James’ Reply: I’m not claiming to own anything. I have no trademarks. But I do have a reputation for innovation and incisive work.
Anyone can say anything. We have free speech in this country. Similarly, I get to share my thoughts, too. Are you concerned that my ideas lack substance? Well, feel free to read any of the many blog posts, articles, class materials, and videos I have made on these subjects.
Given that the people I’m calling out ignore or mischaracterize what I do, I don’t see how “refining” it will help with that. But actually, I’ve been refining it all along, and continue to do that.]
Hi,
I really enjoy these discussions that to me seem to epitomise a fundamental dichotomy in the way our educations systems on the one hand teach us what we are expected to know and on the other hand stifle imagination. It is odd that you go to university to firstly learn what others teach you, then to master that knowledge to the extent that you can teach others and finally if you have proven that you have mastered the knowledge, they may offer you an opportunity to undertake a rational investigation of the truths and principles of being, knowledge, or conduct (Philosopy) or if you like the pursuit of wisdom. I get so many graduates with elaborate degrees in IT proudly displaying their ISTQB foundation certificates who have never taken the time to think. They seem to have never gone outside their front door, never wonder “why it is so” and cannot understand the concept of risk and consequence that is so fundamental to what we do as testers. They are supported in their ignorance by those who paint the world in simple pictures and provide concepts, methodology and processes (and certifications) that make them feel safe. There are marvelous academics out there and great ideas but they are by their nature uncertain, they require that you think, criticise, be sceptical and question and as such they are uncomfortable and ill-fitting. It is much simpler therefore to sell ideas that are comfortable, where you don’t need to think too much. It is much easier to get your bit of paper in two days that says you are capable than to stand there and say you are still learning. It is easy to write a book on exploratory testing for dummies. I will wait for the real thing. “Genius knows its limitations, stupidity is not thus constrained” Oh and the latest from the ISTQB:
Talk the same talk. Everyone will learn accepted, consistent terminology to lessen confusion and misunderstandings.
Become masters of your domain. Mastering fundamental testing principles and practices creates a more productive and cohesive team.
Get “credentials.” “CTFL” at the end of your name not only means that you are an ISTQB™ – Certified Tester, it means you’ve added long-term value to your team and to your company with proven, credible testing standards and processes.
I think CTFL means “Certified Total Fucking Loser”
It is a sad fact that there are people out there who follow the thinking that if you can’t do teach and then try to state that those of us who are trying to change and improve our industry that we do not understand the ideas like ET. What these people are out for is personal gain, not for the industry as a whole. I remember when I worked at Apple Computer with you that it was an educational journey for everyone as we learned how to become better testers as well as improving the industry.
Keep up the good work we who are are in the trenches salute you.
Hi James B,
I agree with you to some extent that YES, what he puts in the book in terms of content was not worth publishing except that of tours technique. I am continuously following you since you had your videos out on Youtube about your lectures. I am slightly confused with the terminologies you used in your book “Lessons learned”. You drifted the topic of Software testing more into Scientific way rather than keeping it to IT science etc.
[James’ Reply: I don’t see that as drifting. I see that as keeping it centered on the essence of the matter. When people think that the skills and practices of testing are something different from that of science in general, I believe THAT is drifting.]
I guess James W had done a tremendous job in correlating the ides of ET with IT.
[James’ Reply: Since he doesn’t understand what ET really is, and has made no deep study of it at all, and also has never held a job as a working tester or test manager, I don’t know what you could possibly be talking about.]
Being a reader for me it’s important that i get most of the stuff what you are writing rather than searching for those philosophical words used by you. My opinion is you both are doing tremendous job and it’s good learning for us. I am looking forward for the day when we would have one book where your ideas are simplified by james W.
[James’ Reply: The philosophical words you are referring to are professional words that any skilled practitioner ought to know. Don’t look for a dumbed down version… raise yourself up.]