Michael Bolton, my brother Jon, and I have produced a new version of our Exploratory Testing Dynamics document. We unveiled it last week at STPCON.
This document describes the elements of exploratory testing as we currently understand them. This new version has not yet been reviewed by Cem Kaner or any of our colleagues who normally have opinions on this, but I don’t anticipate that it will be revised much when they do. It should be fairly stable, for now.
ET cannot be summed up fairly as “unstructured playing around” and if you look at this document you will see why. If nothing else, count the elements in it. Crikey.
The document consists of four lists:
1. Evolving Work Products
Exploratory testing is an approach that emphasizes evolution. We start with something and make it better and better. Often testers can be overly focused on finding bugs, but look at all the things that get better. For instance, you can run your automated regression tests a thousand times and the tests don’t learn and neither do the button pushing testers. But you learn a lot by sapiently retesting– you evolve. One of the products of ET is a better tester.
Some evolution also happens in highly scripted testing, but in ET it’s a primary goal. It’s the point.
2. Skills and Tactics
What does an exploratory tester do? Well, feast your eyes on the Skills and Tactics list. Each of those items are things that seem to us to be reasonably independent, teachable, and observable. Some of them are also skills that a highly scripted tester would need, but they work a little differently for explorers. For instance, the skills of reading and analyzing documents means doing that to prepare for or aid exploration of the product, rather than to prepare detailed instructions for scripted test execution.
3. ET Polarities
The Polarities list is unique to ET. An exploratory tester must manage his own mental process, and that process is about developing ideas and using those ideas to search the product for trouble. We’ve found over years of experience doing this that a tester loses steam if he spends too long in one train or type of thought. So, what can he do?
One answer is alternation. We alternate among complementary activities. Two activities are complementary if performing one of them rejuvenates the other. A great example is reading about a product and directly interacting with it.
4. Test Strategy
The Heuristic Test Strategy model is a set of guideword heuristics that we publish as part of the Rapid Software Testing class. In addition to the general ideas of exploration, evolution, and alternation, the guidewords represent the specific subject matter of testing– the conditions we need to succeed, the things we look at, the things we look for, and the specific test techniques we use to produce tests.
With these lists, I can systematically evaluate a tester and identify strengths and weaknesses. We in the Context-Driven School of testing need models like this, because we focus on skills, rather than techniques and tools. This document allows us to focus our efforts and develop specific exercises for tester training.
James
How about this polarity “Verbal vs Pictorial thinking/representation”. I personally experienced it. Thinking about a verbal representation appear to activate difffent part of brain vs thinking about drawing a picture. Consider about ways to covey an idea – through words or through picture?
We can even add “audio/video” means of representation.
Shrini
[James’ Reply: Great idea!]
#sigh#
Whatever happened to Testers find defects and developers fix them. Kaner, Falk and Nguyen wrote the book, it is still the book. All the rest of this stuff seems to be self-perpetuated to make someone a living or reinvent the bleeding obvious. I am also starting to hear that certification now equips people to be QA testers, QA Test Engineers, QA Test Managers… So now sticking QA on the front suddenly legitimises this nonsense? On the one hand James the enfant terrible bucaneering around screaming THINK and on the other hand exhortations that you don’t need to think if you have completed our 1.5 day course and got your bit of paper. READ THE BLOODY BOOK.
Permit me to say that James has matured nicely out of “enfant terrible” and is now working on “honorable gadfly” status, as far as I can tell. 🙂
Me, I’ve lost count of the times I’ve done more than just find defects. Contributing to documentation being a big thing that doesn’t ordinarily make it onto the official “testing” charter.
I agree that, in almost every context I can think of, the expression “Quality Assurance” is pretty much newspeak or utter bilge where software development is concerned. “Quality Assessment” might not be so bad.
As I might have said here before, one of the things that bugs me is the sometimes popular idea that the “QA” “department” should be one guy — who also runs the daily builds. That one still makes me shake my head.
Thanks Michael,
Not sure about Gadfly, but I will admit his Bach is worse than his bite lately. I think once you get to the stage of seeing the same old rubbish getting chucked over the fence you are bound by concience to try and make things better. After 25 years of testing I am a bit tired of strapping on the armour and chasing windmills. Now if we could find a way to make testing redundant by improving the quality of the deliverable up front… I could ride into the sunset as the last “Real” tester. …… But what would JB do? PS how do they get such excellent security check words at the bottom of the page? They are often very appropriate!
Just been catching up on my well worn copy of Testing Applications on the Web, Hung Q Nguyen. It was somewhat comforting to see ” In operational terms, exploratory testing is an interactive process of concurrent product exploration, test design and test execution.” – James Bach. Hung Q Nguyen along with Kaner and Falk are of course the Authors of “The Book” which makes the attribution to JB pretty close to nomination for Sainthood. So JB stop Bucaneering around and get the definitive work out before I am too old to read it. PS I have been working on my test capability assessment tool which I use to help me assess the capability of an organisation to support testing (usually prior to me setting up a testing centre) and it is inescapable that if an organisation has a low test capability score then any strategic approach to testing or any number of ISTQB certified clones will at best be a very expensive exercise in futility but at the same time as your capability to make the correct decisions is poor you will try all the cheap quick fix solutions. If your Capability score is very high, then chances are you already test really, really well in which case you don’t need guys like me. What I am hinting at is that irrespective of what we strategise in terms of testing, it is just as important to communicate that the organisational capability to do stuff that supports testing is FAR more important tnan the exercise of actually testing. Make my life easier and you won’t need to test!
When we are talking about Test Strategy. Let’s not forget the popular test type i.e repetitive test activity. This form of test activity gives scenarios where applications fails and gives much hypes for quality free software. Accordingly we can plan our Test Strategy. But always we should go for Ad-hoc cycle in a testing phase, this type you can give some leverage to the test engineer to think and more to play with the software.
Riya Gupta
– http://www.testingmantra.com
Hi James,
My name is Gil Zilberfeld, and with my colleague, Roy Osherove, we do a video cast called: This week in testing.
Roy talks a bit about the new article in this episode, so please come and watch. And if you like, talk about us so more people can enjoy.
Thanks,
Gil Zilberfeld
Typemock
[James’ Reply: I like your energy in the video. I like how you expressed some skepticism about “10%”. I want to see more critical questioning. 99% of us (far too high, since the maximum acceptable percentage is 0%) accept whatever metrics anyone wants to make up. Can I get an amen?]
As an early one-year anniversary gift to version 2.2, I offer additions to the ET polarities list:
subjective vs. objective
details vs. big picture
structured vs. free-form
serendipity vs. precision
[James’ Reply: Help me with this: What exactly is the difference between subjective and objective? I need specific examples. It’s such an abstract and misunderstood distinction.
I like details vs. big picture.
Structured vs. free-form is a bit difficult since there is no such thing as free-form– only unrecognized structure– unless you are speaking of completely random variables. Perhaps there is a different way to say this? Closed structure vs. open structure? Rigid structure vs. plastic structure?
Serendipity is not the opposite of precision. I guess it would be the opposite of planned?]
First, I see this list as inspiration; and I don’t think these words can be defined “objectively”, and they don’t have to be strict opposites.
However, I can tell how I see the words, and how they help me (sometimes by switching focus, and sometimes by having both in mind at the same time.)
Subjective is to look at the product with all of your own feelings. Do I like this? What would I like to happen?
Objective is to evaluate the product by external existing values/statements/measurements. E.g. expected result in a test case, requirements, design specifications, standards the product should adhere to, typical behavior for products in that environment.
(Note: Something truly objective, in a philosophical sense, doesn’t exist.)
(Side-note: To me, a good, professional tester is subjective; whereas professional, in Sweden, often means to “leave your feelings at home”.)
You’re right, Serendipity is not the opposite of precision (when re-thinking this, I guess it’s quite close to Focus/De-Focus.)
Structured vs. free-form is also difficult to interpret, and equally well could be expressed as Free vs. planned
We also have the “Opportunity” from Session-Based Notes (which I have changed to Serendipity, because it’s a nicer word.)
Summary: I don’t have a good polarities pair for this, at the moment.