Excellent testing requires skill, but heuristics give structure to that skill. Heuristics help us access our skills under pressure.
A heuristic is a fallible method of solving a problem or making a decision. Cem Kaner and I came to this definition based on an extensive search of papers and books across fifty years of psychology and engineering. Amazingly, we were not able to find a single coherent definition of heuristic in our research (The dictionaries are not much help, here) . Coleridge, Kant, and Polya have all written about heuristic reasoning. But we needed a definition that would bring the issues into focus.
There are two main issues: something that helps you solve a problem without being a guarantee. This immediately suggests the next issue: that heuristics must be applied sapiently (meaning with skill and care).
An excellent example of a heuristic is a hammer. Do you see how a hammer can help a carpenter solve a problem, but does not itself guarantee the solution? I like this example because it’s so easy to see that that a hammer may be critical to a skilled carpenter while being of little use to an unskilled lout who doesn’t know what to pound or how hard to pound it or when to stop pounding.
Heuristics do not replace skill. They don’t make skill unnecessary. But they can make skilled people more productive and reliable.
How Do You Tell if Consultants/Trainers Understand Heuristics?
I typically hear two reactions from rival testing consultants of other schools of thought (especially the Factory and Quality Control schools) that love “best practice” talk:
- “Oh yes, heuristics! That’s another name for ‘best practices’, right?”
- “Oh no, heuristics! That’s just a fancy name for ‘best practices’!”
Obviously both reactions miss the point. Even if these folks rename their ideas of what you should do to call them “heuristics,” they would be leaving out a key idea, which is that skill must rule methods. Talking about methods, focusing on methods, enshrining methods, is only sensible if humans are left in charge. The heuristic nature of engineering is the reason why a “best practice” is an absurdity. Seek not the perfect practice. Seek instead to practice your skills.
My friend Judah Mogilensky once commented that he thought my Heuristic Test Strategy Model was a lot like the Capability Maturity Model that he works with.
“Are people allowed to change or suspend the CMM whenever they see fit?” I asked.
“Oh no,” he replied.
“Then it isn’t being applied responsibly as a heuristic. It’s being treated like a law, or an oath, or something else that places itself above its subjects and binds them.”
So that’s how you tell. Look to see what’s driving things. People, or concepts? Fundamentally, “methodology” can’t control projects. Anytime you seem to see methods in charge, you are actually witnessing a project out of control, or else a project under covert control by shadowy masters.
When someone teaches you a way to solve a problem, check:
- Do they teach you how to tell if it’s working?
- Do they teach you how to tell if it’s going wrong?
- Do they teach you heuristics for stopping?
- Do they teach you heuristics for knowing when to apply it?
- Do they compare it to alternative heuristics?
- Do they show you why it works?
- Do they help you understand when it probably works best?
- Do they help you know how to re-design it, if needed?
- Do they let you own it?
- Do they ask you to practice it?
- Do they tell stories about how it has failed?
- Do they listen to you when you question or challenge it?
- Do they praise you for questioning and challenging it?
James, can you please include a link to your Heuristic Test Strategy Model in this entry? I just think it’s worthwhile for those who have not seen it to have easy access.
I have found it very useful in the past and have made my own version, in excel, to allow me to add comments against each applicable section and use it as a checklist for thoughts.
Do you have other ‘tools of the trade’ you use for other stages in the development lifecycle, for example, requirements review?
[James’ Reply: Most of my public stuff is right here on my website. The appendices to the RST class is linked on the front page of https://www.satisfice.com and the Heuristic Test Strategy Model is the first document.]
your questions sound so much like my idea + interpretation of a ‘leader’ 🙂
Loved this post. I’m going to put your list of questions to use in a work session tomorrow!
I’m involved in refining and expanding a testing process right now and I’m amazed at how many people think that a new/different process is going to make them a better or more successful tester. I think the process can help with things such as organization, communications, reporting, etc. But I’ve seen terrible testers get more terrible in a highly structured process and I’ve seen great testers do wonderful work with no process to speak of.
I agree that skill is skill and heuristics are heuristics and while they inform each other, they don’t have as much influence on each other as people think.
I began to comment on this post (and in particular on Meeta’s response to it), but it turned into a blog post of its own. You can read it here.
http://www.developsense.com/blog/2010/05/heuristics-and-leadership/
Unless the English language has changed in some strange way (again? it happens), a hammer is not a heuristic; hammering is. A hammer is an object. Hammering is a methodology.
[James’ Reply: When “heuristic” was coined in the early nineteenth century, it was the name of a exploratory process of reasoning. It was used in that way in Polya’s 1946 book How to Solve It, too. Sometime in the years that followed, the word became attached to specific patterns of solving problems.
I think this is the ordinary process of reification that goes on in English.
I agree that hammering is a heuristic. That’s what I mean when I say that a hammer is a heuristic. It’s akin to saying that a car is a form of transportation, even though it’s actually motoring that gets us around.]
After Sapient Testing, heuristics looks so real. It defines the ways that people often miss due to their constant focus on processes that they learn in schools. It matches what all should be actually in a leader and aims to achieve what I feel a leader should possess. Great Post!
To meeta’s Point – I think this post re-emphasizes the role of skill in development and application of heuristics in testing (solving testing problems while recognizing “fallibility” of the method”).
I am finding it hard to see the connection to leadership. I think the last set of questions on this post about “checking a method to solve a problem”, might have prompted Meeta to think those are questions that a leader would ask.
I would say they are generic questions that any problem solver or someone who evaluates (or asked to evaluate) a given method to solve a problem. These are not necessarily “leadership” related questions.
Shrini
James,
Another great post. Thanks for doin’ what you do and continuing to share your thought-provoking analysis with the testing community. This post really hit home with me personally and reminded me a lot of our dinner together in Boston at STPCon. I think you grilled me on most of those 13 questions. I thoroughly enjoyed the experience and learned from it.
In reaction to your post, I wrote a quick blog post expanding on your list of 13 questions above in a slightly different context, titled “13 great questions to ask software testing tool vendors.” http://hexawise.wordpress.com/2010/05/31/13-great-questions-to-ask-software-testing-tool-vendors/
– Justin Hunter
Shrini:
I think Vips’s comment about leaders might have been intended to make the point that pure /followers/ don’t make the best testers, and that keeping clear the distinction between algorithms (–“sure things”) and heuristics helps one to not be quite so much of a follower as one who doesn’t make that distinction.
By contrast, I’m sure we both know people who lump all the things they have decided are always good into a category they call “best practices” and then don’t think about them, just [try to] execute them.
Heuristics + Skilled tester ————–> Wisdom
Application
e.g. if a skilled carpenter hits hammer at right place to start the engine, shouldn’t we call that as wisdom?
or if a unskilled carpenter keeps hitting, in an attempt to start the engine.
Like a skilled tester using heuristics finds important bugs, called a testing expert 🙂
[James’ Reply: I don’t feel that’s quite right, because heuristics don’t make you any wiser than you would be without heuristics. Heuristics don’t encapsulate wisdom. What they do is improve your reliability, or reduce the cost and effort associated with solving problems.]
I agree with your point that it improve tester’s reliability or reduce the cost and effort associated with solving problems.
It means heuristics may suggest a wise behavior in that context but they will not make a tester wiser.
Loved this post particularly in an era that enshrines process over reason, common sense, of skill in the domain they are applied. Non-heuristics, what were they when grow up, guarantees?