I told a rival that I don’t use the term “manual testing” and that I prefer the term “sapient testing” because it’s more to the point. This is evident in the first definition of the word “manual” in the Oxford English Dictionary: ” 1. a. Of work, an action, a skill, etc.: of or relating to the hand or hands; done or performed with the hands; involving physical rather than mental exertion. Freq. in manual labour.“ Sapient, on the other hand, means “wise.”
He laughed and said “Bach, you are always making words up.” And then told me that in his opinion manual testing did not evoke the concept of unskilled manual labor. Now, other than establishing that the guy doesn’t have an online account to the O.E.D. (definition of “sweeeeet!” is “an online O.E.D. account.”), or perhaps doesn’t consider dictionaries to be useful sources of information about what words mean, I see something else in his reaction: I blew his mind. What I said doesn’t intersect with anything in his education.
To understand me, the man will have to use his sapience, rather than responding manually (i.e. with his hands).
In other words, I notice that some of my rivals in the testing industry don’t merely disagree with me, they apparently don’t comprehend what I’m saying. Example: after some ten hours of solid debate with me, over several sittings, Stuart Reid (who is working on a software testing standard of all preposterous things), told a colleague of mine that he believed I don’t truly mean the things I said in those debates, but merely said them to “be provocative.” Huh. That’s some serious cognitive dissonance you got going, Stu, when the only way you can process what I’m saying is to declare, essentially, that it was all just a dream.
Of course, I don’t think this is an intelligence problem. I think this is a lack-of-effort-to-use-intelligence problem. It’s not convenient for certain consultants to tear up their out-of-date books and classes and respond to the challenge of, um, the last 30 years of development of the craft. So they continue to teach and preach ideas from the seventies (or create testing standards based on them, because they believe not enough people appreciate testing disco).
Anyway, in the Context-Driven community’s latest attempt to explain the ins and outs and vital subtleties of testing, Michael Bolton has come up with a promising tack. Maybe this will help. He’s drawing a distinction between testing and checking.
Brace yourselves for insight. A lot of what people call testing is actually mere checking. But even checking requires testing intelligence to design and do well. This gives more specifics to my concept of sapient testing. Here are Michael’s seminal posts on the subject:
- http://www.developsense.com/2009/08/testing-vs-checking.html
- http://www.developsense.com/2009/09/transpection-and-three-elements-of.html
- http://www.developsense.com/2009/09/pass-vs-fail-vs-is-there-problem-here.html
- http://www.developsense.com/2009/09/elements-of-testing-and-checking.html
When Michael first made the distinction between testing and checking, I was annoyed. Truly. It blew my mind in that bad way. I thought he was manufacturing a distinction that we didn’t need. I decided to ignore it. Then he called me and asked “So what do you think of my checking vs. testing article?” I had to say I didn’t like it at all. We argued…
…and he convinced me that it was a good idea. Thank you dialectical learning! Debate has saved me again!
I now agree that it’s a practical distinction that can be used as a lens to focus on the quality of a test process. I do have to get used to the words, though. I now see a difference between automated testing and automated checking, for instance: automated testing means testing supported by tools; automated checking means specific operations, observations, and verifications that are carried out entirely with tools. Automated testing may INCLUDE automated checking as one component, however automated checking does NOT include testing.
Making this distinction is exactly like distinguishing between a programmer and a compiler. We do not speak of a compiler “writing a program” in assembly language when it compiles C++ code. We do not think that we can fire the programmers because the compiler provides “automated programming.” The same thing goes for testing. Or… does that blow your mind?
Sadly, the distinction between testing and checking makes describing test-driven development somewhat awkward, because it’s a test when I write it and a check after I run it for the first time. Am I test-driving or check-driving? This only matters because we have to describe what we do in order for others to learn to do it. Still, I have to admit I like the pun of Check-Driven Development, even if it only works in “American English”.
[James’ Reply: It’s an awkwardness that’s good for us, I think. Let me work through the ideas…
Strictly speaking you are “doing testing” by “writing checks”, but not actually “writing tests.” If you run the checks unattended and accept the green bar as is, then that is not testing. It requires absolutely no testing skill to do that, just as you wouldn’t say someone is doing programming just because they invoke a compiler and verify that the compile was successful. If the bar is NOT green, the process of investigating is testing, as well as debugging, programming, etc.
If you watch the tests run or ponder the deeper meaning of the green bar, you are doing testing along with the checking.
Think of “test” as a verb rather than a noun, and it becomes clear that test-driven design is truly test-driven design, although the testing is rather simplistic, based primarily on those little checks. Once the design is done the automated checks become useful as change detectors against the risk of regression. They certainly aid the testing process, despite not being tests.
Checks definitely do NOT drive development. Development is never a rote and non-sapient process. It’s far better to say test-driven, because the design of the checks is a thoughtful process.
I wouldn’t insist that people change their ordinary language. I see no problem calling whales “fish” or spiders “insects” in everyday life. But sometimes it matters to be precise. We should be ABLE to make strict distinctions when trying to help ourselves, or others, master our craft.]
I genuinely laughed out loud when I read “testing disco”.
Aside from that, I like the distinction of checking vs testing and have been trying it out on people recently, with mixed responses. The ‘programmers vs compilers’ analogy seems particularly useful and I’ll be testing people’s response to that when approaching the subject in future.
Actually, I’d have expected you to really *like* the term manual testing, or at least manual checking. We have automated checking, manual checking and sapient testing. The first two use computers and hands but no brains.
[James’ Reply: I suppose “manual checking” fits.]
I appreciate this, James. One thing that gets me is the people hear exactly what we have to say, then respond with something that boils down to shrugging shoulders. Example:
“Matt: How can we say a practice is best? We can only say that it’s the best we know now, or the best we know of for our situation, or that the practice has worked for us in the past in a specific kind of envirionment, right?
Bob: Well, I suppose that’s true. But best practices are a well accepted term.
Matt: But they are a /misleading/ term Bob. They are a marketing term.
Bob: Yeah, well, I think some degree of hyperbole is expected in these things.
Matt: Does that make it okay?
Bob: (shrugs shoulders, walks off)”
The application to the term “test automation” is obvious corollary. I’m not sure what to do with that yet.
[James’ Reply: “Yeah, well, I think some degree of hyperbole is expected in these things.” This is about as close to saying “Look at me! I am stupid!” that someone can come without actually saying it outright. Sheesh. Of course some degree of hyperbole is expected if no one does anything to counteract it. Also some degree of stealing, rape, murder, all kinds of crimes are going to happen– but much more if we do nothing whatsoever to enforce the law.
I believe, in their hearts, the people who insist on using the term “best practices” know that they are dilettantes and fakers. I don’t expect them to admit it, but they must feel sick inside. Perhaps they cover that feeling with alcoholism.]
Yes, “manual checking” works for me too in exactly the way that “manual testing” doesn’t. Nice move, Norval.
—Michael B.
I really like the C++ programmer vs the compiler. So without forcing you into a box do you agree that the production of quality software needs both checking and testing? To what degree etc…. depends upon more variables worth listing.
To further the thinking great testers would use computers and other pieces of software to do the checking for them, so they could spend time, energy and intelligence on testing.
[James’ Reply: No I would not agree to a context-unspecified statement like that. I hope you don’t either. Think it through, man.]
Sorry James, this time I think that Reid is right.
I use each day exploratory testing when testing, but also I remember to ‘check’, if necessary, depending on the context.
Please ‘check’ how many times you have used the word ‘check’ in your articles about exploratory testing.
An example: in your ‘Heuristic test strategy model’, the use of checklists is clearly mentioned (and for wery good reasons). Now, for me, the basic idea about using a checklist is linked to checking (sapiently). But, still – checking!
See also Michael’s definition: ‘Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation.’
So, James, I think your reply to Reid must be ‘placed-in-a-specific-context’ to become true. You must accept that testing includes checking. What, how, when, that depends…on the context. But, definitely, yes, every tester does some checking. Remember API testing. How could I not check first if a function works as specified?
Ladislau Szilagyi
[James’ Reply: Perhaps you’ve misunderstood the concept of checking, as proposed by Michael. It is a specific usage. My use of the word check has not been, up to this point, confined to that usage, and may not always be. However, there are times when strict speaking is important.
What you cite as Michael’s definition is not a definition, but rather a description. A check *is* confirmatory, but that’s not its defining feature. Not all confirmatory actions are checks.
Testing does not necessarily include checking, because checking is, by definition, a non-sapient process. I can well imagine testing done without recourse to non-sapient actions.
In your API example, it’s trivial to see how to test it without checks: write a program to invoke the API and personally review the output. That is a sapient process. It becomes checking only if we link the observations to decision rules such that they are carried out non-sapiently.
I agree that we may use checks in our testing, but we are not required to in order to do testing.]
This discussion of sapient testing is expanding my mind as well. I’ve been doing a great deal of “manual” testing over the last several years, and always find myself wanting to transcend this. OK, it’s a bit stronger – I rebel against it! It goes something like this:
– Attend scrum meetings that treat the test scripts like factory widgets to be completed
– Try to complete them on time
– Always encounter complexities far beyond what was planned
– Play catch-up
The problem is with “trying to complete the scripts, in sequence, on time”. It is impossible.
What is possible is:
– Examine the test scripts, freqeuntly printing them out hardcopy to assist intuitive optimization of the testing
– Think about strategies to optimize the sequence of steps
– Seek help and give help to the team to understand gaps in specification of the system under test
– Seek help and give help to the team to understand gaps in specification of the tests
– Utilize available knowledge-bases for understanding
– Fill in the gaps
– Run as many of the tests as possible
– Utilize little automation-tricks where possible within timeframe given
– Complete as much of the scripts as possible. Repeat above steps within available timeframe to arrive at best possible testing (“satisfice”).
This still takes too long, but takes far less time than trying to run the script in sequence. It is also much more a challenging thinking exercise than the dull, mind numbing clerical work of running a script “manually”. At its best, it can be downright mind-blowing.
Matthew, James: James says it says “Look at me! I am stupid!” — well, maybe.
I also get a whiff of “You can’t fight City Hall” and “Everybody does it”. So my take is:
– {stupid}, and-or
– {beaten down}, and-or
– {that’s-the-way-the-game-is-played complacency / “reasonableness”}.
[James’ Reply: But Michael, there IS no authority. There IS no city hall. So that one seems odd to me. Beaten down? Maybe, but that implies that they were at one time fighting, and I doubt that. Reasonableness? But what is reasonable about gratuitous hyperbole? That’s why I go back to “stupid,” which is the word I use when referring to someone who has intelligence and chooses not to use it.]
That last one, so labeled, actually being /worse than stupid/: in my view it’s likely more like “Everyone knows it’s a thankless and hopeless task, everyone with any PHB* credibility [sic] is in the Best Practices buzzphrase circle jerk together, so why stick to any principle higher than ‘get away with what you can appear to be doing’?” Stupid and functionally CORRUPT, however papered-over.
Salient dramatic anecdote: At Three Mile Island, one of the factors in the near-meltdown was that there was a defect-lockout tag posted on a bad panel element because they were following Best Practices. Unfortunately Best Practices as-read-or-remembered apparently didn’t say anything like “make sure all lockout tags don’t obscure any other critical indicators” (that is, ANY indicators). Whoopsie daisy.
MMB
* Pointy-Haired Boss, after the “Dilbert” cartoon.
James, I don’t quite understand what you don’t understand. The “City Hall” and “reasonableness” I was talking about in my breakdown is the same one, more or less, that I was hypothesizing in my paragraph that begins “That last one, so labeled, actually being /worse than stupid’…” which you left intact and uncommented upon.
[James’ Reply: I didn’t have anything to say about that, that’s why I didn’t comment.]
I gather that you used “stupid” to mean what I would call “willfully stupid”, a distinction that is not instantly apparent to me when I only see the root word. When I see the root word, unmodified, I think it means “dull-witted”.
[James’ Reply: It works either way.]
If you don’t see any point in trying to describe the matter in terms of group dynamics, pleasing, not-rocking-the-boat or making-noises-some-MBA-will like, preferring to label it all “stupid”, you’re free to do so. But at some point “worse than stupid” means something, well, /worse than stupid/ to me. Venal. I can’t tell if you think corrupt is worse than dull-witted. I think it is.
[James’ Reply: It might be. I don’t have an opinion on that.]
Maybe you think stupid (by your definition) is always venal or corrupt.
Very well, but I think the bare word is broader than that.
Re: “beaten down”: perhaps your context under consideration is too small. You’re a proponent of unschooling, I believe. Maybe you don’t think some folks get their independence systematically subverted by things that instill learned helplessness, until “intellectual honesty” is an empty phrase. I suspect such people are more common than not. “Is” is not “should” 🙂
[James’ Reply: You could be right.]
[James’ Reply: “Yeah, well, I think some degree of hyperbole is expected in these things.” This is about as close to saying “Look at me! I am stupid!” that someone can come without actually saying it outright. Sheesh. Of course some degree of hyperbole is expected if no one does anything to counteract it. Also some degree of stealing, rape, murder, all kinds of crimes are going to happen– but much more if we do nothing whatsoever to enforce the law.]
I think that’s a a degree hyperbole that needs “checked”. (Where “checked” means counteracted, and being a process that requires sapience, thus my argument ( http://bit.ly/4faNaX ) about using such a vague word for precision.
[James’ Reply: You got close in your post, but I don’t think you quite understand the distinction we’re trying to make. We’re not trying to distinguish between exploratory testing and non-exploratory testing. We’re distinguishing between what we call testing (whether exploratory or not) , and checking, which we claim is not testing at all. Checking can be part of testing, but is not necessary for testing.
It’s amazing that the testing field, as a whole, has evolved so little in the last thirty years. My colleagues and I are trying to solve that problem, and maybe one big part of the problem is the sloppy use of language. Maybe becoming more precise about certain distinctions will help.]
I think the same sort of pitfall came with using “agile” to describe a specific process — though the larger problem there was attempting to use the word as a brand that meant “whatever process is espoused by certain specific individuals” and I think the fear here is that people are concerned about the marketing-ization of the terms instead of the enumeration of categories.
[James’ Reply: That’s a reasonable fear. I try hard to avoid that. It’s my goal to avoid that. So if you see that in my work, please call me on it.]
The epistemics of whether checking requires even a whiff of sapience appears to be a delicate one, somewhat along the lines of the old “tree falls in a forest” saw.
Pitfalls in thinking about this sort of thing are compounded by the language (and thus, probably, by the Lakoffian underlying multirole circuits that he calls “metaphors”). E.g., a Demingesque meaning for “error”, e.g. {how far the micrometer reading on this dimension of this part is from the specified value} shares the same English word as one used to denote a {failure of sensing + thinking}. But we go to test with the English we have… 🙂
Metaphorically, I have the intuition that a dry film — a residue, if you will — of sapience might be present at and after the creation of checks, but fluid sapience is not present even when checks are being operated.
[James’ Reply: I’m not sure what a dry film of sapience would look like, or how I would detect it. We call an action non-sapient if it does not *require* a skilled human in the loop. Any action that does *require* a skilled human we call sapient. I use the word “skilled” just to avoid the trivial case of a human required who is merely there to simulate a machine using muscle power. Thus, a man pushing a mechanical lawn mower because he doesn’t have a motorized lawn mower available isn’t doing sapient lawn mowing simply by virtue of putting his back into it.]
I can appreciate Matthew’s “(shrugs shoulders, walks off)” discussion – I think it captures the essence of many real discussions that have occurred. But I think it’s dead wrong to brand the shruggers as fakers and assume they know you’re right. In these discussions, the anti-best practices crusader has failed to influence their target.
James, I’ve seen you fail to affect change on several occasions like this. In some of those cases, I think the failure was avoidable.
[James’ Reply: I don’t know what you’re referring to, Danny. My goal is to tell the truth, be good at my job, and be of service to people who also care about doing good work. That’s all. That’s what I do. If people want to listen, I’m glad. If they don’t, I’m sad. But I’ll keep doing going, either way.]
Danny,
Can you be more explicit about your claim or provide examples? I am constantly looking for ways to be more persuasive, but I suspect that each of us has an audience or ‘tribe’ that tunes in to the core of who we are – the part that speaks louder than words.
[James’ Reply: I suspect Danny is just talking about my habit of ridiculing people I think are harming the craft. A lot of people think I would be more effective if I were more polite. But I think the world has enough polite people already, and not enough philosophers.]
I think this blew up my mind. Now I keep asking myself, am I testing or checking?
Today they asked me to ‘test’ a small modification on the way a feature works in the application. The programmer said ‘I’ve already verified everything but I’m sure that when you put your hands on it you’ll find a lot of problems’. And he was write, in fact I explored new ways of using that feature and found some fails, caused by the changes made or exposed by it. Of course I also checked the obvious use cases.
I guess the programmer had checked and the tester (I) had mainly tested.
I started as a programmer twenty three years ago. Since then I had several kind of positions in IT business. I’m new on Testing, started about two years. I’m enjoying it more and more each day and I’m learning a lot with it.
[James’ Reply: Yes, exactly! You got it!]
Some discussions do my head in! I think of testing as many different ways of finding bugs. Some are highly planned, thought out well constructed exercises, some are based on having a poke arround or “suck and see” sometimes testing is intuitive and even instinctive and sometimes it is fun to jump up and down on the keyboard or hit it with a stick (2001 Space Odyssey). There are times when it just smells dodgy. If you have an open mind and apply some sapience along with experience, then what and how to test becomes an enjoyable exercise in puzzle solving. Irrespective of wether it is checking or testing, if it has no chance of finding a bug then it is probably a waste of time. I like these blogs as they challenge us to THINK and dont spoon feed easily digestible pap. (sorry that sounded like my diatribe against ISTQB) A question for James: Do you find that you inadvertantly or sub consciously not only start to test all sorts of things but that they break in anticipation of you finding a bug?
[James’ Reply: Yes. I live and breathe as a tester.]
Bach, you have been a breath of fresh air to the community and I attribute your work has help alot of us but sometimes you use your understanding of the english language and symantics to run around what people say just for the fun of it!
[James’ Reply: Please give me an example of me running around. I don’t understand what you mean.]