Michael Bolton is accused of hand-waving in this thread on LinkedIn. (See the comment by Peter).
Michael and I talk a lot about cognition and exploration. We speak in tropes that come from philosophy and various branches of science. Once in a while, some fellow who understands little of what we say assumes that we just made it all up to impress the ladies.
It brings to mind the Large Hadron Collider. Here’s an excerpt from one of their bulletins:
“The very smooth and fast transition to operation with ions was made possible by very good beam instrumentation performance with a relatively low number of charges per bunch, and magnetic behaviour very similar to operation with protons, as expected. These two factors combined allowed the setting-up operations to be completed very quickly, and stable beam operation, with 2 bunches per beam, was achieved in just a few days.”
Gee, if you know nothing about physics, or the LHC project, this might sound very much like hand-waving, too! In this case, however, if it sounds sketchy, it’s probably more about the receiver than the source. After all, there really is a $10 billion device sitting in the ground with 4000 physicists poring over the data it generates.
As ambitious professionals, we need to be able to speak about complex subjects without constantly going back to kindergarten to bring along the people who refuse to study their own craft.
I don’t mind if an earnest seeker, who happens to be ignorant, asks what seems to be a silly question. I will help everyone who wants to learn. And I don’t mind the assertive dissenter who has done the homework and yet has a different style and judgment from mine.
I’m talking about something different: the willfully ignorant blowhard.
Please don’t be like that.
Peter’s comments have a judgemental tone, which isn’t helping his argument. I get no sense of objectivity, either.
“Can we ship?” seems too rigid. Context is key.
The “great” minds of testing argue semantics while the rest of us actually test.
Check vs Test vs Yes/No vs Maybe/Maybe Not … too funny girls, too funny!
[James’ Reply: I test, too. Except…
I video my testing. I’m accountable for my testing. I can explain my testing. I’m famous for my testing. I’ve testified under oath about my testing. I write about my testing… All Christmas weekend I was up writing code for a semi-automated oracle to analyze test data from medical thermocouples. What were you doing, besides leaving inane comments on blog postings written by people who care about the craft?]
Hmm… Yet another commenter who is willing to deliver abuse, but only while hiding behind initials. The rest of us use our real names, spelled out in full. And we provide a place where people can comment. And we respond to critique.
And yes, I actually test as well. It’s tiresome having to retype it, so I’ll just post a link: http://www.developsense.com/blog/2011/12/scripts-or-no-scripts/comment-page-1/#comment-9819.
—Michael B.
As a newly minted tester cutting my teeth in my first full-time testing gig, I have been amazed at the tenor of the dialog around the testing profession. It guess I expected that people should be able to share dissenting opinions and discuss the merits of ideas without resorting to belittling others (in this case jmm’s comment, although this is just one example of many that I’ve run into as I’ve begun to engage with the community).
I look forward to the time when I no longer feel like an ignorant, but earnest seeker and can engage the testing community with some ideas and experience of mine own. Thanks for the great resources (blog posts, people and article recommendations) that both you and Michael (and others) share.
[James’ Reply: There’s an important difference between dissent on ideas and social rejection that serves to define and police community boundaries.
Dissent, in the sense you seem to mean, is what happens when people share a culture with a strong sense of what it means to have a useful debate. They follow that protocol, whatever it is, while accepting the premise of shared membership in the social group.
But when dissent occurs between people who do not share enough of a culture, it becomes more of a social immune response. Each side rejecting the presence of the other in the community. Community and immunity are linked, that way. That discussion should not be interpreted primarily as a clash of ideas, but rather as mutual cries of “I deserve to exist.” I believe that is the real motivation for JMM comment. If he didn’t feel that way, he would not have commented at all. Meanwhile, my reply to him is essentially a counter cry of “I deserve to exist!” wherein I refused to cede to him the rhetorical badge of “person who gets things done instead of bullshitting.”
I treat people differently if I feel they belong, or seek to belong, in my community, than if they seem to be assaulting the very basis of it. In the case of JMM, he seems to have an allergic reaction to my style, and this, to him, is more interesting and relevant than whatever substance I am trying to communicate. Therefore, he says what he says, and therefore I blow him off. Moreover, I do it publicly, because I want him to be an example to others who might be so outside of my circle that it’s pointless for them to comment.
I also look to my colleagues to help me make good judgments about when I’ve crossed important lines of behavior, because sometimes I get angry and lose perspective.]
There are some really interesting semantic and epistemological things going on in that LinkedIn thread.
There are some people preferring a discussion about definitions over one about meaning:
“Why isn’t it testing? Because you remain unsure if any of the behavior of the system is valid.”
“[…] a test is for the production of an expected value, hence why there is a field for “expected results” in the testing software product or the paper based template one is using.”
So instead of discussing if the distinction between checking and testing is meaningful, some people reply that they define testing differently. So they miss out on the good part of the discussion.
Another great example is this one:
“You claim that testing is not _always_ done to answer a question or to provide yes-or-no answers. I claim that if that were true then testing of that nature will always be insufficient – as the testing of PDFThing so evidently was. In other words Michael, testing is _always_ done to answer some question and testing which cannot be used to answer a yes-or-no question is ersaz [sic].”
There are two definitions of testing in that quote. Testing as an activity in a project (don’t want to use “phase”) is done primarily to answer the question: “Can we ship?” Testing as an activity as such, i.e. the activity of investigating a system, can be a means to provide answers AND a means to provide questions.
In my opinion, recognizing a difference in using certain words and being able to go along with someone else’s definitions to facilitate meaningful discussion, is an important testing skill. The difference may lead you to a defect: what if e.g. different developers assign different meanings to the same word? And being able to temporarily accept someone else’s definitions (suspension of definition? – as in suspension of disbelief) will allow you to understand their perspective, helping you build different models to inform your testing.
There are also some people in dire need of a class in epistemology.
“If the person devising the test does not know the expected result, it is not possible to devise a realistic test, and the test analyst person is obliged to revert to the relevant SME to obtain additional info in order to be able to complete the preparation of the test.”
“The first rule of testing is that one tests against a spec. Without a spec there is nothing to test.”
“I am not a big supporter of the overly philosophical approach to the role of testing, as it is a simple set of black and white principles for me, […]
They are assuming that:
– perfect knowledge of what the system is supposed to do is possible;
– perfect usage of that knowledge to write test cases is possible;
– perfect knowledge of the state of the system under test is possible;
– perfect comparison between observed result and expected result is possible.
I don’t think we’ll ever get close to even just one of those and there are plenty of philosophers to back me up on that one. Immanuel Kant and Karl Popper come to mind. In my opinion, such epistemological hubris (Why stop at one when borrowing words from the Greek? :-D) is dangerous for a tester. It guarantees you’ll miss out on valuable information.
But it gets even better:
“I don’t see why we should know everything about an application before we start testing, any more than we can define a set of requirements completely. However we can (as you have) define some things which are (hopefully) always true and look for counter-examples.”
“My testing definition wasn’t meant to imply we know everything before we start, and exploratory testing is useful to fill that gap. But not as useful as doing upfront analysis and prep. The earlier you find a bug, the cheaper it is to fix, so finding bugs after the code has been cut should never be the first resort.”
Both quotes seem to agree on the fact that it’s impossible to know everything, but neither seem to attach much consequences to it. As if they believe (my speculation here) that perfect knowledge is theoretically possible, but just not (yet) in practice. The problem with that position is that it allows you to think that it’s possible to slowly but surely approach perfect knowledge, if only you try hard enough. So your testing consists of two parts: the pretty part in which you get all the information in advance, write all the checks in advance, then you test and the ugly part in which you have to write some additional checks because the documentation was incomplete. Improving testing then consists of reducing the size of the second part as much as possible.
However, if you accept that perfect knowledge is also theoretically impossible, you’ll realize that it’s futile to try reduce the ugly part of testing to zero, so you’d better get very good at that ugly part. And the good news about that is that in my experience, the ‘ugly’ part is also the more interesting, more fun and more rewarding part of testing. And if we manitain the distinction between checking and testing, perhaps it’s also the only part of testing. 😉
[James’ Reply: After reading this, the thing I would like to have more knowledge about is you. I’m not surprised more people don’t think and analyze in the way you’ve demonstrated. What surprises me is that so few people even aspire to.
I put it to myself, years ago: what do I have to do, say, think, be, in order to find the best bugs as fast as possible and otherwise serve my clients excellently as a tester? It was quickly apparent that narrowly pre-defined expected results could not get me to that.
I’d love to see how you perform on some of my testing exercises. Contact me on Skype instant messaging! My handle is “satisfice”.]
Joep was a participant in that legendary RST class in London, November 2010 that seemed to spur a number of fabulous bloggers. I’d like to see more from him too!
I find the exchange interesting, but I’m growing disgruntled with the wild leaps in logic Peter, Doug, and Nick are making in that thread. It appears they have a lot of misconceptions about what I mean when I type a variety of things. It would be nice if they’d ask questions like a good tester would, but I guess they’re too busy trying to delay the release of my next comment.
[James’ Reply: It’s not easy being a good tester. It’s not just that you have to understand logic and reason, and study it. The bigger problem is you have to get control of your emotions enough to WANT to do those things.
What I see are people who aren’t necessarily unintelligent. It’s more like they are reluctant to push the “be intelligent” button in their brains for fear that the brain door will swing open and they will be crushed under all the clutter that tumbles out. All of us fear the imponderable to some degree. A good tester is like a lion tamer– you can be in the cage with complexity and subtlety because you know how to stay clear of its jaws. Where the uninitiated see only pointless risk, the good tester can perform seeming wonders.]
Good Evening,
I’m still learning in this grand arena of testing and I do have to say that if testing was just a case of ‘Can we ship’ or ‘Just doing checks’ then testing as a profession would be a very boring one. When I test, I’m always looking to learn something new about the product that I’m testing.
[James’ Reply: Boring and also it wouldn’t serve its purpose.]
To me, just doing checks requires no major brain power, it would be no difference from ticking off the list at a supermarket, however proper testing (In my view) requires you to think about so many interconnecting things that I’m usually worried that I’ve forgotten to incorporate something in my testing and not find the defect because of it.
[James’ Reply: The definition of checking is “any information gathering (and verification) activity that could, in principle, be performed by a machine.” There’s no reason a check can’t require a lot of intelligence or be otherwise very difficult to perform. Checking is not a bad thing. It’s part of testing. The point Michael and I try to make by talking about “checking vs. testing” is that no mechanizable process exhausts the vital work of testing. Testing is more than checking, no matter how interesting or useful checking may be.]
Maybe that’s why I like the idea of context driving testing as it makes the tester think about what they are doing rather than blindly following the specifications (Don’t get me wrong, the basis of the specification is a good starting point for a tester). However, just because you don’t have a spec doesn’t mean that you can’t test. If you were in a forest, would you need a map to get from one side to another. No, you may take longer or on the flip side, you might find a quicker route.
[James’ Reply: Sounds good (although the appropriate term is “context-driven.”]
On Peter’s point about testing being part of ‘Can we ship’ – Testing doesn’t do that as there are companies that ship products anyway without tester input.
My opinion on what testing is that we provide information about the state of the product under that particular moment in time.
[James’ Reply: Exactly.]
In all the talking, I find it interesting that the customer’s viewpoint has not been discussed. I personally feel that we are the champions of the customer. We have the unique ability to see it from both sides of the coin (The business and the customer side of things).
[James’ Reply: ANYONE on the project can champion the customer. This is not the exclusive province of the tester. That’s why it’s not something that I tend to emphasize.]
I like a good discussion because good or bad points, it makes me think about what I do and that’s always a good thing.
Have a happy new year.
The discussion was so riddled with misconceptions and vile insults I’m wondering whether to stay in the group at all quite frankly.
The contributors come across as being more interested in belittling anyone that disagree’s with them, rather than sharing information and debating. Which I suppose explains the blinkered approach to testing they refer to in the thread.
I’d love to know what their contexts have been that have caused them to have these opinions and which cause them to share them in that way.
[James’ Reply: That’s why I rarely post on unmoderated forums. But I guess it serves to remind us that testing is hard. We must study and be better and give ourselves some credit for becoming wiser instead of parading willful ignorance.]
Its not rocket science to determine the cause: ENVY, because someone has a bigger popularity. (Some of you might live in a too politically correct environment and might miss this)
ENVY manifests through many (mostly indirect) actions:
– finding something to use as an attack – “Can we ship?” is quite a limited metric and I don’t think its the decision of the tester. No tester is actually deciding on release and if he does that, conflicts might occur.
– jokes (no one tells you anything, but after they find some things for which envy its triggered for them, this is one way of manifesting)
– personal attacks related to personal life, family etc:” I can’t present to conference because I have to take care of my ten kids” (but no one asked you to do that)
– many other forms
Besides envy, we are all subjected to something inter-personal. If someone pays and organizes a conference for me, and in addition is criticizing someone else constantly in front of me, I will tent to import that criticism (adapt it, make it look legit) and also attack my friend’s victim.
One main issue with context driven community is that it creates a little (apparent) chaos, lack or order. Companies might find hard to promote people on organizations.
But I think its still a great community, great principles. We don’t need to stay all the time together, we need to discover, learn and enjoy.
I’m a great fan of exploratory testing, but I worry that you and Michael in your enthusiasm to tease out key attributes seem to belittle the ‘checking’ aspect of it. And I suspect it’s that, rather than any semantic or epistemological deficit, which is provoking the apparent displays of willful ignorance.
[James’ Reply: We are using the distinction to attack a fundamental and widespread problem in our craft: the idolization of checks. It is that *attitude* we are belittling. Also, checking is “little” compared to testing, since testing subsumes checking.]
‘Checking’ is factory – ISTQB, mechanical and boring, ‘testing’ is exploratory – buccaneering, human and exciting.
[James’ Reply: When people gush about mechanical processes as if they will save us and as if that is the all we need to do, or a majority of what we need to do, or even a substantial minority of what we need to do, then I feel the need to smack them upside their epistemological heads, yes.]
And testing in the real
world (as in confirming something works) needs both. Yes, you’ve got to have some ‘testing’ before you can do any ‘checking’ but that
doesn’t make ‘testing’ in some way more important or necessary or better than ‘checking’: regardless of how well the ‘testing’ is done,
[James’ Reply: On the contrary, it absolutely does. Testing already includes checking, so there’s already no danger of “not doing checking” when you testing. Checking does NOT include testing, however. Focus on testing, and checking takes care of itself. Focus on checking, and you completely miss the point.]
if
there’s no ‘checking’ the supplier should no more expect to please their customer than a taxi driver who drives with peerless skill on the
most interesting roads, eschewing all the while the tedium of plotting the best route to the ride’s destination and the consequent
(whisper it low) stupidity of an arrival.
My view is that the pirating of the word tester and implying some of us are ‘testers’ and others ‘checkers’ both irritates those who feel
branded non-sapient by the ‘buccaneers’ and makes the essential work of completing testing (checking results) appear unattractive and
apparently unimportant.
[James’ Reply: We’re not pirating the word testing, we’ve *rescued* it. We’ve gotten it back from the well-meaning clods who lost the thread back in 1972 (the Chapel Hill Conference) and kicked off the Factory School of testing.
If you check without testing, you are doing disservice to your client and the public and I am ashamed of you and you should be ashamed of yourself and your children should also be ashamed of you. So shape up or else be overrun by the sapient testers who find more bugs and better bugs than you will.]
So as an approach it’s more likely to engage the ‘factory’ testing community’s flight or fight response than win their hearts and minds, and is certainly not as useful or persuasive as the ‘What exploratory testing is not’ blog. What, btw, do you think caused that volcano to erupt?
[James’ Reply: If you want to be nice to the marching zombies, that’s your thing. I don’t mind. But it’s not my thing.]
‘Testing already includes checking…’? That’s a great deal to make one word mean.
[James’ Reply: Well, we have words like “piloting” which subsumes “steering.” You can be fully qualified to steer an airplane but not be qualified to pilot one, yet all pilots are qualified to steer. Throughout the world of skilled work, in arts, sciences, engineering, you will find people qualified to do something, and other people (called other things) qualified to do that plus more.
Michael, Cem, and I, in our studies of the things one MUST do and be to consistently find great bugs in complex products, have come to see testing as applied epistemology. Those who pooh-pooh this, however, offer no alternative that works. They promote ritualistic behavior and explain testing effectiveness in terms of kabbala-style metrics. They can’t explain testing skill or talent except with recourse to mystical terminology.
Yes, there is a lot to great testing, and my job is to find out what it is and how to do it, wherever that takes me.]
As this is all about mindlessness, laziness and wilful ignorance, I’d tidy up the ET semantics by using the word ‘drone’, noun and verb, for the ‘check without testing’ metonymy. Would make it easier to find ‘be intelligent’ buttons and harder to misunderstand, wilfully or otherwise. More isomorphic too, often handy later.
Pip Pip!
[James’ Reply: Drone may work. Robot works, too, and fits with the original meaning of the word robot.]
I’m seriously wondering if Nick has read this: http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/
—Michael B.
I was torn… I was learning a lot during the read (not about testing so much, but about people), but once it turned to insults I was out of there.
I commend MB for responding the way he did throughout, and also you JB for the same.
[James’ Reply: Michael has advised that I not keep reading the thread, because he knows I would just get angry. Michael has much more patience with people like that than I do. I love that about Michael.]
Good advice from MB mate. It is just not worth the energy. Besides, with you focusing on other things, we learn more!
Yep, that’s my selfish side coming out. ;0)
I am vérificateur en France, and very much admire your approach. In disputation of ignorance I
am expecting to see little of testing but much of people and am not disappointed! I find in your
words much of significance in the way you and MB deconstruct the semiotics of the ignorant –
redolent of the intellectual tradition francaise – Debord (“Boredom is always counter-revolutionary.
Always”), and Derrida (“Certain readers resented me when they could no longer recognise their territory,
their institution”) – the two are famed for textual and societal analysis and the difficulties they
surpassed to express their ideas.
When they talked less formally of their goals epistomological, they spoke of the need to ‘remove the
urine’ (pardon my poor translation), constantly the refining and distillation, always to approach the ‘pure
ontological essence’, and never to cower from the engagement of critics.
So, selfishly with David I also thank you for your ‘blog’ to take and continuing this tradition, please do not allow anger
with fools to stop you share your ideas with the rest of the world.
A bientôt – Pepé
[James’ Reply: Thank you. I love that the French take philosophy seriously. I haven’t read much of Derrida (I have Limited Inc.) but I like how, for him, sacred cows must be slaughtered.]
I’m a little late in digging into this discussion. I went back to LinkedIn to review the discussion topic and comments only to find the discussion thread has been removed. So, I will do my best to comment from what I remember.
I would love to drive the entire community to view testing as service based industry with our service being information. If we are doing our jobs well, the information we provide is of value to our customers. If this approach were adopted, the “Can we ship?” power, that many seem to think they have, would melt away. It would be replaced with more collaborative informed decision making with those running the business empowered to understand the features and limitations of the product and how it will impact their customer base. The business retains the right to decide to release and they should not abdicate that authority to anyone else. Abdication occurs because the business lacks the confidence to make the decision, which translates directly to a lack of useful information. Yes, there are other reasons to abdicate authority, the most obvious simply being a desire to shuck accountability. But, I believe with any type of power, people want to seize and use that power so long as it does not “destroy’ them (there are exceptions here too). Useful valid information is the best defense when wielding this type of power against the risk of personal injury. Share the information, share the power.
By the way, I think if people applied the information as a service approach with an understanding of their customer when it comes to defect reports, the average report usefulness would improve.
[James’ Reply: Well said.]
I came here to read about ” What Exploratory Testing is Not” but certainly i find interesting to see people are interested and keen to know about the topic and there are something else …
I find few points which are relevant for my growth and i will keep that in mind if i am mentoring someone as well.
One thing i will try to add, there is lot missing here in comment as well as in description from the Professional perspective. As a group you guys are doing good job but it can be better if you can give/suggest something which can be helpful to dig or give a direction for person/organization in Testing areas.
Great Job Guys
SK