[Authors’ note: Others have already made the point we make here: that exploratory testing ought to be called testing. In fact, Michael said that about tests in 2009, and James wrote a blog post in 2010 that seems to say that about testers. Aaron Hodder said it quite directly in 2011, and so did Paul Gerrard. While we have long understood and taught that all testing is exploratory (here’s an example of what James told one student, last year), we have not been ready to make the rhetorical leap away from pushing the term “exploratory testing.” Even now, we are not claiming you should NOT use the term, only that it’s time to begin assuming that testing means exploratory testing, instead of assuming that it means scripted testing that also has exploration in it to some degree.]
[Second author’s note: Some people start reading this with a narrow view of what we mean by the word “script.” We are not referring to text! By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This includes text instructions, but also any form of instructions, or even biases that are not instructions.]
By James Bach and Michael Bolton
In the beginning, there was testing. No one distinguished between exploratory and scripted testing. Jerry Weinberg’s 1961 chapter about testing in his book, Computer Programming Fundamentals, depicted testing as inherently exploratory and expressed caution about formalizing it. He wrote, “It is, of course, difficult to have the machine check how well the program matches the intent of the programmer without giving a great deal of information about that intent. If we had some simple way of presenting that kind of information to the machine for checking, we might just as well have the machine do the coding. Let us not forget that complex logical operations occur through a combination of simple instructions executed by the computer and not by the computer logically deducing or inferring what is desired.”
Jerry understood the division between human work and machine work. But, then the formalizers came and confused everyone. The formalizers—starting officially in 1972 with the publication of the first testing book, Program Test Methods—focused on the forms of testing, rather than its essences. By forms, we mean words, pictures, strings of bits, data files, tables, flowcharts and other explicit forms of modeling. These are things that we can see, read, point to, move from place to place, count, store, retrieve, etc. It is tempting to look at these artifacts and say “Lo! There be testing!” But testing is not in any artifact. Testing, at the intersection of human thought processes and activities, makes use of artifacts. Artifacts of testing without the humans are like state of the art medical clinics without doctors or nurses: at best nearly useless, at worst, a danger to the innocents who try to make use of them.
We don’t blame the innovators. At that time, they were dealing with shiny new conjectures. The sky was their oyster! But formalization and mechanization soon escaped the lab. Reckless talk about “test factories” and poorly designed IEEE standards followed. Soon all “respectable” talk about testing was script-oriented. Informal testing was equated to unprofessional testing. The role of thinking, feeling, communicating humans became displaced.
James joined the fray in 1987 and tried to make sense of all this. He discovered, just by watching testing in progress, that “ad hoc” testing worked well for finding bugs and highly scripted testing did not. (Note: We don’t mean to make this discovery sound easy. It wasn’t. We do mean to say that the non-obvious truths about testing are in evidence all around us, when we put aside folklore and look carefully at how people work each day.) He began writing and speaking about his experiences. A few years into his work as a test manager, mostly while testing compilers and other developer tools, he discovered that Cem Kaner had coined a term—”exploratory testing”—to represent the opposite of scripted testing. In that original passage, just a few pages long, Cem didn’t define the term and barely described it, but he was the first to talk directly about designing tests while performing them.
Thus emerged what we, here, call ET 1.0.
(See The History of Definitions of ET for a chronological guide to our terminology.)
ET 1.0: Rebellion
Testing with and without a script are different experiences. At first, we were mostly drawn to the quality of ideas that emerged from unscripted testing. When we did ET, we found more bugs and better bugs. It just felt like better testing. We hadn’t yet discovered why this was so. Thus, the first iteration of exploratory testing (ET) as rhetoric and theory focused on escaping the straitjacket of the script and making space for that “better testing”. We were facing the attitude that “Ad hoc testing is uncontrolled and unmanageable; something you shouldn’t do.” We were pushing against that idea, and in that context ET was a special activity. So, the crusaders for ET treated it as a technique and advocated using that technique. “Put aside your scripts and look at the product! Interact with it! Find bugs!”
Most of the world still thinks of ET in this way: as a technique and a distinct activity. But we were wrong about characterizing it that way. Doing so, we now realize, marginalizes and misrepresents it. It was okay as a start, but thinking that way leads to a dead end. Many people today, even people who have written books about ET, seem to be happy with that view.
This era of ET 1.0 began to fade in 1995. At that time, there were just a handful of people in the industry actively trying to develop exploratory testing into a discipline, despite the fact that all testers unconsciously or informally pursued it, and always have. For these few people, it was not enough to leave ET in the darkness.
ET 1.5: Explication
Through the late ‘90s, a small community of testers beginning in North America (who eventually grew into the worldwide Context-Driven community, with some jumping over into the Agile testing community) was also struggling with understanding the skills and thought processes that constitute testing work in general. To do that, they pursued two major threads of investigation. One was Jerry Weinberg’s humanist approach to software engineering, combining systems thinking with family psychology. The other was Cem Kaner’s advocacy of cognitive science and Popperian critical rationalism. This work would soon cause us to refactor our notions of scripted and exploratory testing. Why? Because our understanding of the deep structures of testing itself was evolving fast.
When James joined ST Labs in 1995, he was for the first time fully engaged in developing a vision and methodology for software testing. This was when he and Cem began their fifteen-year collaboration. This was when Rapid Software Testing methodology first formed. One of the first big innovations on that path was the introduction of guideword heuristics as one practical way of joining real-time tester thinking with a comprehensive underlying model of the testing process. Lists of test techniques or documentation templates had been around for a long time, but as we developed vocabulary and cognitive models for skilled software testing in general, we started to see exploratory testing in a new light. We began to compare and contrast the important structures of scripted and exploratory testing and the relationships between them, instead of seeing them as activities that merely felt different.
In 1996, James created the first testing class called “Exploratory Testing.” He had been exposed to design patterns thinking and had tried to incorporate that into the class. He identified testing competencies.
Note: During this period, James distinguished between exploratory and ad hoc testing—a distinction we no longer make. ET is an ad hoc process, in the dictionary sense: ad hoc means “to this; to the purpose”. He was really trying to distinguish between skilled and unskilled testing, and today we know better ways to do that. We now recognize unskilled ad hoc testing as ET, just as unskilled cooking is cooking, and unskilled dancing is dancing. The value of the label “exploratory testing” is simply that it is more descriptive of an activity that is, among other things, ad hoc.
In 1999, James was commissioned to define a formalized process of ET for Microsoft. The idea of a “formal ad hoc process” seemed paradoxical, however, and this set up a conflict which would be resolved via a series of constructive debates between James and Cem. Those debates would lead to we here will call ET 2.0.
There was also progress on making ET more friendly to project management. In 2000, inspired by the work for Microsoft, James and Jon Bach developed “Session-Based Test Management” for a group at Hewlett-Packard. In a sense this was a generalized form of the Microsoft process, with the goal of creating a higher level of accountability around informal exploratory work. SBTM was intended to help defend exploratory work from compulsive formalizers who were used to modeling testing in terms of test cases. In one sense, SBTM was quite successful in helping people to recognize that exploratory work was entirely manageable. SBTM helped to transform attitudes from “don’t do that” to “okay, blocks of ET time are things just like test cases are things.”
By 2000, most of the testing world seemed to have heard something about exploratory testing. We were beginning to make the world safe for better testing.
ET 2.0: Integration
The era of ET 2.0 has been a long one, based on a key insight: the exploratory-scripted continuum. This is a sliding bar on which testing ranges from completely exploratory to completely scripted. All testing work falls somewhere on this scale. Having recognized this, we stopped speaking of exploratory testing as a technique, but rather as an approach that applies to techniques (or as Cem likes to say, a “style” of testing).
We could think of testing that way because, unlike ten years earlier, we now had a rich idea of the skills and elements of testing. It was no longer some “creative and mystical” act that some people are born knowing how to do “intuitively”. We saw testing as involving specific structures, models, and cognitive processes other than exploring, so we felt we could separate exploring from testing in a useful way. Much of what we had called exploratory testing in the early 90’s we now began to call “freestyle exploratory testing.”
By 2006, we settled into a simple definition of ET, simultaneous learning, test design, and test execution. To help push the field forward, James and Cem convened a meeting called the Exploratory Testing Research Summit in January 2006. (The participants were James Bach, Jonathan Bach, Scott Barber, Michael Bolton, Elisabeth Hendrickson, Cem Kaner, Mike Kelly, Jonathan Kohl, James Lyndsay, and Rob Sabourin.) As we prepared for that, we made a disturbing discovery: every single participant in the summit agreed with the definition of ET, but few of us agreed on what the definition actually meant. This is a phenomenon we had no name for at the time, but is now called shallow agreement in the CDT community. To combat shallow agreement and promote better understanding of ET, some of us decided to adopt a more evocative and descriptive definition of it, proposed originally by Cem and later edited by several others: “a style of testing that emphasizes the freedom and responsibility of the individual tester to continually optimize the quality of his work by treating test design, test execution, test result interpretation, and learning as mutually supporting activities that continue in parallel throughout the course of the project.” Independently of each other, Jon Bach and Michael had suggested the “freedom and responsibility” part to that definition.
And so we had come to a specific and nuanced idea of exploration and its role in testing. Exploration can mean many things: searching a space, being creative, working without a map, doing things no one has done before, confronting complexity, acting spontaneously, etc. With the advent of the continuum concept (which James’ brother Jon actually called the “tester freedom scale”) and the discussions at the ExTRS peer conference, we realized most of those different notions of exploration are already central to testing, in general. What the adjective “exploratory” added, and how it contrasted with “scripted,” was the dimension of agency. In other words: self-directedness.
The full implications of the new definition became clear in the years that followed, and James and Michael taught and consulted in Rapid Software Testing methodology. We now recognize that by “exploratory testing”, we had been trying to refer to rich, competent testing that is self-directed. In other words, in all respects other than agency, skilled exploratory testing is not distinguishable from skilled scripted testing. Only agency matters, not documentation, nor deliberation, nor elapsed time, nor tools, nor conscious intent. You can be doing scripted testing without any scrap of paper nearby (scripted testing does not require that you follow a literal script). You can be doing scripted testing that has not been in any way pre-planned (someone else may be telling you what to do in real-time as they think of ideas). You can be doing scripted testing at a moment’s notice (someone might have just handed you a script, or you might have just developed one yourself). You can be doing scripted testing with or without tools (tools make testing different, but not necessarily more scripted). You can be doing scripted testing even unconsciously (perhaps you feel you are making free choices, but your models and habits have made an invisible prison for you). The essence of scripted testing is that the tester is not in control, but rather is being controlled by some other agent or process. This one simple, vital idea took us years to apprehend!
In those years we worked further on our notions of the special skills of exploratory testing. James and Jon Bach created the Exploratory Skills and Tactics reference sheet to bring specificity and detail to answer the question “what specifically is exploratory about exploratory testing?”
In 2007, another big slow leap was about to happen. It started small: inspired in part by a book called The Shape of Actions, James began distinguishing between processes that required human judgment and wisdom and those which did not. He called them “sapient” vs. “non-sapient.” This represented a new frontier for us: systematic study and development of tacit knowledge.
In 2009, Michael followed that up by distinguishing between testing and checking. Testing cannot be automated, but checking can be completely automated. Checking is embedded within testing. At first, James objected that, since there was already a concept of sapient testing, the distinction was unnecessary. To him, checking was simply non-sapient testing. But after a few years of applying these ideas in our consulting and training, we came to realize (as neither of us did at first) that checking and testing was a better way to think and speak than sapience and non-sapience. This is because “non-sapience” sounds like “stupid” and therefore it sounded like we were condemning checking by calling it non-sapient.
Do you notice how fine distinctions of language and thought can take years to work out? These ideas are the tools we need to sort out our practical decisions. Yet much like new drugs on the market, it can sometimes take a lot of experience to understand not only benefits, but also potentially harmful side effects of our ideas and terms. That may explain why those of us who’ve been working in the craft a long time are not always patient with colleagues or clients who shrug and tell us that “it’s just semantics.” It is our experience that semantics like these mean the difference between clear communication that motivates action and discipline, and fragile folklore that gets displaced by the next swarm of buzzwords to capture the fancy of management.
ET 3.0: Normalization
In 2011, sociologist Harry Collins began to change everything for us. It started when Michael read Tacit and Explicit Knowledge. We were quickly hooked on Harry’s clear writing and brilliant insight. He had spent many years studying scientists in action, and his ideas about the way science works fit perfectly with what we see in the testing field.
By studying the work of Harry and his colleagues, we learned how to talk about the difference between tacit and explicit knowledge, which allows us to recognize what can and cannot be encoded in a script or other artifacts. He distinguished between behaviour (the observable, describable aspects of an activity) and actions (behaviours with intention) (which had inspired James’ distinction between sapient and non-sapient testing). He untangled the differences between mimeomorphic actions (actions that we want to copy and to perform in the same way every time) and polimorphic actions (actions that we must vary in order to deal with social conditions); in doing that, he helped to identify the extents and limits of automation’s power. He wrote a book (with Trevor Pinch) about how scientific knowledge is constructed; another (with Rob Evans) about expertise; yet another about how scientists decide to evaluate a specific experimental result.
Harry’s work helped lend structure to other ideas that we had gathered along the way.
- McLuhan’s ideas about media and tools
- Karl Weick’s work on sensemaking
- Venkatesh Rao’s notions of tempo which in turn pointed us towards James C. Scott’s notion of legibility
- The realization (brought to our attention by an innocent question from a tester at Barclays Bank) that the “exploratory-scripted continuum” is actually the “formality continuum.” In other words, to formalize an activity means to make it more scripted.
- The realization of the important difference between spontaneous and deliberative testing, which is the degree of reflection that the tester is exercising. (This is not the same as exploratory vs. scripted, which is about the degree of agency.)
- The concept of “responsible tester” (defined as a tester who takes full, personal, responsibility for the quality of his work).
- The advent of the vital distinction between checking and testing, which replaced need to talk about “sapience” in our rhetoric of testing.
- The subsequent redefinition of the term “testing” within the Rapid Software Testing namespace to make these things more explicit (see below).
About That Last Bullet Point
ET 3.0 as a term is a bit paradoxical because what we are working toward, within the Rapid Software Testing methodology, is nothing less than the deprecation of the term “exploratory testing.”
Yes, we are retiring that term, after 22 years. Why?
Because we now define all testing as exploratory. Our definition of testing is now this:
“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”
Where does scripted testing fit, then? By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This does not refer only to specific instructions you are given and that you must follow. Your biases script you. Your ignorance scripts you. Your organization’s culture scripts you. The choices you make and never revisit script you.
By defining testing to be exploratory, scripting becomes a guest in the house of our craft; a potentially useful but foreign element to testing, one that is interesting to talk about and apply as a tactic in specific situations. An excellent tester should not be complacent or dismissive about scripting, any more than a lumberjack can be complacent or dismissive about heavy equipment. This stuff can help you or ruin you, but no serious professional can ignore it.
Are you doing testing? Then you are already doing exploratory testing. Are you doing scripted testing? If you’re doing it responsibly, you are doing exploratory testing with scripting (and perhaps with checking). If you’re only doing “scripted testing,” then you are just doing unmotivated checking, and we would say that you are not really testing. You are trying to behave like a machine, not a responsible tester.
ET 3.0, in a sentence, is the demotion of scripting to a technique, and the promotion of exploratory testing to, simply, testing.
” By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily)”
So we could say, given that our mind processes many heuristic-powered observation systems outside of our realm of choice, that what you used to call the exploratory end of the formality scale is, in fact, powered by scripts.
[James’ Reply: Yes. We are biological machines, after all. But by human convention, we decide as humans to see ourselves as exercising free choice, at least in principle.
Think of the concept of figure and ground. It used to be that all professional testing was seen as formalized (scripted). Scripted was the ground, and exploratory testing was this weird figure in the foreground. We are reversing that. We see exploratory testing as the background– the normal, if you will– and scripted testing as a figure that usefully intrudes into it.]
Scripts could refer to our mental models that shape the way we process our perception into beliefs. Can we say that all tools are partial scripts because although we choose to use them they shape the way we work? All test techniques are partial scripts because although we choose to implement them they influence the our attention and perception. Presumably this is what’s meant by using them “responsibly”? Using the agency we have left after we’ve excluded what we cannot choose.
[James’ Reply: Yes. Nicely put.]
If we subscribe to this change of terminology I wonder what value the term “scripted testing” has any more. How do we refer to the formality scale now that scripts are a necessary part of exploration? Is there now a place for an “externally imposed structure” scale? Or a “self-imposed structure” scale?
[James’ Reply: It’s helpful to distinguish between tacit and explicit scripting. Although there is tacit scripting in all testing, to some degree, there is not necessarily any explicit scripting. Explicit scripting– formalization– is more important than ever, except now we think it’s in its proper perspective. We are now treating scripting as a family of techniques for imposing certain kinds of structure onto the testing. As we have since 2001, we will refer to the formality continuum as a structure for thinking about what things we want to lock down, and what things we need to set free.]
This merge is long overdue. It’s been on my mind for the last year or so, but I haven’t made it explicit. Thanks for doing that.
I keep looking at the course I teach at UTS on software testing, and my own Exploratory Testing course – there’s very little distinction between the two.
I’ve had problems with the current definition of Exploratory Testing :
“a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project”
Apart from it being unwieldy it fails to promote an essential aspect of Exploratory Testing – that is the ability to feed new information gleaned from testing back into your test design. For me, this is the core to what ET is all about and its what differentiates scripted testing from exploratory testing. How does this feed into Exploratory Testing 3.0?
[James’ Reply: Seems to me “mutually supportive activities that run in parallel” means that we are feeding things we learn into new and better testing. In ET 3.0, we call that testing. Testing means to explore and experiment, and that is inherently a cyclic, evolutionary process– to which you can add or affix scripting.]
The back story on ET is the most important learning I’ve had today. Thank you!
Additionally, I see no reason for disagreement on the lexical proposal put forth in the closing statements, but I might wonder if this posting lacks a certain weight in our modern, fragmented, profit-over-all-else marketplace. We have been cleaning up messes in the performance testing space for years regardless of how many different terms or perspectives we adopt, how much energy we put out there to promote change or increased awareness. We’re even trying a new “performance manifesto” to see if that helps fuel the rebel alliance.
What next steps on implementation do you think are important?
[James’ Reply: Lacks a certain weight? I don’t understand. I’m not really concerned about the marketplace, though, except inasmuch as MY market consists of people who want me to teach and consult about excellent testing.
Our plan is to continue to develop a detailed and deep sense of the skills of testing and a rich vocabulary for talking about them and effective methods of teaching them.]
Sorry to be unclear, I meant “lacks a certain weight in our modern, fragmented, profit-over-all-else marketplace” <- meaning, the incompetent people who are abusing the term "tester" or "testing" and degrading the meaning will be nonetheless enthused to start illegitimately calling themselves "exploratory testers" while at the same time "we are testers who do exploratory things" while equally abusing the terminology by employing poor testing practices.
The point I ask is: how does this proposed name change lead us to a path whereby stakeholders and hiring managers might spot an unscrupulous incompetent charlatan tester who will "say whatever you want" just to get a job?
[James’ Reply: I don’t think the name change leads us to any such path. We need to do other things to accomplish that. For me, switching from saying “exploratory testing” to speaking of the “exploratory nature of testing” is a shift of rhetorical center-of-gravity that allows me to talk of ET basically all the time. It’s meant to help me unify my language and thus speak with more coherence.]
Enjoyed the timeline; helped structure & simplify a lot of what I’ve learned about the emergence of testing thought over time.
I can see the value of redefining testing to intrinsically include exploration. ET 3.0 becomes RST ?.?, perhaps? Anyway, it’s an idea I can get behind.
[James’ Reply: I’m talking about RST, yes.]
My question is about advocacy in a broader, industry-wide sense, versus advocacy in a more pragmatic purpose, in a testing role at an organization. When little or only recent progress has been won in an org, on the value of exploratory testing, or if there’s still a bit of a battle about its value, it seems it may take some wind out of the sails to advocate that exploratory is “already a part of testing”. If some org leadership still expects & values little more than “aggregate checking reports” and isn’t interested in the value of our careful distinctions here, it may be hard to secure time & expectations for questioning, studying, modeling, etc. Or may muddy the waters of the sales pitch.
Does that make sense? I guess I’m saying, if some folks only just learned, or still are learning, about ET 2.0, maybe they’re not ready for 3.0 semiotics yet. I suppose the first question is, do you agree? And the second is, do you have thoughts on how to approach that?
[James’ Reply: That’s why I have for years been resisting the occasional student or colleague who suggested that ET is just ordinary testing. I have said the same thing. We need to focus attention on the exploratory aspects of testing because the industry is obsessed with scripting.
By making our claim that all testing is exploratory, we are now talking about ET all the time. When we describe it, demonstrate it, defend it, it’s all ET. This is important because otherwise we are marginalizing it. Instead, we must normalize it.
But… good news! We can still talk about ET as an explicit activity when that feels necessary.]
I can see the value of redefining testing to intrinsically include exploration.
I prefer to think of it as restoring the meaning of testing, rather than redefining it. That’s what this post was about. http://www.developsense.com/blog/2015/02/give-us-back-our-testing/
Thanks James, that is an excellent write up and solves so many of the issues I’ve had with understanding ET. Basically it tells me that some of my constructs (in my mind) are outdated versions instead of valid at the same time. Having it in a timeline and history gives me the context to work in.
I’m still not quite convinced of your ET 3.0 plan but I guess that is work in progress and might change 100 times before it gets close to being something more concrete. The good thing is it gives me something to think about and to observe (how you go through the thought processes).
[James’ Reply: More concrete? Exploratory testing is testing. That’s concrete. We teach it and we pursue it all as an exploratory activity that is conditioned by some degree and some types of scripting.]
The question I have is if you’re overhauling the “exploratory” why not also refactor the “scripted”? Like ad-hoc it is connected strongly to a certain action and way of thinking. What you describe scripting to really be (which I don’t dispute as the core idea) I think deserves another word. I’m still thinking about that but there are words like intent, process, procedure,… which all have some truth to them but don’t quite fit the bill. I think redefining scripting though will send those less familiar with ET into a wrong direction though.
[James’ Reply: What’s wrong with the word scripting? Anyway, regardless of that word, there is already a different word, which I used throughout the post: formalization. Process, procedure, and intent are not words that have anything to do with scripting. When you think process is about scripting, you are speaking specifically of a FORMAL process; when you think procedure is about scripting, you are specifically thinking of a FORMAL procedure; and when you think of intent, you mean a FORMAL intent (rather than a whim). When I speak of scripting, I am speaking of giving up control, which is what formalization is all about. To formalize anything is to say “not just any old way… I will do it in this SPECIFIC way.”]
As for it taking years to come to terms with terms it dawned on me that it takes me years to actually understand those terms too. To ingest what they actually mean. To hit a problem in real life to finally grasp what and why the hypothesis or theory really applies. And that is what I struggle with. It is very hard to keep up or even catch up with progress. There are so many new things to ingest. But thankfully I see this progress and am not caught up in the factory world that is trying to sell me a dead horse. 😉
[James’ Reply: Yeah. Just remember this: we work in a world of ideas. Ideas are our business. It’s only right that our terms be precision instruments that take discipline to wield.]
By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily).
…
[James’ Reply: Yes. We are biological machines, after all. But by human convention, we decide as humans to see ourselves as exercising free choice, at least in principle.
I also feel very uncomfortable with this definition, especially given the context of the specificity in which you are defining many of the terms and language you use. I don’t think your reply sufficiently addressed the issue for me.
[James’ Reply: Well, state your problem…]
I’m going to make what might be a large assumption from your reply above that you agree that free will is a illusion.
Hence, any ‘choice’ is in fact not a choice but rather has some cause outside of our control but in most cases not generally understood or known. Given the above definition I cannot view any testing that anyone does to be anything but scripted, which I would disagree with.
[James’ Reply: I think you’ve misconstrued the word “choice.” We have options in the world and we choose among them. That is an established phenomenon. Don’t pretend you don’t get that. The fact that our choices are biased by many factors, and the fact that we can call those biases a form of scripting, does not eliminate the value or meaning of choosing.]
How do you discern a difference between “Your biases” and everything else that happens in your brain, which is also not in reality your choice?
[James’ Reply: I can study biases. I can manage them to some extent. So can you. It isn’t necessary, of course, to “discern a difference” between biases and anything else that happens in our brains. It’s all a big biased hairball and that’s okay. There are still choices and we make them. Deal with it.]
Why do you make the distinction between those functions in the brain which we to some degree understand the cause and behavior of, and those which we do not yet understand?
[James’ Reply: First, I make no strong distinction between those things. Second, a mild distinction is useful because that helps us become better at making choices.]
“But by human convention, we decide as humans to see ourselves as exercising free choice, at least in principle.” This statement really surprised me. Why is ‘by human convention’ a good reason to do anything? At one point it was human convention to imagine the earth as flat. This did not change the fact that it is not.
[James’ Reply: Because we’re not talking about physics. We’re talking about a social activity: testing. Testing is done by people, for people. And when we speak of scripting we are speaking of the limits of our choices, whether neurological or otherwise. In this social world, we treat each other as agents capable of independent action. That’s a human convention (though probably other animals do that as well). It’s called the “intentional stance” and it’s a powerful heuristic for predicting how other people will behave.]
It seems to me that as we learn more about the brain, how it functions and operates and what causes different types of behavior, the amount of behaviors that you can categories and ‘choice’ will constantly be diminished as they are one after another placed the ‘bias’ column. In this way your definition is kind of like the behaviors of the gaps definition.
[James’ Reply: To script is to remove choice. You know what choices are. You know what it means to have less choice. Are you really worried that your employer will learn enough about your brain to remotely control you? Is that what this is about?]
“I think you’ve misconstrued the word “choice.” We have options in the world and we choose among them. That is an established phenomenon. Don’t pretend you don’t get that. The fact that our choices are biased by many factors, and the fact that we can call those biases a form of scripting, does not eliminate the value or meaning of choosing.”
No we don’t. Not if by choose you mean that we could have done anything but that which we did do. This is what I fundamentally mean when I say free will is an illusion. You think that you made a choice. But in reality, when you inspect all the causes you realise could not have done anything but what you did.
[James’ Reply: True or not, that is irrelevant to us. It does not enter into the social life of humans. If it did, then I would feel free to treat you, right now, like a chatbot, and simply delete your comments. It’s because I think you are not a chatbot that am interested in what you have to say.]
To rephrase, you could not have chosen anything other than what you did. Hence, ‘the choice’ was an illusion of having choice when no choice really existed. You think that you had the choice to go on a killing spree when you got up this morning, that appears to be a choice, but in actual fact the physical state of your brain and the rest of the world this morning meant that you did not ‘make that choice’, which is to say you *could not* have made that choice.
[James’ Reply: I believe that is technically true, and I believe it is irrelevant to life. We perceive that we have choice. Or in other words, whatever that perception of freedom is, I’m going to call choice. That’s what I’ve talking about. There are levels to it. Perhaps at the deepest level there is no freedom. I don’t care. By the way, neither do you. You think you are choosing your replies to me, right now. That’s enough for you, isn’t it?]
I will concede that there may be situations where is may be pragmatic to operate under the assumption free will or choice exist. I do not believe this to be one.
[James’ Reply: If I actually took you seriously about this, I would be logically compelled to dismiss your belief: because you seem to be claiming that you have no choice but to have this belief, and any “belief” that has not been freely chosen is not properly a belief in the epistemic sense, but rather some sort of hallucination or insanity.]
“To script is to remove choice. You know what choices are. You know what it means to have less choice. Are you really worried that your employer will learn enough about your brain to remotely control you? Is that what this is about?”
You cannot remove choice where no choice existed. You cannot have less choice than none. No I am not worried that an employer will learn enough about my brain to remotely control me. That never crossed my mind and I’m not sure why you might think that’s what I was worried about.
[James’ Reply: Choice did exist and does exist. You are choosing not recognize that, right now. If your words are more than a mere recording you memorized, and you expect me to treat as other than a chatbot, then you must have chosen them on some interesting level, even if we agree that on some other level you did not.]
I think there *is* a difference between following a script you have been provided, written, or even just thought of and performing testing where you do not follow a script in the same manner. What I disagree with is the distinction you are making when you say “Your biases script you. Your ignorance scripts you.” and what you call “choice” as if these biases are somehow separate entities distorting the way the rest of the way the brain functions and not just what we call specific identified common behaviors.
[James’ Reply: Our brains are complex. They consist of many interacting systems that inter-distort each other and bias each other. So what? How is that a problem for you? When I say the word “I” I am referring to my sense of myself, but also my body, and also myself as a legal entity. I am referring to the loose collection of thoughts in my mind at this moment, and my memories. “I” is ambiguous, but that is not a problem for us in ordinary life. We can live with that ambiguity.
Why does it trouble you so much that I can and will talk about structural scripting that acts on me via unconscious processes and environmental constraints?
I have been using this simple idea of scripting for a long time now. It works. It is philosophically coherent. Try it.]
And I think I’ve just identified what may be a difference in perception or understanding. A bias is something that we usually associate with groups of people, if not all people. Maybe you haven’t or don’t consider things like biases that exist in only one person to be the same as biases.
[James’ Reply: Cognitive scientists don’t think about biases as belonging mainly to groups of humans, and I don’t either.]
I would argue they are the same and that is all your consciousnesses is made up of. A complex network of biases (as the cognitive neuroscientist Dr. Sam Harris puts it, it’s tumors all the way down).
[James’ Reply: I agree with you. That is also my understanding.]
And when you get down to it, every decision you have ever made can be traced back and said to be caused by one or more or these biases which in turn were caused by a combination of your genetics, environment, and your experiences. None of which you were in control of or had any choice about.
[James’ Reply: Sure. But we don’t have to care about that on the level of social life.]
My concern is about trying to categorise some emergent behaviors of the brain as ‘bias’ and others as ‘choice’ as if to say your biases don’t make your choices for you. If you are saying that your bias is a script you or your brain is following then I would say you have no choice but to follow those scripts, and knowing about and compensating for one or more bias(es) simply means your following a different script, a different ‘bias’ if you will.
[James’ Reply: You cannot credibly deny the human experience of consciousness. It doesn’t matter that it is an epiphenomenon of lower order processes– it exists. We experience the phenomenon (in the sense of phenomenology) of choice. You keep trying to say that choice does not exist, as if choice were some sort of supernatural substance that I am positing. No. We live in the world of physics and everything is bound to physical law. Choice does not exist as a substance or a special category of reality. Choice is simply an experience. We definitely have that experience of choosing. You have it. You know you have it. That experience is what I’m talking about. That experience is influenced by many interesting factors. I’m going to call some of those influences biases (when they influence us not through conscious apprehension). I’ll call others “judgment” or “temperament” or “reasons”… these are all heuristic terms for somewhat plastic and overlapping factors.]
I really think this warrants consideration and I hope you take a minute to consider my view.
[James’ Reply: I would like you to appreciate that I have stayed up most of the night, tonight answering you. I think your views are interesting and challenging enough that I feel, sort of, compelled to respond.
Can I ask you what you think would be a better way to define or delineate scripting?]
p.s. The irony of me having to check a check box saying “I’m not a robot” to submit comments hasn’t escaped me.
“True or not, that is irrelevant to us. It does not enter into the social life of humans. If it did, then I would feel free to treat you, right now, like a chatbot, and simply delete your comments. It’s because I think you are not a chatbot that am interested in what you have to say.”
Generally I would agree. But you are making a very specific distinction about functions of the brain which you say are not choice (scripted) and those which are. You are in fact asserting that one is capable of making a ‘choice’ which is not affected by bias.
[James’ Reply: I am not at all asserting that. I am asserting that the phenomenon of choice is real to us, and that we can coherently speak of biases as separate from choice even though choices are “made of” biases, because from within the experience of choice we can become aware of some factors that are specifically influencing us (as in “I don’t want to make this decision on an empty stomach”) and make meta-choices about those factors that may serve to improve the processes of choosing. It is that landscape of choice and bias that I am referring to. The experience of it is real and important.]
Yes we are all ‘just’ complex chatbots. Ones that generally react poorly to being called or treated like chatbots and become less productive and cooperative. That is why you don’t treat people like one. If people liked being called or treated like a chat bot and responded by becoming more productive and cooperative you would be more inclined to do it.
[James’ Reply: Of course. You like me treating you as if you have choices and are making them. And so I will. Thank you for helping me convince you, human unit called JamesIrving.]
“If I actually took you seriously about this, I would be logically compelled to dismiss your belief: because you seem to be claiming that you have no choice but to have this belief, and any “belief” that has not been freely chosen is not properly a belief in the epistemic sense, but rather some sort of hallucination or insanity.”
I am, but I don’t follow your logic. I did not have any choice but to come to this belief given my experiences, environment, genes, and information available to me. That doesn’t diminish the belief.
[James’ Reply: “Belief” without agency has no meaning. A political lobbyist “believes” there is no global warming because he is paid to “believe” that. His “belief” has no meaning so I have to use quotes when I use that word in reference to him. He’s a walking, talking billboard, not a human. He cannot be convinced or influenced by the logic of an alternative position. Only the “logic” of money will change his “belief.” Maybe, deep down, there is a free part of him that secretly has an opinion, constructed from an honest examination of the issue. We can’t know.]
Logical inconsistency or lack of reason would diminish it. If one has no choice but to come to logical conclusions, the lack of choice to do so does not mean the conclusion is no longer logical or any less valid.
[James’ Reply: That’s only true within a formal system. But we are not operating within a formal system– unless you are referring to physics, chemistry, and the particular state your mass of protons, neutrons, and electrons happen to be in at this moment– and we have no access to that. We have access to the phenomenon of choice, though, and we use that every day, every waking moment.]
“Why does it trouble you so much that I can and will talk about structural scripting that acts on me via unconscious processes and environmental constraints?”
Because in the way you are speaking all testing you would perform would fall into this category. Hence, making it redundant.
[James’ Reply: That can only be true if you deny the phenomenon of choice, which is undeniable by any normal human regardless of the autonomic processes may lie beneath it. Maybe you aren’t a normal human, but I’m going to assume you are until you prove otherwise.
You seem to feel that physics happening on one level makes it unnecessary or meaningless to speak of consciousness happening at a higher level. This is a perfectly self-defeating argument, as I have already pointed out. Out of compassion for the “you” that you are perhaps unknowingly negating, I must continue to treat you as a conscious man who deserves to be treated as more than a walking symbol production machine.]
I am arguing that there is no action you can perform including testing where your bias does not script you and to think otherwise is to misunderstand the nature of our consciousness.
[James’ Reply: I agree with you on that point. Where I disagree is how you deny the phenomenon of choice, and claim that therefore “everything is scripted testing.” That is a composition fallacy. Just because choices are “made of” scripts, ultimately, does not mean they have the same character as scripts. Choices are a higher level construct which, because we humans inhabit that construct, are useful and important to talk about and deal with.
Consciousness of this kind was not exactly “invented” by the human species, but we certainly pushed it farther, with poetry and art and spirituality, than other animals have. Let’s not pretend we don’t have this awareness of choice, even as it is surrounded and influenced by other structures.]
As have agreed with this principle what is the point of a definition of scripted testing which includes cognitive bias if that would put all testing into the scripted testing category on the account our brain is incapable of doing anything else?
[James’ Reply: It does not put all testing in the scripted category, as long as you are able to distinguish the phenomenon of choice from non-choice, which humans can easily do in many cases, have a hard time doing in some cases, and I agree cannot do in still other cases (but that’s okay).
I urge you to read The Shape of Actions, which talks a lot about this.]
“You cannot credibly deny the human experience of consciousness. It doesn’t matter that it is an epiphenomenon of lower order processes– it exists. We experience the phenomenon (in the sense of phenomenology) of choice. You keep trying to say that choice does not exist, as if choice were some sort of supernatural substance that I am positing. No. We live in the world of physics and everything is bound to physical law. Choice does not exist as a substance or a special category of reality. Choice is simply an experience. We definitely have that experience of choosing. You have it. You know you have it. That experience is what I’m talking about. That experience is influenced by many interesting factors. I’m going to call some of those influences biases (when they influence us not through conscious apprehension). I’ll call others “judgment” or “temperament” or “reasons”… these are all heuristic terms for somewhat plastic and overlapping factors.”
I do not deny the experience. But again if you want to talk about bias or ignorance scripting your testing I cannot imagine how you could do anything but scripted testing.
[James’ Reply: You CAN imagine it if you rise out of flatland and enjoy a new dimension. You can resolve this paradox: All testing is scripted, yet all testing is exploratory. (…in different ways and combinations that we can control to an interesting and substantial degree.)]
If you want to talk in general terms we can agree on choice referring to the experience and not the reality. If; however, you want to talk about the functions in our brain that deny us this choice and the same level the bias operates on, you have to include all of it.
[James’ Reply: I do include all of it. I find that including all of does not in any way negate my experience of choice.
I once had a fever. During the fever I “witnessed” the breakdown of my own consciousness. I spent hours listening to my internal organs talking to each other. In that fitful dream, I could understand the language they were using. I remember it being the most boring conversation I had ever heard– like listening to a roomful of men playing poker. I wanted to tell my heart to SHUT UP I’M TRYING TO SLEEP.
When I came out of the fever, I realized what probably happened was a dissolution of some of the barriers that normally keep the lower level biological functions out of my conscious awareness. It gave me a better appreciation for how consciousness is a big show. I am composed of many creatures cooperating. I am not I.
My point is: even having this experience, I have no problem seeing myself as a creature that has and makes choices. I CAN accept “all of it” and still speak of choice and bias.
You’re right that I cannot possibly make an unbiased choice, but that doesn’t mean I don’t have the sense of making choices, and that such a sense is meaningful EVEN WHILE I seek to understand more about my biases and bring more of them under control.
In a sense, all of life is a process of becoming a little more awake, that way.]
“I would like you to appreciate that I have stayed up most of the night, tonight answering you. I think your views are interesting and challenging enough that I feel, sort of, compelled to respond.”
I do. Thank you
“Can I ask you what you think would be a better way to define or delineate scripting?”
Perhaps
“Script”: Any control system or factor that influences your testing and lies outside of your realm of cognitive function
[James’ Reply: How does that account for the obvious case of following a set of instructions? The instructions are not necessarily outside of my cognitive function. I may reflect upon them and understand them in great depth. What makes it a script is that it is outside my realm of choice. I have decided to “submit” to a decision by my slightly younger self to follow the instructions. I have chosen (on one level) not to make new choices (on another level). The fact that I may be influenced to do so, unknowingly, by having low blood sugar at that moment is not necessarily a major concern (although it might become one).]
“I am not at all asserting that. I am asserting that the phenomenon of choice is real to us, and that we can coherently speak of biases as separate from choice even though choices are “made of” biases, because from within the experience of choice we can become aware of some factors that are specifically influencing us (as in “I don’t want to make this decision on an empty stomach”) and make meta-choices about those factors that may serve to improve the processes of choosing. It is that landscape of choice and bias that I am referring to. The experience of it is real and important.”
The only distinction you are making there is between the biases you understand and are aware of, and those which you are not. This doesn’t mean they are two different things.
[James’ Reply: In terms of our experience, it literally does mean that. And that’s what I am referring to.]
This is kind of like looking at a magician and saying his tricks really do appear to be magic so when I say magic is real I’m referring to the experience of magic when you go see a magic show.
[James’ Reply: The whole point of what magicians do is to give you an experience of wonder. If he wants to call that magic, no logic stops him. What you seem to be implying is that he would be behaving absurdly because he wants to use two different definitions of magic at the same time in the same context. That’s the fallacy of equivocation. If he were doing that, that would be self-contradictory, but neither he nor I am necessarily doing that. And I assert that I am not doing that. I am referring to the phenomenon of choice, which you and I experience each waking moment. That fact that we both seem to believe choice is an epiphenomenon built from non-choice elements is irrelevant to my use of the term “choice.”]
Except I know how the first two tricks were done, I understand the mechanisms at work, so I’m going to call those clever tricks. But the rest, well that’s real magic, though I acknowledge that deep down they’re all just clever tricks. I’m fairly confident you cannot come up with an argument for your position that cannot be applied to magic in the same way.
[James’ Reply: What if the magician himself did not know how he did the tricks? In fact that is the case in real life, when you go down to a certain level of somatic tacit knowledge. When I first tried to do a coin vanish, I couldn’t do it, even though I “knew” the trick. But my fingers practiced and my fingers learned to do the trick. Now my fingers have what Daniel Dennett calls “competence without comprehension.” To me, it’s magic when I will my fingers to type these keysssssss ttttthhhaaaatttt I am typing and my fingers obey. It’s real magic, in an important sense, and I relate to it as such. This creates no contradictions for me. But what if a scientist comes around and tells me that he knows why I type slowly instead of twice as fast? Maybe he proves it by feeding me some kind of vitamin serum and lo! my typing speed increases. That’s fine with me. A little more of life has become demystified.
Your argument makes levels of magic seem absurd only because you are implying that the magician is lying to you about magic. If he is not telling lies, it’s just a statement about levels of awareness.]
In general conversation using the term magic is fine, everyone will know what you mean. But if you start trying to separate the magicians trick into those which are trick and those which are magic then I don’t know what you’re talking about. All I see is tricks, some I understand and some I don’t.
[James’ Reply: When I refer to real magic I mean the tricks no one understands how we do. They lie outside our realm of control, hence outside our realm of choice, as well. Stage magic is an illusion whereby one person purposefully deceives others.
This is a useful distinction: the stage magic vs. real magic.]
““Belief” without agency has no meaning. A political lobbyist “believes” there is no global warming because he is paid to “believe” that. His “belief” has no meaning so I have to use quotes when I use that word in reference to him. He’s a walking, talking billboard, not a human. He cannot be convinced or influenced by the logic of an alternative position. Only the “logic” of money will change his “belief.” Maybe, deep down, there is a free part of him that secretly has an opinion, constructed from an honest examination of the issue. We can’t know.”
Trying to ‘convince’ the lobbyist has no meaning in this context as the lobbyist is just a messenger. He is not the owner of the belief. I would not say that he “believes” the message he is delivering (although he may incidentally) He is not required to believe it, just deliver the message.
[James’ Reply: He does not present himself as a messenger. He wishes to deceive us so that we think he has a belief. He may even convince himself that he has a belief. It only becomes clear that he doesn’t when he realizes that no possible evidence will sway him.]
There is a lot to explore in this subject, the concept of belief and different types of belief is a whole other discussion. Here is another way to phrase it. Given the information available to me and the physical structure of my brain I have come to a particular conclusion – the only conclusion I could have come to under those conditions. Given new information (i.e. an argument from you) I may be forced to come to a new conclusion.
[James’ Reply: That’s really not how belief operates outside of a formal system. Even in a formal system there may be many alternatives that have pros and cons. Can you explain how simulated annealing works with that explanation? How about the Prisoner’s Dilemma? How about chaotic systems? Where does judgment come in? How do you account for different qualified people having different judgments? How do you account for incommensurable ontologies (conflicting paradigms)?]
“That’s only true within a formal system. But we are not operating within a formal system– unless you are referring to physics, chemistry, and the particular state your mass of protons, neutrons, and electrons happen to be in at this moment– and we have no access to that. We have access to the phenomenon of choice, though, and we use that every day, every waking moment.”
I disagree. Indeed logic, reason, and evidence are values that I am appealing to. I think it’s a safe assumption you are also appealing to those values. I am functioning under the assumption they are the only values worth appealing to. if we cannot agree join that then this conversation cannot proceed.
[James’ Reply: I’m sorry but I don’t know what you can possibly mean by “I disagree.” Are you saying that when you follow the compulsion of rules of logic you are not operating within a formal system? If so, you are uttering a contradiction. Logic does not allow you to do that. Stop breaking the rules. Perhaps you don’t know what a formal system is? You should look that up before we go further.
Logical compulsion, by definition, only exists within the “game” (or formal system) of logic. Not outside of it.
We are operating within an informal system, in this conversation. We are dealing with what’s called “the logic of real discussion.” We are obliged to use abductive and inductive inference, rather than deduction for most of the work we must do. But only deduction *forces* you to accept specific conclusions.]
After all what evidence or logical argument can you present to demonstrate these are worth valuing to someone who doesn’t value that type of argument? I really don’t know what you mean by ‘access’ to the phenomenon of choice. We are aware of an experience that feels like free will and choice. I cannot understand why you would say that gives you ‘access’. Again that would rather be like saying we have access to the phenomenon of magic.
[James’ Reply: If you want to call choice magic (a word you haven’t defined, so I’m going to assume it means “mysterious process”), that’s fine with me. You already know what I am talking about. We have access to the phenomenon of choosing; the experience of choosing. You make what feel to you like choices. You have acknowledged that, and that is what I’m talking about.]
“You seem to feel that physics happening on one level makes it unnecessary or meaningless to speak of consciousness happening at a higher level. This is a perfectly self-defeating argument, as I have already pointed out. Out of compassion for the “you” that you are perhaps unknowingly negating, I must continue to treat you as a conscious man who deserves to be treated as more than a walking symbol production machine.”
No. My objection is with you making a distinction across different levels. On one level you can say that humans have choice and free will. They seem to act and behave in ways of their choosing. On the next level you can say there is bias: all sorts of software and hardware influences that end up scripting all the behaviours of all humans. You could perhaps argue another level at the atomic or subatomic level which defines the behaviour of the second level but it’s irrelevant. To try and categorise behaviours into categories that act at different layers makes no sense.
[James’ Reply: It makes sense to me. And I don’t see, here, how you have identified any real difficulty with it. Can you please tell me one practical difficulty in the scheme I am using? By the way, I’ve been using this scheme in a way that feels quite productive to me for many years. So, if you think I am self-deceiving, you’ll have to demonstrate at least some consequence that has been or is secretly hurting me, right?]
“I agree with you on that point. Where I disagree is how you deny the phenomenon of choice, and claim that therefore “everything is scripted testing.” That is a composition fallacy. Just because choices are “made of” scripts, ultimately, does not mean they have the same character as scripts. Choices are a higher level construct which, because we humans inhabit that construct, are useful and important to talk about and deal with.”
As you say “Choices are a higher level construct” this is my point. You cannot say Scripts are made of choices, except script A, B, and C because I know how those effect my choices and what those scripts are, so they diminish my choice. But all the other scripts, well that’s just choice.
[James’ Reply: I don’t say that scripts are made of choices, although that wouldn’t be a terrible thing to say, since a lot of choices go into explicit scripting.
Perhaps you meant to write “you cannot say testing is made of choices, except with regard to script A, B, and C because you know how those affect your choices and what those scripts are, but all other testing, well, that’s just choice.”
If you meant to write that, then I disagree with you. I definitely can say that, and I can mean that, and no logical or practical factor or law stands in my way. I can call things that feel like choices CHOICES. I can talk about them, defend, them, etc. And if and when I come to realize that there is some specific structure to those choices which lies OUTSIDE my control I can refactor my analysis and speak of the very same thing as partly (or even entirely) scripted.]
“I urge you to read The Shape of Actions, which talks a lot about this.”
I’m not a great reader of books but I’ll try.
[James’ Reply: Well, I’m impressed that a man who does not read philosophy has such a natural enthusiasm for philosophical debate. But if there is any book you really MUST READ it is Godel, Escher, Bach: The Eternal Golden Braid. Until I read that, I didn’t quite understand logic and formal systems. I believe it will blow your mind in a useful way.
If you love thinking philosophically, you don’t necessarily need to read a lot. Wittengstein famously read almost nothing and he is considered one of the greats. Socrates was dubious about the whole business of literacy. But some things out there are really powerful reads.]
“I do include all of it. I find that including all of does not in any way negate my experience of choice.
I once had a fever. During the fever I “witnessed” the breakdown of my own consciousness. I spent hours listening to my internal organs talking to each other. In that fitful dream, I could understand the language they were using. I remember it being the most boring conversation I had ever heard– like listening to a roomful of men playing poker. I wanted to tell my heart to SHUT UP I’M TRYING TO SLEEP.
When I came out of the fever, I realized what probably happened was a dissolution of some of the barriers that normally keep the lower level biological functions out of my conscious awareness. It gave me a better appreciation for how consciousness is a big show. I am composed of many creatures cooperating. I am not I.
My point is: even having this experience, I have no problem seeing myself as a creature that has and makes choices. I CAN accept “all of it” and still speak of choice and bias.
You’re right that I cannot possibly make an unbiased choice, but that doesn’t mean I don’t have the sense of making choices, and that such a sense is meaningful EVEN WHILE I seek to understand more about my biases and bring more of them under control.
In a sense, all of life is a process of becoming a little more awake, that way.”
But if you include all magic in your category of clever tricks then what is left for magic? Once you acknowledge it’s all tricks then magic has no meaning anymore.
[James’ Reply: Wherever do you get that idea? Magic does have meaning: magic is the stuff I don’t understand yet. Magic is what seems like magic. Again, you haven’t defined your word “magic” so I guess that means I’m allowed to make up whatever meaning I like for it.]
As I said earlier as you understand more and more of your biases, more and more parts of your choices have to be put into the scripted category until eventually we have a total understanding of all bias and the illusion of choice is completely dissolved.
[James’ Reply: That could happen in principle, but not in practice. And it’s surely not happening for us on our projects, today, so, try to relax, man.]
You can certainly talk about magic and clever tricks. As you learn more clever tricks and more about how clever tricks are done less and less seem to be magic until one day every magic show you go to has only tricks you understand and no more magic is left.
[James’ Reply: Except the magic that the magician himself practices yet does not understand. And that includes the feeling of choosing.]
You can strive to learn more about bias and bring more of them ‘under control’ without asserting that they are different entities acting on or creating your choice in a way that all the biases you don’t understand do not. Or asserting that they are somehow separate from the experience of choice you have rather that functions that contribute to that experience.
[James’ Reply: I am asserting there is a useful difference between biases you aware of and biases you are not aware of. The difference is obvious and you already understand it. The difference is this: we can study the biases we are aware of and we can systematically plan to enhance them or mitigate them. That is not so much true about biases we are not aware of.]
“How does that account for the obvious case of following a set of instructions? The instructions are not necessarily outside of my cognitive function. I may reflect upon them and understand them in great depth. What makes it a script is that it is outside my realm of choice. I have decided to “submit” to a decision by my slightly younger self to follow the instructions. I have chosen (on one level) not to make new choices (on another level). The fact that I may be influenced to do so, unknowingly, by having low blood sugar at that moment is not necessarily a major concern (although it might become one).”
I’m not suggesting it’s an easy problem you have refined these terms down to a level that discussions about the nature of what it is to be conscious becomes relevant but I am sure the distinction you are drawing does not exist in any practical way.
[James’ Reply: The distinction I am making does exist in an immediately practical way that I have been applying for years. The fact that you are confused doesn’t stop me from doing things that work, man.]
OK maybe “at the time of performing the test” needs to be added. Hence if you write a test case, you are surrendering your control to your earlier self, as you described. The same applies if you create a test case or test script but just don’t write it down.
“Script”: Any control system or factor that influences your testing and lies outside of your realm of cognitive function at the time of performing the test.
[James’ Reply: It is still not outside your cognitive function if you are thinking about it. What it’s outside of is your CONTROL; your realm of choice. Perhaps it would help for you to read a little about control theory, too? Or do you deny that the word “control” refers to anything important?]
You said “we settled into a simple definition of ET, simultaneous learning, test design, and test execution.” This contrasts scripted testing with the word simultaneous. Scripted testing has learning, test design, and test execution but these are done consecutively rather than in parallel. So during scripted testing you are allowing the product of the test design phase to dictate your behaviour during test execution. I think this definition does that but it’s likely it can still be improved.
[James’ Reply: I think you are warping the definition of “cognitive function” in order to avoid using the word choice.]
“I spent hours listening to my internal organs talking to each other. In that fitful dream, I could understand the language they were using.”
I’m intrigued by this and would love to hear more about it.
What were they saying to each other?
[James’ Reply: They were trading or negotiating or something. Just like people playing poker. In the dream I understood the language, but I remember thinking, wow, this is boring to listen to.]
Were the organs themselves communicating with each other, or were the parts of the brain that control them communicating with each other?
[James’ Reply: Based on my theory that the fever was causing a malfunction of my consciousness circuitry, it would have been regions of the brain responsible for for the organs, I suppose.]
Did you experience the conversation directly in English, or did you translate it?
[James’ Reply: It was definitely not English. It was some sort of code, but I understood it.]
Do you know of any other cases where people have experienced this?
[James’ Reply: See “My Stroke of Insight” by Jill Bolte Taylor for something similar.]
How convinced are you that you were actually listening to them, rather than imagining/dreaming it?
[James’ Reply: I *was* imagining it. That’s what consciousness IS… imagination. Each of us imagines himself one integrated creature. But it seems a very small leap to think that imagining I am listening to boring messages among my own organs is based on actual messages going between those systems. Such messages are happening, so all that’s needed is a little breakdown.
The dream went on for many hours, too. It wasn’t an ordinary dream. And all night I was running a temperature that broke in the morning (just before I had to get up and go teach).]
Thanks for your response James,
I wonder if it would be possible to gain conscious control over the organs, the way we can over our lungs when we become aware of our breathing.
Also, I wonder whether the negotiating organs themselves possess/generate some level of consciousness when the ‘I’ isn’t conscious of them, or if they can only possess/generate consciousness as part of a larger consciousness.
[James’ Reply: I recommend that you look into the techniques used by the people who compete in breath-holding competitions. Also: read Daniel Dennett’s work on consciousness. (I’m currently reading Intuition Pumps and Other Tools of Thinking]]
Jill’s TED talk made me think of altered states of consciousness I’ve experienced:
Dreams, when you just uncritically go along with the situation you find yourself in, without ever thinking how this situation arose. Perhaps this is what it’s like to have the consciousness of a creature such as insect (or a subsystem of the brain we’re not normally aware of).
Voices, when half falling asleep while trying to concentrate on something late at night. Starting off as phrases heard during the day, then turning into something more dreamlike – sounding like they would’ve required a conscious entity to create them, but having not been consciously created by ‘me’.
Migraine, being able to read individual letters but not have them automatically form words — having to read a really simple word many times (while the letters seemed to be changing their order) until the word eventually popped into my head. This also had the same effect on my speech and internal monologue. This was fascinating and fun (though scary the first time!). Haven’t had a migraine in ages, makes me almost nostalgic!
This has really got me thinking and will take a while for me to re-read and fully digest. To help with my learning I’d like to better understand how an explicit script is seen as something that influences my testing, but more specifically, lies outside of my realm of choice.
Using the example I’m doing responsible scripted testing (exploratory testing with scripting); how is the conscious decision which parts of the explicit script I decide to use and to the degree it dominates and influences my testing not seen as a choice?
[James’ Reply: Sure, it’s a choice. It’s a choice on one level. Once you make that choice you enter a lower level. Here’s an example: I can do a lot of things, today, but having agreed to the social contract of civilization, and not wanting to be arrested, I voluntarily submit myself to the laws of Breda, Netherlands. I COULD have stolen someone’s bicycle today, and not broken any law of physics, but I would have violated other rules that now limit my choices. If I had sat with someone in a cafe and played chess, while playing I would have further limited my choices. I would be allowed only the options that are part of that game. We move in and out of “script fields” throughout each day. We do this voluntarily, as well as being compelled, at times.
So, as I test, I might decide to work from a spreadsheet for a while, or I might decide to try every item on a menu. While I am doing those things, I am operating in a “script field” that limits my choices. Of course I could suspend that scripting as a responsible tester, but then I may not achieve what I want to achieve. I pull scripts over me the same way I pull up the covers at night if I feel cold.
By saying testing is inherently exploratory, we’re saying that testing begins with choices, and that to close choice down is a technique we can deploy, rather than a normal state of working.]
So, in essence, all testing is both exploratory and scripted?
[James’ Reply: Yes. In different ways and degrees. I talked about this when you took my class (didn’t you take my class?).]
Yes, I was there. Simply reconfirming.
Given the premise all testing is scripted in some sense and to some degree – why the need for term scripted testing, with script being the modifier?
[James’ Reply: I need the term so I can talk about it. It’s very important for testers to understand that we don’t name these things to put them into a box, we name them to be able to REMOVE them from their obscuring, simplifying boxes. Having understood that all testing is scripted to some degree, we don’t call it a day, but rather we talk about the WAYS it is scripted and the DEGREES to which it is scripted.]
Is it to infer a meta-choice dominated towards explicit scripting in testing? Sure it can’t infer tacit scripting, how would you know?
[James’ Reply: It’s to give us a conceptual handle that allows us to turn the subject around in our minds and control it (once we learn enough about the different ways that testing can be scripted).]
You talked about operating in a “scripted field”, would scripted mode be a suitable alternate description? Do you see a hard delineation between scripting and testing? Or do they blur into one another?
[James’ Reply: Scripted field is a far better term than mode. A scripted field implies that it lies outside your choice. You are making choices within that field. That’s a good way to think about it, I believe.
There is a massive difference between scripting and testing. I presume you mean to refer to scripted testing vs. testing? As we clearly state in our post, we have been teaching that there is NO hard delineation between those things for more than a decade. That’s what the Formality Continuum is.]
This is a great story of the evolution of terminology. Beginning with why it was required, then how it changed, to why it’s no longer required. A very neat history lesson for all testers.
It’s also a great example of how we testers need to evolve and continue to learn. There will be many out there who felt comfortable using the terms ‘exploratory’ and ‘scripted’ who now need to consider dropping ‘exploratory’ and relearning ‘scripted’. A tester’s education is never done.
The one thing I do all the time now is ask what people mean by certain terms, and this is something that I owe to the CDT community (understanding the value of definitions).
I can see this having a flow on impact to discussions around the term ‘context-driven’. Not necessarily including the community or paradigm of thinking, but the approach. I believe that all good testing is context-driven, and some could argue that even bad testing can be driven by context (when thinking about context in the dictionary meaning of the word, not necessarily when considering the 7 principles). So can we look forward to dropping the term ‘context-driven’ sometime in the future, and just calling it ‘testing’, or ‘good testing’? I think that would be a great discussion to have, and would most certainly warrant more time and space than is available in this forum. I mean, we’d have to define the term ‘good’… and that could take a very long time (years in fact)!
Thanks James and Michael.
[James’ Reply: Context-Driven is made necessary by the existence of other schools of thought. When we say we are deprecating exploratory testing as a term, that is within the Rapid Software Testing methodology, which, of course, does not have competitors within itself.]
That’s why I narrowed in on the ‘approach’, not the school of thought. But I can see your point.
Feeling at ease after reading this. Thank you for writing this,James and Michael.
Well, as a RST graduate, RTIO peer advisor and as lifelong student of RST, I’m keen to understand how would it impact the “Scripted and Exploratory Testing Continuum”?
[James’ Reply: We now call it the Formality Continuum. It is not impacted at all.]
Does that still remain relevant if we are to believe that all testing in principle is exploratory? What would change about “Scripted and Exploratory Testing Continuum” i.e. in a way it is explained as a concept?
[James’ Reply: We mix elements that are beyond choice with elements that are within choice. We also exercise “meta-choice” in the sense of choosing what that mix ought to be. Nothing there has changed. What’s changed is that instead of seeing scripting as primary, into which choice intrudes, and instead of seeing scripting and choice as co-equal players, we believe it is most correct and effective to consider choice as primary and scripting as a factor that intrudes upon it.
When we say testing, we mean exploratory testing with some aspects of scripting involved.]
On Twitter, I had a rather snappy tweet;
The history of the craft has bought you here, I just arrived. I don’t need to learn and then unlearn years of cruft.
In a conversation about this.
And I came here to propose a thought experiment that was too long for Twitter, to hopefully express my viewpoint;
What if this article was written in reverse chronological order?
For those who experienced the history of software testing, it makes sense to start at the start.
But for those of us who didn’t, the frame of reference we have means it takes great leaps and bounds to delve into this history – it is actually rather inaccessible. And even in it’s current form, this reads better to me, and aids my understanding more when I read it backwards.
As a thought experiment, how differently would this be if it was framed to begin with ET 3.0, and then went back to explain the roots of these ideas? And what benefit, if any, could that framing provide?
[James’ Reply: Who are you? You are an interesting guy. Why don’t I know about you already? You’ve raised important issues. I will think about what you’ve said.]
Thanks James and Michael for the great blog. It is a good overview of the development of exploratory testing with mindshifts I took myself (with help of others including the two of you) or am still working on to understand. I’m specially happy about the statement there is still a place for scripted testing in testing, some people in the community seem to reject the use of any form of scripting whatsoever.
[James’ Reply: I have never in my career met any such person, and there can’t be any such person in the Context-Driven community, since to reject scripting as a technique is to promote best practice thinking.
There was never any question of there being a place for scripting. The question has been how to frame scripting so as to minimize the danger of it being a fetish. Another problem has been the persistent misunderstanding of what scripting IS. Even now, just based on your statement, I suspect what you mean by scripting is what I would call explicit scripting (rendering the script in the form of a tangible artifact).
I also suspect that when you have encountered people who resisted scripting, they probably thought they were resisting unnecessary explicit scripting. Perhaps you were asking them to formalize testing prematurely.]
I think depending on the situation scripting can be useful, as well as session based testing and freestyle exploratory testing. In my view only a small portion of the tests should be based on scripts, most important in my view is that testers always have to think carefully what way of testing is best in their situation.
[James’ Reply: You are stating the obvious, Jan. What’s not obvious is what do you mean by those terms you just used, and how specifically should context justify or discourage the use of any of those heuristics? That’s what takes study and time. Michael and I tried to show, in our post, that is it not an easy process to learn the fine distinctions that translate into professional responsibility in testing.]
I really like this blog post! But there is one niggly thing that is confusing me the more I read it… It’s this sentence: “By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice”
By that definition, wouldn’t all testing (exploratory) technically fall under that definition of “script”? We make choices based on information. Even information that we learn through exploration. The more information the have, the more we limit our choice down one path (based on risk assessments and our judgement). Our choices are always influenced by whatever information that we have.
[James’ Reply: The fact that we make choices within a field of influences does not change our experience of choosing. We still choose, and we perceive limits to those choices. To some extent we can change the structure of the systems we live within and thus change the nature of our choices.
I’m happy to say that all testing is scripted in some sense and to some degree. But we have a lot of control over the ways that it is so, and it is our responsibility as testers to use that control well.]
Unless I am now picking something up wrong(?)… but it seems as though the only testing that won’t fall under this “script” definition is testing that is completely random without taking any prior information into account (so as to not influence or affect our choices).
[James’ Reply: Just sit down and test something. Anything. Do you notice that you have a lot of choices available to you for how to do it? You can’t do everything at once. You choose what to do first, and second, etc. You are composing a sort of music of testing. There are rules within the structure of the music that constrain you, but there is still a lot of freedom.
See my replies to James Irving for more on this matter.]
I moved from the IT world tot eh software (ISV) world nearly 10 years ago, as a test engineer (I don’t have a computer science background). I work in an R&D division.
This entire time I have had to describe what I do, as my entire career has been what the software testing field has considered exploratory.
It might be internally developed code, it might be externally developed code.
More traditional software testers (ones that work for products teams and the like) have looked at me strangely as they are all focused on automation, specifications, and the like – and I have always been focused on the customer, the experience, the story, and understanding how the systems work and making them ‘right’ for the scenarios.
I explore, I apply my domain knowledge, I challenge the code, I challenge the developers, I challenge the scenarios, I challenge the marketing direction and intent – it is all relevant to the emotion and expectation of quality.
I find it interesting that it has taken this long for the business to wake up and recognize that this is valuable.
[James’ Reply: We’ve been pushing ET for a long, long time, but whereas wisdom requires long fermentation in special casks, ignorance pours from the sky on a daily basis.]
I know that my organization finds it valuable, and even strategic, but I have felt that the industry where my title lives has considered this ‘not’ testing.
[James’ Reply: The industry of FAKE software testing may consider it that way. But not real testers.]
Because I don’t live down in the automation focus that the industry has become.
Thank you for bringing awareness and hopefully waking up the business.
I can only hope that over time folks that work in the discipline of test will be rewarded as well as the folks that develop code.
Michael, James, thanks for this blog post. James already touched upon the subject that all testing is exploratory and that scripting can be a part of testing, earlier this month during the RST course I was on. This post clarified quite a bit.
However, I also got confused reading the paragraph below and I hope you could clarify.
“By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This does not refer only to specific instructions you are given and that you must follow. Your biases script you. Your ignorance scripts you. Your organization’s culture scripts you. The choices you make and never revisit script you.”
So besides formalized scripting, there are other control systems or factors at work when you do scripted testing.
[James’ Reply: There are factors that influence your testing and that are beyond your choice, whether you intend to test in a scripted way or not.]
I assume you mean to point out with this paragraph that biases, ignorance, company culture, etc. are examples of tacit scripting, ingrained in your subconscious? Don’t such factors always influence your testing unconsciously, whether it is exploratory or scripted testing?
[James’ Reply: Yes, they do and that IS scripting. Scripting means to substitute a programmed behavior for a chosen behavior. Any unconsciously programmed behavior is scripting. This is already accounted for in the formality continuum that I showed you in class.]
I’ve always pictured scripted testing as sticking to a formalized step-by-step description of a test procedure and would describe bias, ignorance and culture as the filter we apply to everything we see and do.
[James’ Reply: I think that’s a weak way of thinking about scripting, and that’s not what I talked about in the RST class you took. What you want to call scripting I would call explicit scripting. There is not difference in principle between the effects of explicit and tacit scripting. They have the same effect. There is a difference in terms of your relationship to the scripting, and the shareability and scalability of the scripting, though.]
Thus, any scripted test procedure would contain all of the bias, ignorance etc. of the creator of that test procedure. And the script would later be subjected to all of the biases etc. of the reader.
Paraphrasing: You script your biases, you script your ignorance, etc.
[James’ Reply: Sure. But even if you are given no explicit script at all, your work is conditioned by your mental models and habits and environment, etc.]
Anyway, I think my question is “Why include the part about biases etc. etc., if all of those also apply to exploratory testing?”
[James’ Reply: Joost, that is a nonsensical question! Please pay attention to the formality continuum. If you understand it, then your question will disappear. All testing lies on a continuum from totally unscripted (which does not exist, due to our biases, etc.) and totally scripted (which exists but is not testing). But just in from the ends of the continuum is testing that exists as some combination of scripted and unscripted elements. We talk about biases, etc. because we testers ought to be aware of the elements that might help or hurt our testing. Scripting is MORE THAN JUST FOLLOWING EXPLICIT INSTRUCTIONS.]
Thanks, James. Your answer, revisiting the formality continuum (and admittedly a good night’s sleep), made me realize that my misunderstandig stemmed from my misguided belief that scripted always meant tangible artifacts, not connecting tacit scripting to the formality continuum somehow. Once I let that sink in it started to make sense to me.
Isn’t it the difference between “42” as an answer on ultimate question of life, the universe, and everything or “is she the one” as an answer to the only question you’ve ever wanted an answer to?
The first answer takes a lot of time and computing power but is unsatisfactory.
The second answer takes an improbable journey through a lot of emotions and is fulfilling.
Thank you, both James and Michael for this excellent, thoughtful and thought provoking blog.
On my first read I agreed, disagreed, thought about implications, got various emotions, continued reading – in short, it was similar to what I like best – testing. Any disagreements from me are to foster discussion rather than having a rant.
I agree with the concept of skilled and unskilled ET and see it as important that this distinction is made so that the corporate world still sees a need for improving ET. Right now I’m not sure I see the depiction of scripted testing as implicit in exploratory testing.
[James’ Reply: Testers must learn to catalog the ways in which their testing is or could be constrained. This is not always easy to do. It’s a skill.]
“The essence of scripted testing is that the tester is not in control, but rather is being controlled by some other agent or process.”
I see value in the thought, however it’s not limited to scripted testing but any task a human being does.
[James’ Reply: Yes, that’s true for any task. But we are concerned with testing, specifically.]
While this sentence can be seen as the truth it may not necessarily be helpful. Also, by this definition scripted testing is every limitation that we can think of – outside parameters (project, culture, etc) and inside parameters (experience, skill, etc).
[James’ Reply: It’s very helpful! It means there is always some script somewhere that influences you, so be on the lookout for it, and try to gain more control over your scripts. It’s called self-awareness. We develop self-awareness throughout the course of our lives.]
I’m not sure that we should call the limitations of testing scripted. Especially for the internal parameters we often can’t explicitly express our experience, skills and what else has lead us to follow a specific route through an application or where we got a test idea from.
[James’ Reply: Why not? It works the same way as any other script. The difference is you aren’t aware of it, yet.]
I see a lot of value in thinking about the limitations of testing, both internal and external (to the tester). I wouldn’t connect it to scripted testing though. Scripted to me implies “known” in terms of approach (I plan what steps to take next), way(s) to follow (I know I can navigate one or more ways through an application) and outcome (I have an oracle helping me with an expected outcome) – for discussion.
[James’ Reply: Sounds like you have chosen not to acknowledge tacit scripting under the rubric of “script.” The problem I have with that is that the character of my testing is identical whether I am choosing to, say, type instead of using the mouse, or if I habitually and unconsciously type instead of using the mouse. My experience of it is different (so I will call it tacit scripting instead of explicit scripting) but the end result is the same: my testing is constrained. It’s a script.]
“You can be doing scripted testing that has not been in any way pre-planned (someone else may be telling you what to do in real-time as they think of ideas). ”
I would argue, that this can be testing by the person who is coming up with the test ideas. The second person acts as the “computer” in the old sense, as only giving input to the machine or extending the machine, in practical terms acting as a machine and as such is actually not testing at all (I just read you say so yourself in the last paragraph). Unless the second person comes up with either ideas or oracles as well.
[James’ Reply: If the person truly exercises no judgment or interpretation at all, we would say there is no testing happening by that person. However, as I explained, scripting lies on a continuum. It’s not all or nothing. I can tell you to “open a file” and not tell you which file to open. You are now scripted (if you follow my instruction) but not totally scripted. I have constrained you to some degree.]
The Rapid Testing Methodology is your baby so you can call it whatever you like, I may have found different terms but that’s fine as well.
Thanks for making me think,
Thomas
[James’ Reply: This is exactly the challenge. Whatever language you use, are you able to speak about levels and kinds of freedom? Are you able to defend your choices and your refusal to make choices using specific technical language? If so, cool. But what we keep seeing is people resorting to vague concepts such as “common sense” or “intuition” in situations where they could be usefully more specific and penetrating.]
James,
thanks for the reply. Agreed on all points, self-awareness is what I wanted to express,you hit the nail on the head.
And yes, tacit scripting to me doesn’t fall under scripting, yet, I may get around to it. Where I went wrong is that the self-awareness of ones own limitations is not handed out when signing on as a tester so I assumed everyone is doing that anyway. A blind spot on my part…
What I’m unsure of now is how this change is perceived outside the testing world. Quiet a few Project Managers understood that there are test scripts and this thing called exploratory testing which seems to get results. Saying we merge the two is likely to cause initial confusion with a lot of explaining to do (which is not really a bad thing). It will be interesting to see where that will lead.
[James’ Reply: What we DO is simple, from the perspective of management: get out of the testers’ way and let them get on with it.]
The next step in my view would be to discuss tacit knowledge not only with testers but either the whole team (in agile projects) or other teams like developers, etc as well.
Interesting times. Thanks,
Thomas
Hi James Hi James and Michael,
Thank you for the thought provoking piece. 🙂
After reading, my first reaction was to say out loud “YES!” (literally). 🙂
When i was first introduced to the idea/concept/approach/label of exploratory testing (early 2000’s), i recall searching for a/the technique that went with the label. The problem was that i didn’t understand what i was looking for and upon reflection i was searching for a stronger *control system* (i.e. fill out this template, see this result).
It took time to undo that *training*.
However, the more I tried and the more i studied, the brighter the light bulb in my head became.
At that time, the label – exploratory testing – was *helpful* in that I was able to begin talking about what i was doing and i was able to call it something even when others around me were dismissive of what i was attempting to do.
But i guess it wasn’t until I started to connect with you both (and others in the CDT school) that it dawned on me that what i’ve been doing is…testing (though less formalised perhaps).
A few years ago, I recall tweeting something like “there really isn’t agile testing – its more like we test in an agile context” and getting a little bit of *heat* for that (the samething occured working for a factory school consultancy and discussing formal testing). That realisation came after engaging with you guys, engaging with the local CDT community (and of course attending RST).
I think ET 3.0 makes things clearer though I suspect that those that hold fast to the “one sword” school may resist these new fangled ideas.
Just as thought, have you seen the mindset *switch/realisation* in organisations/projects whereby testing is seen as inherently exploratory even though formalised testing may be the method of choice (just curious as i can think individuals mostly)?
[James’ Reply: We’ve seen it among our clients. And we have enough clients now to feel that it is a trend.]
I tried an experiment yesterday with an important business unit where I’m attempting to have them rely less on explicit scripting methods in their scripted testing. Rather than use the term Exploratory testing, which is often met with much apprehension to those unfamiliar, I simply talked about testing. I went onto describe the formality continuum (only talked about explicit scripting methods) and made recommendations for a session-based approach. This was met with great enthusiasm, and I wonder if I would have had the same outcome had I started the conversation with the term Exploratory testing?
I’d like to take a contrarian view on a statement as I think there is a nugget worth discussing, but I haven’t yet found my own words for my thought. So I hope some disagreement will help put it into focus. In the last paragraph it is stated “If you’re only doing ‘scripted testing,’ then you are just doing unmotivated checking, and we would say that you are not really testing”. Earlier it is stated “By ‘script’ we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily).”
[James’ Reply: Yes.]
Most people define ‘script’ as a detailed, step by step procedure for accomplishing an objective.
[James’ Reply: Yes, but most people haven’t studied much. It’s important that the testing craft can be allowed to benefit from other arts and develop a sophisticated ontology and terminology.
My use of the word “script” is grounded in psychology (http://en.wikipedia.org/wiki/Behavioral_script) and organizational theory (http://www.jstor.org/stable/258285). I adopted it to speak of compulsory behavior twenty years ago, and I’ve been teaching it ever since. A lot of people in the testing field have been exposed to this usage, by now.
But also very important is that my use of the term is backwards compatible with the uneducated use of that word. What I’m doing is extending it past its folk connotations, but I don’t think I’m denying those connotations.]
While appearing subtle, redefining the term script actually has significant implications in the conversation. A ‘script’ still can be a detailed, step by step procedure and fit the new definition as it is intentionally defining a control system that, albeit usually temporary, limits a tester’s choice – e.g. you must follow the script.
[James’ Reply: Yes, we call that an explicit script, or a formal script.]
To me, what is interesting, is first, a script can, by design, limit choice temporarily, since it is used to define what should be done in what order to accomplish a goal. It becomes more of a plan (a plan to accomplish something) prior to test execution.
[James’ Reply: What is the purpose of distinguishing between a script and a plan? Seems to me a plan is a sort of script.]
The ‘planning’ phase is full of both constrained and unconstrained objectives or goals. If the planning of the tests is sufficient, then why would sole scripted testing only be unmotivated checking?
[James’ Reply: All you have to do to understand this issue is watch yourself test. You will find that the only way you can approach being truly scripted is to literally and completely stop caring about what happens during your work. Otherwise, you will be wondering about it and interpreting it, and that wondering and interpretation make you able to detect unexpected problems and is not and cannot be scripted.
I think when you wrote that statement you imagined yourself with a pretty good idea of what you want to do, and you COMPLETELY DISCOUNTED all the lower layers of thinking like a tester that MUST accompany competent testing work.
Part of learning how to think like a tester is to take the moment-to-moment practice of reasoning and curiosity seriously.]
It’s not that I support solely scripted testing, but many people do.
[James’ Reply: No educated person does. Let’s be clear about this: I’m not going to have my ideas or expertise held back or criticized because there exist people who know nothing, who want to know nothing, and think silly thoughts because of that.
You can’t show me “solely scripted testing” because it cannot exist, because mechanical action is not testing.
My attitude about this is completely inline with all of Science, by the way. Scientists use computers to help them perform science, but no one says that what computers do is “sciencing.” Without scientists, and with only a program running, no socially recognizable science can happen.]
If the foundation for arriving at those scripts is sufficiently varied and unconstrained, then the result shouldn’t be categorized so negatively.
[James’ Reply: Why don’t you propose a specific example, so that I can show you how hollow it is? Go ahead. Prove me wrong.]
When I first read the statement, it invoked interpretive feelings that unmotivated checking is bad therefore sole scripted testing is bad.
[James’ Reply: Unmotivated checking is bad, yes. It’s idolatry. It’s ritualistic behavior.]
I think the additional area of exploration on this topic is buried in the relationship between planning and executing. As I write, I’m brainstorming, that perhaps the freedom of thought – that enables exploration and learning – needs injected into the process at some point to ensure testing has the best chance of yielding fruitful results. So perhaps sole scripted isn’t all that bad provided unconstrained freedom has had enough exposure to avoid stunted results.
[James’ Reply: I don’t know what those words mean. Provide an example.]
My final thought on this piece is formed around an analogy of testing to scientific discovery and experimentation. In science, the freedom of explore typically happens with a hypothesis.
[James’ Reply: Exploration long precedes hypothesis. Or you could say that lots of little micro-hypotheses happen along the way before a big one comes into view.]
The experimentation tends to be scripted to ensure control over variables and it facilitates repeatability.
[James’ Reply: That is not how science works. That is a popular myth about science. Just EXACTLY as in testing, scientific research also begins informally before being formalized. Even highly formalized experiments are riddled with informal and subjective bits and pieces. See Harry Collins work on that. I recommend Changing Order (which is all about the myth of reproducibility in experiments)]
I don’t know that issues exist with sole scripted testing in as much as it exists with constrained thinking throughout the testing process.
[James’ Reply: I don’t know what that sentence means.]
My second, and shorter thought, is this. Early in the life of ET maturity and adoption (1.0-2.0), many folks spoke of the benefits of ET. New ideas often need to be ‘sold’ based on those benefits. As is suggested, if testing, by its very nature is exploratory and scripted testing is a foreign element, then it makes me want to work backwards-taking a cue from Morris’ comment above. What then are the benefits of scripted testing?
[James’ Reply: Scripting has both benefits and hazards. It helps create consistency and comprehensibility in our testing, even when the scripts are unconscious. Scripts (of any kind) are the media by which culture is maintained. Scripting makes certain things much more visible. Scripting helps me systematically search a product by progressively locking some things down so that I can search them in a sustained fashion. My progress of testing is largely conditioned on the things I choose to stop choosing, and the things I can’t help but accept.]
By the way, I appreciate the time and thinking you guys put into this topic and appreciate the freedom to pose contrarian questions to elaborate and advance the discussion.
[James’ Reply: I’d like you to record at least five minutes of human testing while following a script– the most scripted you can be– then break it down moment by moment and describe what you did. Then I will look at that and show one of two things: A) that is wasnt’ testing, but rather mechanical behavior, or B) that you did engage in behaviors that were not in your explicit script, and represented interesting thinking and learning that we do not know how to describe in procedural terms, and therefore ought to be treated as exploratory behavior.)]
Instead of writing well thought out, point/counter-point responses, I prefer conversations. I’m doing my best to do that via posting. The reason I make that statement will become evident later on.
[James’ Reply: Okay. I also prefer conversations.]
First, I want to make sure that people understand that I do understand the point that is being made. I am adopting a contrarian view simply for the sake of a exploring a few concepts. I am not against testing being, “Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”
I fully support and understand the statement. I am also a member of the CDT community. I do not support concepts such as 100% automation, or checking as a replacement for testing. I think it is important to clarify my views apart from the discussion point of view I am taking. In many ways, I am attempting to channel typical tester behaviors and thought processes in attempts to make public a conversation that may not otherwise occur. After all, given your stature in the community, it can be daunting to ask a question for fear of being made to look stupid or looked down upon.
[James’ Reply: I understand. Thanks.]
To the first part of the response dealing with the definition of ‘script’. Many testers use the much narrower definition as a way of specifying a sequence of actions to be executed (to paraphrase ISTQB). As an aside, like you, I believe it is important to infuse not only the testing industry, but even our personal lives with knowledge from other fields. In the narrow sense, a script specifies a sequence of actions. A plan tends to become plan that which outlines what (narrow) scripts will be developed. In the broader sense, a plan is a script and a script is a plan. Both lay out some sort of path of action, or ideas, to be followed and are constraints on choice. My only point is distinguishing between to the two in my original response is due to popular convention. For what it is worth, I know that is wrong, but still bring it up for those who don’t yet get it. I think it cannot be said enough that a script is anything, whether written or not, that suggests a sequence that also is, or is expected to be, followed.
[James’ Reply: A script is not necessarily a sequence. A script, in the general sense, includes such things as police tape surrounding a crime scene that forces a pedestrian to go around. We could use a whole bunch of different words for different flavors of constraints upon action. That can be okay. But we also need one word that can be used to refer to the matter in general. I settled on “script” for that, as other people, in the field of psychology, have similarly extended the term.]
That alone goes against the very nature of learning through exploration and experimentation. Hence, the statement that scripting is a foreign guest in the house of testing. That’s not to say there isn’t a place of detailing a sequence of actions. Sometimes you need those to accomplish the broader goals-such as I don’t know how to get to where I want to go from, so I follow a path laid out for me that I have confidence will get me there. As an aside, I found the Gioia and Poole paper interesting; especially the concepts of weak and strong scripts coupled with our tendency to convert behavior into prototypic scripts. Especially, since I just stated “After all, given your stature in the community, it can be daunting to ask a question for fear of being made to look stupid or looked down upon.” This is nothing more than a strong prototypic script, that I am applying, that says, if you don’t view yourself as an expert in a field, when you talk to one, you best not challenge them publically, or you may be put in your place. [Hence, the more conversational and stream of consciousness flow to my writing here]
To another main point – mechanical actions or behaviors. I’m glad you used that phrase because 100% mechanical action fully constrains choice with zero opportunity to explore and experiment. The extent to what you can learn through exploration and experimentation with the system is limited to the ability to actually implement the mechanical behavior. However, you have to also acknowledge that once you have done an action, you cannot experience it for the first time again, you can only repeat the action.
[James’ Reply: In a sense, every action is experienced for the first time, since exact repetition is not humanly possible.]
Therefore, mechanical actions executed by people have very limited learning opportunities. To bring it back to the definition “Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.” Mechanical actions are very limiting and in some circumstances are preventative. Besides, we are not really built for consistent, exact, and, repeatable behavior; but machines are. By extension, the reason 100% automation as the sole testing effort is bad is because it is nothing more than checking and that is not testing [I’d suggest people read your interviews and yours and Michael’s blog posting on the topic].
The one point I made in the original response dealt with sufficiently varied (narrow) scripts being over characterized as negative, you characterized as ritualistic behavior. I too agreed then and now. I was toying with the idea that if a ‘mastermind’ laid out every possible path they want covered, they could script it and have others do it, and it would still be effective.
[James’ Reply: If you actually imagine that happening in detail, you should see how it can’t work that way. Let me put it this way: If you tried to lay out such a plan to a person driving a car, and that person were only going to follow your script and do nothing that you did not tell him to do, you would not be able to construct a plan that actually did lay out every possible path for driving. How could you specifically call out all the possibilities? You wouldn’t even BEGIN to be able to describe them. You would have to refer to the driver’s general driving skills. There is no other way. Either the driver follows a simple explicit script and drives stupidly, OR the driver uses skill and doesn’t follow your script, OR the driver uses skill and follows a very high level script that leaves a lot of work to his own discretion.]
The reality is the moment a ‘mastermind’ handed off his scripts to ‘testers’, through weak prototypic scripting, the ‘testers’ are prevented from exploring and learning. The reality is that the mastermind is no longer an active participant; therefore, the mastermind is prevented from continued learning. Therefore, testing stops at that point. [I do realize that any good tester will take any attempt to explore and learn, even when governed by a script] As an industry, we’ve always been plagued with the test monkey issue – that being we need to ensure anyone can execute our scripts AND anyone can do our job. That thought process is poisonousness and it actually impedes testing.
[James’ Reply: Yes.]
The thoughts of yesterday are more crystalized through this exchange. In summation, I think the message I was looking for is this:
Scripts are very broad and include behavioral, instructional, and other forms of sequencing actions or ideas. Scripts are constraints. Constraints limit testing. Limited testing means limited knowledge.
[James’ Reply: Well, some scripts represent automated behavior– programs. Those can be very specific, of course. They can help testing, even. But they are not testing, itself.]
I apologize that I haven’t responded to every question and suggestion you raised. There are a lot of great side topic conversations that could be had.
Joe
PS: I’d rather have had this as a conversation, but thought there was value in making it public and documenting it.
[James’ Reply: I agree. Thank you.]
Hey,
Lots of questions are about script(s), scripting and scripted testing. Do you have a definition of those?
BTW, I counted variants of the word script and explore in this blog post (and comments). Here are my results:
Script 42
Scripts 39
Scripting 77
Scripted 100
Explore 6
Exploring 4
Exploratory 85
Feels sort of ironic that scripts “won” the word count 🙂
[James’ Reply: How is that ironic? It’s completely in-line with what we are saying. We think exploratory testing is normal, therefore we should not need to talk about it.
Didn’t I define “script” in the body of the post?]
Hi,
I just want to share what I got here. As a responsible tester, I want to learn to recognize, be aware of and finally control the scripted part of my testing to my best ability. Paraphrased, I want to learn to control my testing consciously on all levels, also those that I seem not to control, don’t know I could control, and never completely can.
[James’ Reply: Okay.]
To me, from start of my “professional tester” experience in 2007, no totally scripted testing ever existed (Yes, it would not be testing, but checking, I found out later). My eyes were burning and brain melting when I was expected to “just test” (=do fast & partially detailed & unmotivated checking) with the explicit scripts (which were generally outdated & incomplete, and the tool which they were stored and used with was tedious). So I had to put my feelings aside and try to control my exploration within the test areas where the explicit scripts directed me, that was the best I was able to do then. Sometimes I did well, sometimes not so. I found bugs usually when engaged with the product and not concentrating on “checking boxes”. I felt forced to say “I kind of found these and these problems when “running the test cases””. I slowly started to learn look outside the obvious like scripts, plans and specifications and seeing bigger pictures. So I slowly started having glimpses what caused what, in people and in the systems used. That is still the most interesting part of being a tester, to me.
[James’ Reply: Cool.]
When I found context driven testing in 2009, many odd feelings started making sense, but new arose and still it was and has been a struggle. One of the biggest problems has been finding words and vocabulary I would understand, and even harder problem, to learn how to explain testing issues to colleagues and others in and around business. These years and this post again closes some dark gaps in my thinking. But it also leaves a lot of thinking for future.
Alienation disturbs me sometimes in your otherwise excellent writings, like this one. I mean for example sentences that are phrased “we this, we that” vs “you this, you that”. I think it should be taken as passive voice, but it reminds me of the problems with terms non-sapient and sapient.
[James’ Reply: I don’t understand the problem. That is not passive voice. In passive voice, it is ambiguous who is doing the action.]
In this post, I mean especially about this part that others also seem to have struggled with, but maybe for different reasons:
“By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This does not refer only to specific instructions you are given and that you must follow. Your biases script you. Your ignorance scripts you. Your organization’s culture scripts you. The choices you make and never revisit script you.”
Why not “influences our testing”, “our ignorance scripts us” etc? To say directly, the tone in that paragraph feels a tad “scripty” to me, as if these constrains only concerned the reader. My own biases obviously cause this feeling but I hope my point makes sense to you.
Anyway, big thanks for this great writing.
[James’ Reply: This post was written by me and Michael Bolton. That’s why we wrote “us” and “our.” We were referring to ourselves as authors of the post.
We wrote “you” to imply that we are talking about you, the tester. We could have said “we,” etc. But it sometimes I like the immediacy of referring to you, the reader.
Is there a problem, here, for you? What are you worried about, specifically? You mentioned alienation. I don’t know how any of this alienates you from anything.]
ya It’s also a great example of how we testers need to evolve and continue to learn. very informative and very useful.
Well written post, thanks.
[James’ Reply: Thanks. I wish I could say the same thing back to you about your comment. Unfortunately, it looks like gobbledegook. See below.]
2 things, in summary:
*doing quality for a product is about much more than just finding bugs.
[James’ Reply: One does not “do quality.” I’m going to presume you mean “produce a high quality product.” If so, yes. I agree that doing that requires more than finding bugs. It requires, for instance, developing the dang product.]
*testing over time is important, and for that you need scripts and effective automation.
[James’ Reply: Testing over time is often important, yes. For that, I do NOT necessarily need scripts and automation, but these things– in some form– can be generally useful. I suspect we are about to disagree about the form.]
I like that you’re working to improve practice of manual testing, but IMO in your zeal you’ve lost sight of the fact that measuring product quality is much more than just finding bugs.
[James’ Reply: I don’t recognize “manual testing.” I’m not trying to improve that practice, since it doesn’t exist as such. I’m working to improve TESTING. We testers use tools, just like programmers use tools. You don’t call programmers “automated programmers” or “manual programmers” so don’t call me that.
I’m not measuring quality when I test and neither are you. You can’t measure quality. What you do is assess it. But I think a better term is INVESTIGATE. I investigate the quality of the product. As I do so I primarily look for bugs, since malfunctions are the primary concern of my clients.
I have not lost sight of the fact that testers do more than find bugs, though. I have blogged about that. So, I don’t know what you are talking about. Do you even know what I have and have not sighted?]
Actually, finding bugs is very important but subservient to the overall picture of measuring quality. Without measuring quality in the big picture, you can’t make a good decision whether to ship your software product (in whatever form you do that).
[James’ Reply: If you mean that we investigate the status of the product pursuant to determining the risks associated with releasing the product, yes that’s what I teach testers to do. Finding bugs is the main part of that. Everybody assumes, already, that the product is worth shipping. The important part of our work is to find reasons we might NOT want to ship it.]
Scripted tests (or, test “cases”) are essential to measuring the quality of your SUT over time.
[James’ Reply: That’s total bullshit that you must have read on the back of an ISTQB matchbook. I’ve been in this business a long time, kid. You will have to do better than that.
For one thing, I don’t measure quality. Secondly, your term “scripting” is undefined. So, this statement is utterly vacuous. Or maybe you will say “you know what I mean!” No, I don’t. There are so many things this statement could mean. I’m not going to pick one at random. Then to tell me that this empty idea is ESSENTIAL? Wow.]
I’d even call this a “best practice” (although I know you hate that term) because you have to measure quality with the time factor; if you don’t you’ll never know when or whether to ship.
[James’ Reply: I’m not surprised you would call it a best practice, since I’m starting to get the idea that knowing the meanings of words is not your priority.]
Therefore, manual testers will be running scripts part of the time. The scripts can be “automated” but there’s a significant risk there of leadership thinking that no manual tester needs to look at a GUI or web page because it’s been “automated.” People doing manual testing can notice and characterize all sorts of issues that automation can’t or is better off not even trying to look for.
[James’ Reply: I don’t know what you mean by “running scripts.” Do you mean performing formal checks? Yes, people interacting with the product will see a lot of things that are totally unspecified and unspecifiable in scripts.]
And, testers can or should do exploratory testing part of the time, off-script. This is valuable as well, and very good at finding bugs.
[James’ Reply: No, testing IS exploratory. Fundamentally. Scripting is a word used to describe a wide variety of techniques whereby the tester cedes some control of the testing. From your wording, I suspect you are referring to detailed formalized test procedures. This is a famously abused practice with limited applicability. It finds few bugs. I discourage it, typically.]
So, I think that by trying to rid the world of the phrase “exploratory testing,” without replacing it with anything, you’re throwing the baby out with the bathwater. There’s an important difference here.
[James’ Reply: Perhaps you didn’t read our post. Testing IS exploratory. It’s all baby. There is no bathwater.]
I know you have a more detailed definition for this, but what you call “exploratory testing” sounds like an activity called “bug bash.” The activity and results seem similar.
[James’ Reply: That is a laughably wrong characterization of my test methodology.]
Actually, I think you guys enjoy semantics and arguing changes to usage a bit too much for your own good 🙂
[James’ Reply: We believe in knowing what the fuck we are talking about. You should try it. But I warn you. It’s not easy. And you can work for twenty-five years on a sophisticated and nuanced test methodology, based on cognitive science and epistemology, and hone your terminology to a sharp edge, only to have some kid declare that it’s basically a bug bash…]
I hear you that it helps you express concepts to your clients, but it tends to knock people off balance. Software quality takes many different approaches, depending on timing, SDLC model, SUT, etc. IMO you’re trying to unify and present a structure, but that doesn’t work for the limited scope you’re considering here i.e. manual testing techniques and finding bugs.
[James’ Reply: You have no idea what you are talking about. I mean that literally. You are talking about Rapid Software Testing and you don’t know what’s in it or what it covers. I’m a programmer and I write code routinely to help me test, yet you seem to think I am speaking of “manual” testing. You are stuck in a 1995 understanding of testing. Please see Mssrs. Dunning and Kruger for debrief.]
Interesting stuff.
I’m a fan of the continuum and often use it to talk about the fact exploratory testing isn’t purely ad-hoc in nature: there is a degree of formality with things like Session Based Testing adding levels of process. I also couple it with a “context continuum” that couples formality of testing with formality of context / environment. Clearly a formal environment doesn’t necessitate “formal” testing, but it tends to be the case. Not sure if it is entirely accurate to your thinking, but seems to work nonetheless
Tacit and Explicit Knowledge is the next book on the list!
Hi Jim:
Somewhere on the continuum (that no longer exists) between exploratory and scripted testing, we find a sub-continuum of “predesigned testing”.
[James’ Reply: The continuum still exists. Except now we see testing as an exploratory process into which scripting intrudes. It’s not clear to me how what you are calling “pre-designed testing” is different than scripted testing as we define it.]
At one end (closest to exploratory) we find short lists of specific input values or intentions e.g. maximum value. Further along, we find these lists extended with required responses. Entry into this sub-continuum is the first point where some form of test design documentation is required.
[James’ Reply: How can test design documentation ever be required? That is to say, the formality concept never stipulates the form in which the scripting is to take. To say that test design documentation is required is like saying “test design documentation on vellum, written in India ink, is required.” The choice of whether and how to document test design is not a matter for test methodology, per se. It’s an administrative decision.
The testing issue is whether and how to control a variable. To control a variable is to formalize. To keep that open is to informalize (or merely to remain informal).]
Crossing this transition point enables two things that the points before it do not.: (1) the written communication of required behavior i.e. requirements
[James’ Reply: Written communication is always enabled (by the skill of writing). I suspect you mean written “receipt” of information. But again, that’s not a matter within the scope of test design.]
and (2) a clear demonstration of understanding about required behavior e.g. by a developer. Continuing along the continuum to detailed scripting disables these two activities.
[James’ Reply: I don’t see how clear demonstration of understanding has anything to do with writing, so I’m confused. Writing is a tool we use to solve certain problems. It’s a heuristic. It guarantees nothing about understanding. People copy and paste text they don’t understand, as you know. Two people can understand the same words differently. Just look at the institution of government to see ubiquitous cases of different people understanding the same texts differently.]
If scripting (an end point) becomes a guest in ET 3.0, what happens to the range of predesigned testing activities (and their associated benefits)?
[James’ Reply: You should read the post again. We include a careful definition of scripting. It seems you think by “scripting” we mean detailed written procedures. Absolutely not. When we say scripting we are speaking of essentially any constraint on a tester’s behavior. This has been the way our community has used the term for many years.
So the answer to your question is: nothing happens to scripting, in all or any of its forms. If you say “let’s test over here instead of over there” then you are suggesting a bit of script. This is absolutely fine. You have introduced a constraint. Let’s be aware of how we are constraining ourselves as testers and why we might benefit from doing so.
But let’s not confuse scripting with testing. Testing is the search; the thought processes that enable that search– and that cannot be reduced to an algorithm.]
[.. we see testing as an exploratory process into which scripting intrudes.]
This perspective ignores two other modes of (what many people call) testing, limited release testing (e.g. beta) and built-in testing. These two modes have nothing to do with searching or thought process.
[James’ Reply: That sounds like complete nonsense to me, so I assume I don’t understand what you are trying to say, Dave.
Beta testing, inasmuch as it actually is testing, involves a whole lot of thinking. The users are thinking in the ways they need to in order to put the product to use. Their process of use is exploring the phase space of the product. People monitoring them have to think about who to deploy the beta to in order to get interesting coverage. Then they have to sift through and interpret the feedback. All this is an exploratory process, to a large degree.
Built-in testing is not testing. It is output checking. It’s important not to confuse checking with testing. Checking is part of testing, of course, but we testers do far more than that, just as authors are more than mere typists, and programming is more than compiling.
Built-in checking, when done well, is part of a test process. Both beta testing and built-in checking are most definitely search processes (if something we are looking for moves through the part of the space that is being searched, then we find a problem, don’t we?) and both involve thinking (you have to design the checks, don’t you? you have to interpret the results don’t you? Surely you understand that, Dave, so what are you talking about?)
I don’t know why you said that our claim about testing being an exploratory process into which scripting intrudes leaves these things out, though. It certainly includes both!]
I just realized a possible source of my confusion. When you use the word “testing”, you may be restricting its use to what “testers” do. You may have different labels for “unit testing”, “acceptance testing” and the two modes mentioned above or all these are implicitly excluding from consideration.
[James’ Reply: I use the word testing to mean testing, which is something only humans can do, whether you call them testers or not. I use the term “checking” to refer to algorithmic evaluation processes, this is completely consistent with the history of the term testing and the term checking, although no one had recognized the need to establish a more formal usage that distinguished the notion of fact checking from that of theory testing. A few years ago, Michael Bolton and I realized that we need to have separate terms for these things. Please see my blog post “Testing and Checking Refined” for all about that.
Programmers realized the need to distinguish between compiling and programming, many many years ago. I’m just establishing the same protocol for testing. Imagine if we didn’t use the word “compile” and just used the word “program” for both the human process of programming and the machine process of programming. The confusion would be unnecessary and silly.
Now here you are, apparently a victim of that confusion. So, I recommend that you distinguish between testing and checking from now on. This requires you to take seriously the differences between what machines can do and what tools can do.]
[When we say scripting we are speaking of essentially any constraint on a tester’s behavior. If you say “let’s test over here instead of over there” then you are suggesting a bit of script. This is absolutely fine.]
Defining scripting as any form of constraint on a tester’s behavior, we can distinguish a “persistent script” from a “transient script”. Organizational or project testing standards or personal checklists are persistent scripts (that always intrude). We can also distinguish two types of persistent scripts: comprehensible (to non-testers) and incomprehensible.
[James’ Reply: Sure there are lots of interesting ways we can distinguish. I also use “spontaneous” vs. “deliberative” and “hard” vs. “soft”]
[I don’t see how clear demonstration of understanding has anything to do with writing, so I’m confused]
A tester/developer/customer might be given a testing assignment and asked to create a comprehensible persistent script to demonstrate understanding of the subtleties of the application. Creating such a script will likely raise questions as to exactly how the application should behave in specific situations.
[James’ Reply: That is a heuristic for achieving understanding. But it is not a method for demonstrating understanding. As you well know from writing things down and agreeing on them with others, you may achieve that agreement and then later discover that you didn’t really agree and did not have a common understanding. I know you have experienced contractual disputes before. Just reflect upon that.]
Finally, I do not see the value of a model that makes exploratory testing primary and all other forms invisible or secondary. I value exploratory testing as one of four modes of test that co-exist, have specific characteristics, and supplement each other. These testing modes are verification tactics along with technical reviews, analysis, and measurement. Testing alone is not sufficient to verify quality attributes such as security or safety.
[James’ Reply: The way you just used the term exploratory testing indicates you do not understand what that term means. Let me remind you that I popularized the use of this term before anyone was using it (even the guy who coined it had left the industry). I’ve been explaining this for many, many years, and my understanding of it has evolved over time.
All testing is exploratory. If you try to show me testing that is not exploratory, then I will show you that you are not telling me the truth (by revealing the exploration you were doing without knowing or admitting it) or I will show that what you were doing cannot fairly be called testing (because you have deliberately and for no good reason cut it off from the learning processes that make testing powerful).
We are not in any way making scripting invisible by saying it intrudes into testing. We are in fact trying to do the opposite– we want to highlight it more. We are certainly saying it is secondary. It OBVIOUSLY is secondary, in the same way that programs are secondary to programmers. Killing a programmer is a much more serious crime than killing a program!
There are strong parallels between what I’m talking about and the history and practice of many other thinking disciplines, including science itself. Please take a look at any given book about how scientists work and what it means to “do science.” It is a human process, David. It is not an algorithm.]
For anyone still reading, apropos of some of the conversation about scripting, I wrote a blog post about it.
http://www.developsense.com/blog/2015/07/on-scripting/
—Michael B.
Love it. Find it interesting. Have problems the etymology 🙂
[James’ Reply: Okay.]
To test reminds me to a lot of things, eg “putting a theory/hypothesis to test”. Rather often this happens “by constructing an experiment”. Here is you script – your bias.
[James’ Reply: It’s far more than constructing an experiment itself, it’s all the social and intellectual context of doing so. See the book “Changing order: replication and induction in scientific practice” by Harry Collins for more on that.]
My etymological dictionary says that the word comes from “examine the correctness”. The latin “testor” describes “one who testifies”. So, from the etymology point of view: test –> Checking
[James’ Reply: I guess if you ignore the entire history of the word test and just look at the Latin, you might think that. But let’s not ignore the history of this word in scientific use.]
On the other hand, explore stems from explorare, “to investigate, to examine”. To me that is the learning part.
[James’ Reply: I agree that exploring is a word that means learning. But more specifically, it means self-controlled learning. If that’s not in your etymology, then it must have connotations that go beyond etymology.]
So, instead of retiring “exploratory” one could retire “testing”
But in the end, this is only a play with words.
[James’ Reply: Right. So, let’s not merely play with words. But let’s use words in reasonable, powerful ways.]
Great fun to read. And, on a totally different subject: I hate to admit it that I got the first invite on “using 29119 correctly” yesterday – yuck. I hate it, what I am wrong
[James’ Reply: Folly, thy number is 29119.]
For small, static teams this makes some sense. But I can’t see how it scales.
[James’ Reply: Oh? Do you also believe management doesn’t scale? Do you believe that lawyering or doctoring doesn’t scale? Mechanical engineering? Which of the many many skilled occupations where people are required to think their way through problems do you see “not scaling?”
Of course, you should answer this: None of them scale AND THAT’S OKAY. “Scaling” is not an issue. Scaling is a fantasy. Testers and testing are not fungible. We are not commodities. Show me an activity that smoothly scales and I’ll show you unskilled labor that can easily be replaced by robots.]
As projects get larger and more complex .. it is unlikely that you will find high quality testers, able to think critically and who enjoy this type of work, to place in all he areas.
[James’ Reply: Of course the same is said of all other skilled workers.]
Some areas will be high quality, others will suffer. Because you don’t have the artifacts, you are left with discussions to improve quality, but no ability to provide assurance to customers that even though the last 2 releases had bugs, you are sure the next will be golden.
[James’ Reply: Why don’t you have artifacts? Where in my post did I tell you not to create artifacts? I think I am saying testing cannot be encoded, so you can’t have artifacts that “do testing” or that “are testing” but of course we can and do take reasonable notes and maintain reasonable data and toolsets. Perhaps you think we don’t do this because you have assumed I am talking about irresponsible testing?]
While you may have 1-2 really good testers initially, as the company grows to 15+ testers, most managers will pull off the 2 good ones to mentor the others (hope you found 13 great people with the passion and capability to do that work)
[James’ Reply: The key is, as any elite military organization can tell you, to have a system for training people. Skilled testers train the next generation. We reproduce ourselves. Institutionalizing this is part of the job.
How do you think good testers come to be? By accident? By miracle? I create them, systematically. Any consistently good organization must do this.]
Also, with the average length of employment for the typical engineer at 1-3 years, thats a lot of domain knowledge walking out the door.
To me its a multi-pronged approach. You need to structured test cases (and some people are really good at writing these) to provide assurance that the core workflows and random bits of the application still work.
[James’ Reply: There are no best practices. There is no way to say what any project needs without understanding the context of that project. Furthermore, all testing is structured, so I assume when you say “structured” what you really mean is formalized. Of course informal testing is structured in many ways, and an appropriately skilled tester manipulates his structures according the needs of the situation. The decision to formalize should not be made lightly or prematurely, because of the danger of over-focusing.
Having said that, I typically use guideword heuristics (such as the Heuristic Test Strategy Model) and prepare a product coverage outline to help guide and report the testing process. Those particular structures are quite flexible. I may do many more things, too. I guess the next biggest thing for me is to produce test data and to line up my test tools.]
But you also need your free thinkers and those focused on Exploratory Testing and the performance piece.
[James’ Reply: Those who are not free thinkers cannot be responsible software testers. In RST methodology we call unfree thinkers who help test “supporting testers” and expect them to do no important work unless under active supervision. A robot– or human pretending to be a robot– is not self-managing.]
A smart manager will understand these roles (along with automation) and build a layered QA organization, instead of a cookie cutter mold.
[James’ Reply: A smart manager knows that excellent testing cannot be scripted, as such. Scripting plays a strictly supporting role in testing, and we must bear in mind its dangers.]
I might be a bit late to the party:
I embrace ET 3.0 as somehow it felt since a long time weird to me that some testing should be exploratory and others not.
Testing is the exploration. By that some sub-tasks might be less explorative and others more. But finally I as human explore a product. No matter e.g. how much support I have by tools.
By that I see many of this “test automation” advocates overlooking that even just the development of machine checks (especial those on system/UI level) include much exploration. As the “test automaters” often the first people exploring the system more in depth.