Today I am writing about the Context-Driven school of software testing. This is a community that initially coalesced around Cem Kaner’s Los Altos Workshops on Software Testing (the LAWST meetings), in the late 90’s. I am a founding member of that community.
In the last two years, more people have joined the community of active context-driven thinkers, helping to develop the notion of what “context-driven” means for testing. I feel like we have a critical mass.
Schools of Thought
In this context, “school” relates to this definition from Merriam-Webster’s Dictionary:
4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) the Aristotelian school; also: the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also: the shared opinions or behavior of such a group other schools of thought
This definition is a good start, but we are referring to a stronger sense of school: a paradigm; an ontology; a way of organizing, experiencing, and thinking about the entire field. In other words, in the way I use the term, it’s not possible to belong to more than one school of testing at a time. If you tried to be in two schools at once, the school I think you would actually belong to would be a new school called Confused.
The value of the school concept is that it allows us to make progress in our work without constantly having the same fruitless arguments with people who are pursuing a fundamentally different view of the field.
I see the testing world in terms of six major schools of testing. These include the “breakthrough� schools of Analytical, Control, and Test-Driven Design, and the “pragmatic� schools of Oblivious, Factory, and Context-Driven. The breakthrough schools generally hope to reduce or eliminate the need for independent testing. The pragmatic schools generally accept the idea of dedicated software testers and deal with the problem of how testers should get the job done.
To the best of my knowledge, only my school has named itself. Bret Pettichord, Cem Kaner, and I, came up with the names for the other schools. More on the first five in another post, right now I want to focus on the world from the Context-Driven school perspective.
The Context Driven Basics
Cem Kaner came up with the term “context-driven”, in 1999. Cem and I drafted the principles of the Context-Driven School in 2001. Context-driven is all about people solving problems.
“Context-driven” means to match your solutions to your problems as your problems change. The principles of my particular flavor of Context-Driven methodology can be found here.
Consider this rant:
Stop memorizing things people say are solutions, then applying them without understanding how or if they actually solve the problems you face.Don’t pretend to solve problems– really solve them. Don’t solve pretend problems– really solve real problems. And you know what? Those problems are going to shift and change, sometimes day by day or minute by minute.
If you think you can do good work by following some guru’s scripted Best Practice, you might as well blindfold yourself, take a dozen sleeping pills, and drive the Daytona 500. But the nice thing about driving blind in the Daytona 500 is that you know when you’ve crashed and burned. The hard thing about software testing is that you can do terrible work for years and not realize it until some uber-bug that you could have and should have found nails some poor user right in his “GUI components.” Even then, you’ll probably just blame the developers for putting the bug there in the first place.
If you want to take responsibility for the quality of your own work, practices won’t save you: what saves you is your skill, plus your ambition to use your skill. I’m not saying you should ignore practices, nor should you hate them. Just don’t worship them.
To learn to test is to learn to solve testing problems in every situation (we call them contexts) that you encounter. If you do that, your process will unfold naturally, your methods will be fluid, your practice balanced and flexible. So ask yourself, do you have the skills you need? Ask yourself, is your testing fitting your context?
(That’s the end of the rant.)
I like ranting, sometimes. In a rant, I can bring the world into sharp focus. I can say bold, definite things. Unfortunately, the rant above, while advocating the context-driven attitude, is not itself an example of that attitude. Do you see why?
Let me explain…
Four Attitudes Toward Context
When I say “context” I mean the totality of a situation that influences the success or failure of an enterprise. If I want my testing to be successful, I need to behave, somehow, in harmony with the aspects of my world that make testing success possible. For instance, I’m not going to introduce the test technique of cause-effect graphing into the context of a team that has no idea what that means or how to do it, and no time or resources to figure it out.
Everybody copes with context. I see four interesting flavors of coping:
- Context-Oblivious
To be context-oblivious is to be unaware of the relationships between the way you work and the many aspects of the context in which you work. A context-oblivious practitioner is aware of doing things, but can’t tell us why it’s done that way. This is typical of novices, of course, but also of experienced people who do not like to think about methodology (a methodology being a system of methods; a method being some way of doing something).
Completely context-oblivious testers simply don’t read blogs like this. What would be the point?
- Context-Specific
To be context-specific is to be adapted to a context and to know that you are adapted. You know what problems you face, and you solve them. But those problems don’t change, and therefore you don’t need to adjust your solutions.
Most of us Americans would say we know how to drive a car. When we say that we are thinking of a particular context: driving on ordinary public roads in ordinary weather in ordinary cars. Any ordinary car driver in America would know that our driving skill in that context probably does not carry over to driving a race car in the Daytona 500 (even without a blindfold). Still less does it carry over to “driving” an airplane, even though there are many elements in common between airplanes and cars.
I don’t need to learn how to drive a race car, because I’m not in that context, and I don’t even need to learn how to drive in unfamiliar situations, because I simply don’t drive in unfamiliar situations.
It’s easy to believe that being context-specific is good enough, but beware: What if you suddenly lose half your team, or part of your testing get outsourced, or features of your product change radically, or you have to adapt to use a new tool. You may not have the option of adapting to one context only.
- Context-Imperial
To be context-imperial is to keep your solutions the same while striving to change the context around you to fit those solutions.
A classic example is when a tester refuses to test without a “complete specification” instead of learning how to test with or without a spec.
Another classic example is the typical testing consultant who asserts we should all use some Best Practice he has in his mind. When it doesn’t work, he says we must not be using “engineering discipline” to follow the golden path to success, or maybe we don’t have “upper management support” to force us to do the right thing. The solution can’t be wrong, so the context must be.
A modern example is the Agilist who insists that test-driven design is the only good way to write software, and the all software must therefore be written in an object-oriented language, and that 100% of all tests must be automated, because otherwise… well… it’s just not Agile! Kent Beck has spoken! Don’t anger the gods of Agile!
It’s the context-imperialists who bug me most. They are like doctors who prescribe medicine to patients they’ve never examined.
- Context-Driven
To be context-driven is to behave as if there is literally no such thing as a best practice. All we can hope for is good practices in context. Context-driven methodologists are very interested in learning about practices. Then they decompose and recompose them to fit the situation at hand.
To summarize the attitudes of practitioners…
- The context-oblivious simply follow convention.
- The context-specific cultivate the skill to succeed in one stable corner of the world.
- The context-imperial insist that the world change to suit their own skill.
- The context-driven cultivate the skill to adapt to a changing world.
These Attitudes Are Not Mutually Exclusive
I said the four types above were attitudes, not schools. (Note that I’m using the same label “context-driven” as both the name of an attitude toward context and the name of my school of testing thought which is centered on that attitude.) That’s because these attitudes, unlike schools, can mix together.
Even the most ardent context-driven methodologist (and I suspect I am that person) cannot be aware of all the aspects of context that matter in every situation. For instance, if a tester is secretly planning to sabotage the project, I don’t necessarily take that into account when planning that project. In that respect, I am context-oblivious.
I am also partly context-specific, because all my thinking about testing is colored and biased by the kinds of projects I have worked on in the past. Since I have done little military work, for instance, my ideas and calculations are probably different than they would be if I had grown up as a tester in that world.
I am also context-imperial, in some ways. If you want me to design a test process for a commercial product with the requirement that every single test be procedurally specified and placed into a test case management system, I would probably tell you that it sounds like a formula for bad testing and I would try to get you to change your mind. I’m not confident I can be successful in that context.
Despite all that, my self-image and my pride as a tester lies mostly with my belief that I can and I should be able to see a wide range of solutions to nearly any problem. That’s what being skilled in this craft of testing means to me.
Look Again at the Rant
See the unqualified assertions in the rant (e.g. “Those problems are going to shift and change…�)? When directed to a person in a context I know nothing of, they are the hallmarks of the context-imperial attitude. The context-driven way is to make speak in conditionals (“If your problems shift and change…�) and qualifiers (“…then you might find it useful to learn how to adapt your solutions to fit.�), or else stick to speaking about my own experience (“I find it rewarding to study the dynamics of testing problems in general, so that I can solve them no matter how they shift and change. This is because, as a consultant, I am asked to give advice in a wide range of situations.�)
If you look at my writing on this blog closely, you will notice that I use conditionals, qualifiers and personal experiences a great deal. Now you know why. It’s the price of belonging to the Context-Driven school.
Context-Driven Challenge
Any decent context-driven thinker can cite at least three scenarios where taking the context-driven attitude would be a bad idea. Can you think of them?
One scenario, as the Context-Driven methodology site inadvertently points out, there are some high risk projects that must follow some sort of “Best Practice� (a.k.a. regulations) to cover peoples’ jobs and/or freedom. A tester cannot go into these situations with some liberal/free thinking testing heuristics. The customer would be exposed to certain catastrophic failures.
[James’ Reply: Okay folks, pay attention, because I’m just going to reject every other comment like this that comes in. Since this was the first, I’ll make my response, and I will just say it once.
High risk projects require better thinking, not worse. More thinking, not less. Free thinkers– the ones who follow reason wherever it leads– are exactly the people I need on intense and important projects. It is extremely irresponsible to imply that some government office or standard will protect you from having to do good work by giving you permission not to use your best lights to solve problems on critical projects.
You are intimating that there is an alternative to using heuristic approaches to test software. There isn’t. In my experience, there are basically two kinds of people who say that we can mindlessly follow “regulations” that will lead us to glory: the kind of people who have absolutely no knowledge or skill, and the kind of people whose ambition is to profit from the ignorance of the first kind.
Which kind are you?]Â
Ouch! I am still trying to pick myself up off the ground after that response. Don’t get me wrong, I am definitely a fan of your work such as Lessons Learned in Software Testing, Risk-Base Testing, Exploratory Testing, and Heuristics Test Strategy Model to name a few. I am fairly new software tester, 4 years, and was pointed to your works by a colleague.
What I translated from your response (after filtering out the ‘you’re an idiot’ innuendoes) is to not limit the testing to just meet regulations and that testing heuristics is needed in conjunction.
I have no experience testing missiles, airplanes, etc. I attempted to imagine what it would be like if I was thrown into such a situation. I would be able to apply testing heuristics, but due to my inexperience, I could possibly miss an important function or what could also be known as a regulation. I do understand that testing cannot be limited to these regulations but do not understand why we should do without regulations. Perhaps your experience has shown you that not-so-bright people are creating these regulations and they are worthless anyways.
I do appreciate your initial response and will continue to keep studying to hopefully someday be a great free thinking/heuristics software tester. Don’t count me out just yet!
[James’ reply: I’m pleasantly surprised that you responded so graciously. I may have misjudged you from your first comment. If that is the case, please accept my apology. There are specific reasons why your comment provoked my response, however.
Let’s start with what a heuristic is: A heuristic is a fallible method for solving a problem or making a decision. ALL testing practices and regulations are heuristic. In other words, there is no “guaranteed, just turn the algorithmic crank” way of testing that will find every important bug. When we are talking about which practices to apply, the discipline that matters is the discipline of working the problem, not the discipline of following instructions. Only once we have decided on a course of action, it’s relevant to speak of following the plan.
One reason I reacted strongly to your comment is that your comment implies that someone else, wiser than you, has decided for you what you should do, and that your role is not to question, not to “think freely” as you put it. But this is not the case (unless you meant to say that your boss has set these regulations for you, and you must do as he says, for he is defining success for you). There are regulations in some contexts, true, but none of those regulations state that you are free to do bad work as long as you follow the regulations. They do not release you from responsibility to think. In other words, the regulations are also heuristics.
There is certainly no support for the claim that hazards would be created by not thoughtlessly following some testing regulation. I would claim that hazards are obviously created by thoughtlessness itself. Skill is the key; skill and knowledge. Not compliance.
If you face a regulation, then you must take it seriously. That may be an important aspect of your context. But there’s no way to generalize from the mere fact that a regulation exists that it is good, or necessary, or not harmful.]
I would call myself intuitively-context-driven. I very easily internalize/soak-in concepts and knowledge, and once I accept something as true, I intuitively make decisions based on that. The problem for me is that I often have the output (a decision on what to do in a particular context), but it is hard for me to explain the thinking I used to come to that decision. Even if I think I have successfully explained it, I have internalized too many concepts, and thus don’t really speak the same language as a novice. (They are able to understand only part of what I say, and the rest is lost).
I think a general case probably applies – it is a mistake to (try to) explain things in a context-driven way when doing on-the-job training. Just tell the person what to do, and maybe drop some hints on the context. If they have the aptitude to learn, then they will take the initiative and follow up on those hints on their own.
Here’s one scenario – two different situations.
Install testing of a Windows application versus testing a website. Here’s the situations:
1. Install testing. When testing an install, it’s important to have and maintain a known state in regards to what dlls exist on a workstation before and after the install assuming the install includes dlls and assuming the testing includes checking the date/time/version of the dlls on the PC. The context I am specifically envisioning is a Windows application that uses shared MFC dlls not just proprietary dlls.
2. Website testing. When testing a site it might be important to know how the website functions with Windows Update turned on. The Windows Update may impact the site’s functionality and skipping an update might be skipping what customers will experience. Better is not updating automatically but updating at decided intervals. What I’m suggesting is turning Windows Update off and only after a review of each update and knowing what is being updated accepting the update and then test the relevant or potentially impacted areas of a website. Of course, what is supposed to be impacted versus what ends up being impacted is not always the same. Different story.
Maintaining state versus not maintaining state. Switching out dlls during testing an install would be reckless but then not testing with Windows Update turned on may not simulate real world usage when testing a production website. I’ve done a fair amount of install testing and the state of a test PC is a condition to be mindful of … I’ve blogged about the state of PC ….
In sum:
To test with Windows Update on would strike me as poor practice for install testing and applications where MFC dlls are involved. (Unless of course, you want to test on a “dirty” and uncontrolled PC to see how the install handles … see how the context shifts and I have to keep thinking about what it is I’m doing, what it is I want to accomplish.)
To test with Windows Update off for a website strikes me as being too clean to simulate real world experience. (But then knowing how the site functioned before and after Windows Update is good knowledge to have … again what is the context, what is the goal …)
Two different contexts. Two different answers. Or more.
[James’ Reply: What a rich example. There’s enough detail here for us to have a conversation. You are also signalling me, with an example like this, that you A) have a command of details and B) have relevant experience with these details. To be context-driven requires that we routinely speak this way amongst ourselves and with our clients.]
James,
Can I consider context driven thinking as collective thinking of the all other schools of testing as described by Bret with context assuming the prime significance?
What is your opinion about following equation?Context Driven thinking = Analytic thinking + Quality Thinking + Factory thinking + CONTEXT (with first three components of varying proportions depending upon the context and in some cases, one or more at ZERO)
Shrini
[James’ Reply: No, the schools are independent belief systems. Each stands alone. I don’t do “factory thinking.” I might select a factory-like method, but not for the same reason as a Factory school partisan.
Do not confuse a school with the practices it tends to use. A Christian and a Muslim and a Hindu may all drive the same kind of car, but that indicates nothing about the overlap of their belief systems.]
Steve Campbell drops a heuristic labeled as general:
bq. I think a general case probably applies – it is a mistake to (try to) explain things in a context-driven way when doing on-the-job training. Just tell the person what to do, and maybe drop some hints on the context. If they have the aptitude to learn, then they will take the initiative and follow up on those hints on their own.
Not to rake you o’er the coals, Steve, but I think it still depends. A couple of quick examples:
If I am running a business (such as a fast food franchise), rather than conducting an apprenticeship (such as once was the case in craft guilds), the balance of time spent explaining context should be different. If I am explaining “don’t cross the street without holding someone’s hand”, I expect how I explain it will differ if my subject person is 2, 22 or 95 years old.
I prefer apprenticeship contexts to follow-the-rules franchise/Harvard MBA ones, especially when the mentor says “just try it my way for {x} period and we can discuss what we both see as an outcome”.
And yes, many apprenticeships are-were run as “just follow the rules” because world-lessons (when actually learned) sink in in a way that word-lessons do not. I think that’s at the heart of your approach; am I right? I’d add that another reason is that people are neither perfect learners nor perfect teachers, and time and attention are limited resources.
Classical 20th-Century business school taught managers to make “humanresources” (I always run the words together to remind me the term is-was Newspeak) as fungible as possible. “There are no irreplaceable people”. True, but but but… not every business context is an assembly line. I’m sure you agree!
I took high school physics from a teacher whose attitude toward my problem-solving was “That’s just wrong. Points off for not being right. Full stop.” It was not through any effort on his part that I took Physics as my major in college. If Feynman had been my instructor, I have the feeling that the tone would have been more like “Hmm, well, I can see where you might think that, but here’s what we actually see, so the effect you’re talking about is probably vanishingly small, and nobody has reliably produced it yet. So we don’t find your approach useful.”
Big difference, same domain (“teaching physics”); different (meta?)context.
[James’ Reply: Good point. That is, actually, one of the contexts in which I discourage context-driven thinking: the context in which you are, as yet, oblivious to context, and you are under the supervision of someone who take responsibility for the value of your work.
Remember the line from The Hunt For Red October?
]
I your Challenge, you did not define “a bad idea” do you 🙂 ?
[James Reply: I leave that definition to you.]
Consider an automatic transmission – changing gears automated based on some defined gear range values (best practice, huh). While it does not consider the context it reduce fuel efficiency and power compared to manual. And still most cars sold in the United States since the 1950s have been equipped with an automatic transmission. Is Europe a context ?
[James’ Reply: Are you saying that it increases or decreases fuel efficiency? In any case I’m not sure what that example indicates, because it is easy to argue that automatic transmission is good or bad, depending on context. I prefer to drive exclusively manual transmissions (it’s more fun), but my car is an automatic due to my wife’s fear of having to start the car on a hill. We both can drive automatics, hence that is what we have. Fuel efficiency doesn’t matter at all, to me.]
I also consider myself to have a context-driven attitude. However I like testers who cultivate the skill to succeed in one stable corner of the world and as a manager try to give them tasks to suit their own skill and seldom – to expand their world.
[James’ Reply: What’s better about that kind of tester than the kind who adapts? Are you saying that you would not hire someone like yourself?]
I’ve recently written a testing chapter for methodology guide for our company. As I’ve blogged previously I don’t like to document what and how to be done. Still I accepted this challenge. In this chapter I used a lot words like “By default, it is a common approach to, it is recommended to, typicallyâ€? to describe what someone could call the best practice.
[James’ Reply: When I write articles, I avoid using those phrases without also talking about alternatives. If it’s the default to do X, why is that the default, and what other choices are acceptable? I don’t say “X is recommended”, I say something like “I recommend X, in situation Y, because Z.” I try to talk about problems, solutions, and the problems those solutions could create.
However, writing articles is different from writing a methodology guide for a company. When I do that, I prefer to record policy and reference information. I don’t try to write down instructions to convey skill and knowledge. That is much better conveyed through person instruction and supervision over time.]
Sometimes I describe few different practices which “depends on software type/test type/project phase/etc.” . And stressed out in the most significant places that “It is up to the Project Manager to decide…” . While doing this – I don’t represent the context-driven school while documenting the best practices, do I? And still there are certain benefits of having the document. If only you avoid two things: 1) obey the document 2) hope to improve process by changing the document
[James’ Reply: I find there’s art and discipline to writing a guide to methodology. It might help us to have a context-driven methodologists guide to giving advice. Maybe I’ll write one.]
As I read it, the challenge was:
‘Any decent context-driven thinker can cite at least three scenarios where taking the context-driven attitude would be a bad idea. Can you think of them?’
Offhand …
1) Some companies care more about being stable, predictable, and repeatable than they care about being _good_. Some have even made a reputation offering products and services that may not be great, but are the same, every time. Most Brand-Name American Fast Food comes to mind. No matter where you eat it, a Big Mac is the SAME – every time. Trying to examine the situation and determine the best thing to do is not what the manager wants you to do; he wants you to follow the operations book.
[James’ Reply: That argues for context-oblivious. The reason it’s a reasonable argument, to me, is that someone else is taking responsibility for the quality of the results.]Â
(#2 is not politically correct, but it is my life experience …)
2) While the American Front Line Combat Soldier is one of the best trained in the world, there are times when the solider needs to obey all orders immediately and to the best of his ability, because there simply is no time to think. Two things that come to mind are when the squad leader points and yells “Run!” (which probably means incoming mortars or artillery) or the invasion of Normandy.
Actually, those situations are rare; the American fighting soldier is trained to think and act quickly, to be situationally aware. That said, there will always be a time when an NCO makes a statement with authority and it needs to be obeyed unquestionably or people may die. (This is where trust comes in)
[James’ Reply: This is a good example. Though there are exceptions (a soldier is not required to follow an unlawful order, for instance), this is also a situation where it may be correct to be context-oblivious. The non-commissioned officers are probably context-specific. The commissioned officers probably need to be context-driven. The great generals of history certainly were.]
3) Sometimes authors and speakers need to use a bit of hyperbole when making a point. For example “‘Any decent context-driven thinker can cite at least three” sounds a lot more emotionally appealing that “It has been my experience that any decent …” or “All of the decent context-driven thinkers who I have met have …” We see the point. (Ok, that one was for grins.)
[James’ Reply: I wouldn’t call that hyperbole. I was not trying to make an generalization about the whole world based on my experience of a piece of it. I was speaking my opinion as to something that defines the attribute of “decent” as it pertains to a context-driven thinker. Since I didn’t identify it as my opinion, I guess you could say it’s a context-imperial statement, i.e. “if you don’t think this way, I suggest that you start thinking this way.” Is it a good thing to do that? I don’t think it’s necessarily bad, but neither is it necessarily good.]
4) If someone else is falling into a trap of one of the other schools, engaging them in a discussion may fail. What you can do that point is to take a logical position from a different context and adopt it into the current problem. The other person may immediately recognize that you are making a mistake, criticize you for it … and have an “aha� moment.
[James’ Reply: That’s an interesting example! Non context-driven behavior as reductio ad absurdum.]Â
5) Sometimes I like to do context-oblivious manual labor to relax my brain.
[James’ Reply: Not really an argument, though, is it? No matter how sleepy I feel, if I’m driving my car, I’m not allowed to go to sleep before I park it. Just because you don’t feel like doing a good job doesn’t necessarily release you from the responsibility of doing it.]Â
6) There are some things to which we are context-specific, like “I am breathing oxygen�, that, most of the time are not worth investing our time in thinking about. Of course, throw me in a lake and bind my legs, and my thinking will change.
[James’ Reply: Often we are oblivious to that. Recently, two college students died when they crawled into a large helium-filled balloon. They didn’t know that helium, although non-toxic, does not solve the same problem that air does. But if you do SCUBA diving then it breathing becomes context-specific. I know why I am puffing on this regulator. I am trained in it.
Clearly, Matt, you are onto a method, here. All you have to do to imagine ways in which it may be good NOT to be context-driven is to imagine how the OTHER attitudes might sometimes be a good thing.]Â
Automated transmission is context-oblivious while manual could be context-driven if you are a context-driven driver. With my example I try to show when context-driven could be either good or not so good depending on stakeholder (driver) needs (does the fuel efficiency matter), skills (start the car on a hill) and even attitude (to have fun).
[James’ Reply: I don’t see how automatic transmission is context-oblivious. It seems that you are saying it is context-oblivious because the driver is not involved in the transmission decisions. The problem I have with that is that the driver is also not involved in the transmission at all. If you are doing something and you don’t know why you are doing it that way, that’s context-oblivious. But if you aren’t involved at all in a task that is happening, that’s not context-oblivious, that’s task-oblivious. For instance, in all kinds of modern vehicles, the driver also does not control valve timing.
Can you think of a task that the driver does in a certain way, but might not know why?]
I have been thinking about this and thought about the military example. I guess Matt was a bit faster replying than myself.
One other example.
Context-driven attitude would not be a good example in situations where the ability to think is reduced. Two examples come to mind.
A) My 5 year old does not have the ability yet to distinguish context as well as I do. While I train him to recognize context, I still prefer that he would follow our specific instructions in most cases.
[James’ Reply: But you expect– and you need– your son to adjust to context in all situations where you have not specified a program to guide his behavior. You expect, for instance, that he will not follow instructions given him by strangers on the street, unless the stranger is a police officer, etc. Furthermore, your son, like all children, learns from playing with rules.
With regard to behavior that you intend to control, you take responsibility for the quality of that outcome, that’s what makes it reasonable. That is one of the big justifications for context-oblivious behavior.]
B) I have noticed for myself that my ability to think gets reduced in extreme stress situations.
When there is a fire in my office building I will follow the evacuation guide. While many other options might be better to evacuate the building a standard evacuation process would avoid chaos.
[James’ Reply: The purpose of training to deal with emergency situations is to maximize the chance that reflex behavior, which remains available to us under stress, will also be effective problem-solving behavior. It’s essentially context-specific training. However, another aspect of dealing with emergencies is to recognize and deal with important context variables.
Recently, my older brother was in a situation where the emergency checklist for his MD-80 airliner advised him to shut down an engine that instrumentation reported to be on fire. He and the other pilot decided that it would be more dangerous to follow that instruction than to skip it, since a one-engine landing in a heavy airplane at high altitude on a hot day is itself quite dangerous. They made this decision because there were no “secondary indicators” of a fire, the assessment of which is a matter of skill and insight. The airline convened a board of inquiry which questioned the pilots deviation from the approved procedure, but ultimately approved their behavior. That’s why we have pilots in airliners!]
I think a core idea is that Agile is about changing the context for testing so an iterative/incremental process can succeed, but the impression I get from “Context-Driven Testing” is that CDT is about adapting to an existing context, not changing it.
[James’ Reply: Hi Keith. That’s right. I’m a tester, not the project manager, not the CEO, not the programmer. If someone wants to follow XP, that’s fine. I’ll adapt to that.
To be context-driven is not to be uninterested or to have no opinion about what would be better working conditions. It’s just that I know that I’m not in charge. And it’s not only me. Realize that the project manager, etc. also have some serious problems if they expect to drag an unwilling or unskilled team along with them on an Agile adventure. Agile development, as you know, is a personal discipline.
I’ve seen a description of context-driven testing on the web, written by no one I’ve heard of, which calls the Context-Driven school a kind of agile testing. While there are similarities between them, they are absolutely not the same.]
My scenario is any instance where ignorance would be considered a virtue.
I believe this is true with usability testing. If I am testing a website that has a diverse audience from many different backgrounds, I will want testers that know the least about the Web or using a computer. In this case, knowledge of the context would be a detrement to a tester because it is difficult to pretend not to have knowledge once you have it.
Another example would be beginner tester’s luck. I’ve seen testers with a fresh perspective on an application find defects that the hardened, knowledgable, seasoned testers didn’t find in 5 years of testing it. In these cases,puting a new tester in front of the application with no context whatsoever might be more beneficial than bringing that tester up to speed on the context.
[James’ Reply:Â I like these examples. In both of them, notice that the tester is acting as a tool in someone else’s process machinery. In such situations, obliviousness is no sin, and may be a virtue.]Â
I think I can add to John McConda’s last comment and consider scenarios where you might want to test the documentation or Help for a product.
Someone already familiar with a product will be able to tell you whether the documentation or Help is correct, incomplete etc. However only a new user to the product (and thus oblivious – assuming they haven’t received any prior training) will be able to tell you whether the documentation or Help does just that and actually helps them.
Depending upon the experience your new user has of your type of product (is this context specific – they might know how to use your competitor’s product?) or whether they are totally new to the product type, the way in which they use the Help will be different.
One of the difficulties I’ve faced as a software developer has to do with the overlapping and often inconsistent contexts in which the code is deployed. One assumes that in a number of “identical” or comparable instances of a system, that the same administrative procedures have been practiced on the systems that makes them equivalent.
Upon deploying code to an environment that inherently differs from another, we are shocked with the lack of conformity to our specs. We came to realize that a number of random accidents in the software development, quality assurance, deployment, and management of systems meant that there were several combinations of systems that resulted in poor achievement of the original objectives.
It reminds me of Edward Albee’s observations that a play might have to be author-proof, actor-proof, and audience-proof in order to succeed.
[James’ Reply: Good point, and well said.]
Call me confused.
But couldn’t the school of testing that you follow depend on the software you are testing.
For example, If I am testing a bomb, might I want to follow an Analytic School’s approach. Making sure I have precise and detailed specs and the bomb works accordingly? Or would you say a context approach might be yes you need to test the specs to the last detail, but hey, maybe you should test what happens if you touch the red button not listed in the specs?
If a Quality School asks the question “Are we following a good practice?�. Is it a bad thing to ask evaluate the defects that were missed and why? And in the end we can determine if we followed a good testing procedure? But if we also learn from our mistakes, would that be more of the context school.
Maybe I have too much of a simple mind to get all this philosophy. Maybe I should just say I see software therefore I test…
[James’ Reply: I think you are confusing practices with schools. Any school could use any practice.]
Hi, James.
One simple, quick question. If I say the context-specific attitude is half-a-portion of context-driven attitude, what say you?
[James’ Reply: I’d say I don’t agree. It’s not at all the context-driven attitude. If you are able to solve a particular math problem, correctly, that is not the same, and certainly not “half” of being a skilled mathematician.]
Hi James
I came across this article and wanted to share a few scenarios that popped in to my head
1.) When doing UAT, the testers may not want to know the details of how the application was implemented and test it truly from a end user perspective.
[James’ Reply: That is a technique of testing, then: ignorance-based testing. However, if the tester is responsible for the quality of his work, he would have to understand when and how to use this technique.]
2.) I recently started to learn swimming and have this anxiety and fear about it and struggled with it for quite sometime. I wonder if I had no context what so ever about it the outcome would have been different? I would just do as told by my coach and not let fear over take me. While in my head I argued it could be argued as context oblivious I have understood that the process of learning to swim is very different for every individual and in my case the less I knew may have been more helpful.
[James’ Reply: If you are not responsible for yourself, then context-driven behavior does not so much apply.]