While testing at eBay, recently, it occurred to me that we need a deeper account of what testers find. It’s not just bugs. Here’s my experimental list:
Testers find bugs. In other words, we look for anything that threatens the value of the product. (This ties directly into Jerry Weinberg’s famous dictum that quality means value to some person, at some time, who matters.) Some people like to say that testers find “defects.” That is also true, but I avoid that word. It tends to make programmers and lawyers upset, and I have trouble enough. Example: a list of countries in a form is missing “France.”
Testers also find risks. We notice situations that seem likely to produce bugs. We notice behaviors of the product that look likely to go wrong in important ways, even if we haven’t yet seen that happen. Example: A web form is using a deprecated HTML tag, which works fine in current browsers, but may stop working in future browsers. This suggests that we ought to do a validation scan. Maybe there are more things like that on the site.
Testers find issues. An issue is something that threatens the value of the project, rather than the product itself. Example: There’s a lot of real-time content on eBay. Ads and such. Am I supposed to test that stuff? How should I test it?
Testers find testability problems. It’s a kind of issue, but it’s worth highlighting. Testers should point out aspects of the product that make it hard to observe and hard to control. There may be small things that the developers can do (adding scriptable interfaces and log files, for instance) that can improve testability. And if you don’t ask for testability, it’s your fault that you don’t get it. Example: You’re staring at a readout that changes five times a second, wondering how to tell if it’s presenting accurate figures. For that, you need a log file.
Testers find artifacts, too. Also a kind of issue, but also worth highlighting, we sometimes see things that look like problems, but turn out to be manifestations of how we happen to be testing. Example: I’m getting certificate errors on the site, but it turns out to be an interaction between the site and Burp Proxy, which is my recording tool.
Testers find curios. We notice surprising and interesting things about our products that don’t threaten the value of the product, but may suggest hidden features or interesting ways of using the product. Some of them may represent features that the programmers themselved don’t know about. They may also suggest new ways of testing. Example: Hmm. I notice that a lot of complex content is stored in Iframes on eBay. Maybe I can do a scan for Iframes and systematically discover important scripts that I need to test.
Maybe there are other things you think should be added to this list. The point is that the outcomes of testing can be quite diverse. Keep your eyes and your mind open.
“Issues” does probably lend itself to a lot of subdividing. A few things that probably fall under issues that often seem to deserve special attention (in my experience, of course) are:
confusion–I don’t understand what the application is trying to tell me…or maybe I understand what it’s telling me, but it doesn’t make sense to me why it’s telling me that right now.
inconsistency–There are two ways to do the same thing…but they differ in some apparently significant way. Or perhaps tooltip instructions don’t match help documentation, or something of that ilk.
redundancy–two controls, fields, dialogues, etc. that theoretically accomplish the same task–do we need both? Are they separate code paths and could they therefore diverge? That one is potentially a testability problem, too.
Maybe I’m getting down to too fine-grained a level, though. Obviously they can all be subdivided in a really large number of ways.
I really like this topic–lots of people have, at one time or another, had to answer the “yeah, but what do you *do*?” question, and it’s pretty helpful, I think, to be able to articulate it in terms of “things”, at least as a means to help people see the value. (Of course, sometimes, people will take that and run create a bad metric for productivity from it, but people who work that way were probably going to find something to measure anyway.)
Testers find tests. Often times the most effective tests are created while running another test.
I really like this topic–lots of people have, at one time or another, had to answer the “yeah, but what do you *do*?” question, and it’s pretty helpful, I think, to be able to articulate it in terms of “things”, at least as a means to help people see the value.
I agree with Steve, and will add that this kind of thinking also helps guide the tester’s interaction with stakeholders. In the past, I’ve felt trapped in trying to explain everything I find in terms of a “bug.” Developing a vocabulary like this opens up how we can talk about these findings with stakeholders. For example, instead of “this isn’t quite a bug, but…” I can say “we might have a testability problem here…”
Sometimes I try to distinguish (in my head, at least) between “obstacle” bugs and “black hole” bugs:
obstacle bug – A threat to the value of the product, but one that users, tech support, or administrators can work around with training, documentation, or configuration and/or data management. The product is still capable of solving the problem it was intended to solve, if not always in the specific way intended.
black hole bug – A threat to the value of the product with no escape. The only way around this type of problem is with a code fix.
This distinction can be important on projects where code fixes are highly undesirable after product delivery (for example, when very expensive, not allowed, or damaging to the company’s reputation).
Testers find boundries. Testers push software to the edge to answer “what is the minimum or maximum amount of x that this software handle and still be useful”.
Testers find dependancies. Software can have unknown dependancies or perceived dependancies that don’t exist. Example: Finding that an application requires a specific port open to function.
James,
First of all, to me, testers ask “important” and “interesting” questions about the product/service being tested. Questions about claims, questions about normal/abnormal usages, patterns, questions about platform, various contexts in which product and service operates – economical, social, cultural, political etc. These questions in turn result into bugs, issues, risks, testability concerns etc.
I believe testers are in a unique position to ask these questions (I could be biased here). Programmers can’t as they are all the time focused on constructing things – taking spec to working code – unless something hits their road – no questions asked. Business analysts are often blinded by “their knowledge’ – anything that is outside their domain of knowledge – no questions asked again. For a PM, questions about schedule, cost, effort, environment, sign off etc are more important than anything else. Good testers keep themselves free from such operational, process related and cognitive constraints. Hence are better placed to ask questions that others either did not (or could not think) of or did not have time to think about.
Another thing testers produce is different perspectives, they think like users, different kinds of users in different situations. Thus they light-up areas that are in dark – which none knew that they existed.
>>> An issue is something that threatens the value of the project, rather than the product itself. Example: There’s a lot of real-time content on eBay. Ads and such. Am I supposed to test that stuff? How should I test it?
How do you define “value” ? I often struggled to define this term. It is an abstraction and relationship. Like other abstractions, value often finds itself a victim of reification. When testers are testing they should be considering threats to value right? What kind of value? to whom? when? I have the answers to these questions – we should first think about (and “like”) who matters and they value (in what terms) and when? We should also think about conflicting values? Do you think we often without our knowledge “reify” value?
Shrini
[James’ Reply: Value is the condition of desire. “I value that” means “I want that.” It means I will sacrifice something for it. I will take risks for it.
What kind of value? Any kind.
To whom? To the people we value.
When? When we’re testing.
Yes, we reify value. That’s okay. We also need to abstract it.]
Bugs, risks, issues, problems, artifacts, curios. It feels like that’s one side of a boundary. Testers also identify the conditions under which the product works as expected. Is it that there’s some sort of fuzzy boundary between the space where the product is clearly working as expected and the space where it clearly isn’t working, and testing helps us understand that boundary?
Testers find issues that turn out not to be problems at all. I like the fact that exploratory testing is a process used to gain domain knowledge which can be fed back into itself. For every project there are plenty of issues raised, early on, that upon being revisited turn out not to be quite as they seemed, but were misunderstandings by the tester.
This also highlights that good systems testers don’t just test software, they test ‘systems’.
Testers find:
– what tool is better in testing a feature
– usability bugs
– what can and cannot be automated.
James,
would like you to elaborate more on “Value”
>>> Value is the condition of desire. “I value that” means “I want that.” It means I will sacrifice something for it. I will take risks for it.
So, value might mean “money” some times and might mean “confort” other times, it might mean “speed” or might mean “cool”. So, to me value is a lable for some “want” by someone from someone at some point of time. Given the fluidity nature of this “value” – I find it problematic to use the phrases like “deliver business value”, “threat to value”, “value add” or “value added service” without qualifying it. To me, use of phrase “value” has to be with qualifiers such as “related parties in question – value to whom? given by whom? percieved by whom, the concrete, materialistig object or attribute or “want” that value points to (value is a lable or abstration pointing to some concrete thing) and finally the element of context (time and place).
[James’ Reply: I don’t see any problem, here. And neither do you. What you’re calling a problem is a feature, Shrini. The word value is not supposed to define anything exact. It’s what we call a concept. It’s a pattern. You need to process it in any real life situation. Don’t eat it raw.
Anything about a product that threatens its value is a bug. Your job, then, is to tell a compelling story that connects a situation in the world with the concept of value.]
>>> Yes, we reify value. That’s okay. We also need to abstract it
Why do you think there is a need to abstract it? for effective delivery of message or intent? Or since concrete things that value as an abstraction might be too intimidating ( I want “value” might be preffered to a direct “ask” such as I want “money/profit”)?
[James’ Reply: We abstract in order to create possibilities. We reify in order make a possibility into a reality.]
Do you think people tend to hide or conceal their “wants” or “intent” behind abstractions like “value” or “quality” so that “wants” remain somehow “vague”?
[James’ Reply: Yes. This is because there are a million things I want. Saying them all explicitly is impossible, so I will create an abstraction that defines the set.]
Why do you think “reification” of value is OK? Do you think we have no option but to reifiy?
[James’ Reply: To reify is to create or point at an example that is consistent with an abstraction. Software, executed, is the reification of the specification.]
Shrini
I had too many ideas spinning off this post, so I’ve started to test them here: http://www.developsense.com/blog/2011/03/more-of-what-testers-find/
—Michael B.
Few years ago I was preparing my trip to China and I took a short course mandarin language.
What I’ve found very curious was that the words “question” and “problem” in mandarin are one the same word.
Meaning that, if someone says in mandarin that he has a problem, he is actually saying that he has a question.
To me it signifies that we as testers are looking for questions.
[James’ Reply: How do you know that question and problem are one and the same word? Maybe the first people who learned both languages just misunderstood that, and they’ve been telling people the wrong thing ever since.
Seriously though, language translation is not the same as mapping one word onto another word. Indeed, in English, when I say that a practice is “questionable” I generally mean that it has a problem. And a “problem” on a test is really just a question you are being asked. So, it’s ambiguous even for native English speakers speaking only in English.]
James,
I’m afraid that I was misunderstood.
Please forgive me if I am wrong.
I will try to clarify that I’ve meant.
In my previous comment, I wasn’t talking about translating (mapping) words between languages but showing how people see and express ideas differently and how they put things in perspective.
[James’ Reply: I know you are talking about that. What I was talking about is mis-translation. Let’s consider a simple example: Can you translate the English word “problem” into the English word “problem”? My answer is “not necessarily.” I might use the word “problem” in a way that you don’t understand. But since you speak English, you might believe that you understand it.
The idea of mapping words is an oversimplification, of course, but even without oversimplifying the problem is still there. I just don’t know what it means to say that in Chinese problem and question are the same thing. For me, it’s incomprehensible that they could be the same thing, in some contexts. But maybe in those contexts the Chinese don’t use their word “problem/question” but rather some other word.]
I think that “putting things in perspective” is what we testers do.
[James’ Reply: Yes.]
Here is another example:
An English expression “to land (a plane)” is “landing”.
In French, “landing” depends where you are landing on:
Landing on the ground is “atterrir”. “Atterrir” contains a word “terres” which means “land”
Landing on the sea is “amerrir”. “Amerrir” contains a word “mer” which means “sea”.
Landing on the moon is “alunir”. “Alunir” contains a word “lune” which means “moon”.
To a french it is unthinkable to say “landing” if you are landing on the sea, moon or anything but land because there is no land to land on. In english, it makes no difference where you are landing on.
(http://translate.google.com/#fr|en|amerrir%0Aatterrir%0Aalunir%0Aterres%0Alune)
[James’ Reply: Isn’t that just a matter of lexical trivia? You could as easily say that “Apple” is unthinkable to a French person, because it doesn’t mean anything in their language. But of course, they can translate into French. Landing can also be translated into French. And it could be translated in many ways, including “Coming down from the sky” or “ending the flight.”]
English make a differentiation between “problem” and a “question” while for Chinese it is unthinkable to differentiate those ideas.
(And what do I mean here by “english” and “chinese”? People or a language?)
I’ve learned that there are at least approx. 900 million persons that are seeing problem and a question as the same thing.
(66% mandarin speakers of china)
http://translate.google.com/#en|zh-CN|problem%0Aquestion%0A
What can I learn from that fact?
[James’ Reply: I don’t see how you can judge whether a Chinese person would find it unthinkable to distinguish between a question and a problem. Just looking at language seems not enough to carry that case. Furthermore, it may be that English speaking people also don’t distinguish between question and problem, but only think that they do because the words are different.]
As a tester, I used to seek problems. (bugs)
Learning that a problem can be seen as a question I am now consciously seeking questions for which there are no clear answers.
(If there is an answer, then there is no question left, isn’t there?)
[James’ Reply: Again that seems like an illusion. An answer is the illusion of settling a question. Questions of fact cannot be settled with certainty.]
And after all, it is much easier to me to present a question to a customer than a problem.
They start to feel involved, less defensive and appreciative.
Now, I am not a native English, French nor Chinese speaker so I hope that I could express my thoughts understandably.
Cheers,
–Asmir
Yes, testers do find a lot more then just ‘bugs’. Here are some things I have found:
I always find a lot of ideas. Example: cool, we can now search for a user by last name. Would it be an idea to also allow search on first name? Or a combination of both?
I also find improvements. Or al least… stuff that I assume/ hope will end up to be an improvement the end users.
Example: Gee, I can create a user by clicking “OK” 3 times, perhaps if we merge those 3 screens to one the user only has got to click once.
Sometimes I find a really “good feeling”. This can happen if you’ve tested something, and found no bugs (of only a few trivial ones), even though you’ve been really ‘evil’ to the software. You get this really happy and good feeling on the inside and all of a sudden it feels really ok to release the software. And when you tell others they start to feel the same way. (Wheter this “feeling” is a good thing and how reliable it is, is a discussion of it’s own…).
Sometimes I find a ‘workaround’. This happens when you test, run into a ‘bug’ and try to continue testing around it. You might find a way around the bug, and if it’s a good workaround it may even end up to be the official (new) way to do it.
I wanted to chime in re: the concept of value and quality… I am not referring to the definition of quality as it pertains to testing, but more so the definition of quality itself. I am fond of the book, Zen and the Art of Motorcycle Maintenance, which is not about Zen and not really about maintaining motorcycles. I won’t spoil it for the few who may pick it up… but I think it might make you think and question….
Enjoy!
k
[James’ Reply: I read that years ago. I don’t recall that he provided a coherent definition of quality. My definition of quality is value to some person.]
We also identify inconsistent oracles (inconsistencies in general–program to requirements, requirements to oral tradition, one stakeholder’s email against another’s). Often these are not bugs until we dig deeper. Sometimes we can’t dig deeper right now. Some inconsistencies remain unresolved–becoming risks or features on the backlog, etc.
We identify problems of value (more than just bugs), questions of value (tests or test ideas), qualities of value (quality criteria), and so forth.
Here is a short list of things I notice myself doing:
* Recognize uncertainty and dig deeper (e.g. I recognize when others are uncertain, as well as myself. If people pause to think when they answer a question, I notice and dig deeper).
* Recognize Confusion (internal and external)
* Recognize questions
* Recognize values (anything which guides our judgment, priorities, decisions, and efforts) (this includes goals, metrics, purpose, etc)
* Recognize context (stakeholders, oracles, purpose, etc). Since some of these aren’t obvious, perhaps we should document them?
As far as this discussion goes, we’re talking about /things/ we notice. This seems to be a focus on the nouns that we can document. Other posts have focused on specific verbs, or lists of actions that testers do. What other categories, I wonder, can we delve into to understand the craft more?
This is brilliant, both the post and comments.
A must read for anyone who think test results can be reported as Pass/Fail.
Testers find ‘human nature’. Seriously, we have to deal with so many different people, disciplines, cultures, that for me, testing is also about human nature. Gotta love it.
Love this article,
I think testers also find for lack of a better term “correctness”
When for example a change in the program works exactly as expected this can also be counted as a finding because it is a piece of data coming forth from our investigation of the package.
Every finding is a finding even if it proves “correctness”,
[James’ Reply: Show me an example of “proving” correctness. I don’t believe you can do that. Not even people who write journal articles about proving correctness are really proving correctness, because they are making a bunch of assumptions about their systems and tools that they believe are safe, and yet may not be.
Ordinary testers are certainly doing nothing like proving correctness. What they can say is that they haven’t yet seen a problem, given what they have tested and what they have looked at.
However, you do have a useful point. When you fail to find a problem, and you have looked for it in some reasonable effective way, there is some probability that it’s really there and you missed it (“type II error” or false negative). The question for testers is what is the collective probability of all possible type II errors you could have made while testing RELATIVE to the value of those bugs that you might have missed (we don’t care about not finding completely irrelevant “bugs”). For example, there is a probability that there is a colony of small animals living in your refrigerator and that they successfully hide from you every time to open the door. But you may have a rational belief that the probability of not seeing ANY of those animals (if they exist) is so low, that your failure to see them much more likely to be because they do not exist.
That’s basically how “correctness” reasoning works: “if the product WERE incorrect, then I WOULD have detected it, therefore it is not incorrect, and THEREFORE it is correct.
But note that I used the refrigerator example because some “animals” are so small that they can infect your refrigerator without being detected, until you get food poisoning or have a food safety expert test it. My point is that if we don’t use the right kind of testing, finding “correctness” may NOT be an indicator that the product IS correct.]
I agree partially with you James but I said we find correctness. That does not mean we have proven it.
[James’ Reply: If you haven’t proven it, I don’t see how “correctness” is a helpful idea. There is such a thing as correctness testing, but when I do that, I’m looking for incorrectness, not correctness. To say that a tester finds correctness is like saying a detective proceeds by looking for everyone who is NOT committing a crime.]
I think that what we call “finding a bug” is that you can prove and reproduce a specific behaviour of the investigated subject. The bug is called thus because it is troublesome to someone important.
However we can also by the same methods prove that we can reproduce wanted behaviour of the investigated subject. This would also be a finding.
[James’ Reply: It’s at most a finding of little interest, since our customers already ASSUME it works. At worst it subverts our senses, since in order to maximize the chance of finding a bug we must be actively looking for one. My inability to find a bug, if I’m doing my job well, already implies that I “discovered correctness.” Meanwhile, a finding of correctness in any given moment does not say much about what will happen in the next moment, yet I keep hearing people say things like “I tried it and it works!” which is wrong. You can’t tell if something works. You can only tell that it APPEARS to HAVE WORKED. Maybe it is no longer working. Maybe it secretly failed. You can’t know.]
Would the wanted behaviour act out differently when variables change. Probably, but this is also true for bug reproduction.
The “finding” in my point is only the outcome of our investigation flawed to some degree as it may be. (and probably is). The conclusion we take off that is a different thing all together I think.
[James’ Reply: My point is, it is both epistemically shaky and socially and psychologically unhelpful for a tester to focus on “finding correctness” instead of finding problems.]