Regular readers know that I’m dissatisfied with the state of the testing industry. It’s a shambles, and will continue to be as long as middle managers in big companies continue to be fat juicy targets for scam-artists (large tool vendors, consulting firms, and certain “professional” organizations) and well-meaning cargo cultists (such as those who think learning testing is the same as memorizing definitions of words and filling in templates).
What I can do about it is to develop my personal excellence, and associate myself with others who wish to do likewise. Someday, perhaps we will attain a critical mass. Perhaps the studious will inherit the Earth.
In that spirit, I’m constantly looking for colleagues, and bouncing ideas off of them to make us all better. I challenge people, and to me this is a virtue. It’s how I separate those who will help the craft from those who probably won’t. Sometimes people don’t react well to my challenges. Sometimes that’s because they are bad people (in my estimation); sometimes it’s because they are good people having a bad day; sometimes it’s because I’m having a bad day; or may it’s because I’m a bad person (in their estimation).
Nevertheless, this is a big part of what I do, and I will continue to do it. You have been warned. Also, you have been cheerfully invited to participate.
An Example Challenge and What Came of It:
Lanette Creamer, unlike me, is not a werewolf (though she describes me by the politically correct term “hairy”). If you read her tweets and her blog then you also know she’s, uh, what’s the opposite of brutal? Anyway, I bet she owns at least one calendar featuring pictures of kittens in hilarious costumes.
I met Lanette a few years ago but as I do with most people, I forgot about her (fun fact: I suffer from a mild case of associative prosopagnosia which, for instance, is why I didn’t recognize my own wife consistently until a few months after we were married). Then I met Lanette again at PNSQC, last year, where she made an impression on me as someone easy to talk to. I checked out her blog and liked what I saw.
2009-11-12 11:51:49 jamesmarcusbach: @lanettecream I’ll go look at your blog.
2009-11-12 12:16:23 jamesmarcusbach: Another must read blog for testers. This one by Lanette Creamer (@lanettecream). It’s the attack of the tester ladies. http://bit.ly/3JT65a
2009-11-12 12:19:22 jamesmarcusbach: I think I encouraged @lanettecream to blog a couple of years ago, and then forgot to follow-up. Guess that worked out.
One thing I liked is that she identified herself, on her blog, as being a member of the Context-Driven School of testing. It means that I can reasonably expect such a person to be self-critical and to accept a challenge from me– a leader in that school.
A couple of days later I happened to see a paper Lanette wrote about “reducing test case bloat.” It was sitting on the desk of Matt Osborn while I was visiting him at Microsoft. I flipped through it and found a definition of “test case” that bugged me.
“Clinically defined a test case is an input and an expected result. For my purposes it doesn’t matter if a test case is automated or manual so long as it is a planned test. For the purpose of reducing test case bloat, I’d go further and say that it is a test you plan to execute a minimum of once in the product lifecycle.”
Lanette was referencing the IEEE with her definition. I hate the IEEE definition of test case. If I ever meet the guy who wrote it, I will bite him on the nose. It’s a narrow-minded supercilious idea of test cases, straight from Colonel Blimp. I prefer a broad definition that encompasses the actual field use of the term. For instance: “An instance or variation of a test or test idea.” By my definition, you can point to a list of test ideas, in bullet form, and call them test cases, just like real people at real companies already do, and not be committing a crime against terminology. Also, my definition does not attempt to enforce a specific organization’s notion of what a test must look like. It has to have inputs or it’s not a test? It has to have specific planned expected result? Not when I test, buddy.
I also didn’t like that Lanette presented it as if it were a universally accepted definition. That’s an appeal to authority, which we in the Context-Driven community do our best to avoid.
From Twitter:
[sadly, I cannot obtain Lanette’s side of the conversation because Twitter sucks in that particular way…]2009-11-14 19:40:16 jamesmarcusbach: @lanettecream Yes, listening to the IEEE is fine if you’re not a true student of testing. But people like us ARE the IEEE (or better).
2009-11-14 19:41:10 jamesmarcusbach: @lanettecream I followed the IEEE, too, for a few years, and then realized that whoever came up with those defs wasn’t very thoughtful.
2009-11-14 19:41:44 jamesmarcusbach: @lanettecream Welcome to software testing leadership, where there is no appeal to authority allowed.
2009-11-14 19:43:22 jamesmarcusbach: @lanettecream The reason I bring it up is that I’ve generalized it myself, and I’m curious if your analysis will reveal something new.
2009-11-14 19:45:18 jamesmarcusbach: @lanettecream One way to frame the question: What exactly do you mean by “input?” What exactly do you mean by “expectation?”
2009-11-14 19:46:50 jamesmarcusbach: @lanettecream I think it’s shallow. I think you can do a lot better. Anyway, I’d be interested to see your analysis of that definition.
2009-11-14 19:48:34 jamesmarcusbach: @lanettecream Another way to say it: maybe that definition is okay– but what does it MEAN? Do you know? Have you really thought it through?
2009-11-14 19:50:49 jamesmarcusbach: @lanettecream IEEE is not a person we can cross-examine. It doesn’t think anything. But for the record, it’s totally wrong about planning!
2009-11-14 19:51:22 jamesmarcusbach: @lanettecream That planning stuff is just propaganda. Ask yourself “what does planning MEAN?”
2009-11-14 19:51:57 jamesmarcusbach: @lanettecream They throw around a lot of words without really thinking about them, it seems to me.
2009-11-14 19:52:41 jamesmarcusbach: @lanettecream I can tell you my opinions of all this. But I’d really love to see you blog about it, first. I’m following your blog now.
I did worry a little bit that Lanette would freak out and think I was attacking her. I’m a little nervous when engaging women this way, especially, since I have more concern about being seen as a big bully. (A man might see me that way, too, but as a fellow man I would have little sympathy. He just has to learn to cope.)
Dialog with Michael Bolton
While waiting to see what Lanette would come up with, I decided to transpect with Michael Bolton on the same topic in the hope that our good natured arguing would help Lanette feel better about the challenge.
James Bach: hey, to help Lanette, could we transpect through IM?
Michael Bolton: Heh.
Michael Bolton: Sure.
James Bach: then I can show her the transcript
Michael Bolton: If you like. Pray, proceed.
Michael subsequently edited and published the transcript of that conversation.
During that interaction I came up with a thought experiment with which to question the Lanette/IEEE view of test cases. Can you test a clock when you can’t give it input?
I have since used this scenario to help explain to my students what I mean by a test.
Lanette’s Response
To my surprise, she wrote two entries. The first one worried me: What Did I Say a Test Case Was?
I went into damage control mode on Twitter…
2009-11-15 03:27:16 jamesmarcusbach: @lanettecream Your post seems a bit defensive. I wasn’t attacking you, I was trying to find out what you meant by what you said.
2009-11-15 03:30:44 jamesmarcusbach: @lanettecream I want to help real testers, too, and when I seek clarity in myself and other testers, it’s because that helps us avoid waste.
2009-11-15 05:09:55 jamesmarcusbach: @lanettecream I feel better hearing that. Questioning you is, from me, a sign of respect. But I don’t mean to push too hard.
2009-11-15 05:22:46 jamesmarcusbach: @lanettecream From your blog, I can tell you are talented. I’m eager to help your talent blossom. One requirement for that is confidence.
2009-11-15 05:25:55 jamesmarcusbach: @lanettecream One source of confidence is to practice working through ideas with your colleagues.
Lanette tried again. Her second post embraced the spirit of my challenge: What is a Test?
Notice how her second post is in the classic form of an exploratory essay. That’s perfect! I wasn’t asking for an ultimate argument and perfect analysis. I was looking for inquiry, insight, and self-examination.
Why should anyone put up with my challenges?
Well, how about career advancement? This can happen in a couple of ways. First, by publicly accepting and responding to my challenge, she improved her reputation for all to see. She shows that she is someone to be taken seriously, because she takes her own learning seriously. Second, she gained the first level of my gratitude and respect, and these things can be redeemed for professional favors of various kinds. When you are part of a community in good standing, you may holler for support and your fellow citizens will turn out in force to help you. When Lanette puts out a question on Twitter, lots of people will try to answer. It’s a great feeling to know you aren’t alone in the industry.
Plus Lanette was later interviewed by uTest. That was partly from how she impressed some of us on Twitter. I also profiled her in my talk on “buccaneer-testers” at Star East.
I hear that Lanette and my brother are collaborating on something together. I’m eager to see what comes out of that.
Another reason people should put up with challenges is that it makes the industry better. We practice our rhetoric and rapid learning. We grow. I’ve said it many times: the major reason all the terrible misconceptions about testing persist after all these years is that there is a worldwide conspiracy among testing writers and consultants not to debate with each other. Live and let live. Don’t rock the boat that feeds you, etc. Yech.
Finally, there’s personal pride. You feel good about yourself when you can take the heat.
When People Run Away
I don’t mind when people say no to a challenge, unless they are claiming to be expert testers. When a consultant or writer in the field won’t engage me, then I have to dismiss him. I can’t take him seriously. Just as I would not expect to be taken seriously if I held myself above the duty of defending my ideas in public. There’s a pretty substantial list of well known people who are professionally non-existent to me, but I don’t know how else to deal with them. We have to have intellectual standards or we can’t get anywhere.
(I know of a couple of exceptions to that rule, both women, whom I won’t name here. They are people who have strong aversions to debate (at least to debating me) and yet have great ideas and have contributed lots of good to the field. I can never be a close colleague of people like that, but I’m glad that they’re out there.)
Remembering Anna Allison
All this reminds me of Anna Allison. She was a rising star in 2001. I had dinner with her after she approached me at a conference and begged for a conversation (anyone can talk to me, at any time, if they give me food). At dinner, she mentioned that she was a bug metrics expert. I rolled my eyes and drew a bug metrics graph, daring her to tell me what it meant. What followed was a tour de force of questioning and analysis. She uncovered every trap that I had put into the graph. I told her she should write an article about our conversation and she did!
Tragically, she was on one of the planes that went into the Twin Towers on 9/11, on her way to a consulting gig in LA. This affected me more than I expected it to, because while I didn’t know her well, personally, professionally she was one of the few people I’ve known for whom debate was great fun. The Context-Driven community lost a happy tigress in her. We need more leaders like that. We really couldn’t spare her when she left us, and no one like her has yet stepped up: a non-threatening personality who is a role model for debate. I think that may be why I have high hopes for Lanette. (Also for Meeta Prakash, BTW.)
What happened yesterday?
Yesterday I issued a challenge to new blogger Michael Alexander. He responded promptly and in admirable fashion.
Lanette subsequently did a video blog about why she reacted to my challenge so constructively.
These events inspired me to explain all this. And so, I call upon all testers to challenge me, challenge yourselves, and challenge each other. Let’s blow out the cobwebs. Let’s be testers, not followers.
I can see that you like to meet people on your terms, and that people whose terms are significantly different from yours, or people who can’t adapt themselves to your terms, will have a hard time finding themselves in camaraderie with you.
[James’ Reply: Yes, that’s true for testing. I’m pretty established in testing. I have a well-established philosophy and I’m interested in exploring and deepening it.]
I don’t mind having a sharp discussion with someone if I have already built some bonds of congeniality with them. My brother is the sharpest tester I know (yeah I know, tester-brothers, it’s been done) and because of preexisting bonds he and I challenge each other, often bluntly, fairly regularly. But if someone I didn’t know challenged me as bluntly, I’d be a little put off and probably wouldn’t engage him. You might argue that I’m missing out on an opportunity to grow — and perhaps I would be — but my terms involve building at least thin congenial bonds first. That’s how I know that the challenge is meant to be in our mutual best interest.
[James’ Reply: Yeah, I can see that. Well, you don’t have to engage anyone. I’m just trying to explain why I engage (even with strangers, most times) and why I respect those who do and why I suggest you might want to, as well. Take it or leave it, as you like. It’s simply a requirement to be respected among the people I hang out with.
The other thing happening with me is that I’m pretty well known. So it’s usually not a situation of “who the hell is this crazy guy?”]
So how do two people with styles that don’t mesh, as I suspect yours and mine don’t, learn from each other? Who gives in? Shouldn’t both adapt at least a bit, moving more toward the center? Oh, wait, there’s my style again.
[James’ Reply: Questions like this don’t have answers outside of a context. I don’t know what your style is or what you want. I don’t know what you are offering me or what I have to offer you. I don’t know what “center” means. But if you read the article you are commenting upon, you will see that I did adjust my style when I was afraid I had hurt Lanette’s feelings. And I use Lanette as an example precisely because she has a very different style from mine.]
I’m working on my first ever blog post ever as we speak. It’s got a catchy title “Teach Yourself Testing: Ten Year Crash Course”.
It’s going to be slightly based on this article by Peter Norvig. http://norvig.com/21-days.html
Nice post, James. I’m very much a believer in challenging myself, my assumptions, my peers, my industry and my work. I firmly believe that’s part of how I grow and certainly it’s part of why I enjoy my work.
Some of this, I’ll be honest, I attribute to you. An aspect of my grounding in testing and test theory was taking the beta of a class you were teaching on exploratory testing in 1996/1997 and I really enjoyed not just what you taught but the ability to question, provide another view and even argue with you. I liked the fact that testing was not a formula and required active participation and constantly changing ideas and thoughts. My nature is that of someone who explores the inside of every box presented to me and then tries to get outside it for a different view, maybe with different glasses. I may not be “right” or even “win” but I come away with new knowledge every time.
[James’ Reply: Thanks Maura. (Maura was one of the first people to take the class that turned in Rapid Software Testing. She went on to write the book Testing Code Security.)]
This exploration, to me, is a sort of synthesis. I can take what I have, lay it out before someone else and have them question things I never thought to question, then look at what they have asked and try to fill in the blanks. I can examine other people’s collections (knowledge, experience, approach, etc) and see if parts of those connect to my own collections in some new and interesting ways. Probably not the best analogy ever, but there you go 🙂
[James’ reply: This is exactly what we call transpection.]
You and I have somewhat similar styles, though I’m not as prone to biting, but I’ve never been afraid of you nor of having a lively discussion about something we disagree on. In part because I have a relatively thick skin and in part because I like the process of the questioning and discussion. But my style appears to be different than Lanette’s and I know it’s different than your brother’s. I adjust mine also, as needed by the audience, but I don’t stop being me and I don’t stop my basic desire to question and learn.
I am really NOT a “go along” person. If I disagree, I will say so. If I agree, I will also say so. Wouldn’t I do a disservice to whomever I was having a discussion with if I was merely a “yes woman”? It would no longer be a discussion. I’m polite, I’m professional, I’m nice. But I’m not someone to sit quietly in a corner and listen to anyone pontificate.
As to camaraderie, the actual definition of it implies being light-hearted and convivial but I’d prefer to take it back to its roots of “comrade” and look at that. Are not many of us, indeed, comrades. We are comrades in an ongoing desire to improve our profession, our industry and our practices. I don’t need to be completely in agreement with someone for that. Heck, sometimes I’m not completely in agreement with myself.
I agree with most of this post but I’m going to play the sexist card. I’m a guy. I would expect the same courtesy to be extended to me as to the women you won’t bully. Not just for my sake, though. I don’t want testing to be a two-tiered profession because of our . . . um . . . tackle.
(I cope by posting comments on blogs.)
[James’ Reply: It’s not my intention to bully anyone. I just don’t consider it bullying to come at you with the full force of my authentic personality. You should be able to take it. If you can’t, then you’ll learn. I expect from you only what I expect from myself, because I believe you are like me.
I don’t assume that women are like me until they demonstrate that they want to or need to be treated that way.]
I like the concept of the thought experiment where you test a clock without giving it input. However, I’d like to question its’ legitimacy.
[James’ Reply: The legitimacy of the example? What’s illegitimate about a clock under a glass dome?]
At some point in the past that clock has been given input. The input maybe in the form of someone adding batteries, winding up or adjusting the mechanism. The difference is that a) someone else may have provided the input and b) that input may have happened some time ago. Either way those input provide the basis of the test case. And if you so much as look for an expected result (e.g. checking the time), even if YOU don’t provide input, then you are conforming to the IEEE definition of the term ‘test case’ because someone else has provided the input.
[James’ Reply: If you are going to be that liberal with your interpretations, why not throw the definition away? What is it doing for you? Anyhow if you consider it input (I wouldn’t) then you must also believe that such input is part of the test. But how can it be, since the tester has no idea what that input was and therefore can’t fully know what the test is that he is supposedly performing?
Also, I prefer not to use the term “expected result” because it’s become infected with the idea that an expectation must be established in advance of test execution. That’s why I prefer to use the term “oracle”, which means “a principle or mechanism by which we recognize a problem.”
I reject the IEEE’s wheezy, bloated, and strangely judgmental idea of what a test case is. For a few years I tried to work with those definitions. I have long since given up.]
Can you test a clock when you can’t give it input?
Well how about these as “inputs”
* Turn off the light – maybe the clock is photo-voltaic?
* Change the temperature in the room.
But then again when has “Don’t Touch That!” really stopped a serious tester from testing?
It’s also somewhat gratifying to know that I am not the only one who does test cases in bullet form.
[James’ reply: Good thinking. It’s important to find ways around restrictions that are put in our way. Now what if I asked you to test a start that is 10 light years away? Is there any way to do that, say, within a two year span of time?]
Thanks for posting James.
I like being challenged. I don’t assume that I know everything. I see that could be other/better/different ways out there. It doesn’t faze me how the challenge occurs unless it gets personal. Then I won’t hear the message – I just hear noise.
[James’ Reply: Tools cut both ways. Getting personal can motivate or destroy motivation.]
I’m a student of the craft. I’m learning. And to paraphrase Bruce Lee, “I keep what is useful and reject what is useless”. That may not happen unless I’m challenged (either by myself or by others) so that I can determine what is useful and what is useless (to me)!
Recently I asked you a question on how to build communities? You gave some excellent insight and ideas on a peer conference (e.g. LAWST – http://www.lawst.com ). I liked that idea. Similar minded peers discussing/debating topics of interest.
You then CHALLENGED me to start a similar type conference here in New Zealand which you then tweeted!
[22/06/2010 2:43:19 p.m.] James Bach: So I just outed you by tweet.
[22/06/2010 2:43:24 p.m.] James Bach: Now you have to do it
[22/06/2010 2:43:28 p.m.] James Bach: heh heh
Will it help testers in New Zealand? I believe it will! Will it be a challenge before/during/after the conference? You bet! BUT i can see that the challenge can help our craft become much stronger.
[James’ Reply: Please keep us posted on what you do.]
I think being challenged and taking up challenges is a necessary part of a thinking tester.
It’s like when you try to teach/talk to a colleague about a topic and the “why” and “what do you mean by that” questions come up.
It forces you to go deeper or re-assess your approach (either you haven’t got to the right level of detail/background yet or your approach is making an assumption about how the other person thinks/responds…)
So, either way it forces you to think, analyse what your own pre-conceptions are and understand more detail of what you’re trying to say. This self-evaluation is key.
For any thinking tester this is necessary to help their own learning and understanding about a topic – those are the tools that will help them ask better questions and even look for assumptions that others are making (whether it’s about a product feature, claim or bug.)
So, challenges per se are a “must” or a /good/ thing. Of course, how people respond to them is a more personal thing – sometimes it’s instant and sometimes they need their own space, go away and think about it. Ruminate, cogitate and then respond, maybe.
But it’s the thinking that’s important – and even showing your thinking.
A good while ago, when doing maths at school and college “showing your working” was actually quite important – it was an insight into your assumptions and map-making to come to a solution, idea, hypothesis, conjecture etc.
I try to re-use that idea – not only because it can help others understand my point or question – but I might re-visit the idea sometime later and then it’s a useful reminder about what my own assumptions were “at the time” – they might not hold true longer or been forgotten. Plus it helps to stop things getting lost in translation – even when you have the same language!
I’ll stop before this turns into a blog post on your comment page. Thanks for the post – it prompted me to think a bit about the topic – and why I think challenges are a good thing, but responding to challenges and “showing your thinking” are even better!
A few comments…
“you can point to a list of test ideas, in bullet form, and call them test cases, just like real people at real companies already do, and not be committing a crime against terminology”
Nice! I want my team to test. I don’t want them to spend any more time than necessary writing test cases (or writing anything that isn’t absolutely necessary, for that matter).
In many cases, that means jotting down bullet-point test ideas, rather than paragraph-form prose. All I really ask is that they put their work products on our intranet site so that we can all share and learn from it.
I get some nervous feedback from newer members of my team (“what template should I use?”, “is that really all you want me to write?”), but eventually they learn that I trust them to do whatever works for them, as long as it works for the company as well.
“All this reminds me of Anna Allison.”
Thanks for the reminder. I think of Anna occasionally as well. We used to work together on the QA team of a now-defunct software company. While she was a person full of terrific ideas, she was always open to new ones.
The first task I was given when joining the company was to implement automated testing, and at the time, Anna was the company’s biggest skeptic concerning test automation. I knew I had cleared a major milestone when I was able to “convert” Anna, by helping her team begin to automate some of their test efforts.
Anna and I remained friends when the company went downhill and we each moved on, and we often had some great discussions about software and testing. We had lunch shortly before Sept 11.
I still miss her – our profession is poorer for her absence.
Hi James,
Just had the CITCON (Continuous integration & Testing Conference, http://citconf.com) in Wellington. It is an unconference and I put the topic up “Test Automation != Quality & != Testing”. This was probably the most visited session of the conf and the most hotly discussed. It digressed quickly to how we do testing today and where everyone was coming from.
1hr is certainly not much to discuss this topic but what was immediately apparent was:
1) There were at least 45 people absolutely passionate about testing
2) They were all in for the “fight” and knew how to
3) They all had good arguments (even if I didn’t agree with most)
4) Developers and testers do have different languages (misunderstandings were not immediately evident)
I think we all took away heaps of thoughts from this discussion and it made me proud that there is so much enthusiasm in NZ on this topic. Be it developer or tester.
I think there are now 45 people more that have realised that testing is not just as simple as some make it out to be, that it can be far more complex than anticipated and -most importantly- it IS open for debate.
All this just by prodding “it” with a stick(ie). So James the confrontational route really works and inspires. Even those that might not see the light the first time round might be back for more l8r on. 😉
Oliver
I had that question given to me at an interview at Microsoft in 1995, not sure if you learned about it there… but I laughed out loud when I read your post. It’s a great problem, and I think I answered it rather poorly, but I was young then and wet behind the ears.
[James’ Reply: Which question? The clock? I made that one up. Did you really get that as a question at MS? Or are you talking about a different question?]
As to your post, I’ve never taken criticism personally, but I think many do.. it’s a shame really, since I never learned by following the lemmings off the cliff – I’ve learned fundamentally from being challenged and dropped on my head. If everyone told me that my approach or ideas were gospel, well.. then, I’d be …. king of the universe or something. A good discord is great for the soul.
Not really on this topic, I’m curious what your take is on BDD using tools like jBehave or Cucumber. I seems to me an early offshoot of Keyword Test Design (which is my baby). I’ve been experimenting a ton with it lately, and it really is some neat stuff. In many respects it offers some additional context to test design development that maybe keyword automation is lacking, but I’m curious how or if you’ve played with these tools/method and what your opinion is as to it’s ability to support using ET methods of test creation and it’s eventual execution automatically.
[James’ Reply: Actually I find the whole concept troubling and misguided. Perhaps I have not seen it demonstrated properly, yet.
When I invented data driven test automation at Apple in 1988 (note: I subsequently learned it had been invented before and since independently by just about every ambitious test automation guy out there) I was excited about the idea of separating the semantics of testing from the technology of test automation. But I learned through that experience that you can’t really separate those things without attentuating testing in important ways. The technology can only do certain clunky things in clunky ways, looping constructs are weak or non-existent or else unresponsive to emerging states, and the oracles are always narrow. When I construct data-driven automation (I think of keyword-driven as a flavor of data-driven) framework, I create an efficient way to cover certain things, but often many other things are covered poorly if at all. (Exception: API testing is perfect for DD automation)
Cucumber seems worse than that, because it seems to be touted as a way for people who aren’t programmers to write test automation. This is dangerous because people who aren’t programmers are much less likely to understand the limitations or capabilities of the tool. The tool DOESN’T understand English, so the ability to write “english-like” statements and have them executed as tests creates the risk that non-technical testers believe the tool will automatically understand and handle things that it can’t understand and can’t handle.
I have not seen examples of really interesting tests written in Cucumber. But the more interesting the test is, the more I would worry about people not understanding what the computer is actually doing when it runs the test. I’d rather see terse tables of test data driven by hoary scripts presided over by a toolsmith who knows what he’s doing.
I have the strong impression that whoever wrote Cucumber is really excited about programming and really uninterested in testing.]
Thanks for the reply James.. maybe I can show you some examples I’ve done recently.. but I would suspect that it would fall under your general impressions about the tool/approach (maybe I’ll see you at starwest and I can show you some stuff)
As for the question, yes, not exactly asked that way, but very similar.. the question was.. (not exactly this, but I’m paraphrasing here) “Here is a clock, you can’t do anything to the clock (anything I assumed implied touching, moving, etc), how would you test the clock to make sure it keeps time accurately. (something to that affect)… it definitely threw me for a loop, but it was fun none the less trying to answer it.. I can’t remember exactly how I answered it… but I remember a lot of silence on the other end of the phone interview (I still was asked to come back for another round.. but didn’t take them up on it).
[James’ Reply: Interesting. I hadn’t heard about that. I guess it’s a natural sort of example, though.]
James, how about a little more rigor, a bit more precision, in defining the product under test? In this post, you illustrate with a clock under a glass dome, and you also make reference to the same item in responding to William Echlin’s comment. In your transpection with Michael Bolton, however, you imagined “…an acrylic box. Inside the box runs a clock.” This is testing two different items.
[James’ Reply: Part of the exercise is for the student to ask questions necessary to clarify that. Just as you are doing.]
Mentally, I envisioned your acrylic box with the clock inside: a large, six-sided, ticking die. One of the first tests I imagined was measuring whether the clock would operate identically regardless of the orientation of the box. Valid test: I cannot interact with the clock, but nothing I read in your requirements limited my interaction with the box.
[James’ Reply: I would reply that you cannot approach the clock or interact with it, physically, in any other way than to look at it from a short distance away.]
Returning to your blog, some of my test cases are irrelevant because the item under test has morphed from an acrylic box to a glass dome. With one product, measuring the clock’s accuracy while it is upside down is a legitimate test case; with the other product, it’s an ID-10T error.
[James’ Reply: It’s nice that you are thinking of ways around the restrictions. I would reward you for that, and then re-emphasize the restrictions, because the point I’m getting at is how to test something that you CAN’T interact with. Of course, a secondary point is that you ought to think of ways around any restrictions that inhibit good testing.]
Or, as you said to Michael in the transpection, “You can’t just make things up, my friend. It helps to have definite ideas about what words mean.” The product is NOT the clock, contrary to what you stated. The product is either the clock in the acrylic box, or the clock in the glass dome; either way, these are two different items which will have different tests.
[James’ Reply: Let’s say that the product is the clock. The glass dome, or box, represents condition limiting your ability to test it.]
James, my challenging colleague, it still seems like you’re asking for two different things.
1. “…the point I’m trying to get at is how to test something that you CAN’T interact with.”
2. “The glass dome, or box, represents condition limiting your ability to test it.”
[James’ Reply: What’s different about these? I don’t see anything contradictory, here. Perhaps we have difference definitions of interaction. See below…]
There’s a reason the clock’s container is glass/acrylic: I have to have some sensory input. Only if I have sensory input that can be traced to the object do I have interaction with the object under test.
[James’ reply: I would say you don’t have interaction. You have the ability to observe. An astronomer can observe a star, but we would not say he is interacting with it.]
Sensory input is to some degree quantifiable, and can be compared to similar input from available oracles: this, to me, qualifies as a test.
[James’ Reply: I agree. That’s part of solving the puzzle. You don’t need to interact with something in order to test it in SOME way. You may need to interact for some kinds of tests, but not all kinds.]
A conditional limitation is not a complete absence of interaction; that would be an unconditional limitation [or an unlimited condition?]. I contend that you cannot test something for which you have no knowledge, no awareness, no sensory stimulation.
Give me sensory input and an oracle, and I can test.
[James’ Reply: I concur. Good job. The point of the exercise is to pose what seems to be a dilemma, and then get the tester to refine his idea of testing to the point where he can see his way through it. You have just done so.]
OK, I’m going to take a swing at the “clock under a dome” scenario:
1. turn on/off the room lights – maybe it’s solar powered – Can I cause the batteries to drain/charge? Can I force the display to “sleep”? Can I see if the display is lit/backlit in the dark? Can I tell if the display changes gets brighter/dimmer with lights on/off?
2. inducing radio frequencies – maybe it’s “atomic” powered – can I interefere with the radio reception that controls the settings? Can I inject changes with radio waves of my own?
3. magnetism – can I induce a magnetic field that interrupts the clocks mechanism? If it’s digital, possible? How about mechanical?
4. simple comparison – can I compare the clock under the glass with a similar make and model that I can manipulate/touch/feel/shake? (maybe using the clock in the glass as a control?) How about comparing it with my wristwatch or cellphone?
5. temperature – if I change the temperature in the room (hotter/cooler) can I detect any changes in the clock?
6. pressure – if I change the pressure in the room can I detect any changes in the clock?
7. back to the comparison – can I go home and find a similar make and model that I can interact with?
[James’ Reply: Good ideas. For those ideas that involve subjecting the clock to different conditions, I would praise your ingenuity and agree that when confronted with limitation we should try to find a way around them. But then I would re-assert the notion that we can’t influence the clock. If necessary I would say “What if the clock were a pulsar, 100 light years away? Can that be tested in less than 200 years?”
For those ideas that didn’t involve manipulating the clock (comparing it to wristwatch) I would ask whether not giving the clock input meant you were still testing it? Testing requires configuration, too. Can you test without configuring the clock?
Testing a model of the clock is especially clever. I would ask you how we could use what we learned to test the original clock?]
James, taking your pulsar example (even though I’m not an astronomer) couldn’t I evaluate the object like scientists do? They learn about, examine, determine the rules, etc. even though they can’t directly interact with it. In this case, testing isn’t so much about breaking the object or even seeing if it matches a predetermined set of specifications. I would argue that, through observation and non-destructive means, we are actually testing (maybe learning is a better word) what the specifications (rules) are.
[James’ Reply: I agree. That means you can test something without INTERacting with it. As long as it’s doing something, and you can observe it in some way, and you have some way of interpreting the results so as to say that it seems to have a “problem” if it does have one, then that sounds like testing to me.
But I suppose in that specific case, we would say that a scientist is using the pulsar as test data to test a theory he has about pulsars, because the pulsar itself would not be thought of as “defective” if it didn’t match the theory.]
If I’m observing a pulsar from 100 light years away, everything is forced into an observation-only mode. AND if I know nothing of pulsars, then I start out like the blind men investigating the elephant. My initial assertions may be flawed, but as I continue to observe and gather data I have continued opportunities to refine or rewrite my hypothesis. I would say that this is where astronomers, cosmologists and the like differ from other scientists – they don’t always get to benefit from direct interaction or manipulation of the items under study.
[James’ Reply: Yeah.]
Taking what little I’ve learned and forming it into a model, whether concrete and physical or a simulation on a computer, I can start to interact in a more physical manner and applying direct inputs, different conditions, etc. A good friend of mine, who worked in the aerospace industry, has often commented on how they built satellite and rocket systems that had all of the components tested without actually putting something into space. This form of testing relied heavily on models and simulators to determine things like flight characteristics, system interfaces, etc. Not saying that the aerospace industry is perfect, but they have managed to put many kinds of complex objects in the air and into orbit, successfully, with this kind of approach to testing.
[James’ Reply: It’s a great idea. I think it can be applied in software testing, too.]
I think we test, in many cases, without knowing what the configuration is or having the ability to change it. I can think of times where I have had to do large scale integrations of systems, many of which I didn’t own OR have control of. In which case, it didn’t limit me from using them to evaluate other systems that I did have control. What it did, however, was force me into a position of a risk-based decision – A) do no testing or evaluation of any system because I cannot control one or B) call out the risk and evaluate what I do have under my control. Either position has risks, but option B gives me some opportunity to learn about the systems, even though it may not be complete.
[James’ Reply: There’s a difference between not knowing the complete configuration and not knowing any of it. Obviously, you know that the system is “running” as opposed to “uninstalled.” Or at least you THINK you know that. Otherwise you wouldn’t bother to test at all.]
I am not much into testing as for now but your video(on Google) and this blog has inculcated deep sense of respect for the field of Software Testing in my heart.
I do alot of questioning and analyzing at work, work that my peers do or work done in other departments and it’s so much fun and above all our work becomes more credible.
I read about “Meeta Prakash” on this blog post and came to know that we work for the same company.
Just waiting for more of your blog posts!
Hi James,
It is an interesting logical clock puzzle you have put forth.
I’m going to take you up on it, but only if you meet my own challenge to stick to your definition of the clock and restrictions surrounding it. Any solution is invalid if you change the base definition. If I misunderstood your base definition I will concede defeat, but if my solution applies, you will have to concede. That is the rules.
[James’ Reply: There is no ultimate right or wrong solution in my puzzles. The point is how you think them through. That’s the learning we can use.
This particularly is not a logic puzzle (not like a chess puzzle, for instance), because there is nothing well-defined about it. To solve it is not a matter of logic, but definitely a matter of inquiry, modeling, and reasoning.]
Your puzzle is defined as:
“James: So, let’s imagine an acrylic box. Inside the box runs a clock. You can see the clock face. You can’t interact with the clock (except to look at it in normal light). Can the clock be tested?”
Solution:
I would ask my friend Joe to open the acrylic box by using solvents such as di- or trichloromethane to remove the acrylic.
Then I will instruct Joe how to handle and perform any test deemed necessary.
At no point in time will (I) (see: YOU can’t interact with the clock) be directly interacting with the clock.
Q.E.D.
[James’ Reply: Okay, you’ve done something that all good testers ought to be able to do when confronted by an obstacle– try to overcome it. That’s good.
Can you also solve the puzzle WITHOUT overcoming this obstacle? What if no one is allowed to interact with the clock? What if, say, the “clock” is light years away (you can see it in a telescope… it’s very big and bright) but you want to do tests in less than a year?
My intent in posing the puzzle is to create a situation where no one can interact with the clock. You can see it. That’s all. Is it possible to test in that situation? Hence, your solution is not “wrong”, but I’d like to hear more.]
Hi James,
Yes, it is possible to test the clock when you can not interact with the clock except to see it. I believe many readers have already suggested ways to test the clock without handling it physically. In fact, it is even possible to test the clock without the clock even existing.
Testing the clock is as much about the clock as it is about the tester of the clock. The quality of the clock is determined (measured) by the tester (end user) based upon certain criteria or preconceived expectations.
[James’ Reply: I agree. What I hope to do when I present this puzzle to a student is to drive an examination of what the word “testing” might mean, and also to help them see that although some testing is not possible in this situation, other testing may be.]
Coming back to the non-existing clock, I’m going to ask the help of my 5-year old daughter to test your clock that doesn’t exist yet. I told her all about this wonderful new clock that lives in a glass jar. I further told her that my friend is building a factory to build this glass jar clocks but the first one will only be available in 2 years. She only asked me one question: “Daddy, is it pink?”. I had to confess, sorry, it only comes out in black. She decided that she doesn’t want one, since she loves pink.
[James’ Reply: Nice example!]
In the example above you may translate it to a market survey.
In conclusion, testing is always possible, but there might be limitations to the amount or kind of testing that may be performed. Anticipating and compensating for possible violations of expectation is key.
[James’ Reply: Thank you for providing an excellent example of what it looks like to go through the sort of puzzles that I believe help us be better applied philosophers of epistemology (a fancy way of saying “tester”).]
James, I would say that your example of a clock under glass, while thought provoking, does not invalidate the definition of a test case as input and expected response. In this case, the expected response is that it display the correct time regardless of input, short of input that damages the clock. So test that – test that it display the correct time.
[James’ Reply: Yeah, that is testing it without giving it input.]
Confirm that an initial input was to start or wind the clock. Ask whether any physical stimuli to the clock case was within the scope of testing. Confirm time zone. Monitor accuracy over a period of time, and ask over what period of time you will be allowed or expected to test the clock.
[James’ Reply: You can try to confirm as you please, but you can’t give it any input.]
BTW, I come from an Electrical Engineering and hardware test background, so I am more comfortable with the IEEE definition. However, I can definitely understand your desire to use a different definition that you feel is more comprehensive and usable.
[James’ Reply: I come from a science and philosophy background, so I like definitions that stand up to critical scrutiny. I think you should want that, too. Look, you can’t give the clock input. You seem to want to include all conceivable past input provided by anything or anyone in your “test case.” Aren’t you working a bit hard to maintain your definition? You might as well call all of history a test case and be done with it.
Besides, I can construct a situation where the product takes no input any time in its history. It’s just a mechanism that performs. It seems to me it is still possible to test such a thing.]
Finally, you seem to have a very different style than I do. However, I will say two things: Test is almost by definition challenging the correctness/completeness/speed/etc. of an object. And the path I take to accepting an idea or point of view generally involves challenging it.
[James’ Reply: I’m glad to hear that. That’s crucial to the testing spirit. If you and I were in the meeting where the IEEE guys were debating the definition– wait, I’m sorry, I’ve been in IEEE standards meetings (For IEEE-829). There is no serious debate. No one has the stomach for it.]
Hi, James. I’m pretty new to the blogosphere and I must admit I’m big fan of yours. I’ve writing for couple of months now (in Finnish) and when I revised my earlier posts I noticed not all my posts have the facts there need be. I read your blog a while ago and I got the spark to challence myself! So I wrote a post about testing my own bloggings against my own expectations when I wrote them. Although I challence myself (someone’s shouting “Biased!”) I learned a lot and I hope it makes my future blogging more mature and more “user fiendly”. The reason why I posted this comment here is that I cited you on my post. I just tought it would be appreciated if I let you know myself.
[James’ Reply: Hi Pekka, thanks for writing. From what I can tell through the Google translator, you seem to be a “sapient tester” blogger. I’m glad to see that.
I hope we meet the next time I’m in Finland (which would be the first time).]
Hi James.
Interesting problem, testing a clock (or, if you like, a pulsar) that you cannot interact with. One question I have is “test it against what, exactly?”. On the surface, you would just take a (purportedly) accurate timepiece and verify that the clock’s output (movement of hands) is accurate with relation to your device.
Unfortunately, any clock you use for verification is just another instance of the very thing you’re testing. Who can say that it is accurate? All it leaves us with is that we can verify with a degree of certainty that the hands (or, indeed numerals) are moving (or changing). We cannot accurately measure whether or not they are moving or changing at the correct rate.
Applying the same reasoning to the testing of a pulsar, the whole exercise would seem to be topsy-turvy. You would be better testing the operation of your timepiece against the output from the pulsar. Although, proving that would be difficult since we have had to use the same flawed measurement devices (however accurate we regard them) to measure the spin rate of the pulsar. Knotty one, that.
We also have to remember that hours, minutes, seconds et al are something we have made up to try to encompass the concept of ‘time’ and it’s value to us. It was never necessary until the church started trying to have regular services. Additionally, since time and space are so securely linked and we know that gravity affects the ‘shape’ of space, the proximity to a gravitational body would affect the output.
I guess it’s one of those questions. No real answer but that’s OK, it’s the path you take that’s important.
One last thing, you give the example of a pulsar 100 light years distant and suggest that the distance makes it impossible to interact with. Well, every object that has mass (ie. all of the particles that make up ‘us’) have a gravitational effect. Gravity is a weak force but it has incredible (and arguably infinite) reach, meaning that we can’t NOT interact with it, however feebly. Just being there has a miniscule but very real effect.
[James’ Reply: Very nice analysis. Have I met you? I’m not surprised that you work for Test Partners.]