[This post is here for historical reasons. However, I no longer use the term “test automation.” I refuse to utter that phrase any more because people keep thinking it means that testing can be automated– as if testing is digging a hole in the ground or some other mindless activity. It contributes to a de-skilling and de-motivating culture that discourages testing expertise. If you insist on using the term then read on, otherwise, consider talking about tools instead of automation. We all use tools. What you call automation is simply the use of tools.]
There seems to be a lot of confusion about this.
Test automation is any use of tools to aid testing. Test automation has been around since DAY ONE of the computing industry. And never in that history has test automation been an “automatic way of doing what testers were doing before”, unless you ignore a lot of what testers actually do.
For the same reason, a space probe is not “an automatic way of doing what planetary scientists were doing before”, but rather a tool that extends the reach of what scientists can do. Test automation means extending the reach of testers.
Test automation is not at all new. What’s comparatively new is the idea of a tester. Long ago, in the late 40’s, dedicated testers were almost unknown. Programmers tested software. Throughout the sixties, papers about testing, such as the proceedings of the IFIPS conferences, almost always assumed that programmers tested the software they built. Testing was often not distinguished from debugging. As larger and more complex systems arose, the idea of dedicated software testers came into vogue. In 1972, at Chapel Hill, the first conference on testing software was held, and the proceedings of that conference show that testing was beginning to be thought of as a discipline worthy of study apart from programming.
At that conference, I think they took a wrong turn. There was much hope and enthusiasm for the future of test automation. That hope has not panned out. Not for lack of trying. More for lack of understanding.
What they didn’t understand, and what many contemporary programmers and testers also don’t get, is that good testing is an inherently a human process– not incidentally, not accidentally, but INHERENTLY. It’s highly social and psychological. The more complex software is, the more important that humans engage intellectually to identify and solve testing problems. But the Chapel Hill conference was dominated by men trained as programmers and electrical engineers, not people who professionally thought about thinking or who trained people to think.
(Who did, you ask? Jerry Weinberg. His 1965 Ph.D. thesis on problem solving is fabulous. He had written The Psychology of Computer Programming in 1970, a number of papers about testing during the sixties, including a section on testing in his 1961 book, Fundamentals of Computer Programming. He taught problem solving classes during the 60’s too, culminating in his 1974 book Introduction to General Systems Thinking. I regret that Jerry didn’t keynote at the Chapel Hill conference, but he will at the next CAST conference, in Toronto.)
The idea of a dedicated trained tester is newer than the idea of test automation, but unlike test automation, it’s an idea that hasn’t been tried on a large scale, because tester training is so terrible.
Pretending that testing is a simple technical process of making API calls doesn’t make the boogie beasts go away. They are still there, Microsoft. My wife still needs me to troubleshoot Microsoft Office, a product produced increasingly, I’m told, by programmers who work intensively on test tools because they never learned how to test. (My colleague Michael Bolton recently ran a testing class at Microsoft, though, so perhaps there is hope.)
Test automation cannot reproduce the thinking that testers do when they conceive of tests, control tests, modify tests, and observe and evaluate the product. Test automation cannot perform sapient testing. Therefore, automation of testing does NOT mean automation of the service provided by the software tester.
In summary, test automation means applying tools to testing. Test automation is ancient, non-programming testers are a newer idea, but what the industry has not yet really tried, except on a small and local scale, is systematically training the minds of testers, instead of just calling them “test engineers” or “SDETs”, giving them tools they don’t know how to use, and hoping for the best.
(BTW, I’m a programmer. I was hand-coding machine language on my Apple II before I ever heard of things called “assemblers” that automated the process. I also ran the Borland Turbo Debugger test team on the Borland C++ project, in the early 90’s. Before that I ran a test tool development team at Apple Computer. When it comes to programmer testing, GUI test automation, and non-GUI test automation, I’ve been there and done that.
It is my experience of the profound limitations of test automation that makes me a bit impatient with the way new generations of testers and programmers who seem to think no one ever thought of test tools before they came along.)
Bravo Jim! (If nothing else you saved me the effort of crafting a long response to triggering blog entry).
I would like to enhance your post with one addition. In the embedded software world circa 1960 on through to *at least* 2003 when last I knew for sure in places like Draper Laboratories (http://www.draper.com who designs and builds embedded military, space, biomedical, and stuff I never held a high enough clearance to know about) reserve testing for the *most* senior and talented engineers on staff. In that world, software and hardware engineers alike strive to someday be good enough to become known and respected as a “tester”.
Granted, these testers do things like test using oscilloscopes & multimeters and when they talk about building a “test jig”, they are talking about soldering wires, chips and LED’s onto a circuit board, but even so, I always like to point out that there is a pretty significant precedent out there for testers being the top of the ladder as opposed to what we frequently see today where testing is too often seen as the stepping stool to reach the bottom rung of the ladder.
—
Scott Barber
President & Chief Technologist, PerfTestPlus, Inc.
Executive Director, Association for Software Testing
http://www.perftestplus.com
http://www.associationforsoftwaretesting.org
“If you can see it in your mind…
you will find it in your life.”
Way to go, James! Maybe we can get this whole business started again–testing as a real profession, based on a true interdisciplinary art and science.
[James’ Reply: There are a few of us working on it. Don’t slow down, Jerry, ’cause we’re following in your footsteps.]Â
I hope I can make it to Toronto. Jerry, are you sticking around for the whole conference?
As a student, long ago, I bought a hardcover copy of _Psychology of Computer Programming_ because what you were saying seemed so important.
It still does. Thanks.
Michael,
Yes, I’m planning to stick around for the whole CAST conference. I’m not one of those “fly-in, tell-them-how-smart-I-am, fly-out” guys. If a conference is great, I want to learn, too. And if it’s not great, I’m not going to go there.
And thanks for buying one of my books. I really appreciate it.
nice article. you made some good points.
however, I would like to offer you some advice on your writing. You tend to make lots sweeping accusations without ever giving evidence.. such as:
“giving them tools they don’t know how to use, and hoping for the best.”
or:
“I suspect he’s hanging around with insufficiently critical colleagues.”
your personal conjectures tend to overshadow any substance you provide. Hopefully you will take this as constructive criticism.
regards,
-Corey
[James’ Reply: Corey, do you understand that this is my blog? In my own blog, I believe I’m allowed to make all the personal conjectures I want to make. In my case, my personal conjectures are informed by considerable experience consulting, testing, programming, etc. I think I’m making plausible conjectures, but of course I may be persuaded to change my mind, if you have a counterargument to offer.
If you have Google where you live, try using it to find out something of the many articles I’ve written and talks I’ve given. You can even read my book, if you want. Then you can decide, based on evidence, whether you consider me to be personally credible.]
I think you misunderstood my position. I am describing test automation, not testing. The two are far from synonomous. Test automation *is* a replacement for what testers once did, but it isn’t a replacement for all of what testers do. Test automation is merely one piece of the puzzle. Used correctly, it can free up testers to look at more interesting parts of the problem. Without it, testers are stuck doing the same things over and over and the products fail to get better.
[James’ Reply: As Don Norman argues in Things That Make Us Smart, adding a tool to a process always changes the process. The invention of the assembly line profoundly changed the craft of manufacturing. You are not replacing what the testers did with your automation, Steve, you are creating a different test process with that automation. We disagree because you don’t think you are changing it in a material way, whereas I think you probably are changing it in a material way.
I’m not saying that it’s a bad thing to do what you’re doing. I also use tools in my work; I am not against tools. What I’m worried about is that you seem not to be aware of, or concerned about, the many things that go on in a good testers mind when a good tester is performing those “repetitive tasks”. You apparently see testing as the performance of a physical task (didn’t you say something in your post about how using the interface is often the hardest part of testing? I read that and almost choked!), whereas I see testing as a process of continual observation and inference. For you, perhaps it’s something like a road rally, where the object is to get through checkpoints as fast as possible, whereas for me, it’s not about speed, but richness of experience. I worry about losing that when I apply tools. Aren’t you worried, too?
Have you heard about “automation bias”? It afflicts pilots who come to place too much trust in their instrumentation, and lose their ability to reason about their aircraft. There are a number of interesting papers about it. Also, there are beginning to be some interesting incidents, especially in Europe, where people get so into their GPS navigation systems that they take outrageous routes to get to where they are going, and even drive into rivers.
In order to use automation safely, it’s important to remind ourselves of what is lost as well as what is gained when we add cruise control to testing.]
I try to avoid saying “test automation”, because I think it’s quite reasonable for “test automation” to refer to automating testing. I prefer something like “tool-assisted testing” for the general case of using tools to help with any testing effort. And I’ll often say “automated test execution” to refer to what most people think of when they read “test automation” (or the horrid term “automation testing”).
You mentioned how testing and debugging used to go hand-in-hand. It’s interesting to see how Glen Myers challenged programmers to separate the two processes, but now I’m telling testers to get more involved in debugging (http://tejasconsulting.com/papers/bug-isolation-pnsqc-2004.pdf).
[James’ Reply: I see a sharp dividing line between debugging and bug isolation. In the latter, we are dealing only with getting a better description of the behavior. We are not trying to fix it, but rather to sufficiently inform our clients about it. Debugging means discovering the underlying fault and then removing it. It is occasionally the case that precisely describing a bug also happens to identify the fault, such as a bug titled “The FILE menu is spelled FLIE”.
I don’t see any huge problem with mixing debugging and testing, though.]
>>> I don’t see any huge problem with mixing debugging and testing, though.]
This is interesting comment – James. Does this mean mixing debugging and testing could a problem but not a huge problem ? You suggest (in the same comment above) that debugging is process of discovering the underlying fault and removing it. So If “mix” testing and “debugging” – will I not be giving “incorrect” feeling or idea that fixing or removing the fault is part of testing ?
A precisely logged bug might be considered “near equivalent” of “removing the bug – but some one has to fix the code – however small it is and that “small” fix can have its own good or bad side effects.
Am I misinterpreting your words?
Shrini
[James’ Reply: I’m not terribly worried to discover that a tester participates in debugging, or performs debugging. I don’t see that debugging will necessarily harm testing. An example of mixing testing and debugging is the process of copyediting an article, for instance.
I’m not saying that the two activities should be confused. Don’t confuse them, okay? :-)]
Gerald Weinberg said:
bq. And thanks for buying one of my books. I really appreciate it.
Heh. Only the first of several. 🙂
Steve Rowe said: Test automation *is* a replacement for what testers once did, but it isn’t a replacement for all of what testers do.
I am not sure if Steve believes that “test automation” is replacement for EVEN what testers ONCE did. Even for once, test automation can replace what a human did while executing a test. Can it? I don’t think so. What is automation script replicated is “one possible” threads of “thinking and action” of what human tester did ONCE.
I believe it is not about doing it once or capturing it in part or whole. No automation tool, I know of can capture what a human tester does while executing a test.
Steve Rowe said “Without it, testers are stuck doing the same things over and over and the products fail to get better.â€?
What types of testers we are talking about? A skilled tester even when repeating tests can each time focus, observe, model, infer, compare, evaluate, question, hypothesize, check various dimensions, inputs, outputs, interactions of the test. You can not push automation in the pretext that there are tasks, which need to be executed repeatedly without any variation and assessment.
I can think of tasks where I think automation would help – say creating a large data set – I would use tool like perlclip. But this is a tester’s explicit choice that other parameters need to be observed.
Shrini
I once tried to codify a scheme for test automation that was intended to produce a report of “interesting” (that is, other-than-predicted/expected) test case results, rather than announcing “pass” and “fail” at high granularity. I was hoping I could automate or at least assist “Tell me the _surprising_ stuff”.
The idea was, in part, to deal with problems such as
(1) Semantic overloading of the terms “pass and fail” — “the product definitely shouldn’t do {x}” — is that a “FAIL” that is really a “PASS”?
(2) The fact that in an evolving product with disputations in specification, the test automation framework can contain test cases that _should_ act one way but don’t yet — is it meaningful to speak of “PASS” and FAIL” when everyone has MIP agreement (“mentioned-in-passing” corridor consensus) that the test case is valid but the product is going to not behave right at present?
Anyone care to guess what my results were? And why?
[James’ Reply: I like the phrase “corridor consensus”.]Â
James wrote: I’m not saying that the two activities should be confused. Don’t confuse them, okay?
The biggest problems I’ve seen come from managers confusing the two activities when estimating and tracking projects. For instance, it’s great when testers and developers cooperate in debugging, but when the testers are spending all day helping developers fix bugs, it screws up estimates based on the assumption that they’re spending all day searching for additional bugs to fix (or ignore).
You can see this problem when there’s a box called “testing” at the end of a project plan, when actually the box ought to be labeled “testing, locating, and fixing, then repeating the process an unknown number of times.” At least then people would realize that there’s no such thing as a waterfall project–only waterfall descriptions of how some managers hope a project will work (for the very first time that ever happened).
IOW, if you’re a tester and don’t want to be blamed for every project screw-up, follow James’s advice and don’t confuse testing and debugging. This is really very important.
[James’ Reply: I like to say there is no such thing as a testing phase– it’s really the fixing phase. I know this because they always ship the product when they think it’s good enough, but not necessarily when they think it’s tested enough.
Good point about estimation. Actually this applies to any project team that divides itself into roles, and then expects everyone to work together, which necessarily involves a blurring of roles. It’s mythical man-month stuff.]
Mythical man-month stuff is also, I think, a component of making all testers qualify as SWET (Software Engineer(s) in Test, or whatever the fad name of the month is). “We can always assign these folks to code maintenance after they burn out doing testing. And we can give then _anything_ to do other than test and they’ll be eternally grateful!”
Of course, giving burned-out + inexperienced + “eager to impress” people maintenance jobs can be a good way to get more examples of Jerry’s “Universal Law of Large Software Errors”.
[Hmmm, I just Goog’d that phrase and came up empty-handed. Am I remembering it wrong, or is it time for me to add a Wikipedia entry? 🙂 ]
Another thing I see in help-wanted ads is that small software places want to hire one person to do (all non-unit?) testing *and* be their “buildmeister” — that is, run their periodic code merge and compile-build process.
I’m not sure what all the factors are in that decision, but I wonder if one of them isn’t “Hey, ‘make’ is automated, tests ought to be automated; it’s a perfect skill set match”?
[James’ Reply: I don’t know which Weinberg law you’re referring to. If you Skype me, I bet we an figure it out.]Â
I once tried to codify a scheme for test automation that was intended to produce a report of “interestingâ€? (that is, other-than-predicted/expected) test case results, rather than announcing “passâ€? and “failâ€? at high granularity. I was hoping I could automate or at least assist “Tell me the _surprising_ stuffâ€?. … Anyone care to guess what my results were? And why?
I’ll offer the conjecture that you had a hard time programming your test automation to experience surprise, or to recognize anything as surprising. In my experience, machines react in an equally blasé way to the normal and to the shocking.
In your later post, I believe that you might be referring to “The Universal Pattern of Huge Losses”, found in Chapter 10 of Quality Software Management Volume 2: First Order Measurement. I could be wrong in my belief, since that pattern has little to do with testers and much, much more to do with managers. (In the same paragraph, there’s also the aphorism to the effect that a loss of X dollars is always the responsibility of an executive whose fiscal authority exceeds X dollars.)
Mr. Bolton: It has plenty to do with the entire effort at shipping quality product and constraining unknown unknowns.
I would dispute your claim that the pattern — thanks for the correction of the name, and the citation — has little to do with testers. In my opinion, the pattern is a consequence of epistemological scotoma that can be experienced by anyone who decides something is “trivial” or “straightforward” — they’re not thinking about the consequences if they are wrong.
Or haven’t you ever met people who call themselves testers who do that? Even Junit testers? 🙂
Regarding the other valuable aphorism you mention: rather than quote scripture (since my copy of Jerry’s book is not at hand at the moment), let me ask the question:
What’s the most pragmatic way of wording it? “is always”? Or “should always be regarded as”?
Because, in the world I have lived in, that finger has very rarely been pointed. I believe the world might be a better place if it were.
But given “Putt’s Law” (the whole book, not just the law), it seems that management reality is defined as what you can get away with. I see managers getting away with an awful lot, just as if that latter aphorism is not at all well understood nor effectively held in mind by those in a position to hold them accountable.
Hey Jim,
Shouldn’t that be “development tool test team” at Apple? 🙂 BTW, I greatly enjoy reading your blog when I get a chance which, alas, is all too infrequently. Always thought provoking.
Mark Cleveland
“Basically, we killed assembler.”
[James’ Reply: Hi Mark! There was a reorg in ’89, sometime around the time of the earthquake. After that, I managed the SPAM team (“Special Projects and Methods”) which was basically a test tool team. Kevin McEntee was on that team. I believe he’s Vice President of Engineering at Netflix, now. He was the architect of our greatest triumph, for which you were the main customer: Tesla, the test language compiler. Remember that? I wish I could have taken that with me when I left Apple. They had no idea what to do with it. Of course, today I’d just use Perl.]Â
Wow, Kevin is at Netflix? Gawrsh.
“TESLA: Less Filing. Tests Great.”
Ah yes. Today it’d probably be some sort of spreadsheet/html mashup.
[James’ Reply: Actually, yes. These days it’s called FIT.]Â
James, I just found your blog this evening, sorry for the delay. I have been reading your articles, and the comments on them. There is a lot here so I did not get through all of them, however I do seem to keep asking myself a single question in every artile and in every comment. I was hoping that you could help me answer it. Why do we test? In one sentance can this question be answered? Maybe the question is too simple. Maybe I need to open a dialog to actually come to an answer here. Do you think you can help me with this?
Bob
[James’ Reply: Different people may test for different reasons. Testing may have a variety of missions. For my projects it’s usually this: testing is the headlights of the project. We want test because we need to know the status of the product as it travels the “highway” of the project and into the wider world.
Asking me why I test is like asking me why I stare out the windshield while driving my car. I don’t want to hit stuff. That’s why.
In the absence of risk to the project, I usually don’t test.]
James,
Thanks for the reply. In an effort to stay with your analogy I would like to ask you the following. If you are driving down the freeway, you do not need a windshield to see. If there was not a windshield there you could still see. Motorcyclist does without them all the time. They are however nice to have so your eyeballs do not dry out. My point is this, I am not 100% convinced that testing is necessary. I know it is ‘evil’ to say such things. However, I would like to know if a team of developers put in rules or guidelines could they stop the need for testing? Is it possible to drive a car without a windshield?
Bob
[James’ Reply: It’s not wind protection I was thinking of, dude. It’s the seeing. Try driving without having any idea where you are or what you are going to run into, eh? That’s my point. In general systems terms, a software project is a cybernetic system. Although it’s possible in principle to produce good software without looking at it, it’s not practical (try and see).
Now, I’ll make the counter-point you probably wanted to make: Is it possible to “see” without “testing”? That depends on what you mean by testing. To me, they are pretty much synonymous. In other words, programmers do a lot of things that are essentially testing activities. Your customers do things that are essentially testing activities, too. It’s not evil to suggest that your programmers or your customers may be able to see enough with their own natural testing that you don’t need independent testers.
Having suggested it, does it make sense not to have independent and dedicated testers? That depends on the risk. How worried are you? It also depends on your testers. Are they any good? For some companies, they don’t bring testers on board until after there’s been a disaster of some kind.]
Bob
A few more pairs of focused testing eyes are necessary given that developers in today’s pressurized work environment test at a minimum. We see this all the time. A feature would be code complete and delivered to the QA environment after necessary unit testing and still you find at-least 5-10 bugs with varying severity.
The skills required are different in both these disciplines and one cannot ignore the other.
My 2 cents.
I like this: “Programmers tested the application before we have Testers.”
Programmers tend to forget that they need to test their own application.
So automated test is for programmers (TDD). It should be called auto checked, auto debug or whatever. And Programmers need to maintain it as part of programming cost. Nothing to do with QA or tester here.
Testers are new, and they do different kind of tests. They will ask for automation if they need. And this should be considered as different problem (QA tool, process…)
There is no magic 🙂
I’m long overdue to read this and just stumbled on it but thank you so much for your insight James!
I also like that you use everyday life analogies to support the role of testing and why it is essential. Those asking the question “Why do we test” or “Do we need to test” should look no further than the experiences in their daily lives to answer that question. You need not look any further than our 5 senses to know that we are pre-occupied every waking minute with testing. A little rudimentary I know but try drinking a coffee your colleague just bought you? Well, you know a few things about the coffee: 1) Your colleague just went to the coffee shop; 2) You know how long it took them; 3) You can ascertain the coffee has cooled down a little; 4) You are handed the coffee cup and can immediately determine or have a sense of the temperature but you still don’t know for sure and; 4) you might go for it and just guzzle if you’re brave enough because you feel you’ve properly assessed the situation however most of us….will TEST it first! And every coffee experience will be different when you are not the one who provided it for yourself.
[James’ Reply: Well said!]