I don’t understand the mentality of bloggers like this guy. His view of the history of testing is a fantasy that seems designed to insult people who study testing. It applies at most to certain companies, not to the field itself.
He says we need a better way to test. Those of us who are serious testers have actually been developing and demonstrating better ways to test for decades, as we keep up with technology. Where have you been, Steve? Get out much do ya?
He thinks automation is the answer. What a surprise that a programmer would say that. But the same thing was said in 1972 at the Chapel Hill Symposium. We’ve tried that already. Many many times we’ve tried it.
We know why automation is not the grand solution to the testing problem.
As a board member of AST, I should mention the upcoming CAST Conference— the most advanced practitioner’s testing conference I know. Go to CAST, Steve, and tell Jerry Weinberg to his face (the programmer who started the first independent test group, made up of programmers) all about your theory of testing history.
Also, Jerry’s new book Perfect Software and Other Illusions About Testing, will be available soon. It addresses misconceptions like “Just automate the testing!” along with many others. Jerry is not just an old man of testing. He’s the oldest among us.
James, calm down. Can you imagine Jerry Weinberg posting something like this? You’re a highly intelligent person. Why do you have to be so nasty?
[James’ Reply: Well, friends of mine don’t post anonymous comments. Also, my friends know that I’m not just intelligent, but fiery. I think there’s a need for that in this industry.
I respect Jerry’s style. It’s not my style.]
Be careful before you pigeon-hole someone as a supporter of test automation as the entireity of the solution. You are mischaracterizing my beliefs. I am in agreement with you about the failure of automation to solve everything and the (implied) need for manual testing. The trouble is, manual testing by itself doesn’t scale to multiple product cycles. The answer must be a blend of the two. Further, the answer to many sorts of testing must be to utilize the power of the computer to carry some of the load of testing.
[James’ Reply: I’m taking reasonable care, I believe. Still I could be wrong. Sooooo can you point to anything you’ve written about software testing that shows you’ve made a serious study of it? What new ideas have you proposed in non-automated testing? I see you talk a lot in vague and sweeping terms about manual testing, but nothing that suggests to me you have spent significant time really contemplating what testing is all about, and what skills are required to do it well. This is the core of my concern about what you say on your blog. Because if you don’t study testing as a sapient process in the first place, your ideas about test automation are all built on a swamp.]
As for the history, I’m well aware that testing started with coders but pointy haired bosses also quickly co-opted that and threw non-coders into the breach. Most companies I’m aware of went through a phase of massive manual testing before they spent much effort on automation. The history I presented is a bit fanciful, but I believe captures the essence of how testing actually was rolled out across the industry at large.
[James’ Reply: Have you STUDIED testing history? If so, what materials from the 60’s, 70’s, and 80’s can you cite?]
HA…You got him James….Nice job…Big Fan of yours
I believe automation of as much testing as possible is critical to the success of any sufficiently large project. BUT… let me qualify that: You have to have good testers to know what to test AND you need to write the software so that it can be easily tested in an automated fashion.
In the end, you can’t automate everything, but you should automate as much as possible and have good testers who know how to find the edge cases and fill in the cracks the automation leaves behind.
I have had success using this approach on several previous projects and it’s worked very well. It freed up the testers to concentrate on the hard problems and not spend all their time doing manual monkey-test regressions and other things that should’ve been automated from the get-go.
We involved the testers from story creation all the way to final acceptance rather than just tossing them a big ball of mud and saying ‘Here, test this’. They helped developers develop the acceptance criteria, identify edge cases, and come up with a strategy for automating what was appropriate to automate and manually testing what was appropriate to manually test.
[James’ Reply: I would say it differently. I would not say that we must automate as MUCH as possible. It is certainly possible to obsess over automation to the exclusion of learning how to test and having a decent test strategy. Haven’t you seen that? I’ve seen that a LOT. You might as well say “to be a good human you must eat as much as possible.” This is just not helpful advice. Speaking in terms of quantity or tonnage of testing implicitly broadcasts that you think testing is equivalent to hard labor.
What I would say is that, on difficult projects, it is increasingly important to make excellent use of tools in testing. This has nothing to do with quantity or proportion. It has everything to do with understanding what tools can do, what humans can do, and how to test something well.]
James,
I guess I don’t see the big deal here.
Steve’s post basically says very little except “it is time to leverage the computer to do more work on behalf of the tester.” Uhm, ok. As far as I can tell, I’ve been doing just that for the past 20 or so years.
He’s throws in the occasional “paradigm shift” of course, but everyone who wants to sound profound does that. And he attempts to squeeze in a torutured analogy about surface area that clearly has nothing to do with the real world.
In the blogging world I see hundreds of such articles without content. “The past isn’t good enough, things are changing, blah, blah, blah. No I can’t tell you what this paradigm shift is, but it’s a-comin’, just you wait and see!”
I keep reading, hoping that at least one will contain some real substance. But until that happens, I’ll just throw this one in the pile with the rest.
So James – What’s the beef?
And Steve – Where’s the beef?
-joe
[James’ Reply: I like how you ended that! 🙂 My beef is that the hot air about automation is yet more of the tapestry of gunk that obscures what we have done, are doing, and need to do more of in this industry. Sometimes I just roll my eyes. Once in a while I react to it, especially when it is so over the top, as in Steve’s post.]
I don’t blame Steve for not knowing testing history. I disagree with the points, not the authority. James, use your extensive knowledge to prove him wrong, not dismiss him outright. And he’s not really arguing about testing history, he’s arguing about what’s next – history was just a backdrop for the argument.
Here’s the problems I see with Steve’s argument:
-Testers already use a great deal of computer assistance to keep up with developers.
-Regression testing is typically (depends on context) one of the lowest value areas of testing. Making it go faster isn’t on the top of the list of tester’s concerns. We already have some pretty good methods and ideas about regression testing.
-Computer-generated tests will be a tool that will help catch some shallow bugs in places we forgot to look. They’ll be helpful. But they’ll be just another tool – they won’t actually reduce anyone’s workload. Likely it’ll increase one developer’s workload to maintain the tool, as these computer generated tests usually act at the unit-test level. They’ll provide similar value as static analysis tools, even though they (might be able to) examine running code.
If your testers are black box manual testers who don’t use scripts and tools to keep up with developers, or if your testers can’t keep up with regression tests, or if nobody has thought of using static analysis tools – the solution isn’t to wait for a silver bullet. It’s to talk to your test manager ASAP!
-Ben Walther
PS. Some test managers use the increasing regression test argument to argue for additional testers or time. It’s a lot simpler than explaining the combinatorics of adding new functionality. That is, every new feature you add, the complexity of testing increases. The problem isn’t so much regression testing the ever increasing pile of old code, or keeping up with developers, but testing dependencies and interactions between new code and old code.
PPS. One of my best test managers told me that today’s testers are spoiled. Back in HER day, testing was harder, because requirements concerned individual bits and bytes (ie, use 2 digits for the year rather than four). So don’t be so sure that developers have raced ahead of the capabilities of testers, and that we’re struggling to keep up. It may have made our lives a lot easier!
[James’ Reply: Hey, good points. I need to get to know you better, Ben.]
if you dropped the arrogance and name calling, you might be a better blogger yourself.
[James’ Reply: Do you come to my blog to drink tea and eat little scones? Did my hard words make you spill your tea? This is a rough blog for people who like straight talk, mister anonymous.
BTW, if you read what I wrote, you’ll see that the only names I called anyone were “this guy” (is that a name?) and “a programmer” (not only is that not an insult, I ALSO am a programmer, and proud of it.)]
Few questions to Steve … (did not know where to make these points …. I am posting them here as since I can quote what he said with references to non-coders)
>>>I’m well aware that testing started with coders but pointy haired bosses also quickly co-opted that and threw non-coders into the breach.
Why did that transition from coders to non coders happen? Just because of pointy haired bosses? what motivation did these bosses had to bring non coders into picture? Because coders are just not good at testing their own code (bais of knowing “how it works”) or because some coders did not like testing (human testing)? or because keeping coders at testing was too expensive or not working? Or just some “intelligent” point haired boss figured out that “Testing” is not all about just running some automation and required some intelligent thinking beyond automation.
Some pointy haired boss figured out that software is a complex creature – too complex to be viewed as only set of binary files of *your* application – it runs and depends upon host of other software and hardware platforms. Human users and their behaviour patterns were just seemed out of “usage” as per “user guide”.
Some pointy haired boss realised – it would be too risky to rely on only automated tests developed and run by a coder.
BTW – what are the real problems that you forsee with the entry of “non coders” in testing arena? Scalability? Cost? repeatability?
>>>The trouble is, manual testing by itself doesn’t scale to multiple product cycles. The answer must be a blend of the two.
Fair enough – let us talk about your suggestions to improve manual testing …
Shrini
Keep it up James.
I believe that too often people can just spout off all sorts of rubbish. I worked somewhere once where I was told that back in the day you didn’t need testers because we’d mathematically prove our code and that was the correct way to test. It was a big pile of rubbish. We proved him wrong through example of improving the quality of the systems that were produced through the selection of different techniques at different times and situations.
I think that you are completely correct to blog on things that you are passionate about.
There is always a misconception by the “Development team or so called CODERS” that ‘ Testing is easy’, Huuuuuu wish testing was so easy as they said !!.
Me being a passionate tester, think in many ways as to how to make the application more robust.
If the “coders” were a BIT more careful in what they did or developed then why would testers be there in the first place?
What really gets to me is the proliferation of articles/blogs/advertisements from people & companies offering up Test Automation as a solution to the so called testing dilemma — this is one part of the puzzle. The next piece of the puzzle is, that more often than not, the people that read these sites/articles, or attend the seminars are people that have a direct influence on the testing strategy within their company, thus the cycle is never broken. It is only testers who know there is a smarter way to test that seek out alternatives — thus sites like satisfice.com, kanner.com, developsense.com and the sorts get buried by the onslaught of these blogs/advertisements and slick promises of automation fantasies.
I am fortunate to be involved in a project where the testers (including the test manager) are allowed to “think” and implement a strategy which works for the product under test. We have not been mandated by a test director to implement strategy “x” because that’s the way it has been done on every other project. Now, does our strategy involve automation? You bet it does, but it is used in the approach we want to use it — not just for automating every bug for a regression suite. Our black box testers have a host of “tools” (i.e. automation) that extends the reach of their testing arm. They are not stuck in the mundane: write test case, run test case, report result of test case merry-go-round. The testers are actually “present” when testing, and because of that our testing group has been receiving rave results from all the other teams involved in our project.
Perhaps what we need is an onslaught of context-driven advertising — the more testers we educate/inform, the better chance we have of stopping the insanity! I’ll donate the first $100. 😉
It’s not that Steve is advocating automation over humans. There are links at the end of his blog entry on TestingReflections that point to a few other articles about the risks of having an “automate everything” mindset. I read them and found some things with which I could agree.
But James is bristling to Steve’s title: “We Need a Better Way to Test” … and so am I.
Steve’s premise seems to be: “Developers are leveraging better programming models such as object-oriented code, larger code libraries, greater code re-use, and more efficient languages to get more done with less code. Unfortunately, this merely increases the surface area for testers to have to cover.”
From this, Steve believes there will be a need for more and more automation:
“… it is time to leverage the computer to do more work on behalf of the tester.”
But that Steve considers this a “better way to test” (his blog title) annoys me as much as it does James. It’s a reckless title, and not thought out like I think a skilled tester would think it out. It’s confusing too, if you read Steve’s other entries that say automation isn’t necessarily better.
So I thought Steve’s other blog entry, “When to Test Manually and When to Automate” would help me understand what appeared to be an inconsistency. I thought I wasn’t getting his message and I’d see many robust, context-driven reasons and considerations for automation being “better” and several ways to understand what “better” might mean.
Unfortunately, I didn’t find anything but generalities. I was looking for Steve to have a tester mindset to anticipate my questions:
What does he really mean by “manual”? Doesn’t automation needs human interaction?
What does he really mean by “automation”? Does the machine replace me in this test?
Better under what conditions?
Better as opposed to what?
Better how?
Better when?
How might automation *not* be better, but worse?
“Better” in the eyes of whom?
“Better” measured against what oracles?
Better for how long?
After reading that entry, I had no understanding what variety of contexts that Steve himself would need to determine when to do which. Not a huge crime, I guess, but focusing on the context of what “better” means did help me understand why James chosethe title “We Need Better Testing Bloggers.”
Regarding your title of this post.
We are three former colleagues that recently started a blog about testing – “Thoughts from the Test Eye”.
Our ambition is to share our thoughts regarding testing; but it is also a way for us to keep in touch and sharpen our minds since we don’t work together anymore. Just seeing eachother at conferences isn’t enough…
Previous posts include various subjects such as “The Superstitious Bug Reporters”, “I like Adhocracy, therefore I am an Adhocrat”, “The flexible testing team”, etc.
Read and discuss at http://www.thetesteye.com
Regards,
Henrik Emilsson
Thanks for the heads-up on the book, I’ll add it to my queue! 😉
While it is very tough to multiplex different mindsets into different ideologies, I am sure everybody will appreciate two aspects:
1. Test Productivity improvement through maturity in Testers: This will include sharing knowledge and techniques, studying the common problems in software to guess/find the bugs, disciplined exploratory testing and so on. I am sure people like James Bach and many others (like qaforums.com moderators) have dedicated themselves on this topic.
2. Test Productivity improvement through maturity in Tools: Ensuring that we use computers for what they are good at, for our testing purposes. This will include using Test Automation solutions in the market very carefully by evaluating them on several parameters, and in cases where they dont serve the purpose (which is the major percentage of cases), at least use an array of tiny tools that increase the Test Productivity. The former or the latter parts must include using tools where applicable in the areas Test Design/Construction, Test Data Generation, Test Diagnosis etc besides conventional area of Test Execution.
If 1 & 2 are clear, we all belong to the same school, and its name is ‘school of peace and productivity’.
For me “future of software test automation” means automating the “reach” of 1 & 2 to the world of Software Testers, and that is what we have started to do at http://www.testersdesk.com.
Well I am in agreement with “Automation is not the total solution for testing”. A machine cannot do anything unless and untill external mind (manual)get involved. We always talk of automation and automation… but automation is important and necessary only to some extent depending on the nature and stability of product or project. There is always a need for manual testing as only human mind think logically and able to find functional & logical defects into the system. You cannot make automation tool to think like human. So my view is that without manual testing one cannot produce a quality product.
“Automation is good if your defect for releases is less and your metrics shows that product is stable, then think of automation (for minor releases) but for major releases do combine manual as well as automation”.
“We know why automation is not the grand solution to the testing problem.”
Why?
[James’ Reply: RTFM, dude. Please see the articles on this site and the other entries on this blog.]
It may not be the best blog in the world but at least he’s tried to make it his own…I just found a web site where my blog’s content is been copied verbatim!
I am trying hard to be flattered…
[James’ Reply: Yeah there’s a guy in South America who does that to me. He passes it off as his own blog. What is the motivation for that?]
Why “The Pointy Haired Guy” does what he does —
— I’m humbly posting on a tangent to this comment:
Shrini Kulkarni Says:
June 1st, 2008 at 10:15 am
Few questions to Steve … (did not know where to make these points …. I am posting them here as since I can quote what he said with references to non-coders)
>>>I’m well aware that testing started with coders but pointy haired bosses also quickly co-opted that and threw non-coders into the breach<<<
Why did that transition from coders to non coders happen?
Another reason for this transition is because end users who do the work every day using the software produced have a unique and valuable perspective that often isn’t seen by the developmernt and testing team. Example – Entering an order into an application is a whole different experience when you are accustomed to taking orders and talking to customers all day long and watching a call queue etc, than when you are focused on ensuring the application takes orders in the way described in the requirements document and correctly does what it does in the background. Each type of experience and perspective is as important/valuable as the other in my own experience.
Hi,
I am not sure if the author of the article has ever been at the bleeding edge of testing but having made my home here whilst James was still a tester in short pants, I can say we have been chewing over automation for years. Back in the early 90’s I worked on an engagement methodology from go to whoa for OO (smalltalk) that aimed to automate everything. I added a lot of the Test/QA strategy and assisted with such arcane things as a proper grammar of action, use case notation etc… The engagement process was to deliver tightly packaged solutions and was adopted by IBM as a World Wide Engagement Methodology. Unfortunately, outside of this type of engagement I find that automation is something I might consider when I want a fast way to add some data.
If I decide to start an association of “real testers” I will not have any IT grads, Anyone certified by the “Ponds Institute” of test crap, anyone who uses metaphor or paradigm in the one sentance and anyone who has not read Kaner, Falk and Nguyen. We will not discuss automation but will compare our just enough testing engagements. (just enough to get through to beer o’clock).
Peter wrote:
“If I decide to start an association of “real testers” I will not have any IT grads”
Hmm. Isn’t Cem Kaner a computer science professor? I guess his students would not be eligible to join your association then. But a quick web search shows me that his undergraduate degree was not in IT – so perhaps he’s ok then. But maybe not Hung Nguyen.
Hi James. Good to see some healthy debate going on and minds being stretched.
I have a couple of thoughts that built while I was reading this string…..
Firstly, I’m saddened that testing is still looked upon by some as the poor cousin in the software world. It strikes emotional chords with me for several reasons, but not least because I’ve put, and I’ve seen a lot of others put, a great deal of effort and life energy into changing things, building things and trying to move things forward.
I perceive that one of the things that holds us back is that some testers do the same thing year after year, regardless of if it works, could be improved or is just plain dumb.
As I read through people’s thoughts on automation I wondered why we’ve stagnated here. If we’ve been using it for years and we’ve all had so much experience then why is this still a debate? Why don’t we all know when to automate and when not to? The value that can be created, the typical problems and the way to balance it with real people skills and human endeavour. Why is anyone even attempting to give us a call to action such as in the cited post? Why hasnt an education programme or research programme or publishing programme taken us past this kind of stuff that would be illconceived in the minds of a 1st year undergraduate?
Secondly, where can I find more information on the history of testing in a coherent form? I’d be really keen to find some sources that help piece it all together. I have experiences from ten years in the field, but they are not rounded, not comprehensive and not necessarily coherent in the overall context of our industry. I’d be very interesting in finding some true historians to our field and think it could add a lot to our community. Can you point me in the right direction?
I’d like to think that we could build a knowledge community that can help us all move things along with a coherent bulk that we havent seen before. I dont yet know how to do it – but I think this article is sparking my interest in trying to find some solutions.
Stew
We do need better testing bloggers – by better I mean folks who are really thinking about the professional, technical practice of software testing. More folks sharing the ideas and insights they have based on experience and research.
Better for me isn’t people who regurgitate the same old material in some rote manner to convince people they’re well versed. Better is people who often I disagree with, who challenge my own thinking, who dare to say the established way of thinking is skewed or down right wrong.
The problem is most people don’t really study and learn this profession. Like most, testers are guilty of skim reading and not understanding, not questioning and it shows. It shows in conversation, blogs, forums, CVs, everywhere.
Like many I have my own blog (the only comment is from you James! At least someone’s reading 😉 and I write discussion papers. What narks me is all the time I’ve been posting on forums, sending emails with these papers attached or blogs I posted to I can count on one hand the amount of folks who’ve called me on what I’ve written, and looking back I’ve written some ill-informed rubbish.
Getting bloggers we think are better than now will mean we’re getting better thinkers than the handful we have now, which will mean we pull ourself up the professional standings and start advancing our profession instead of being led.
Till the world gets better testing bloggers, I would love to see great testing bloggers like you, blog more often.
Hello James,
Again you have an interesting post. I have to admit that I had to read it more then once (including the comments) and also in relation with the concerning article.
Somehow I can find myself in the phrase: “We Need A Better Way To Test”. Only I would not make this statement in general. It is sort of a drive for me to see what I/we can do better at the customer site. For me, that better way consists more on existing methods and knowledge and how that can be adapted in the customer’s process.
Therefore I understand your response on it:
[Quote=”James”]
He says we need a better way to test. Those of us who are serious testers have actually been developing and demonstrating better ways to test for decades, as we keep up with technology. Where have you been, Steve? Get out much do ya?
He thinks automation is the answer. What a surprise that a programmer would say that. But the same thing was said in 1972 at the Chapel Hill Symposium. We’ve tried that already. Many many times we’ve tried it.[/Quote]
One of the interesting points you are mentioning is that we are developing and demonstrating better ways to test for decades. I think our profession is one of the dynamic professions in the world. And I think we keep on developing and demonstrating for more decades, as I strongly believe we want to maintain our adaptability.
In this context I disagree with the phrase: “We need a better way to test”. It should be more like: “We need another good way to test”. As our adaptability will lead to a new way of defining approaches in their context.
Is Test Automation the solution to all questions? I don’t think so. When I started testing you had a few number of test tools. They all mentioned that they are the answer on all problems in software testing. If you take a look at the number of tools now: there are too much to make a good pick. If there is market for all those tools, though they claim to be THE Solution, why are there still more tools coming to the market. To me it seems that the existing tools are not that suitable enough. As long as this is the situation, how can test automation be the answer to challenges in manual testing.
Would I use such a statement? Yes, I might only to trigger other people think in other context, as I think there is more in testing then just performing you well know method. Would I use such a statement in the context that everything else we have learned was good and now we need something different? No way. Is test automation the answer? Also a negative answer. Though can it help? Yes it can, only when wisely used and when we know and are aware about the way the testing process can be under control. Are there better ways of testing and will we chasing them? Of course as we want to maintain our adaptibility.
[James’ Reply: Thanks for the comment.]
There are a few fundamentals that always appear to be missed in attempting to apply any kind of tool-assisted automation to the process of software testing, at least in my experience. For the purposes of illustration I’ll use the scenario of a scriptable/macro-based tool that can control a GUI-based application. Think of Selenium if you know it:
* You don’t get much value out of having an automatic script until the GUI of the application, as a minimum, is stable. Scripts are very fragile: if just the name or placement of an element moves between builds, the process tends to fail at that point. Are you a tester or a script wizard? Do we use the tools or do they use us?
Therefore…
* Since application stability only tends to come towards the end of the project, automation can’t be finalised (or even started) until you’re near the end of your testing time on a project anyway.
Therefore…
* Time spent setting up, refining, and running the automated test scripts is hardly ever recouped from what would have been achieved from just doing it manually. Even if you get good at writing scripts, it will come towards the end of a project not the beginning so you can’t leverage the speed/efficiency savings.
(An honourable exception goes to load-testing software.)
Things that don’t follow but have been observed:
* Even the most worthy, open-source, open-standards, access-to-big-library tools still require you to learn what is at a minimum another (dialect of a) scripting language. Operationally there’s danger that working knowledge will be concentrated in one or two of your testers. As that happens, they will slowly become testing tool maintainers and not testers.
* What is the true possibility that these scripts are at all portable, editable, or reusable between your projects? Every project is different. Remember “reusable software”? Exactly.
If you can negotiate this minefield you might find /some/ test automation techniques useful. It’s not all bad. I might expand this into a blog post myself one day but in the meantime I’ll ride on James’s coat-tails.
Then I discovered you wrote https://www.satisfice.com/articles/test_automation_snake_oil.pdf 🙂
Your August 28th 11:08 post here still stands as a very nice short piece written by someone not named Bach 🙂 Good work!
>>> You don’t get much value out of having an automatic script until the GUI of the application, as a minimum, is stable
Paul – what makes you to believe that Automation can be applied only at GUI level (though you have used “minimum” in your sentence)? As per James – automation is “use of tools – other software programs to support any aspect of testing” – so GUI centric or based autoamtion is one of many ways to think about automation.
What if I automation using a commad line interface, APIs, unit tests using JUnit framework? All these are examples on Non GUI automation. What is I use a tool such as perlclip to generate some random test data and use them for data driven tests? What if I write a program to scan a long test report to identify errors? – examples of automation application in testing tasks other than execution”
>>> Since application stability only tends to come towards the end of the project, automation can’t be finalised (or even started) until you’re near the end of your testing time on a project anyway.
From the examples I quoted above — automation can happen at any stage of testing even when application is not even ready to test — only you need to broaden the definition of automation – anything that helps human to do better testing is an application of automation
Agree?
Shrini
I really enjoyed this debate!!
YES. being fiery is needed in this industry!!
to answer these critics i have only one answer being a tester i will learn more and more and let my skills talk not need to answer such idiots .let skills speak
Learn learn and only Learn and grow