This posting by Corey Goldberg illustrates an interesting and all too common kind of confusion people get into when discussing methods and practices. It’s worth pondering.
On SQAForums, someone stated:
“ISEB defines automated tested as useful only in mature testing environments and where functionality is not changing i.e. at regression testing.”
to which Corey replied:
“…and ISEB would be completely wrong on that point. web services testing should be fully automated, as there is no UI, just an API.”
Let’s analyze these statements. The first writer seems to be under the sway of ISEB, which immediately induces a heavy sigh in the pit of my soul.
(There are now thousands of people who might be called “certification zombies” lurching around in an ISEB or ISTQB-induced fog, trying to apply what they learned in a few days of memorizing to the complex reality of testing.)
When the first writer says that ISEB “defines” automation as useful only in a certain context, that’s a perfect example of the inability to separate context and method. To think clearly about methodology, you must be able to sift these things apart. Best practice thinking can’t help you do this, and in fact discourages you from trying.
I don’t know if ISEB actually defines or discusses test automation in that way, but if it does, I can tell you what ISEB is probably thinking.
(BTW, one of the big problems with certification programs is the depersonalization of convictions. I say “ISEB” when what I want to say is Dorothy Graham or one of those people who support and edit the ISEB syllabus. You can’t argue with a document. Only people can have a point of view. To argue with ISEB itself is to argue with an anonymous sock puppet. But that’s the way they want it. Certificationists quite purposefully create a bureaucratic buffer of paper between themselves and any dissenters. To pick someone whom I believe advocates the ISEB way, I will choose Dorothy Graham.)
If Dot advocates that belief, then she is probably thinking about GUI-level automation of some aspects of test execution; a set of detailed scripted actions programmed into a software agent to exercise a system under test. If so then it is indeed likely that modifying the system under test in certain ways will break the test automation. This often leads to a situation where you are constantly trying to fix the automation instead of enjoying the benefits of it. This is especially a problem when the testing is happening via a GUI, because little changes that don’t bother a human will instantly disable a script.
So, even though the first writer appears to be reading off the ISEB script, there is some validity to his claim, in some context.
Now look at Corey’s reply. Corey is not under the sway of ISEB, but I worry that he may be under the sway of a typical affliction common among programmers who talk about testing: the reification fallacy. This is the tendency to think of an abstraction or an emergent process as if it were a fixed concrete thing. Hence if a programmer sees me punch a few keys in the course of my testing, and writes a program that punches those same keys in the same order, he might announce that he as “automated the test”, as if the test were nothing more than a pattern of input and output. Certainly, it is possible to automate some aspects of testing, but the aspect of it that requires human reflection cannot be automated. In fact, it can’t even be precisely duplicated by another human. It is an emergent phenomenon.
(Some would say that I am splitting hairs too finely, and that imprecise duplication may be close enough. I agree that it may be close enough in certain contexts. What I caution against is taking the attitude that most of what is valuable about testing, most of the time, is easy to automate. When I have seen that attitude in practice, the resulting automation has generally been too expensive and too shallow. Rich, interesting, cost-effective test automation, in my experience, is a constructive partnership between human thinkers and their tools. I believe, based on my knowledge of Corey, that he actually is interacting constructively with his tools. But in this case, he’s not talking that way.)
What Corey can do is use tools to interact with a system under test. He uses his un-automatable human mind to program those tools to provide certain input and look for certain output. His tools will be able to reveal certain bugs. His tools in conjunction with un-automatable human assistance during and after execution and un-automatable human assistance to re-program the tests as needed will reveal many more bugs.
The reification fallacy leads to certain absurdities when you consider different frames of reference. Corey points out that a web service has no “user interface”, and therefore is accessible only via a tool, and anything that is accessible only by a tool must therefore require “fully automated” testing. By that reasoning, we can say that all testing is always fully automated because in all cases there is some kind of hardware or software that mediates our access to the object of our test. Therefore, the fact that I am using a keyboard to type this blog posting and a screen to view it, by Cory’s logic, must be fully automated writing! I wonder what will be written next by my magic keyboard?
From one frame of reference, a web service has no user interface. From another frame of reference we can say that it does have a user interface, just not a human interface– its user is another program. How we test such a thing is to write or employ a program that does have a human interface to manipulate the web service. We can operate this interface in batch mode: write a program to submit data, run it, review the results, and re-write the program as needed. Or we can operate the interface interactively: write a program to submit data, present results, then wait for us to type in a new query.
Corey and the first writer are not in a helpful dialog, because they are talking about different things. I would tell the first writer to treat ISEB as having no authority or wisdom, and to instead learn to reason for himself. The relevant reasoning here, I think, is to wonder what kind of tool we could find or write that would allow us to interact with the web service. At the same time, we need to consider how the web service interface might change. We might stick to highly interactive testing for a while, instead of investing in a batching system with lot of automatic oracles, if we feel that the interface and functionality is changing too fast. On the other hand, one of the nice things about testing through an API is that it is often rather inexpensive to script sequences and batches and simple oracles; and consequently inexpensive to fix them when the system under test changes. I suspect that belief informed Corey’s response, although I wish he would make that belief more apparent to people who are used to thinking of testing as a human-driven process.
As a programmer, I am aware of the urge, sometimes, to say “I didn’t do it, my program did.” In testing this naturally turns into “I didn’t test that, my program I wrote to test that did.” The crucial difficulty with this way of speaking, when it comes to testing, is the way it obscures the many, many choices the programmer made while designing the program, as if the program itself made those choices, or as if there were no choices to be made. The thing is, I don’t care, for a regular program, how many other ways it could have been written, or how any other things it could have done. But these are vital concerns when the program is meant to test another program.
Jim,
You wrote: “How we test such a thing is to write or employ a program that does have a human interface to manipulate the web service”. Good one mate, I’d agree with that. Remember we spoke about Agile Performance Testing last year? One of the techniques I spoke about was “automatic manual testingâ€?. This involved creating a web application which the developers, deployers, and testers could use to load data, run scripts, check the system was integrated, etc. So, some tests were partially automated like this, but the real trick was that this system enabled quicker manual testing.
And we were testing… a web-services API. Hey presto.
For the rest, I am not sure if you are splitting hairs. Many developers are absolutely obsessed with automatic testing. I am not so keen myself, it’s a context thing, I certainly don’t bleat out like a sheep from Orwell’s Animal Farm “automated good, manual baaaaaaaaaaadâ€?. But, you testers are much better at saying what should be automated and when. I think the agile lot have a lot to answer for here. You ever read Lisa Crispin’s book about testing on an agile project? One of the chapters was called “manual testingâ€? and it was about one line long “don’t do itâ€?. That pretty naïve approach has formed into a strong cultural meme now, especially amongst developers…
Good luck trying to get rid of it.
Jamie.
[James’ Reply: Yes, I’m familiar with Lisa Crispin’s book. It’s a depressing example of the problem. I’ve met Lisa and I’ve tried to talk with her about this… didn’t help.]
Hi,
I am the author of the original comment and quote from ISEB.
In the UK it is a very good thing to have the ISEB qualification as it is a requirement for
most testing jobs. I know it is heavily based on V model and is based on theory not practice. I dont think I have ever worked on a project that has followed more than a few bits of it.
MY point was to get a discuusion going on how we (my company) are going to test as our organisation has just switched from v model to SOA and agile. None of us have experience of this so I am trying to feel my way.
I dont think I’m ever going to get my head around automating 100% of testing and never doing any manual testing. 99% of testing maybe:) but I find defects when I dont follow scripts manual or automated so the human element should remain.
I have recently wrote a reply to a tester on the original forum and I would be greatfull if you could post any helpfull info there so I dont need to follow loads of bloggs as well 🙂
I’m going to decline to reply to Corey as I have found his replies confrontational and very un-helpful so far.
[James’ Reply: Corey and I just finished a private conversation that did not go well. He seems to think I completely misunderstood your message and his reply, but he will not tell me what and how specifically I have misunderstood. Oh well. My assessment is anyone who says that good testing can be fully automated is deeply confused and needs counseling.]Â
Hi James,
I too have read Lisa’s book and started a thread last year on a yahoo forum group that she was posting to regarding the chapter on test automation. The general consensus there I believe was that test automation was really the only way to move forward on iterative projects. I disagree.
[James’ Reply: Yeah, I disagree, too! Sounds nutty.]
We’ve been running a programme for over two years where there is very little automation, but we rely on skilled testers and session based testing. This has proved to be far more benificial. However, it is appropriate to mention that Lisa does follow her chapter on manual testing with another one that explains her view point. There are of course other issues around test automation such as they only ever run the same tests, so the subset of inputs and expected outputs is never really varied.
[James’ Reply: If you know anywhere that Lisa has discussed testing skills, promoted testing skills, or exhibited testing skills, I’d like to see that.]
On another point of note, I know that you class a court stating that you are a software testing expert as being better than a testing certification programme. I don’t disagree. However, you should bear in mind that in the UK Professor Roy Meadows founded the theory that three cot deaths meant murder and a great many people went to prison for murders that they may not have committed. This so called “expert” has now been discredited and the CPS is having to review many of the cases. Personally I’m not in favour of the ISEB, I’ve sat the exam and my grandmother could have passed it.
Olly
[James’ Reply: Well, if I gave the impression that being declared by a court to be an expert is better than ISEB, then I apologize. Neither have substance. The court was not qualified to declare me an expert, in any deep sense of the word. No one is qualified to do that, in my opinion. The people I respect most in the industry will tell you that they are not equipped for that. What we have is an industry full of self-promoters. I also am a self-promoter. I hope that the way I promote myself– which includes arming my clients against over-estimating me– is more credible in the long run, and helps me avoid complacency.]Â
Just to jump back into the debate. James Gooda: I am sorry to hear that you need this qualification for your test jobs. That sucks arse tremendously. This gold star and certificate thinking is all bs.
About testing in an iterative manner. It is friday and I am bit tired so I will refer you to me blog for more info. Hope this is OK with you James B. You may also like to read a paper of mine about agile performance testing, it will answer many of your questions I think. There is not much literature about v-model to iterative transitions. James (Bach) may suggest some more. If you drop me a line I will get a copy of the paper to you, I can’t post it anywhere because IEEE have the copy right.
http://jamiedobson.co.uk/?q=node/36
Olly. Automation is an interesting subject. The context really does matter. If you are going to test an armchair for strength you may want to use a machine to help you. If you want to check it for comfort a selection of user doing ‘manual’ tests can help. The real trick, that makes most people shit in their pants, is that you must use your brain and judgment to decide what to do and when. This takes skill, patience, and a smart mind. No certification in the world can give you this.
Sounds like you’ve done alright with our testing. Gonna write anything about it?
I wish you all nice weekends. Nice conversation.
Jamie
Jim,
People with emotional issues are confused and in need of counseling. Not people who assert all API tests should be automated. I don’t think all API tests should be automated. It’s just an opinion. Maybe we need to have a discussion about this and not name call. Unless this is a private joke between you and Corey. Is it?
Is it that people maybe get a machine talking to a program confused with automation?
Right, am really going now. Good weekend all. J
[James’ Reply: I do get sardonic when confronting bullies. Corey should know better than to use phrases like “fully automated” in relation to testing. And he should not ridicule people who are trying faithfully to reconcile the human element with the tool element unless he wants to be publicly scolded. That said, I do actually believe that he needs counseling– test methodology counseling.]
I hope italics work in WordPress.
Corey points out that a web service has no “user interface�, and therefore is accessible only via a tool, and anything that is accessible only by a tool must therefore require “fully automated� testing.
Just to be clear on this (and I don’t want to make an enemy of Corey by saying so), I’d contend that a web service does have a user interface, and a human user interface at that. The interface is something that people call its “application program interface”, or API. When people call it that, there are some words missing but implicit. I’ll try clumsily to replace them here: a means by which an application program may interface with the web service in a way that some human requires. No program is worth testing–or writing–unless it provides some benefit to some human at some point. As you’ve argued, James, all human interaction with a computer is to some degree automated. But it works the other way too; whenever a program uses an API, it is doing so in the ultimate service of some human.
I frequently test my own code and library code via an interpreter (Ruby’s, for instance). That code may not have a GUI, but it for sure has a UI.
—Michael B.
In a reply to Jamies comment about context, that was the exact point that I was making, context is the key to deciding what test method one should use, not just blindly follow automation, or manual testing as many would have you believe.
With regards to ISEB, I’d like to point out that I never put it on my CV. I took the course to find out why so many “qualified” people were frankly so useless. The answer is simple, the course teaches nothing I couldn’t find out in about ten minutes of talking to a good tester and in my experience has been taken by people who want to get more money out of their careers. The qualification does more bad than good to our industry. I would prefer a fresh graduate who is eager to learn than an ISEB qualified tester. I’m feeling queezy about even thinking about interviewing another ISEB candidate.
ISEB Foundation Certificate is a basic primer in software testing terminology. It’s sadly necessary because 1 most CS graduates have never been educated in any way about testing and 2 because most project managers have not either (in my humble experience).
For someone who wants to make a career in software testing in the UK I’d say the investment in buying Cem’s book, reading it then passing the ISEB exam (total cost, less than 150 GBP) is the best investment in your early career you’ll make. Not because passing the ISEB exam will teach you anything (but reading Cem’s book will) but because it will get your foot in the door somewhere. Perhaps this is a lame situation, but there you go.
Who knows, with a sympathetic manager you can get sent off to EuroSTAR and have the chance to meet testers from all over the world and if you’re lucky have a chat with James.
Let’s not pretend that ISEB qualification is a complete education but it’s a useful first step at least. The problem I guess is thinking that it’s more than that.
[James’ Reply: Hi Chris. As I have blogged in an earlier blog against certification, I don’t even accept this faint praise defense of ISEB. If someone has been ISEB trained, and I have to work with them, then my feeling is that they are worse off than if they had no training at all. I think ISEB is promoting the equivalent of 18th century medical practices. No doctor today would suggest that a “good start” to being a doctor is to read about trauma care circa 1785.
What’s more, I have found the ISEB people to be immune both to criticism, and to any ideas produced in the last 25 years about how to do testing. The ones I’ve talked to seem to consider platitudes and sarcastic dismissals to be the height of brilliant reasoning. In one case, an ISEB trainer argued that the fact that their ideas don’t change is a sign that their ideas are correct and that the fact that I am constantly tinkering with my own methodology is a sign that it’s wrong. That’s just the argument used against Galileo…]Â
James
I can see you’ve had some run-ins with ISEB.
I can see the parallels between your philosophy of testing and Bruce Lee’s philosophy of martial arts.
It’s a sadly repeated human story – knowledge gets crystallised, and passed on without critical examination. Soon ‘the system’ is more important than the people, and defended unthinkingly.
The happy ending is this : Bruce Lee transformed martial arts forever, although it takes constant vigilance to stop the descent back into orthodoxy.
[James’ Reply: Speaking of that, Antony Marcano gives a wonderful rant about Bruce Lee and what he accomplished. I have great admiration for Bruce Lee… and Antony.]Â