Adam Goucher noticed that I recently laid a curse upon commercial test tool vendors (with the exception of Hexawise, Blueberry Consultants, and Atlassian). He wondered to me how a tool vendor might avoid my curse.
First, I’m flattered that he would even care who I curse. But, it’s a good question. Here’s my answer:
Test tool vendors that bug me:
- Any vendor who wants me to pay for every machine I use their tool upon. Guys, the nature of testing is that I need to work with a lot of machines. Sell me the tool for whatever you want to charge, but you are harming my testing by putting obstacles between me and my test lab.
- Any vendor that sell tools conceived and designed by a goddamn developer who hates to goddamn test. How do I know about the developer of a test tool? Well, when I’m looking at a tool and I find myself asking “Have these vendor bozos ever actually had to test something in their lives? Did they actually want a tool like this to help them? I bet this tool will triple the amount of time and energy I have to put into testing, and make me hate every minute of it” then I begin to suspect there are no great lovers of testing in the house. This was my experience when I worked with Rational Test Manager , in 2001. I met the designer of that tool: a kid barely out of MIT with no testing or test management experience who informed me that I, a Silicon Valley test management veteran, wasn’t qualified to criticize his design.
- Any vendor selling me the opportunity, at great cost, to simulate a dim-witted test executioner. Most tool vendors don’t understand the difference between testing and checking, and they think what I want is a way to “test while I sleep.” Yes, I do want the ability to extend my power as a tester, but that doesn’t mean I’m happy to continually tweak and maintain a brittle set of checks that have weak oracles and weak coverage.
- Any vendor who designs tools by guessing what will impress top managers in large companies who know nothing about testing. In other words: tools to support ceremonial software testing. Cem and I once got a breathless briefing about a “risk-based test management” tool from Compuware. Cem left the meeting early, in disgust. I lingered and tried to tell them why their tool was worthless. (Have you ever said that to someone, and they reacted by saying “I know it’s not perfect” and you replied by saying “Yes, it’s not perfect. I said it’s worthless, therefore it would follow that it’s also not perfect. You could not pay me to use this tool. This tool further erodes my faith in the American public education system, and by extension the American experiment itself. I’m saying that you just ruined America with your stupid stupid tool. So yeah, it’s not perfect.”) I think what bugged Cem and me the most is that these guys were happy to get our endorsement, if we wanted to give it, but they were not at all interested in our advice about how the tool could be re-designed into being a genuine risk-based testing tool. Ugh, marketers.
- Vendors who want to sell me a tool that I can code up in Perl in a day. I don’t see the value of Cucumber. I don’t need FIT (although to his credit, the creator of FIT also doesn’t see the big deal of FIT). But if I did want something like that, it’s no big deal to write a tool in Perl. And both of those tools require that you write code, anyway. They are not tools that take coding out of our hands. So why not DIY?
Tool vendors I like:
- Vendors who care what working testers think of their tools and make changes to impress them. Blueberry, Hexawise, and Sirius Software have all done that.
- Vendors who have tools that give me vast new powers. I love the idea of virtual test labs. VMWare, for instance.
- Vendors who don’t shackle me to restrictive licenses. I love ActivePerl, which I can use all over the place. And I happily pay for things like their development kit.
- Vendors who enjoy testing. Justin Hunter, of Hexawise, is like that. He’s the only vendor speaking at CAST, this year, you know.
For me, Cucumber has value, though it’s certainly not my foremost tool. At my current company, we’re dealing with a lot of cultural issues – departmental silos, development almost completely located in a far distant time zone with almost no regular work hour overlap, lack of good documentation, and under all of it, a lack of trust. Improvements are being made, but it’s a slow process. In my test automation, I use Cucumber as a thin wrapper around the code I’m already writing to make the actual scenarios more accessible as documentation of our app. One portion of the test suite is based in Cucumber (though other portions skip that and go to the underlying code). As we explore the app, talk to the various teams and generally find out how things should be working, we can write Cucumber scenarios to track that knowledge and share with others. Sure, I could do this in other ways, but it does look like it will work for us (it hasn’t gotten far enough along to really catch on yet). If better documentation already existed elsewhere in ways that didn’t get stale too quickly, it would be of less value to me, certainly, but in my context, the extra time on Cucumber scenarios does add value.
“I’m saying that you just ruined America with your stupid stupid tool.”
No, I’ve never said that. Have you? If so, why?
(I do agree with most of what you write here.)
[James’ Reply: I have not literally said this. I’m conveying a feeling via comic hyperbole.
I do feel that HP Quality Center is basically ruinous. This is because it promotes a silly way to think about testing. I believe people have died, and will die, because of the Factory School mentality it embodies.]
Unfortunately, most tools are designed by people who know nothing about testing.
Vendors should not only try to consult testers, they should also consult their customers!
Sadly, many software vendors don’t want developers to talk directly to customers (“They might promise them something!”).
Instead, the communications path is more like:
end user -> IT mgr -> IT architect -> IT procurement guy -> vendor sales guy -> dev architect -> dev mgr -> developer
cubicle office ivory tower golf course/steakhouse/gentleman’s club ivory tower office cubicle
Another stinking idea that comes with HP Quality center – “Business process testing” – which horribly trivializes the very idea of business process and testing of the same and also propagates a very bad approach to think about (black box) test design.
I have personally at many time confronted with the “sillyness” of the idea. Many business leadeas and test consultants who bet on the idea refuse to listen to my line of argument. They are constantly failing HP’s ploy to link QC and their automation tool and make money with doing lot of harm to testing, test design, automation and to very idea of business process testing.
Sadly, most of COTS tools catering to “testing” or “test management” – are designed with executives and high profile business leaders in mind. I can say for sure they are not designed tester in the mind. These tools are designed to be tools in the hands of business leaders to make nice-looking presentation with graphs and laden with bogus metrics to tell some story about testing and how they saved money. These tools not for testers at all – they are for the bosses and managers having a different agenda on testing.
Shrini
Nice points with totally relevant data. To me as long as the tool allows a tester to extend the framework or be able to import libraries, I am fine with it.
I just don’t understand why tools would have individual and site licenses separately, except to cater a few consultants who maybe work out of their own office space. This definitely is something that bothers me too.
According to me test tools can be used only to run regression tests, they cannot test for one. Overnight testing, work while you sleep are all catchy words that may impress management who don’t know what exactly is being told.
The things that I look out in test tool:
1. Language thats quick to pick up (most scripting languages are similar except for a few nuances)
2. Extensibility (which allows me add my libraries so that I can add more functionality which cannot be achieved by the tool itself)
3. Utilities like ObjectSpy, Code coverage are ones that I find very useful to use along with the tool itself.
“I don’t see the value of Cucumber. I don’t need FIT. But if I did want something like that, it’s no big deal to write a tool in Perl. ”
Those open source frameworks are promoting some way of working. I don’t know if we can use those here as examples of test tools. Those frameworks relate to some social way of developing some runnable specs. Those won’t remove work, just reshape it.
Hi! I just started working in a large organization as a tester. Our virtual machines run on Linux and are built in Java. What sort of functional testing tools do you recommend ? Currently we only use Jmeter and LOADUI.
[James’ Reply: I can use any tool to help test a product, in one way or another… First figure out what problem you want to solve, then consider a tool that will help you solve it.]