The Factory School is that community of process people who believe testing benefits from eliminating the human element as much as possible. They wish to mechanize testing, and to condition the humans within it to see themselves as machines and emulate machines as much as possible. It’s an idea that has a number of advantages, with the important caveat that it makes good software testing impossible by leaving no room in the process for skill and thinking.
Sometimes when I complain about the Factory School of software testing, people think I’m exaggerating or making it up.
But check out this quote from Yaron Sinai, CEO of Elementools:
At first when I saw this, I thought it was a joke news story. Apparently not. (For the sake of Mr. Sinai, I hope he was misquoted. I will gladly post a retraction if that is the case.)
The man’s tool is called Test Case, which is emblematic right there. It’s focus is on test cases, not good testing. His view of test cases appears to be test procedure steps, and he wants testers to stop thinking, dammit, and just follow steps. Like factory robots. Robots that don’t question or talk back. Nice. Saaafe. Rooooooboootttts.
I haven’t found much information about Mr. Sinai online. He apparently hasn’t written about testing, per se. At least he is consistent: he has contributed no ideas to the testing craft, and now he sells an idea prevention system in the guise of a test management tool.
I want to ask Mr. Sinai whether, as a CEO, he faithfully follows his “CEO cases” each day, written for him by other, smarter CEO’s. Or does he, gasp, think for himself? I bet he would reply that although he is a smart CEO, not all CEO’s are smart enough to make their own decisions, and thus it’s only reasonable that their work should be scripted. When I suggest that a minimum requirement to be a CEO should be the ability to think about business problems and make decisions on their own behalf, I’m sure he will say “that’s just not practical.” By which he will mean, of course, that HE doesn’t know how to do it.
The Factory School promotes the antithesis of engineering, while often using the word engineering as if it were some corpse impaled on a stick. Their approach to managing an engineering process is to kill it. Indeed, a dead process, like a dead horse, is much easier to manage once you get used to the smell.
And a lot of top managers buy this crap because the demos are simple, they are unaware of alternatives that would actually help, and the perfume on their cravats tends to mask the stench of tool vendors who don’t know anything about the craft.
My wife actually is in the same field and her current client uses Elementool for defect tracking. The system is pretty flexible, has a clean looking UI and easy to use. That being said, I’ve registered for the free account and have looked at their Test Cases Module. It is simplistic with “Test Name, Description, Step Description, Expected Results, Example Data, Severity, Priority, Assigned To, Submitted By” free text fields (alpha-numeric). The ironic thing is, I created a test group, then I didn’t create a test. I clicked view test and received the error message below. To re-confirm, I logged out, cleared my cache, logged back in, and simply clicked View Tests. The error re-appeared.
“Unexpected Error
This may be due to resource unavailability or a technical glitch.
The problem has been logged and our support team has been notified.
Please try your request again. We are sorry for any inconvenience this may have caused you.
If you cannot access your account please send an email to 911@elementool.com
Error Details:
Category: System.NullReferenceException
Error Code: 5475″
I will assume testers tested this before releasing it to the public. Given that, I have seen this type of situation before. Testers focus on the complex functionality within the system, such as adding user defined fields, email triggers, etc… All of these, and bulk of the tests run against this module will most likely require data and certain configurations to preexist in the system. Sometimes, testers forget what the first time user might experience when using an application.
Or maybe they just followed the steps, because that was all that needed to be done.
The important thing is to bounce the stick up and down, James. I’m told it’s very good at warding off crows from the cornfield.
It would seem that they ate their own dog food and used their product when they did their testing. Too bad they spent time specifying and limiting the test exercises to guarantee minimum coverage.
I’ll bet they have a test case for creating a test group and another one for entering a test script. If they follow best practices they unlink tests – the newly created test group is never used and test scripts are only entered for existing test groups. Minimal chance of finding any defects that way.
Just wondering how much Jeff Feinman got paid to write the ad.
I don’t read the SD Times, but if this is indicative of their standard fare, I’m not about to start. I note that there is a distinct lack of outraged replies. I suspect that their readership are the sort of people who are right now telling their chief quality monkey to go test drive the tool. It would appear not to be the sort of tester/test manager who finds the idea behind such a tool repugnant.
Time for a tangential rant.
The sad thing is that a company like this can be successful at all. I think it’s down to human nature and the tendency to value the easy short-term option over the more beneficial, but generally far less sexy longer term one. As long as the quick and easy, cheap solution is available, why make an effort? You don’t need a real gold nugget – it’s much easier to paint a turd gold and sell that instead. It’s why so many people are morbidly obese. It’s one reason why software quality sucks – one could go on and on, but until there is some sort of paradigm shift in understanding of the false economy that is the ‘easy option’, then the sort of change you want to see will remain limited to an enlightened (probably frustrated) few.
Just got this today… typical spam bait, but looking at the name they had addressed me in relation to the deceased was especially funny and relevant to this thread.
“Dear Jon Testing,
I am Barrister John Mensah,a solicitor at law.I am the personal attorney to ENGR.J.P.Testing,a national of your country, who is a Contractor and have spent most of his life in my country (Togo) Here in after shall be referred to as my client.On the 30TH of April 2000, my client,his wife and their only son were involved in a car accident along bagida express road.All occupants of the vehicle unfortunately lost their lives. Since then I have made several enquiries to your embassy to locate any of my clients extended relatives,this has also proved unsuccessful. ..”
Ben – for your tangential rant: you can’t stop evolution.
Everyone has the opportunity to try and make up something useful. Mostly, it will be useful for themselves and if they’re real happy with it, they will market it (I guess that’s where all the ‘best practices’ comes from).
But in evolution there’s an element of survival. Probably not many testers will let these products survive, but they will still exist (~be created) and they will still arrive on the market at a steady rate – only to succumb. After all, most of all species on this planet have gone extinct – a lot of testing tools has gone already and more will go the same way.
Still, in niches, some of these tools will be useful and save the day for someone.
Interesting that Sinai was the mountain where Moses was given the ten commandments. Perhaps our Mr. Sinai is just playing on his namesake!?
James,
I am not surprised and so will be many in India. Day in and out we see people (our managers) making such statements and pushing testing totally out of anything related to “thinking” or questioning.
I have seen many people keeping so much FAITH on test cases and some even can not think of testing without test cases at all.
On way I often counter these folks is by saying “if a tester were to simply follow STEPS and report what happened – then you want a tester? Any one would do?” If testing is about following steps – why bother testing at all as it looks very silly to pay so much to simply follow steps. I am sure there is a simple way out without using any human”
Shrini
G’day Carsten.
I was more lamenting the fact that our addiction to the quick fix has allowed us to circumvent evolution (to extend your metaphor).
In nature, if you choose the wrong evolutionary path, you die. We don’t do that much anymore.
Ordering a big-mac may seem to some like a good short-term solution for hunger. Do that every day though, and you end up fatter and slower than the rest of the herd. The penalty for that is far less immediate than say being a fat slow guy would have been 40,000 years ago.
Same in software testing, it seems. Unfortunately, when a stupid/ignorant/(etc) manager gets it wrong, you’re not allowed to kill them. Instead, you get to live with the pain of their evolutionary retardation. The quick fix is a credit card where the interest rate is a high percentage of inevitable pain. Luckily (for some), if you’re in management you can delegate the suffering.
Welcome to the graveyard of testing. Of Autocracy! Where thinking is banned. Testers are like dead bodies running to the finish line or made to and without their ticking brains which they have to leave behind lest they impact the orderliness of test case based testing. What crap!
Even if test cases were written by thinking brilliant people, will they ever be complete – for God’s sake! This is like becoming an ape from a man as Ben rightly suggests. Kill the passion, kills the brain and a man (read the tester who wants to reveal… please… only reveal) becomes a rat waiting for directions to take this or that direction dependent on someone else’s instructions. How subjugating! How stifling!
About 18 months ago I started working for a large multi-national software consultancy (whom I will not name!) who actively promoted this idea to their clients, along with the fallacy that automation will make testers even more productive and reduce testing time.
I wish I had known this before I joined them. I walked within 3 months.
aha!…I’ve finally found the source of dribble from my 3rd-from-last manager…I almost quit because he was making us fill out this spreadsheet, and demanding that for any particular module, he expected 700-1000 test cases! And, astonishing, he allowed, no he demanded the assignment of a test step as a case!! Inflating the count of cases, of course, and boosting his productivity graph line. I say almost walked, because he jumped to a new job (at a presumably higher salary), and our next manager took 3 weeks before he jumped too…
On that project, I pushed hard for the abandonment of the useless 6000+ test cases in that spreadsheet, and was successful in promoting a functional and system approach to reducing the number of cases while enhancing the coverage and effectiveness of the testing, and getting pretty ok with our process, and efficency.
Now, a year after that, I am out of a job as well…
I’m almost giving up, but then I get angry…and anger is a really good souce of energy, if it is recognized as such…just pure energy, and take control of the direction the energy pushes in, and I feel very pushy!
[James’ Reply: I’m lucky never to have worked for a manager like that.
I’m glad that you are awake and kicking back. Thank you.]
Not too long ago I had to give a demo of a GUI-Driving “test automation tool” at a conference as part of my speech; I wanted to show how brittle the GUI was and what problems you’d encounter. To do this, I needed to build up a non-trivial test suite with it, see how long it took me to write, what the challenges were, etc.
My wife, who has a wonderful education in liberal arts (BA in philosophy from an ivy-level college) walked by an saw the icons flashing and the screen running, and said “that’s awesome.”
I suspect that explains the success of some large percentage of GUI-driving tool sales: To people inexperienced with testing, having stuff fly by the screen is, in fact, “driving it”, is the same thing as testing. And it’s cool, right?
I had an HOUR to explain the real challenges with this at ST&P Conf. I had some success.
Sadly, Not everybody goes to ST&PConf. (But you can still read my column for free, every month, downloadable as a PDF from http://www.stpmag.com ! 🙂