Many years ago I took a management class. One of the exercises we did was on achieving consensus. My group did not reach an agreement because I wouldn’t lower my standards. I wanted to discuss the matter further, but the other guys grew tired of arguing with me and declared “consensus” over my objections. This befuddled me, at first. The whole point of the exercise was to reach a common decision, and we had failed, by definition, to do that– so why declare consensus at all? It’s like getting checkmated in chess and then declaring that, well, you still won the part of the game that you cared about… the part before the checkmate.
Later I realized this is not so bizarre. What they had effectively done is ostracize me from the team. They had changed the players in the game. The remaining team did come to consensus. In the years since, I have found that changing the boundaries or membership of a community is indeed an important pillar of consensus building. I have used this tactic many times to avoid unhelpful debate. It is one reason why I say that I’m a member of the Context-Driven School of Testing. My school does not represent all schools, and the other schools do not represent mine. Therefore, we don’t need consensus with them.
Then what about ISO 29119?
The ISO organization claims to have a new standard for software testing. But ISO 29119 is not a standard for testing. It cannot be a standard for testing.
A standard for testing would have to reflect the values and practices of the world community of testers. Yet, the concerns of the Context-Driven School of thought, which has been in development for at least 15 years have been ignored and our values shredded by this so-called standard and the process used to create it. They have done this by excluding us. There are two organizations explicitly devoted to Context-Driven values (AST and ISST) and our community holds several major conferences a year. Members of our community speak at all the major practitioners conferences, and our ideas are widely cited. Some of the most famous testers in the the world, including me, are Context-Driven testers. We exist, and together with the Agilists, we are the source of nearly every new idea in testing in the last decade.
The reason they have excluded us is that they know we won’t agree to any simplistic standard based on templates or simple formulae. We know those things look pretty but they don’t help. If ISO doesn’t exclude us, they worry they will never finish. They know we will challenge their evidence, and even their ethics and basic competence. This is why I say the craft is not ready for standards. It will be years before all the recognized experts in testing can come together and agree on anything substantial.
The people running the ISO effort know exactly who we are. I personally have had multiple public debates with Stuart Reid, on stage. He cannot pretend we don’t exist. He cannot pretend we are some sort of lunatic fringe. Tens of thousands of testers have watched my video lectures or bought my books. This is not a case where ISO can simply declare us to be outsiders.
The Burden of Proof
The Context-Driven community stands for excellence in testing. This is why we must reject this depraved attempt by ISO to grab power and assert control over our craft. Our craft is still an open marketplace of ideas, and it is full of strong debates. We must protect that marketplace and allow it to evolve. I want the fair chance to put my competitors out of business (or get them to change their business) with the high quality of my work. Context-Driven testing has been growing in strength and numbers over the years. Whereas this ISO effort appears to be a job protection program for people who can’t stomach debate. They can’t win the debate so they want to remake the rules.
The burden of proof is not on me or any of us to show that the standard is wrong, nor is it our job to make it right. The burden is on those who claim that the craft can be standardized to study the craft and recognize and resolve the deep differences among us. Failing that, there can be no ethical or rational basis for standardization.
This blog post puts me on record as opposing the ISO 29119 standard. Together with my colleagues, we constitute a determined and sustained and principled opposition.
Thanks James. This sort of contribution is important. You are talking about exactly the problem that struck me when I looked closer into what ISO were doing. ISO has clear rules about the need to achieve consensus, which is defined as “general agreement, characterized by the absence of sustained opposition… by any important part of the concerned interests”.
The working group attempted to resolve the need for consensus, as you say, by defining the opponents as irrelevant. The Context Driven School, in their eyes, is not an “important part of the concerned interests”.
That stance might look plausible to people who know nothing about the testing profession, but it doesn’t bear a moment’s scrutiny from anyone who understands how testing has progressed over the last couple of decades.
The ISO 29119 working group might claim that the need for consensus applied only to the fine detail being discussed by those who were already committed in principle to a standard in that form. That stance would assume that all reasonable, professional testers were agreed on the need for ISO 29119. This would be a fine example of a circular argument and would lack any serious credibility. How can a standard be generally applicable in a diverse profession when the only people who developed it were from a single faction? How can consensus apply only to those who were already committed to agreeing about the big important issues?
So, what is ISO going to do? Will it allow the ISO 29119 working group to ignore ISO’s own rules? Will it continue to pretend that consensus has been achieved when that is quite obviously not the case? Or will it concede? It is a difficult problem for them, but it is a problem entirely of their own making.
Hi James,
Today I got curious about ISO/IEC/IEEE 29119 and why it seems to make some testers I know (of) very upset. I almost bought it since I am dying to know what’s in it! In my mind I was sitting with the paper while trying to make sense of each and every sentence in it. I wanted proof that it was good or bad for me as a tester!
Here’s about the time when I ended up on this page to read your blog and stopped feeling the urge to go buy the standard. I also came to think of the time I tried to use IEEE-829 to write a test plan (ISTQB class told me to) for a customer who asked for a test plan. It turned out that they didn’t care at all about my test plan because what they wanted help with was the actual testing. (To get ideas on how help them with the actual testing I got a good help from satisfice.com thank you very much!).
What really caught my interest here was what you write about “The Burden of Proof” and I don’t fully understand it. I am more curious about what you mean with “The burden of proof is not on me or any of us to show that the standard is wrong, nor is it our job to make it right.” If you don’t mind, please help me to understand how you motivate this.
[James’ Reply: “Burden of proof” is a philosophical idea. It refers to who is required to make the case. For instance, if you claim that I stole money from you, I am not required to prove that I didn’t. The burden is on you to show that I did. And if you claim that there are chickens on Mars, I am not obliged to accept it as true unless you provide sufficient evidence. In fact if you claim that there are definitely NOT chickens on Mars, you still have a burden of proof (albeit an easy one to attain). Only if you say “I don’t know whether or not there are chickens on Mars” is there no burden of proof on you, because you are reporting an internal mental state.
If somebody claims that they have a standard that covers the work of all software testers, they have a burden to demonstrate that their standard is acceptable to credible people in the craft. It’s not like these standardizers were elected to represent us and can commit us to standards without consultation. Nobody elected them. They have no special standing.]
You also wrote “The burden is on those who claim that the craft can be standardized to study the craft and recognize and resolve the deep differences among us.” Why is that?
[James’ Reply: Because that is a claim they are making about US, Annie! They can’t simply presume to know us, can they? We don’t presume that they understand us and how we work, do we?]
Best regards,
Annie Rydholm
I feel so strongly about ISO29119! I started my career trying to follow standards and living by ISTQB rules – it was when I stepped away from this and stopped fighting against my brain which was desperate to do some thinking that I think I became a better tester. Not only a better tester but one with some actual integrity. Adding rules to the existing state of confusion is going to slow even more testers down in ‘discovering themselves’.
[James’ Reply: Congratulations on your birth as a thinking tester.]
I really don’t understand how you could get a group of people together in a room (or a metaphorical room) to discuss how to make things better and come up with something like this. The idea that we should use the same ‘best practices’ no matter what the organisation, no matter what the project, no matter the mix of testers (I could go on) is laughable.
I only have one best practice that I will follow religiously – I hire intelligent testers who can think for themselves to actually assess the project they are on and do what they think is best.
Do you remember Esperanto? It was a language that was ‘invented’ that you could use anywhere because all countries would use it as their second language to create a universal language….sounds ridiculous and never caught on? Surprising… (although I think some people still learn it just incase, sounds familiar)
[James’ Reply: Mi parolas Esperente, iomete. La hotelo estas luksa kaj grandaj…]
Thanks for the post, it is reassuring to see so many testers agreeing on this topic!
Sarah (@girlstest2)
@JC – They will do what they have always done and simply proceed with their stated plan as it stands. If you look at the principals involved they all have a vested interest in perpetuating certain myths about software testing and what constitutes professionalism and “best practices” within it. These myths include but are not limited to regulatory compliance, the role of V&V activities, and the promulgation of command and control structures for cost savings. Just as with the ISTQB BOK the framers of this “standard” have apparently gone out of their way to ensure the documents are so arcane as to require specialized instruction and coaching if you wish to adhere. It also just so happens many of them offer this sort of service either as a primary means of income or as a supplementary income source.
In order to sell this idea and thereby sell their services, they must convince the people with the checkbooks they have the right of it. If they come out and say it is but one of several methods for testing software many of those checkbooks are likely to snap closed. In order to keep the pyramid growing and the cash flowing upward they MUST make the claim they are the keepers of the true knowledge and insight. Anyone else can be branded a heretic or an apostate or a hysteric or simply foolish.
@James – Evocative as always. I think we’ve both done this and had this done to us in our professional careers. It irritates me to no end when the snake oil salesmen and saleswomen with large advertising budgets pollute my personal work and reputation. I’ve had customers adamantly insist I be [insert standard] compliant and be able to show said compliance while working. More often than not these same clients had no idea what that meant since they themselves had never read the documents. All they knew is there was some sort of international standard out there and they wanted to make sure anything they did was compliant with it since they were sort of afraid of admitting they didn’t know what it was they really wanted from me. I’ve had commissioned articles refused because they were critical of organizations who either directly advertised or whose members were major buyers of advertisements. I’m glad there is a growing organization forming to fight this pyramid scheme. I only wish I could have been there at CAST this year as I had originally planned.
I didn’t attend because my company sponsor pulled their backing at the last minute. The stated reason was essentially that the development engineer had never heard of CAST. They suggested I attend one of the STAR conferences instead and perhaps get some training on the ISTQB while there.
[James’ Reply: In the words of Bugs Bunny, “What a maroon!”]
I was wondering what took you so long to criticize this new standard!
[James’ Reply: As I get older I am trying to get my students to take up the job of saving the world.]
I think it is OK to have a rule book and then we also break the rules like real testers.
[James’ Reply: Whose rule book? My rule book?]
You may want to clarify if you are just disagreeing with the Test Process and Documentation part of the standard or if you are against all parts of the standards.
[James’ Reply: I’m not disagreeing with the standard… THERE IS NO STANDARD. In order for me to disagree with it, it would have to first exist. What exists is a bunch of ignorant opinions from shadowy people.]
In my opinion some part of the standard is still good to have since it gives a base definitions for testing and few test design techniques.
[James’ Reply: That is not an argument for it being good. You can’t say that a document should be a standard because it has definitions in it. Please have a higher bar for your craft than that! I have definitions, too. Why don’t you call my body of work a standard?]
It will help fresh grads to get introduced to testing.
[James’ Reply: That’s terrible! We don’t want fresh grads getting introduced to testing on the false belief that this “standard” represents our craft.]
When the word Standard is added, it obviously brings in more attention to the fresh grads who wants to start from basics (believe me….when i say grab and read a book on testing techniques people hardly listen, but when i say go and read this ISO standard they mostly make the attempt).
[James’ Reply: That’s why we have to discredit that junk.]
I have also interviewed many ISTQB certified testers who are not worth employable as testers. It’s sad part that not may universities teach the testing ideas in most part of the world. So i think, its also an important responsibility for the senior testers to introduce the fresh testers with the context driven testing approach(like my seniors did). People who deserve to be good testers will start appreciating context driven testing as they mature in the art. It doesn’t matter if we are being a good cop or bad cop, what matters is if we are able to nail down the tough culprits (Defects) for which we are being paid for!
James, where can be downloaded the debated ISO 29119?
At http://www.softwaretestingstandard.org/ I found only a table of contents with no details.
Aha, found it, it requires $194 for a copy. Damn.
[James’ Reply: This is part of what we mean by “rent-seeking.” ISO first seeks to declare an invalid standard, then they force us to pay them money to learn about the standard, whether that be to critique it or comply with it.]
Gokul: “when i say grab and read a book on testing techniques people hardly listen” in this case maybe tell them to pick up a book called “The Agile Samurai”. It’s a short read and inspires thinking, rather than NOT thinking.
“It’s sad part that not may universities teach the testing ideas in most part of the world.” I would disagree with this. EVERY university, regardless of a students declared major or course of study, teaches you how to think. And thinking is our craft.
James, thank you so much for this blog and for your work to promote thinking in testing. My previous manager very much promoted brainless “testing”. While under his tutelage I obtained the ISTQB foundation certification and I’m quite embarrassed to even talk about it. The more I think about it the more I realize how useless it is. “Let’s all make sure we are only looking at a problem from a single viewpoint so as to avoid finding issues that may come up from other viewpoints.”
[James’ Reply: We forgive you. Lots of testers go through this.]
As I read this blog and become more informed about what this ISO 29119 “standard” is doing, it occurs to me that prescribed standards are one of the reasons I left the field of education and entered technology, specifically software testing. The human mind is still, by far, the most powerful and interesting tool that exists to interact with, explore, and test software. The nuance of the human mind provides us as thinking, context-driven testers, the unique ability to learn about our environment and make critical decisions based upon our awareness of the environment. To me, it is not unlike a continuation of the evolution of thought, observation, and decision making that has brought humanity this far.
I will not make a diatribe here about the multiple comparisons (and failures) that I imagine happening in software testing if ISO 29119 is indeed accepted as “standard.” In ten years as an educator in two different states (in schools serving underprivileged children), I watched as an effort toward standardization of content attempted to (and in some cases completely succeeded) in destroying teacher and student creativity at the expense of something “looking pretty” for outsiders. If this happens, as I said, I anticipate software testing and software testers going in the direction of education in the United States: a pretty book, full of ideal standards, that kill creativity, innovation, and exploration (substance) for the sake of shallow agreement.
[James’ Reply: Thank you, Jessica. Of course, I bounced out of the school world after only 11 years in it. I couldn’t fit into their standard ways.]
I think it is OK to have a rule book and then we also break the rules like real testers.
[James’ Reply: Whose rule book? My rule book?]
[Gokul] I meant the standard as the rule book because if the organization will be evaluated for their testing standard against a check list which is pathetic! It will be fun to watch organization following all these standards and still having customer complaints(this is what i meant by breaking their rules)!
[James’ Reply: This does not answer my question. Whose rule book? Everything “standard” is produced by someone. Either that is adopted by other people or it isn’t. The Metric system is a standard, but so is the Imperial system. We don’t follow the metric system in the USA. It’s not our standard.
I see your point that even following those rules won’t help them, but I don’t want to let some random child create standards for me just because it’s good for their education to fail.]
In my opinion some part of the standard is still good to have since it gives a base definitions for testing and few test design techniques.
[James’ Reply: That is not an argument for it being good. You can’t say that a document should be a standard because it has definitions in it. Please have a higher bar for your craft than that! I have definitions, too. Why don’t you call my body of work a standard?]
[Gokul] I like this comment! I think they are using the word standard just to reap commercial benefits for themselves.
“It’s sad part that not may universities teach the testing ideas in most part of the world.”
[Miller.Y]I would disagree with this. EVERY university, regardless of a students declared major or course of study, teaches you how to think. And thinking is our craft.
[Gokul] It is not just about thinking, its also about thinking as a tester, followed by documenting and justifying your thought process (which is important for business while working in regulated environment like mine). How many times have you said this to yourself that “why this developer is not thinking about a simple scenario and end-up providing a defective software to you to test!” Does it mean the developer had not taken a course of study? I think that is what is missing in our university teaching system today.
Hi James,
So when do you think that the ISO will introduce their new standard for writing novels? There must be hundreds of big name authors who would love to lend credibility to such an exercise, and I think that we can all agree that the process of writing novels would benefit from standardisation. After all this would help new writers get started, and give them a common language when speaking amongst themselves, and dealing with agents and publishers.
I think that we can all agree that the above paragraph sounds absolutely ridiculous, but after reading around this topic for the past couple of weeks, these seem to be the arguments that are being trotted out by those in favour of this standard.
These arguments do not stand up to even basic scrutiny if you think of testing as a creative process.
ISO 29119 looks like to be another of ISOs effort to bring in a standard for testing like all its other standards which are achieved by organizations through certifications only(most of the time). Like James said, it increases the burden of proof/evidence and the documentation efforts would be higher than than the effort spent on intelligent testing. In my last job, apart from testing,I was also an MR for the ISO 9001 QMS and found it to be taxing and really did not see any big enthusiasm from any of the PMs/TLs to follow the same other than merely for the sake of Audits and certification. The process compliance was always seen as an additional project overhead. There was always a big gap between what they actually did to produce quality work and what they were supposed to do as per defined process in which they did not find any value..
Then I turned to the other alternative school of thought for testing and is recommending context based testing to clients who really listen. Being in a service industry, sometimes we are forced to follow what our clients believe in and mandate as they are all aware of only what they see as “International standards”.
And , actually from my experience, not everyone of the clients really check whether you really follow the process in every phase of Development / testing for their projects. They are satisfied as long as they see you as CMMI certified/ ISO certified etc..Only we are burdened with additional efforts of documentations, Audits, Proofs, etc..
This 29119 has to be stopped as ISO as an Organization may not be the Industry experts on testing to define the standards on their own. If this is being implemented without consensus, then it would become a classic example of why processes always fail. Whenever a process is implemented without the buy in from those who are expected to use/execute the process and when the value adds from the process is not visible to them, a process would certainly fail…
What this ISO aims at is creating a monopoly where only companies which have money but cannot compete goes and purchase these certificates to entice new customers.If you can’t beat competition kill it seems to be the mantra.
My take :
Do software testing exist even without these so called standards? Answer is BIGGGG YESSSSS ..
Do these standards prescribed work for all sets of organizations ,for instance I know of a company which excels in testing Gaming software,they don’t have any standards which are prescribed by this ISO body. The customers are happy with the end result of their testing and that is all that matters and not the test case or metrics or templates or over documentatioon.How on earth can anyone doucment Gaming software or rather prepare test steps or rather for flight simulation software to just name a few.
Testing is an art and a craft honed by skilled craftsmen which doesn’t require certifcates but what it requires is passion to test which is ampy exemplified by many many great tester’s whom I have admired and on whose blogs I write or comment on.
If following certain set of standards (so called standards) prescribed by ISO body could result in a defect free quality software then shouln’t there be a standard for developer’s too so that they give a defect free code.
This ISO can never ever work for simple reason that no user of a software /hardware or anything worth it will read a user manual on how to use the same.
So the ISO could well try to implement standard USER MANUAL for user to read and comply with disclaimer that aything he/she does beyond what is stated in USER MANUAL and any defect or issues raised therein would not be accepted.
If at all the ISO body wants to do anyting for betterment of the testing community they should stop fiddling with this.
The biggest issue that I have with the ISO 29119 standard is the lack of openness.
[James’ Reply: That’s the biggest issue? Not the fact that it is completely illegitimate??]
Surely it is in the communities best interest to have standards open and transparent?
[James’ Reply: I agree, but first the standard has to be a standard at all, right?]
I imagine the way a company could then make money from the standard is through the auditing process or training. However if it was an open standard, then the auditing process would be open for competition and there might be a loss of income but this doesn’t seem like ethical and professional behaviour. It’s like me saying, “nuh-nuh” *tauntingly dangles standards document just out of reach of a tester* “I have the best trade secrets and I’ll show them to you. For a fee, how do I know they’re the best? pay me to find out”
[James’ Reply: You keep using the word standard, but then you talk as if it’s not a standard, but rather an open source methodology. I don’t have a problem with people creating open source methodologies.]
I was having a conversation with someone at work about this issue, they were concerned that if this standard was taken seriously by business than more testing jobs would be offshore to a cheaper labour force and we would be out of work. Would that necessarily be a bad thing if there were more jobs in places like India where they seem to have larger rates of Poverty compared to Australia?
[James’ Reply: The point is not that we don’t want them to have jobs. They can have all the jobs they want. The point is when that happens by dumbing down our craft and by consulting companies promoting fraudulent and damaging “standards.”
I want you to be free to practice your craft and show an employer that perhaps you are a better tester than some faceless guy in a consulting company 10,000 miles away. So, let’s oppose efforts to make the craft unsafe for anyone except Factory school robot men.]
I have been testing for about 11 years now and have worked both as per standards (during the initial ISTQB days, and yes I have done that certification) and as a context-driven-tester (though unknowingly). As soon as I arrived at a stage where I had to lead a team of testers and plan testing for a project/product, I lost the connection with these so called standards. I found every project had different requirements, every client had some pre-conceived notions of kind of testing they would like to get done and every test team had their own idea about the ‘best practice’. I have worked in Service based organisation throughout and therefore often my ideas about best testing approach for a specific project have been rejected because they do not conform to Organisational standards.
Standards curb the creativity and oppose changes. People who promote these standards often oppose context-driven-testing because they hide behind them to cover their own incapability.
I would just like to comment that I completely understand ‘your stand’ on our craft!
Real standards are open. Real standards are to serve the community in which they are accepted and applied.
From what I read on this blog and the comments, I can only say the ISO workgroup responsible for creating this standard are just practicing mental masturbation
(http://www.urbandictionary.com/define.php?term=Mental+Masturbation ).
So you haven’t even read the standard???
[James’ Reply: It is not necessary to read a fake standard in order to understand that it is fake. On the other hand, it is necessary to read and respond to the arguments of your opponents in order to be taken seriously in a debate. This is something you have not yet done.]
I think you’re being a little bone headed about this.
[James’ Reply: Oh am I? Let’s see whether you can explain or justify this remark…]
The ISO 29119 standard is a good basis for building a testing capability from the ground up, using an accepted basis for a framework.
[James’ Reply: This is an assertion hanging in the air. So far it doesn’t address my concerns, nor is it supported with evidence. Perhaps that’s coming next…?]
However it’s a given that risk based assessment a la ISO 25010 is a good addition, as is the concept of user centered design static testing and user centered test design.
[James’ Reply: No, kiddo, it’s not a given. But if you think it is, you have to support that with evidence or some sort of logic. Who has given this? How do you know it’s given? That’s how debate works. It’s not just a string of assertions.]
These are not mutually exclusive, and to get all precious about 29119 being the basis for all evil is somewhat laughable.
[James’ Reply: I think the precious thing is that you are writing as if you sincerely believe you are making an argument. In a child, that would be cute. I’m concerned because I suspect you are an adult. We expect more of adults than for them to confuse bland, vague assertions for incisive discourse.
This is a big problem with our industry: the complete inability of so-called professionals to construct a coherent reasoning process in support of their ideas. In this blog post, I made an argument. And you can try to respond to it, if you want to. What you have written is not a response– it’s a bleat. You are bleating. Stop bleating and start talking, please.]
I know one of the contributors on the committee and I don’t think they were thinking that 29119 would be a hard and fast exclusive standard, rather its the basis around which a number of other standards and new concepts can be applied, and given the breadth of my own career (anything from multi billion dollar defence projects to small mobile app projects) its clear you can use 29119 as a framework, around which you can fit just about any testing method, including agile, UCD, exhaustive testing and even ad hoc user experience assessments.
[James’ Reply: That is irrelevant to the point I made in this blog post.
So, far, it sounds like your argument is “I personally like 29119 therefore anyone who disagrees with me is a bonehead.” Well, you get to have an opinion. But excuse me if I can’t take your poorly reasoned opinion seriously. I’ve been in the industry 33 years, and I think 29119 is an insult. It is an insult to you as well as me. I’m embarrassed that I must defend you from being insulted by this bankrupt “standardization” process. You should have the good sense to know that standardization on ANY level must be sensitive to concerns of working practitioners, and that your personal good feelings do not justify overriding that of an entire community of serious professionals in this field. ISO 29119 ignored our concerns (at least 1000 of us who signed the petition) and did so in an obviously deliberate, cynical fashion.]
I’d like to see a little more exploratory thinking from my testing brethren and less closed mindedness and pure opposition. If you don’t like it, stay away from it, but trying to tear it down isn’t a great look guys and girls.
[James’ Reply: So, when we offer arguments– which are not even addressed by your side of the debate– WE are close-minded? What am I supposed to think of you? How can you imagine that I could treat you or any of your cronies with respect when this kind of non-responsive arrogant braying is your idea of a comment? How do you live with yourself?]
I am not familiar with ISO 29119 (although I do have some awareness of other ISO standards) but I think you have written a very interesting article here.
I am a little confused by your comment “…ISO 29119 is not a standard for testing…” when, as far as I can tell, it *is* A standard for testing. Had you worded it “…is not THE standard…” I would completely agree.
[James’ Reply: It is not a standard except in two senses: it has the form of a standard, and it is called a standard by the commercial organization that created it and profits from saying so. But those are trivial senses. Surely, the far more important issue is whether practitioners follow it, whether they think it ought to be followed, and whether it was produced using a process that was responsible and acceptable.
If you are going to call it a standard by the first two senses, then lets also elevate every other process document in use in the industry to the status of standard. If the bar is that low– that anything stamped with the word “standard” and that makes any sort of recommendation about practice gets to be called a standard by the rest of us– then 29119 will be immediately swamped by a thousand rivals, including my own methodology: Rapid Software Testing. Of course, I do not claim my methodology is or should be an industry standard. But, that’s only because I’m a responsible professional. I thought we were all supposed to be responsible.]
From what James Christie wrote, it sounds like ISO have a bit of a problem gaining concensus (although I cannot tell if the definition he provided is ISO’s definition or someone else’s), and I can see that this is an issue – and probably your biggest concern. As per your introduction, had they simply decided to base the standard on less than, say, 10% of interested parties objecting they would probably be in the clear (and to be clear: “10%” was plucked from thin air; nor do I attempt to define “interested parties”).
[James’ Reply: No, they would not be in the clear, for reasons I have stated clearly in my post.]
The page “http://context-driven-testing.com/” gives an interesting hypothetical example of the testing needs for control software for an aircraft vs. those of a word processor. In this case it is very clear that taking a context-based approach is by far the best (and only realistic) approach. However (and without knowing the standards needed for testing an aircraft’s systems) it seems plausible that working to ISO 29119 *in that context* is appropriate. The ISO standard will not guarantee perfect results, nothing can; but it could be considered that following ISO provides a step, possibly one of many thousandd, in the right direction to help prevent aircraft falling out the sky.
[James’ Reply: If you want to follow the Context-Driven ethic then you cannot say whether it is appropriate or not. By your own admission, you don’t know enough about the standard or the context to judge that. I know enough about the content of the standard to know that it cannot be appropriate– any more than astrology or Feng Shui is appropriate. ISO 29119 was not created according to any responsible research process. The framers of that document made no attempt to understand or respond to modern testing practice (which would have been a social science research project), nor to base their work on any other scientific or evidence-based foundation. In fact, in response to all challenges to their work they say– nothing. They have no rebuttal. They sulk like children and complain about being attacked. They have consistently refused to debate the content of their document with the prominent and well-reputed professionals, such as I, who have called them out as frauds. To this moment, they have made no official response to the ISST petition and detailed statement of concerns.
I may be right or I may be wrong in my judgments about testing, but I ANSWER any and all serious critique of my views. That’s what serious people do.]
If you have ever been involved with outsourcing to any of the off-shore body-shops (and I would be very surpirsed if you have not, at least to some extent) you will know that it can be very difficult to predict the quality of staff you are allocated.
[James’ Reply: Of course.]
(And when you later find out you have some duffers on the team you will already have thrown cash at them and will realise that changing the team is costly, usually in terms of timescales.) Requiring staff to be have some sort of standards-based qualification makes no guarantees that they are good, but can help weed out some (but by no means all) of the duffers.
[James’ Reply: No, it won’t. And of course you have no evidence that it will. It’s your opinion, but your opinion is wrong. As a man who has worked in three different outsourcing companies and also been on the client end of that equation and also been sent to India to train such testers on behalf of my clients and also been involved in the development of one of the first certification programs for testers, I assure you that “standards” don’t solve the problem of human variability. Furthermore, human variability is not even a problem. Incompetence is a problem, but that can be readily recognized and dealt with through ordinary management processes. Given that I disagree with you, then, at the very least we have a controversy here that is worth investigating, not a great reason for a piece of shit like 29119.
Also, outsourcing is a often a bad idea. If there were a standard, the standard should say “don’t do that.” Boeing outsourced much of the work on the 787 and paid many billions over budget for that delusion. It can work, of course, but for the most part it is a hideous waste of money. Furthermore, where I have seen it work well, the success had nothing whatsoever to do with standards.]
This is certainly one of the benefits of ISO 9000, of which I have much more experience. There are, of course, other reasons why standards are beneficial – often to help inexperienced staff work to a more predictable standard.
[James’ Reply: I also have experience with ISO 9000. Since the early 90’s I have encountered quite a few ISO 9000 companies that have me consult or teach. I have not once witnessed any positive benefit of ISO 9000 that related to any everyday work process. ISO 9000 is completely ignored by people who do work. I can almost say that for ISO 13485, but I have seen actually seen 13485-based SOPs being acknowledged by workers at one of the medical device projects I experienced. I was on that project for a couple of months, though, before anyone even showed me one of those so-called “standard operating practices.” By comparison, I have not yet encountered a client who told me that anything I was teaching them about testing would have to be reconciled with ISO 9000-3.
Not all standards I have encountered are worthless charades. I admire the ISO 60601 series and have used it. I have found ISO 9126 to have useful information in it.]
In my various roles (generally in software delivery and project management) I call on my training and experience in PRINCE2, MSP, CSM, Risk Management, ISO 9001/TickIT and dozens of other areas to provide the best service I can in every situation. I would expect ISO 29119 to be another area of knowledge that I could call on to address the problem at hand. If so, it sounds like ISO 29119 is simply another string to the bow of providing a context based service.
[James’ Reply: I think you should be a lot more skeptical about bullshit being promoted as a standard. In any case, testing is my area of expertise. ISO 29119 claims to be a testing standard. So I’m not sure what you think you are accomplishing by telling me, an expert in the field who is warning you about 29119, that you think it’s a valid string to your bow.]
Perhaps it just boils down to the term “standard” being the issue here? Had ISO referred to it as (just another) “working process”, that would have kept it open to be used as and when appropriate, rather than being seen as something that would fit every situation.
Thanks for getting the discussion going and the thought processes flowing!
Hi James. I agree with you, this can’t be a standard if it doesn’t take into account the opinions of the profession as a whole. I have some questions:
1) Is it worth to fight it?
[James’ Reply: Of course it is!]
If it is such a piece of garbage, wouldn’t the smart people realize this and ignore it?
[James’ Reply: If by “smart” you mean true experts in software testing, of course you are right. But that’s not the problem. The problem is we have false experts running around all over the place, and we have non-experts who run companies and government agencies. This “standard” is and will be used as a weapon to suppress testing expertise in our craft.
Also, whether it is or is not garbage is irrelevant to my point. My point is that it is not a standard. It’s an illegitimate power play.]
I guess a customer will come eventually and ask “Do you follow ISO 29119?” and I might lose the business if I answer no, but would I want to work with them in the first place?
[James’ Reply: Of course you would have wanted to work with them. They don’t know what they are doing. You do (in this hypothetical, anyway). But you won’t get the chance to show them a better way because they have been poisoned by a cynical marketing campaign by your rivals in the industry who don’t believe in a level playing field.]
It might be a good opportunity to educate them. If they are stubborn and want to follow the document, walk away.
[James’ Reply: Yes, it’s a good opportunity to educate them. But if you reject the standard that they assume is good, will you get that chance? Will you have to tell a lie and say you support it?]
2) This might sound facetious, but would you be OK if it had another name? If it made clear that it doesn’t cover the complete practice of testing?
[James’ Reply: Anyone is free to publish any garbage they want, and many people do. The problem here is that ISO is a standards organization. Their standards are not supposed to be the political opinions of one sub-group of a craft.]
3) Do you think a testing standard is possible? You wrote “the craft is not ready for standards” but will it ever be? It might be that a testing standard is actually impossible.
[James’ Reply: A standard is possible when we, as a craft, come to agreement about how to do our work. We are nowhere near that, and furthermore, no one has even seriously tried to forge such an agreement– except maybe me and my community. We’ve made our heartfelt and well researched suggestions and arguments. These have been ignored by the consulting companies, because they see our ideas as a threat to their profits.]
Thanks in advance
—
David
Hi James, Thank you for this very thoughtful blog. I agree with you that 1) this is not a standard because among other things it excludes an important school of thought and practice-CDT from the conversation and 2) because it proposes a “one size fits all” approach to testing, test organizational structures and methods without a meaningful basis other than “because that is what we’ve done before” (see the comments on the IEEE 829 supersession) on the ISO site, and 3) because I do agree that most negative comments in support of the 29119 are either circular or pseudo reasoning (also agree this is a big problem in software testing in general) and therefore invalid. What I would have liked to see from the WG was an acknowledgement that this compilation can serve as a set of guidelines that, coupled with the advice of experienced, informed testing professionals who accept CDT as a viable approach, can serve as the foundation for building a viable, customer, client, in-house relevant testing practice. What the 29119 WG is asking practitioners to do is what we in the military refer to as “drink the Kool-Aid” and it is a horrible idea. Thanks again for your insightful blog post on this issue.
Good Morning James –
I’m new to this field, and I have just completed a new Incident Management program for emergency response scenarios. I’ve been asked about validation testing. If I don’t use 29119, is there any ‘standard’ to follow?
Sorry for the ignorance…
[James’ Reply: First, you should not be making this decision. Just that you ask this question shows that you are not qualified. It’s not your fault– you’ve been placed in this position without being given training or supervision that you need.
But I want to help you specifically, and not just brush you off. So here are some clues:
“Validation testing” does not mean “test according to a testing standard.” It can mean a few different things, though. It could mean “testing” (validation might be a redundant word). It could mean “validation as opposed to verification in the context of performing what regulatory people would call ‘verification and validation'” (if so, then it means to determine by testing or whatever means necessary that the product fulfills its stated purposes, as opposed to testing against a low-level technical specification.) It could mean to test it for compliance to a specific technology standard (this may be an industry-specific spec such as an Internet RFC). Or it could mean “test the capabilities of the product as opposed to reliability, performance, usability, or other so-called non-functional attributes” (validate that the product can work).
I have heard all these things called “validation testing.”
I have never seen a general testing standard that was worth reading, let alone following. Such “standards” are written by arrogant, sloppy consultants. I’ve met many of them. You will find that my name is on the IEEE-829 standard– a standard that I repudiate. But, yes, I was on that committee and saw that awful process, first hand.
What you probably need to do is learn your product thoroughly, then create whatever test data and fixtures are needed to run experiments that exercise the product and allow you to detect important problems in its behavior. You really don’t need a standard to tell you that, do you?]
Bill