Paul Gerrard believes there are irrefutable testing axioms. This is not surprising, since all axioms are by definition irrefutable. To call something an axiom is to say you will cover your ears and hum whenever someone calls that principle into question. An axiom is a fundamental assumption on which the rest of your reasoning will be based.
They are not universal axioms for our field. Instead they are articles of Paul’s philosophy. As such, I’m glad to see them. I wish more testing authors would put their cards on the table that way.
I think what Paul means is that not that his axioms are irrefutable, but that they are necessary and sufficient as a basis for understanding what he considers to be good testing. In other words, they define his school of software testing. They are the result of many choices Paul has made that he could have made differently. For instance, he could have treated testing as an activity rather than speaking of tests as artifacts. He went with the artifact option, which is why one of his axioms speaks of test sequencing. I don’t think in terms of test artifacts, primarily, so I don’t speak of sequencing tests, usually. Usually, I speak of chartering test sessions and focusing test attention.
Sometimes people complain that declaring a school of testing fragments the craft. But I think the craft is already fragmented, and we should explore and understand the various philosophies that are out there. Paul’s proposed axioms seem a pretty fair representation of what I sometimes call the Chapel Hill School, since the Chapel Hill Symposium in 1972 was the organizing moment for many of those ideas, perhaps all of them. The book Program Test Methods, by Bill Hetzel, was the first book dedicated to testing. It came out of that symposium.
The Chapel Hill School is usually called “traditional testing”, but it’s important to understand that this tradition was not well established before 1972. Jerry Weinberg’s writings on testing, in his authoritative 1961 textbook on programming, presented a more flexible view. I think the Chapel Hill school has not achieved its vision, it was largely in dissatisfaction with it that the Context-Driven school was created.
One of his axioms is “5. The Coverage Axiom: You must have a mechanism to define a target for the quantity of testing, measure progress towards that goal and assess the thoroughness in a quantifiable way.” This is not an axiom for me. I rarely quantify coverage. I think quantification that is not grounded in measurement theory is no better than using numerology or star signs to run your projects. I generally use narrative and qualitative assessment, instead.
For you context-driven hounds out there, practice your art by picking one of his axioms and showing how it is possible to have good testing, in some context, while rejecting that principle. Post your analysis as a comment to this blog, if you want.
In any social activity (as opposed to a mathematical or physical system), any attempt to say “this is what it must be” boils down to a question of values or definitions. The Context-Driven community declared our values with our seven principles. But we don’t call our principles irrefutable. We simply say here is one school of thought, and we like it better than any other, for the moment.
Axiom 0 : stop and use your brain ! The rest are details that might be helpful, or not.
Gerrard mentions ‘defending your position’; I think the only defense worth having is to communicate your analysis of what to do in your circumstances. Stating that ‘I followed Gerrard’s axioms’ is not a defense.
[James’ Reply: Good point. I wonder how many testers will try to use it as a defense?]
Hello James,
Interesting view you posted here. As it made me think what “school” I’m belonging to. I wished I could say immediately I belong to the Context-Driven School or Paul Gerrard’s School. Though it made me think. And I believe articles/postings/books who trigger me to think are good postings.
These articles made me write some thoughts of mine down in my blog. And during this writing I noticed that Axioms as stated by Paul Gerrard could also be seen as the starting of a new dictionary. Only not the explanation what those words mean, instead how to fulfill the meaning of it. And calling them Axioms might be too strong for them, as I understood axioms are taken granted for being the truth. And somehow I have problems with that. As I strongly believe it depends on the situation you are into.
Perhaps I interpreted both articles the wrong way. Then the benefit of it that I can learn from it, and within some time I can acknowledge I was wrong.
Thanks again for sharing your thoughts about this. I triggered me to learn something new and keep on looking for more information related to these topics.
With regards,
Jeroen
[James’ Reply: The “axioms” can have heuristic value, even if they aren’t really axioms or theorems or for that matter– true!]
Hi James:
As one of the catalysts for Paul’s axioms, I feel I should post my initial reactions to this now, though I assume “this one will run and run”, and I will post more when I’ve time (also I need to post on Paul’s own blog).
1. You may remember when we discussed this in west London a few years ago – you persuaded me that “principles” is a better word than “practices” for what I was saying then, and I accept that.
Paul has now tried to go further with axioms, and I believe others have already tried this before Paul.
But my original idea was a set of “always-good practices/principles” each of which is context-interpretable on a formal-to-informal scale. I was saying (and still am) that these are all the good things you should consider, and address in some way – but the way you do them will vary considerably (and after due consideration!). And I used Goldratt-Dettmer as a mechanism for that, which I still believe works.
[James’ Reply: It sounds like you are talking about heuristics. But heuristics are also context-specific. I just don’t understand the mentality of people who want to say “this principle is inherently good regardless of context”, unless they are declaring a school of thought, in which case they know they don’t speak for everyone and all situations. What’s so hard or unpleasant about acknowledging context, discussing it, and talking about the skills you need to apply the ideas appropriately?]
2. If we use Bret Pettichord’s quoted set of schools (which I think is a useful analysis), they are Analytic, Routine, Quality and Context-Driven. But I say: if Context-Driven is a school, and if all other schools are not context-driven, then arguably by definition the other schools *all* “cover their ears and hum when someone calls their principles into question”. But I hope and believe not – by definition of the word ‘school’, they should all continue to learn. Otherwise Bret would have called (some of them them) asyla?
[James’ Reply: I think you really need to think about what a school is, Neil. A school is what Kuhn called a paradigm (at least, it’s one of the things he called a paradigm). It’s a set of ideas accepted as basic and true for a community of people. Of course schools learn, but they learn within their own system. For instance, the Cartesian paradigm of physics did not transition into the Newtonian paradigm. Both schools fought hard for dominance, and eventually the Cartesian adherents died out. A school is in many ways like a religion.
It’s not just a set of fixed beliefs– it’s a set of foundational fixed beliefs. You can say everyone should keep learning, but I suspect Paul Gerrard is not going to let go of his belief that quantification is axiomatically good. He learns, but he learn about other things. Similarly, I’m not going to “learn” that there are best practices or always good practices in the world. I can’t learn that. To take that position would be the equivalent of 2+2=5, for me.
When a group of people adhere to a stable set of beliefs in opposition to other groups of people with other beliefs, you have a school. My view of the schools out there is somewhat different than Bret. Bret likes to use “routine” because he thinks it’s less insulting than “factory”. I say Factory school because that’s exactly the analogy that the Factory school people are pursuing. They want to mechanize and dehumanize the craft. “Routine” strikes me as too shallow a word for that.]Â
3. I am pleased you refer to the Chapel Hill 1972 book – I have been showing my copy over the years to anyone who comes too close, and they cannot believe (or do not care) that I am that old. But I argue you cannot now call Chapel Hill a school – in its day it was the entire education system for software testing! We have all evolved from that, even you?
[James’ Reply: Of course it’s a school, if it’s a set of ideas that has believers who don’t question those ideas. And no we didn’t all evolve from it. I never accepted it. I briefly flirted with the Crosby school of No Defects, and then abandoned that for context-driven (although at the time I called it risk-based, and later market-driven). Furthermore, if you look at testing writing before 1972, it was generally more open-minded and open-ended. I’m referring specifically to the proceedings of the IFIP conferences, and to Jerry’s work.]Â
4. I agree that this topic of ‘test coverage’ is one of our biggest challenges, but I’m intrigued by your apparent choice between ‘quantification grounded in measurement theory’ and ‘narrative & qualitative assessment’. Are you saying some people can aspire to the former but the remainder have to languish in the latter? Or can we come up with some hybrid? They might seem different but perhaps they’re reconcilable via balaned scorecards or similar.
[James’ Reply: I’m saying that no one can quantify test coverage in a comprehensive way, because the rules of measurement theory require that you have a model that relates the scale to attribute you want to measure. What we want to measure, really, is “testedness” or “practical testedness”, but there is no such thing. So we go for surrogate measures like code coverage. Surrogate measures can be useful in specific situations, and to a limited degree, but to declare that everyone must use them would be consider, in my community, extremely irresponsible.
By the way, storytelling is not inferior to quantification. They are different things useful for different purposes.]
Remembering the bounce!
Cordially,
Neil.
I posted one query on Gerrad’s blog asking about the basis of he choosing these 12 items as axioms and the meaning of the word “axiom” that he implied in that post.
If (and if) his meaning of axiom is “a universal truth, unquestionable thing, one that does not requires ANY proof” …. (or BTW, does that called “common sense”?)
My question is why he thinks that these things do not require any proof ? What made him to come to that conclusion? Is that by inductive reason?
Why these are “irrefutable ” in the first place? – I am not talking about providing a counter example to prove that an axiom here is really not an axiom but I am questioning the basis of calling these as axioms (unless Gerrard’s connotation of an axiom is “prove me wrong, I will either argue as why the proof is incorrect or if convienced, I will withdraw the axiom”)
Shrini
[James’ Reply: I suspect Paul has not subjected his axioms to testing. But it’s just a blog entry, so perhaps this commentary process is the testing process.]
Having axioms itself is not a bad thing – great things have been achieved by adopting an axiomic system; for example, Euclidean geometry. As long as you
a) define the context under which these axioms hold true
and
b) open to the possibility of other axioms, other systems, because they equally hold great values.
[James’ Reply: Yes, I’d like to see Paul satisfy these conditions, plus two more: choose axioms that are less ambiguous (we can only guess what they mean), and explain what the benefits are of adopting them (the Euclidian axioms let us do interesting things).]
Yan Said … “a) define the context under which these axioms hold true”
Paul seems to be very “determined” not to use the word “context” or say anything that remotely implies something that could be confused with the word “context”.
In a blog post prior to this post of axiom, Paul states ….
“From the perspective of any software tester, they are self-evidently true
The axioms apply to any test approach from an end to end perspective to the perspective of an individual doing some testing
The axioms are distinct from guidelines or principles that reflect a particular context. They are context insensitive.
”
Paul seems to driving to a world of testing contexts do not exist – there are some “suggestions” that are available for tester to consider (as they are good things) and you do not consider them there can be “serious” consequences. Paul then goes on to say, “well, it is up to you to use or not use these – but make sure you use your brain (there is no axiom for it”.
As it is clear from few of his clarifications to my questions — He seems to be not taking the route of “principles”, “laws” or “rule” but some kind of “suggestions”. Use of the word “axiom” confuses me .. may be that is wrong word to describe what Paul’s idea. More than anything else, it “context insensitiveness” of these “axioms” worries me.
Shrini
[James’ Reply: It’s interesting what lengths Paul is going to avoid established notions such as context, heuristics, and schools of thought, while at the same time borrowing the term axiom from logic and using it, incorrectly, as a synonym for heuristic.
I’ve seen other people do this, and it seems to be because they think it’s some sort of buzzword bingo we’re trying to play, for marketing purposes, rather than a straightforward application of long-understood engineering ideas. Even Myamoto Musashi, who wrote the Book of Five Rings, in 1643 or thereabouts, discussed the importance of context and heuristics (the translator used different words, but the concepts are there).
When we hatched the idea of declaring our school, we expected that people would attack our school, and we expected that people would declare alternative schools. We did not expect so many people to behave as if different communities and philosophies of testing don’t exist! Such a view is so anti-intellectual, anti-scientific, anti-historical, and anti-reality, that it boggles my mind. I have to wonder, what do these people read? We have access to hundreds of years of engineering and scientific history. How can a person consider himself educated in our craft and be frightened of the word “heuristic” or fail to see the fundamental social and psychological contingencies of testing? I want waste as little time as possible on such people. The schools concept let’s me do that.
One reason to deny schools, unfortunately, is that it interferes with the marketing strategy of people who wish to appear as if they speak for us all.]Â
>> cover your ears and hum ….
Interestingly, there is a pranayama (systematic breathing technique) called Bhramari that involves the practice of closing your ears and hum…
Benefits claimed are —
“Besides having a profound effect on ears, nose, eyes and mouth, it enlivens your looks; improves the glamour of the face. Its benefits do not end here – it improves one’s concentration levels, relieves migraine pains, reduces stress and mental agitation, hypertension, and successfully combats and helps prevent many a disease.”
http://myblogonbeauty.blogspot.com/2007/04/bhramari-pranayam.html
I think, your word of use the word is to indicate “compulsive and deliberate ignorance of external thoughts” to remain “calm” … am I right?
Shrini
[James’ Reply: Yes. This is not a bad thing, though, Shrini. I have no problem with people wanting to focus on one way of thinking and exclude other ways. There is only a problem if they also want to deny that they are excluding anything!]
Even laws of physics is bound to context. For example, Newton’s three laws of motion, see below:
“These three laws hold to a good approximation for macroscopic objects under everyday conditions. However, Newton’s laws (combined with Universal Gravitation and Classical Electrodynamics) are inappropriate for use in certain circumstances, most notably at very small scales, very high speeds (in special relativity, the Lorentz factor must be included in the expression for momentum along with rest mass and velocity) or very strong gravitational fields. Therefore, the laws cannot be used to explain phenomena such as conduction of electricity in a semiconductor, optical properties of substances, errors in non-relativistically corrected GPS systems and superconductivity.”
I would be very surprised that laws of testing is not bound to some kind of context.
[James’ Reply: Most people don’t deny the role of context, outright. What they deny is the practical value of considering context. If you spend all your life indoors, you never need to know when to put on sunglasses. A lot of test methodologists seem to think they live in cozy cottages of unchanging context.]
As James points out Paul has misappropriated the word axiom here. Axioms are just self-evident/mutually agreeable truths/statement/assertions that need no justification. They are statements that form the basis of some further conclusions. Furthermore, there must be underlying those statements some mutual understanding of terms to be used in the ensuing discourse.
Even with Euclid’s axioms we couldn’t have a geometry class if we didn’t all understand the meaning of what it means for a point to lie on a line or for one point to be between two other points or for two angles to be congruent. Paul’s assertions while not necessarily ambiguous are open to interpretation, so my doubt that we can consider them axioms lies at least as much in their form as their validity.
Particularly puzzling/troubling is decision to invent his own definition of what axioms are. He even goes as far as to say “A testing approach must adhere to or implement the axioms or be deemed ‘incomplete’.” Yeah, tell that to Kurt Godel, and while you’re at it show me a “complete” testing methodology.
Paul qualified this whole endeavor as a straw man. He seems to have given us a good target.
[James’ Reply: He backed off a lot in his reply to me, but I still feel that he wants it both ways: he wants to put out a strong statement of his beliefs that he sees as foundational not just to him but to the entire craft, and ALSO he wants to tell us that he’s not part of any school, as if he has no foundational beliefs, no political agenda, and aw shucks he’s just one little ol’ consultant and certainly not someone anyone could confuse with an expert software tester. But my experience with Paul is that he wants his clients to take him seriously, the same way I want mine to do. For that, he, like me, needs his clients to agree with him on the fundamental problems and elements of testing. He wants to do that without seeming to be parochial. He will discover there’s no way to do that, except by lying about the state of the craft.]
“I generally use narrative and qualitative assessment, instead.”
I’m not sure exactly what this means, but it sounds like something I would like to learn more about.
What are examples of statements based on narrative and qualitative assessment that are useful to testers and project management?
[James’ Reply: Numbers are just a substitute for a conversation. When you have a well-defined model of what you want to talk about, numbers are marvelously efficient. But it’s problematic for testing, because the things we want to measure are so complex. Numbers look scientific, but usually aren’t. So, I developed a qualitative testing dashboard, detailed here: https://www.satisfice.com/presentations/dashboard.pdf
The dashboard has the property that there is no raw data on it. Raw data must be interpreted from a position of already knowing what’s going on in the project. Instead of raw data, the dashboard presents subjective (but defensible) judgments. The idea is that each cell in the dashboard can be explained. That’s where narrative comes in.
By narrative I mean explanations. I mean I tell a persuasive story about testing. So a dashboard is just a structured way to conveying a testing story.
Example testing stories can be found here: https://www.satisfice.com/sbtm/demo/reports/s_by_datetime.htm
Click on the links under the “Sessions” column. Each one is a session report. In the notes section of each report you’ll find a little testing narrative.
Notice the difference between that and a bunch of context-free numbers. “32 bugs open” tells you nothing, unless you know what the bugs are. But “no stop ship bugs” tells you something meaningful. “There are 32 bugs open and the triage team believes we need to fix ten of them and that they probably will be ready for tomorrow’s build” also tells you something useful. That’s a narrative that also uses numbers.]
I would tend to agree with what Yan said. Like the laws of physics, the laws of testing are also bound by a common thread of context. We cannot disengage ourselves from context and operate in isolation.