In a recent Twitter conversation, yet again someone who should know better claimed that testing improves quality, and yet again Michael Bolton and I rose to speak against that notion. Our correspondent wants to reduce it to a simple matter of probability:
If you omit testing, the quality (typically) will be A) Better B) Worse C) Indifferent? Only one sensible answer exists. This makes testing necessary, but not sufficient, cause of quality. https://t.co/GvUWr3D8Ow
— Mika Mäntylä (@mmantyla) February 8, 2018
My answer to this question is to reject it. It rests on certain false premises:
- That we sufficiently agree about what quality is.
- That we sufficiently agree about what testing is.
- That we agree on what is typical.
- That we agree that discussing human choices in terms of brute causality is helpful in this context.
I’m not saying that questions like this are necessarily bad. I’m sure I have used a similar construction in some other argument. The problem here is that the writer of the Tweet arrived at this simplifying formula after dismissing, as irrelevant, my social and ethical objections to his entire line of reasoning.
I say that testing does not improve quality, first, because it is obviously true that learning facts about code does not change code. Second, and much more importantly, I say it because it is both practical and ethical for testers to respect the agency of the people who control the code. Those people may not fix the bugs I report. It is critical to my discipline as a tester to understand that. Otherwise, I risk losing my credibility and influence. I risk adopting an attitude that desensitizes me to the kinds of problems my clients need me to find.
Agency. It’s dawning on me that all of my projects are connected by this thread.
- I am opposed to “best practices” because that phrase is just a ploy to avoid responsibility for decisions about how to work.
- I am opposed to professional certification programs, such as the ISTQB, because it is just a ploy to profit from the fear and ignorance of testers and managers; insidious manipulation via snake oil.
- I am opposed to “standards” that are crafted by consulting companies to justify expensive, useless services.
- I don’t use the phrase “test automation” because that encourages thinking about testing as a set of mechanical actions instead of a set of choices, interpretations, and explorations.
- I insist on distinguishing testing from mere fact checking. Checking can be done by machines that make no choices, but testing requires socially competent judgments and a whole raft of choices which must be made and to which testers must be accountable.
- I am opposed to self-driving cars not because they are unsafe, but because they aim to change without deliberation or consent the social contract that humans have with each other about the uses of public roads and accountability for what happens on them.
- Although I am loud and opinionated, I tell my students that my goal is to turn them into colleagues, not followers. People who parrot what I say are no use to me.
- My son was born at home, because my wife wanted control over the entire birth process and was not satisfied that the hospital would respect that. So, I read a few books on midwifery, hired illegal midwives to help, and made plans in case of emergency. (To my surprise, I found research which showed that home births were safer for low-risk pregnancies.)
- I could not tolerate school. I was assigned tasks without my consent.
- I am opposed to forced schooling, because good education is a personal journey of self development. (My son dropped out after 6th grade.)
- The biggest conflict of my marriage centered around money. I solved it (after a few years to get my pride under control) by giving my wife control of our finances and giving her 70% of the company I started. I got complete creative freedom and she has cheerfully been my assistant for 19 years and counting. Lesson: giving away one kind of power can give you back another kind of power.
Agency is the capacity of an actor to act; the ability to make choices. I have a lot of intellectual attitudes and arguments about testing. But my emotion and motivation about testing comes from my feelings about agency.
I am a tester because I want to set people free. I am a teacher because I want to set people free. I am a husband and father because I want to keep my family free.
Fight the algorithm.
I am opposed to self-driving cars not because they are unsafe, but because they aim to change without deliberation or consent the social contract that humans have with each other about the uses of public roads and accountability for what happens on them.
Speaking of agency, it’s not the self-driving cars themselves that do that. People who carelessly or even recklessly promote self-driving cars do that.
I’m happy to let go of my own agency for transportation—when trained, skilled, capable humans are in charge (as when I’m on an airplane or in a taxi), OR when the problem space is sufficiently constrained (as in the case of the automated shuttles that run between the inner and outer terminal buildings in airports, or even the subway system in Copenhagen). But the less constrained the environment is, the more I want responsible humans actively supervising things.
I don’t want to fight the algorithm. I want people to use algorithms powerfully while remaining in control of them. I do want to fight any move to abdicate our responsibilities for the algorithms—that is, our responsibilities to ourselves and to each other.
[James’ Reply: “Fight the algorithm” is better for a bumper sticker than “Fight against situations whereby people hide behind algorithms to avoid responsibility for their decisions.”]
“Fight the algorific!” “Fight the algopressors!” “Fight the algorisible!”
And “Best Practice” is a better bumper sticker than “A pretty solid practice that is typically (relatively) inexpensive to implement and that has worked somewhat successfully for a majority of our clients with minor modifications (which may or may not have resulted in additional, unplanned costs) and, to our knowledge, has not resulted in death or dismemberment.”
[James’ Reply: I guess it’s better in the same sense that “I’m an idiot” is better than “I choose to say silly things about process that deny agency and misplace credit, such as ‘best practice’, instead of longer phrases that also deny agency and misplace credit, because I don’t give a damn about my craft.” In other words, both are wrong (I don’t believe you are an idiot and I don’t believe you don’t give a damn) but one is shorter.
Your second statement, despite its additional qualifiers, still commits the same error: you seem to think that quality lies inside the practice itself. It doesn’t. The quality of a practice in the social world (testing is a supremely social process) comes from the skill of the practitioner as he solves the problem while negotiating the context of that problem.]
As a publishing musician (and software tester), I have to “fight the algorithm” constantly on my creative side.
All of the distribution services use these things to detect sampled music. Since I occasionally sample past works in new pieces, I am regularly being accused of sampling myself. It detects that the music is the “same” as my earlier work, but is too dumb to look up the writer/publisher, which is always the same name as me. I receive an automated rejection, but a simple email to the moderation team always clears it up. It’s what I regard as an unnecessary manual step. Because “automation”.
Fairly recently, I uploaded some music that was eventually rejected by a distribution service. I was accused of uploading a podcast, and they don’t publish podcasts. A few months prior, I had uploaded a snippet of that track, then “in progress”, to my podcast of my music that I publish through a different company that only hosts podcasts. That’s how I was able to discover that their algorithm or sniffer or whatever was scanning podcasts. A simple email to the moderation team cleared that up. Lesson learned: don’t publish music to your podcast before publishing it to commercial distribution services. Because algorithms.
One of the major music retailers on the internet (their name rhymes with “Snapple”) dislikes “blurry” artwork, and apparently has agreements with distributors to not let any “blurry” artwork onto their system. They have sniffers for that. (Major label artists can do it, but not indie artists.) So this company is judging and rejecting artwork by algorithm. Because they can.
[James’ Reply: This is fascinating. Thanks.]
I’m interested in your statement, that agency is a thread that connects all your projects.
The word “agency” describes (1) “the capacity, condition, or state of acting or of exerting power”. (Webster’s 9th New Collegiate Dictionary) It also (2) describes instrumentality, i.e. “a person or thing through which power is exerted or an end is achieved”.
When you “respect the agency of the people who control the code”, are you acknowledging that agency is personal, and therefore capable of both error and insight?
[James’ Reply: Yes, although instead of personal I would use the word “creative.”]
How does that change how you test?
[James’ Reply: It changes what I’m looking for and how I’m reporting it, because I see myself as an agent for the people who create quality. Thus, I need to anticipate what they would consider important and report what they would consider important. I am also going to challenge their notion of what is important if I think it is not serving what I believe is their best interest, but ultimately only my clients control that.]
Secondly, saying that testing does not improve quality overlooks simultaneous, reciprocal collaboration by pairs or even groups. Test-based development also improves quality. Thoughts?
[James’ Reply: I don’t think I’m overlooking that, because I don’t think it is “testing” which is improving quality. Testing is not development. Testing is discovering the truth. Development is creating new truth, and THAT is where quality is actually made.
Furthermore, there is no such thing as “test-based” development. If you are referring to TDD, then that is more properly called “output-driven development” because that is literally all it is: a programmer conceives of a desired output and then writes code to produce that output. This has nothing to do with testing except in the most trivial of respects. But even if it were proper to call that testing, it is not the testing that creates quality, is it? The quality comes from the programmer who knows what code is worth writing.]
I suppose saying that testing improves quality is like saying that diagnoses’ improve medical conditions before any medication has been taken or surgery performed.
“Development is creating new truth, and THAT is where quality is actually made.”
I know i am late into this, but, I do think it is important with words when we are discussing testing and quality, since quality is subjective, i wouldn’t say that the quality automagically appears just because some lines of code are written.
[James’ Reply: I didn’t say that it automagically did anything. Quality is created by the developer. Quality is a relationship, not a substance, and of course the relationship can be undone by forces outside the control of the developer. But the fact remains that the developer, by definition, is developing that relationship in the first place.]
I’d like to change the statement to:
Development is creating new truth, and that is where quality ‘Could’ be made.
[James’ Reply: I shouldn’t have used the word “truth.” Perhaps I meant pattern. And doesn’t it seem coy to say that quality could be made, when that is the express purpose of the whole process? It’s not an accident that quality happens. When I began this sentence, I was intending to write something that you could understand. The fact that this sentence continues that thought in a coherent way is by design, not accident. Assuming I succeed in communicating with you, I deserve credit for creating that message, not merely the possibility of a message.]
I want to change the statement because:
Unless anyone finds value in the solution, then quality is not achieved.
My thoughts of quality are based of Jerry Weinberg’s definition: “Quality is value to some person(s)”+(who matter) By yourself and Michael Bolton.
Yeah, you are right, you didn’t, it was me who interpreted what you wrote that way, my misstake.
I like your explaination of it and i do agree (if we count ‘missunderstandings’ in as well).
(“Quality is a relationship, not a substance.” – I like that line 🙂 )
Hmm, yeah it does…
I’ve rewritten this answer like 20 times now and that is becasue every time i read “Quality is created by the developer” it seems simplified and more information is needed, like you just gave to me.
I can’t put my finger on it but it gives me the feeling that by saying that:
1, Quality is a 100% certain outcome.
2, Development is a one-man job.
As you were talking about, there are things that can affect the dev -> quality relation.
So i guess my conclusion from this is that i would not say “Quality is created by the developer” (even though it is correct) without a follow up.
Cheers!
Thanks for the reply!
[James’ Reply: This is a fair and reasonable comment. A point worth thinking about.]
Hi James,
A few years back, we had a good email exchange about namespaces and testing/checking. While I don’t think we completely agree (which is fine), I came to realize that agency was the part of the model you cared most about. We did some monster bashing on the test/check concept, but based upon our discussion, I found agency was the actual key idea rather than the minutia we debated.
With that said, I wonder if you have spent much time reflecting on the Schneider Culture Model (Example: http://agilitrix.com/wp-content/uploads/2011/03/Schneider-Culture-Model.jpg )? In reading about some of what the agile community did with the model, it appears that there may be some commonality. For example, while a pretty limited study, it appears that Agile teams are less excited by control culture (http://collectiveedgecoaching.com/2010/07/agile__culture/ ).
[James’ Reply: I haven’t seen this before now, and it’s not a great fit for what I think it going on, but it doesn’t look particularly dangerous at first glance, either.
I guess the thing that strikes me about it most is that cultivation and competence are put into different quadrants. That makes no sense according to my worldview, because competence is achieved VIA cultivation. You don’t get it any other way. People are not built like bots– they grow like trees. Also, I don’t understand the distinction between people-oriented and company-oriented. A company is a group of people. Perhaps, the intention to speak of a “ludic” orientation– which means a focus on a system as formalism; a game that we play. It’s a mechanistic and reductionistic orientation; teleological rather than organic.
I also don’t see why collaboration is more “reality-oriented.” If I were to name a quadrant that was focused on the here and now and established solutions (reality rather than possibility) and also people-oriented I would name it “ritual” or perhaps “common sense” rather collaboration, and the tagline might be “we do what seems right.”]
It seems to me when you talk about agency v algorithms, I get the sense it is part of a larger theme, perhaps like the cultural model Schneider proposed.
[James’ Reply: I don’t see how the Schneider model represents anything larger. It looks to me like it’s exploring exactly the same space. Algorithms are mechanistic; agency is humanistic (that’s the horizontal dimension in the Schneider system). The vertical dimension has to do with convergent versus divergent processes (in other words, ones where an algorithmic and optimizing approach can solve the problem vs. one where a heuristic and satisficing approach must be used).]
Perhaps you simply desire more freedom than cultures that desire control want to give.
[James’ Reply: The nature of the work we do requires freedom. It’s no mere personal preference. I just wrapped up a two year expert witness gig. It would have been literally impossible for my clients to “control” my process without destroying it, because they had no idea what needed to be done and how until I told them– and I had no GOOD idea until after I had explored some plausible options.]
Cultures who desire top-down control desire security, order and clarity of work by having control (or an illusion of control) they think they achieve their goals. The very fact that an algorithm has control, one which you can’t negotiate with or bypassed*, has no interest to partner with and is uncaring and cold seems to disturb you.
[James’ Reply: Only because A) it often doesn’t work, and B) it’s deeply irresponsible. Other than that, I don’t care. There is a fundamental morality involved here. You say “cultures who…” but you don’t even mean that, do you? You mean “people who…” (cultures have no agency, so it is not proper to say that they “do” things) and those people ought to be held accountable for their actions. As long as there is accountability (which is all about agency) people can do whatever dopey things they want to do, as long as they don’t harm others while doing it.]
Is this an accurate, even if limited, model? Do you feel that “Agency vs. Algorithm” is the outer most layer of your conflicts? I ask in part because I’m grappling with some of my own questions around culture and control.
[James’ Reply: I think the issue of agency vs. algorithm is the most fundamental of all human questions. It is the essence of religion, free will, liberty, justice, the meaning of being alive, etc. Are we process or are we substance or are we data? When I chose to vote for Hillary Clinton was I influenced by Russian bots? Was I tricked by advertising? Was I virtue signalling? Was I just doing my wife’s bidding? When I eat a hamburger, is it because my micro-biome “wants” me to or because “I” want me to? This is a sentence I am writing, but is it a choice I made to write it or was it somehow pre-determined?]
* Excluding bugs/exploits, but not like human laws where you can choose to bypass them for good cause, like jay walking because it’s clearly safe.
– JCD
P.S. The “I’m not a robot” algorithm timed out on me after 5ish minutes and demanded I re-check the checkbox and click images twice. Once while I gave myself some think time before posting and again while writing this P.S. The second time wouldn’t even let me check “I’m not a robot”.
I admit I only found the model very recently, so my thinking isn’t particularly deep on it yet. However, I think I can explain some of your issues with it, from my own interpretation. Competence is about individuals who are highly competent in their own respective field, but don’t play much with others. The sort of ‘cowboy coder’ or ‘hero’ mentality. This is part of what is meant by ‘company oriented’ in that a hero sacrifices themselves to solve problems late into the night to make a deadline for the company.
[James’ Reply: I don’t see how that addresses my issue with it. You can’t separate competence from the means by which people attain competence– it is cultivated. Engineers don’t arrive on a clam shell from the sea. If the guy is talking about how competence is attained he should say that. So maybe the split is not “cultivation vs. competence” but rather “company-supported cultivation vs. personal cultivation.” But then you’ve have to reverse the polarity and say that company-supported cultivation is “company-oriented” and personal cultivation is “people-oriented.” Still that would be weird, since they both are kind of both-oriented.]
I also suspect that those organizations have a top-down approach, but unlike control culture, the way leadership gets people motivated is by the ‘just do it’ demands rather than the more scripted style the control culture uses. Competence oriented grow as individuals but they don’t, by default, help other members of the team get more done. I think as a teacher you probably see cultivation and competence together because you cultivate people by trying to teach them craftsmanship. I appreciate how there is no distinction for you. I don’t claim the model is perfect or even correct, I just found it useful and wonder if you had used it as a stepping stone as well.
[James’ Reply: As a teacher I know that competence is always a cultivation process, just as botanists know that plants are always grown and never assembled in a factory. I also wrote a book on self-education, wherein I spoke a lot about personal cultivation (although I didn’t use that phrase).]
James Said: You say “cultures who…” but you don’t even mean that, do you? You mean “people who…” (cultures have no agency, so it is not proper to say that they “do” things)
This reminds me of the Austrian school of economics, where they deny anything but individuals exist, ignoring any ‘herd’ responses. I suppose in your view angry mobs don’t really exist, it’s just a group of individuals, each personally angry, not impacting each other. I can respect that model, it has useful aspects, even if I don’t subscribe to it.
[James’ Reply: Sigh. Come on man. You can listen and summarize better than that. I want to tell you off because you are making an absurd straw-man argument, but I believe you are not malicious in this case, just mistaken. Let me correct you about what I believe:
James Said: I think the issue of agency vs. algorithm is the most fundamental of all human questions.
Ah, so you think this is essentially a question of free will.
[James’ Reply: No, I think the fundamental question is one of agency, which is what I said. I think of it as more fundamental because it is closer to human experience. I don’t experience free will; I ASSUME I have free will. I experience agency, though. The issue of free will is part of that, obviously, but if you consider it more fundamental than that’s okay with me.]
I agree that is important. However, the philosophical question of free will (mechanism, quantum randomness, etc.) can be separated from the internal feeling of agency or the lack of agency. That is to say, the universe may truly be random and/or mechanistic, but it doesn’t mean we don’t still feel internal needs for personal control, autonomy or agency. Is your personal primary concern about the mechanism of the universe (or lack thereof) or the internal feelings of agency?
[James’ Reply: Yes, I agree that the questions are separate. I have little or no concern about free will; it’s agency that matters. But feelings of agency can be criticized on the basis of being mere mechanical illusions, so that’s why conversations about free will matter.]
I don’t expect an answer to that, just revealing a question (https://www.ribbonfarm.com/2017/08/10/questions-are-not-just-for-asking/ ).
Thanks James, I appreciate your time and your thinking. It can be good to be questioned! It helped shake out some of the bugs in my thinking. I’m sure more still exist.
[James’ Reply: I appreciate you for your questioning. And anyone who reads Ribbonfarm is likely to be a friend of mine in a far deeper way than any momentary annoyance I have with him will touch.]
– JCD
If only there were more people in the world who so valued agency in themselves AND others, so that more humans had the opportunity to realise their own potential, dreams and true selves, and could learn to recognise and respect the contributions made by others. You support liberty with vast integrity, and I have the greatest respect for that.
[James’ Reply: Amazingly, not so many managers and employers value it. But I’ll keep trying.]
On the topic of self driving cars, I am amazed at the amount of investment into this so called prophecy which claims to end 90% of traffic accidents (because we the humans won’t be driving lol), and yet after all of these years there is not a single product which is available that claims to be fully self driving. I have read that some manufactures are 80% of the way there but what I haven’t seen is any form of business case? For example of the 1 million cars sold in Australia we estimate that 10% sold will be self driving in the future which means it will take 1000 years to make a profit based on the trillion dollar investment. I’m also interested in what your thoughts are around other automation topics there are many studies claiming that 60% of jobs will disappear due to automation, I don’t understand where they get these ridiculous numbers from which are completely baseless!
Evaluate the following: “I could not tolerate school. I was assigned tasks without my consent.” & “I am opposed to self-driving cars not because they are unsafe, but because they aim to change without deliberation or consent the social contract that humans have with each other about the uses of public roads and accountability for what happens on them.”
Do you see the irony?
[James’ Reply: No. But feel free to explain, if you think you see some.]
I’m very interested in how you define and discuss the concept of agency. Do you have any resources (books, blogs, etc.) that you could recommend to further one’s understanding of “agency” ?
[James’ Reply: Here are some sources I found useful. Note that agency can be considered on the level of your moment to moment experience of speaking and moving, and also on a sociological level, which is more what I am talking about– but they are linked.
https://en.wikipedia.org/wiki/Agency_(sociology)
https://www.thoughtco.com/agency-definition-3026036
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5002400/
https://pdfs.semanticscholar.org/4868/1826475e599dff70c873ef98dfa756d537d8.pdf
]
Very agreed with you James, as I’m believing that Human will always be human, and man will always be man, as what the role of our minds if we set and rely on riles, standards, best practices, …. for all things
as by this way no need for Human in life as whole, so only to be in safe way we should only rely on our minds.