One of the things that makes it hard to talk about quality software is that we first must overcome the dominating myth about quality, which goes like this: The quality of a product is built into it by its development team. They create quality by following disciplined engineering practices to engineer the source code so that it will fulfill the requirements of the user.
This is a myth, not a lie. It’s a simplified story that helps us make sense of our experience. Myths like this can serve a useful purpose, but we must take care not to believe in them as if they were the great and hoary truth.
Here are some of the limitations of the myth:
- Quality is not a thing and it is not built. To think of it as a thing is to commit the “reification fallacy” that my colleague Michael Bolton loves to hate. Instead, quality is a relationship. Excellent quality is a wonderful sort of relationship. Instead of “building” quality, it’s more coherent to say we arrange for it. Of course you are thinking “what’s the difference between arrange and build? A carpenter could be said to arrange wood into the form of a cabinet. So what?” I like the word arrange because it shifts our attention to relationships and because arrangement suggests less permanence. This is important because in technology we are obliged to work with many elements that are subject to imprecision, ambiguity and drift.
- A “practice” is not the whole story of how things get done. To say that we accomplish things by following “practices” or “methods” is to use a figure of speech called a synecdoche– the substitution of a part for the whole. What we call practices are the public face of a lot of shadowy behavior that we don’t normally count as part of the way we work. For instance, joking around, or eating a salad at your desk, or choosing which email to read next, and which to ignore. A social researcher examining a project in progress would look carefully at who talks to whom, how they talk and what they talk about. How is status gained or lost? How do people decide what to do next? What are the dominant beliefs about how to behave in the office? How are documents created and marketed around the team? In what ways do people on the team exert or accept control?
- Source code is not the product. The product is the experience that the user receives. That experience comes from the source code in conjunction with numerous other components that are outside the control and sometimes even the knowledge of product developers. It also comes from documentation and support. And that experience plays out over time on what is probably a chaotic multi-tasking computing environment.
- “Requirements” are not the requirements, and the “users” are not the users. I don’t know what my requirements are for any of the software I have ever used. I mean, I do know some things. But for anything I think I know, I’m aware that someone else may suggest something that is different that might please me better. Or maybe they will show me how something I thought was important is actually harmful. I don’t know my own requirements for certain. Instead, I make good guesses. Everyone tries to do that. People learn, as they see and work with products, more about what they want. Furthermore, what they want actually changes with their experiences. People change. The users you think you are targeting may not be the users you get.
- Fulfillment is not forever and everywhere. The state of the world drifts. A requirement fulfilled today may no longer be fulfilled tomorrow, because of a new patch to the operating system, or because a new competing product has been released. Another reason we can’t count on a requirement being fulfilled is that can does not mean will. What I see working with one data set on one computer may not work with other data on another computer.
These factors make certain conversations about quality unhelpful. For instance, I’m impatient when someone claims that unit testing or review will guarantee a great product, because unit testing and review do not account for system level effects, or transient data occurring in the field, or long chains of connected transactions, or intermittent failure of third-party components. Unit testing and review focus on source code. But source code is not the product. So they can be useful, but they are still mere heuristic devices. They provide no guarantee.
Once in a while, I come across a yoho who thinks that a logical specification language like “Z” is the great solution. Because then your specification can be “proven correct.” The big problems with that, of course, is that correctness in this case simply means self-consistency. It does not mean that the specification corresponds to the needs of the customer, nor that it corresponds to the product that is ultimately built.
I’m taking an expansive view of products and projects and quality, because I believe my job is to help people get what they want. Some people, mainly those who go on and on about “disciplined engineering processes” and wish to quantify quality, take a narrower view of their job. I think that’s because their overriding wish is that any problems not be “their fault” but rather YOUR fault. As in, “Hey, I followed the formal spec. If you put the wrong things in the formal spec, that’s YOUR problem, stupid.”
My Take on the Quality Story
Let me offer a more nuanced version of the quality story– still a myth, yes– but one more useful to professionals:
A product is a dynamic arrangement, like a garden that is subject to the elements. A high quality product takes skillful tending and weeding over time. Just like real gardeners, we are not all powerful or all knowing as we grow our crop. We review the conditions and the status of our product as we go. We try to anticipate problems, and we react to solve the problems that occur. We try to understand what our art can and cannot do, and we manage the expectations of our customers accordingly. We know that our product is always subject to decay, and that the tastes of our customers vary. We also know that even the most perfect crop can be spoiled later by a bad chef. Quality, to a significant degree, is out of our hands.
After many years of seeing things work and fail (or work and THEN fail), I think of quality as ephemeral. It may be good enough, at times. It may be better than good enough. But it fades; it always fades, like something natural.
Or like sculpture by Andy Goldsworthy. (Check out this video.)
This is true for all software, but the degree to which it is a problem will vary. Some systems have been built that work well over time. That is the result of excellent thinking and problem solving on the part of the development team. But I would argue it is also the result of favorable conditions in the surrounding environment. Those conditions are subject to change without notice.
A really incisive piece of writing that offers excellent insights which go well beyond the conventional narrative spun in software development circles.
The only thing that I’d question is: “The product is the experience that the user receives”. I’m not sure this is the full story. A user is an important stakeholder but is not always the most important stakeholder in some situations. I work on corporate software and in many cases the emphasis has been too much on the user experience at the expense of concerns of cost of ownership over time and intersystem interoperability. That’s to say applications end up as pretty balls of mud before they’re even shipped. I would say that a product exists an interstitial space between its users, its environment (eg enterprise) and time.
[James’ Reply: Okay, that sounds good.]
James,
I see many fail to consider that quality is lives OUTSIDE the product – in a virtual world that connects product to the user – it is a relationship”.
Hence, Bug counts, all sorts of metrics of software that attempt to look for quality inside the product (defect density, defect leakage %, testing effectiveness) make a futile attempt.
They horribly trivialize and simplify the notion of quality as intrinsic attribute of the product – giving quality a flavor of absoluteness. Relation ship by definition involves two entities. One is the product and other is user. Hence quality is a very personal … and software metrics attempt to remove “personalization” and generalize quality as a something that is built into the product.
Shrini
Quality can be percived on many levels of project. Source code for customer is not product, but internally, in eyes of developers, source code must have quality at least so much that project can be mantained without to much reingeniering/refactoring. In outcome, quality of source code means less time spend on developers side, less money spend on customer side meaning happier customer. More quality source code/project/product/people involved have, word “less” in this context means more – less. 🙂
Customer must know what he wants. I’m bound by his wishes that are formalized in “the spec”. What I can do is suggest what will work, what won’t. It’s up to customer to decide, I can only make his decision more informed.
Sometimes something that I percived is without quality (i.e. quick dirty code) for customer means opposite – it was there in time, it worked and in “big picture” made sense. Like – one man’s trash, another man’s treasure – kind of thing 🙂
Just on your second item in the series I realized one point, where I’m not sure you’re going to adress it in the next few “episodes”. Stating that quality is dead means, that quality must have lived at a time in the past. Your recent entry in the series made me aware of this, since quality is not built, it never was; rather it lived beside us and we managed to pass it away.
One note I would like on Shrinis comment:
People are trying to measure quality in the ways you described in order to see process in their daily work. This is based on the assumption that measured intrinsic quality will lead to extrinsic quality experiences as well. This assumption violates the oracle behind this. It’s just a heuristic. All these measures are placed in order to see when something bad happens. The other way round is not valid all the time – i.e. my measurements show that I have improved, but customer’s perceived quality is different from that.
Another side note: Measurements usually focus on technical aspects of the software. Most users don’t deal with the technical part of your products.
“quality is a relationship”
Wondering if you have read Zen and the Art of Motorcycle Maintenance? Get this thought from there?
[James’ Reply: I did read it. At the time it confused me, because I thought it was obvious what quality is. (I was young.) Later, I read Lila, the sequel, and that did the trick.]
Hi James,
Nice post! In particular, I like the shifting of focus onto the notion of relationships.
If I may, I’d also point out that qualities such as “quality”, the values embodied by practices, and the evolving nature of relationships over time all come down to intent. Amongst other things, without a clean connection to intent those other things devolve into static, abstract notionals devoid of life.
Rock on,
John
[James’ Reply: Rock on.]
>>>quality is lives OUTSIDE the product – in a virtual world that connects product to the user – it is a relationship
A colleague challenged my statement as above and said … if it is a relationship and personal between the user and product … then how to measure quality?
I answered him “considering quality as something built into the product is a MYTH. A myth sometimes provides a primitive model to understand complex things but we should not confuse myth to be the reality. We need to externalize the notion of quality as something that resides in a virtual space between product and its user, its environment (thanks to Neil). If you know and you are sensitive about external manifestation of quality – you will change the methods and tools to measure and interpret the quality”
James, I hope you have better answer “how to measure quality”
Shrini
[James’ Reply: We can measure lots of things. Quality itself cannot be measured, because the word “quality” is a placeholder for lots of different ideas that may be integrated in different ways. But certainly we can measure things that we believe, in some particular circumstance, are factors in quality.
There are people out there who believe that quality can truly be measured. I would like to invite them to try to measure the quality of mortgage-backed securities. Lots of smart people thought they could do that and turned out to be very VERY wrong. In fact, the whole idea of a market is founded, to a large degree, on how different people can look at the same commodities and see different quality. Some buy what others sell.
We gather information about quality. We reason. We make judgments. That’s how it works. It’s not a scale, but a mind that does the weighing.]
You have iterated the viewpoint in your post in a much shorter format, “Quality is value to some person (who matters).” I wholeheartedly agree with you. Quality is simply the relationship between the product and the value placed upon that product by the person who matters. To take your point into a physical world, if software ended up being a building, then the following relationships could exist, each with their own definition of value (and therefore quality): the building inspector who wants to make sure it meets safety codes, the bank who wants to make sure that someone other than the original buyer finds the same monetary value in the property, and the actual buyer who commissioned the build in the first place. In none of these situations is quality built into the building; as no relationships can be built into a physical thing. However, consistent processes that cause the built product to achieve the same value to one of those stakeholders can be built. That is what Quality Assurance is about. It is about consistently providing value to those who matter. We all have different ideas on what quality means for any given product, because each of our relationships is different with the product and what we value from the product is different. QA is really about identifying stakeholders, identifying what they value, and developing processes/improving consistency in a way that helps the product achieve the value to the stakeholders. This is why QA can be difficult job; because there can be stakeholders who have disparate values that sometime conflict with each other.
I think often times the statement that “quality is built into it” is useful in delineating job roles. The Quality Assurance umbrella encompasses more than just testing a product. It is about establishing repeatable processes with the goal of ensuring any product following that process will have consistency and thereby value. Testing is just one tool under the QA umbrella. In a nutshell, testing is a quality control function. The product is compared to some oracle and the results noted. This job and the information it provides is important, but it is just one part of a bigger picture.
So, I agree, the statement that “quality is built into” something is a myth that should not be overindulged. That’s why I limit that statement to help delineate the different roles between someone in QA and someone in testing.
[James’ Reply: Remember that a “repeatable process” is yet another ephemeral arrangement, to some degree. In essence, it could be said that no process is repeatable, or that every process is repeatable, depending on where you draw the line.
So, when somebody tells me about their repeatable process, I immediately look for the parts of it that don’t repeat or the parts of it that shouldn’t repeat. I look for residue and side effect– what’s happening that is not being featured or spoken of.]
James,
I think your ABS example is bad one. There is no ambiguity about ‘quality’ in this case. The ‘placeholder’ that you mention was a number – the hazard rate (risk of default).
[James’ Reply: That number is just the end result of a giant unknowable abstraction. It’s that abstraction that I was referring to.
But I suppose you’re right, in the sense that the purpose of credit default swaps is to transform something with complexity and meaning (a particular mortgage) into something that has liquidity and seeming simplicity. So, perhaps I’m referring to the quality of the decision to buy such securities, rather than the security itself.]
Well, James, I liked the post and found it good reading. Then I got to the part at the end about the death of quality.
I’ve heard this expressed a number of ways – Entropy, the Second Law of Thermodynamics, “All good things must come to an end.”, “Rome didn’t fall in a day – but it did fall.” – It makes me sad.
My father in law, David Ellinghausen, who was a manufacuturing quality engineer, once told me that “all it takes is one very dumb vice president to mess everything up. That’s why you have quality engineers. We institutionalize the system so that he can’t destroy it all with a poorly placed order.”
oh, he used different words, but I think I got the main point right.
All this makes me very sad. I, however, am a privateer, and I only accept letters of marquee from organizations that are trying to build something great.
In America, we still have the resounding echoes the freedom that our forefathers fought for us in the 1700’s. Building great software may be a losing battle to time … but I’m in the fight anyway.
Rock On, Indeed.
While I enjoyed this blog entry and truly gained much insight, I’m a little bothered by the tone which suggests that unit testing, TDD, automated customer testing, and even BDD are harmful. I believe that you are saying that relying on any one and only one is a myth, but I assert software ‘built’ using TDD will be of ‘higher quality’ than software built sans-TDD.
[James’ Reply: When you claim that things are better with TDD, etc., I assume you also intend to say “all other things being equal.” But all other things may not be equal. I routinely encounter Agilists who seem to pursue their notion of Agile testing while disdaining or refusing to study sapient system testing skills.
However, there is some solid value in the practices you cite, if we make all the customary charitable assumptions. I’m not denying that. I just think it doesn’t solve the problem, because the problem is out of our hands.]
You call out all the potshots organizations and individuals have taken at getting to higher quality, but you don’t give them the credit for trying to produce higher quality. After all, Porsche adopted some of these pedantic measure, called Lean Manufacturing, and now are able to produce defect free cars at the end of the assembly line. See “Lean Thinking.”
[James’ Reply: Nothing can be “defect free” except in the imagination of people with insufficient imagination. But all you have to do is talk about a consistently pleasing experience (a lower standard, perhaps, but its what I’ve been talking about). Even my Saturn gives me a consistently pleasing experience. My desktop computing environment does not.]
I think we should toss the word ‘quality’, period. Instead, ask people to describe what qualitues they believe make good software. As a word, we’re a little more familiar with good, and we can all say what we want out of our software, next to the requirements. Requirements are what we want it to do, and we want those requirements to follow certain guidelines: not break. Complete with errors. Tell us when we done somethin wrong. Give us a chance to back out. Not tank the system. Stuff like that.
I think Paul has the right attitude, but I don’t think he’s got the answer (if there is one). /Tossing out/ the word doesn’t help. Pressing for details, not being satisfied with fuzz, does–as long as it’s understood you’re not just being a d*** by so doing. At some points/times discussions by people who are making a difference need to be very specific. On other occasions, if the word “Quality” is prohibited, someone will invent or adapt some other word to mean roughly the same thing: “value” or “correctness” or what-have-you.
Re: “giv[ing] us a chance to back out”… Aza Raskin has written about the importance of “undo”. But there are classes or aspects of software and software “side effects” that are not supposed to *have* undo–Sarbanes-Oxley audit trails and the like. Persistent store is a tricky thing, Especially when so much software sucks.