I am thinking and talking a lot about ethics, lately. Maybe that’s because Context-Driven testing is getting better traction, now. More testers are approaching me, asking for guidance. More testers are challenging fake testing in their organizations.
Or maybe I’m just noticing it more, because Cem Kaner used to take the brunt of all this. In that respect, I have historically been more a follower than a leader in our community. Recently, I’ve stepped forward to be more of the role model that I am expected by my colleagues to be.
(Note: Ethics is a guaranteed sore subject. Whenever I talk about ethics, I can’t avoid implying that I believe some people are not as ethical as they should be. But, you know, that’s the burden of leadership. To those who avoid politically difficult subjects, enjoy your shadowy existence. Lurk on, yon lurkers.)
I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.
I need my own, because these codes above don’t directly deal with certain common testing ethics issues.
So, I’m the content-owner for the 2nd Kiwi Workshop on Software Testing, tomorrow morning, here in Wellington, New Zealand. Our topic is ethics. To get ready for it, I thought I would write out some of the ethical principles by which I strive to operate. Here they are:
- Know what a test is. Avoid labeling an activity as a “test” unless it represents a sincere effort to discover a problem in a product.
- Maintain a reasonable impartiality. The purpose of testing is to cast light on the status of the product and its context, in the service of my clients. I may play multiple roles on a project, but my purpose, insofar as I am a tester, is not to design or improve the product.
- Do not claim to assure, ensure, or control quality. I don’t control anything about the product: a tester is a witness. In that capacity, I strive to assist the quality creation process.
- Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.
- Apply test methods that are appropriate to the level of risk in the product and the context of the project.
- Alert my clients to anything that may impair my ability to test.
- Recuse myself from any project if I feel unable to give reasonable and workman-like effort.
- Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.
- Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way. (e.g. I will not keep test case metrics, because they are damaging in almost any context)
- If I do not understand or accept my mission, it shall be my urgent priority to discover it or renegotiate it.
- Do not deceive my clients about my work, nor help others to perpetrate deception.
- Do not accept tasks for which I am not reasonably prepared or possess sufficient competence to perform, unless I am under the direction and supervision of someone who can guide me.
- Study my craft. Be alert to better solutions and better ways of working.
Hi James!
You said “–my purpose, insofar as I am a tester, is not to design or improve the product.” Do you mean that when confronted with a poor decision you just bring the solution up and say “this might be a poor decision”? If you have the competence to give improvement suggestions, should you still leave them unvoiced? “Are you then in the role of a tester” might be a question, you’d ask.
[James’ Reply: Voice whatever you like. But when you voice things that are outside your role as a tester, you run the risk of neglecting your own role, confusing others about your role, and invading the turf of those people.
I give my opinions, but I am careful to couch my opinions as that of an interested amateur. When I’m talking about problems, as such, then I am speaking from within my role and expertise.]
“Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.” What if you report a bug that the developers aren’t willing to fix? I heard this theory that you should only report stuff that is getting fixed. How do you feel about that?
[James’ Reply: That’s a ridiculous theory. How can I know what they are willing to fix until I talk to them about it? How can I know if they might wish to change their minds?]
“Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way.” In the case of a client asking for a metric derived from test cases, do you just say “no” or do you consult the client to improve their measuring? Or do you just reject the case?
[James’ Reply: I can do any of those things. Usually, I talk a lot.]
Sometimes the client has a process that needs to be followed. The process might be “stupid” or inefficient. Do you help the client to make the process better or would you just ignore it and do it the way you think would be most productive and efficient? Let’s say the client uses the ISTQB foundation testing process by the book. Would you change your testing approach to fit the process or would you require the process to be changed?
– Pekka
[James’ Reply: I will not violate my principle. But it’s a matter of judgment as to whether something that is not a very good way to do something is also a bad way to do it. There’s a lot of gray area, and I have to navigate it. I’m on a project right now where I’m being asked to follow a template. I’m going to change the template, however– but not totally. If the customer wants to change it back, they will have to hire someone else to do that.]
With the possible exception of one requested clarification (below), this is fabulous.
*EVERYONE*… and I mean *EVERYONE*… should have and follow a personal code of ethics. Its not for me to say what the details of another persons code should or should not include (that’s the context part), but I’d be willing to consider calling “have a personal code of ethics that you actually follow” a “best practice” (right alongside the other things I’d be willing to consider calling “best practices” like, “nourish your body”, “educate your mind”, “strive for acceptable balance in your life”, and “try not to be an idiot too often”).
The requested clarification is around the second bullet. I get the important distinction about role here, and I’m pretty sure I follow (and agree with) the intent. I’d like to hear more of your thoughts on why, when you are serving in a testing role, might not periodically be operating with the purpose of improving the product.
I’m thinking specifically about experiences I’ve had “pairing” (we didn’t call it that, but that’s what it was) with developers/architects/analysts very early in the project explicitly to test their ideas/concepts/prototypes/unit test design frameworks/etc. with the sole purpose of helping to build a better product.
I could argue that isn’t “product improvement” because there was not yet a product… I could argue that what I was testing was someone’s “work product” vs. “the end product” and thus the impartiality applies to the work product and the end product is a context factor… I could also argue that what I was doing in those moments was actually serving in a coaching role and not in a tester role. So my intent is not to debate or negate the point, my intent is to hear your thinking on the matter (though, as always, I reserve the right to debate or negate based on your answer — and I don’t think you’d have much respect for me if I didn’t reserve that right .
Again, great post. I’m hard pressed to see how an ethical person could take offense to this one. 🙂
—
Scott Barber
System Performance Strategist and CTO; PerfTestPlus, Inc.
Author; Web Load Testing for Dummies
Co-Author; Performance Testing Guidance for Web Applications
Contributing Author; Beautiful Testing
Contributing Author; How to Reduce the Cost of Software Testing
http://www.perftestplus.com
http://about.me/scott.barber
“If you can see it in your mind…
you will find it in your life.”
[James’ Reply: Hi Scott. I’m not trying to say that it’s wrong for a goalie to score a goal, but simply that a goalie who scores a goal on purpose is probably neglecting his duty AS a goalie. A tester may help improve the product on his own time, sure. But AS a tester, it’s not in your scope to improve the product, or for that matter, to do the taxes of the company you work for.
Can you recommend a modification of the language to clarify this?]
James – this seems to not be an ethical disposition (e.g. moral philosophy, discernment between right or wrong behavior) but instead a statement of how you might personally operate (and suggest others may choose to operate) in their profession with integrity.
[James’ Reply: Each one of them is an ethical issue, not simply how I operate. To violate any of them is to cross a line that harms or may reasonably be expected to harm my client.]
An ethical translation for “Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.” would be that, “It is wrong to consciously cover-up any mistakes that have high-impact to the client.” The tone of this alternate language seems like a broadcasting of moral judgement to me; stronger and self-righteous.
[James’ Reply: Oh, is THAT what you are saying? I wrote it that way to mimic the ACM Code of Ethics style.]
My moral philosophy states: “It is right to be self-righteous about behaviors that are beneficial to one’s self and society.”
[James’ Reply: I agree! I have been told that I sound arrogant and sanctimonious when I talk about ethics like this. My reply is “I think it is right to invite people who would discourage ethical conversation to meet me in hell.”]
Thanks for a great post!
James, what do you mean by product? Is it just the released software that can be “tested” or the source code as well? What about non-code artifacts like requirements & specs, or the opinions & assumptions & facts held about a product in people’s minds?
[James’ Reply: I mean the released product, in all its aspects; the thing we are creating for our customers.]
Many testers are given some control over the products they work on.
[James’ Reply: If so, then they are not testers. Nothing in the concept of “tester” or “testing” is the idea of creating a product. Someone who is a tester may certainly create a product, of course: in that case, he is a tester AND designer. You can call him a tester, but you would be speaking of only one of his roles.
The point of my principle is that my responsibility as a tester requires me to avoid doing things that will interfere with that responsibility. If, informally or formally, I have other roles– let’s say that I am also the receptionist, or the company cook– then I must take care that I can perform all my roles.
I once was in charge of testing AND technical support, when I worked at SmartPatents. I soon gave up my tech support management role, however, when I determined that I could not be a responsible technical support leader while at the same time take care of the testing.]
Some teams/companies regard product improvements made by testers to be an important part of assisting the quality creation process. Do you think those testers should push back on this or is it that your principles are designed with test consultant roles in mind more than permanent roles within product development teams?
[James’ Reply: I think whenever a tester is expected to improve a product, that expectation will probably interrupt and diminish their ability to test. I am a testing expert. I think testing is useful and important. I am also a programmer. I know that I cannot be a good programmer and a good tester at the same time. They are very different activities. Maybe that means I have a very high standard of programming and testing. However you think about it: yes I would probably push back if my employer expected me to be competent at development as well as competent at testing.]
Re: testing vs product development
Here at ProChain I’ve had several opportunities to do product design, including developing as a side project a set of Microsoft Project functional macros that helped our consultants speed up some tedious work. I took great pride in creating these so-called Expert Tools, and they have become part of our regular tool set. But the time I spent developing them was time I did not spend testing the rest of our products, and to this day I’m noticeably worse testing these tools when we modify them than I am testing other products that I did not design.
Hey Andrew,
Do you mean Acceptance Test Testers? i.e. Testers from the client that test the “fit for purpose” of the software/product? They could have the mandate to suggest improvements that are missing for proper business use and/or defects that need to be fixed. James’ question still remains though if these people really don’t go over and beyond what a tester’s mandate usually is. So we might be stretching the term tester a bit.
Oliver
Ethics is surely important but in my opinion not as a topic to talk about.
Of course we can talk ethically, but a “Code of Ethics” is, in my opinion, always the wrong way to go.
[James’ Reply: When you say “wrong way to go” are you proposing an ethical standard? Or, are you making a pragmatic suggestion?]
Let me quote Wittgenstein’s Tractatus Logico-Philosophicus in order to explain my point of view. “It is clear that ethics cannot be articulated” (6.421) and also “The first thought in setting up an ethical law of the form “thou shalt . . .” is: And what if I do not do it? […] It is clear however, that ethics has nothing to do with punishment and reward in the usual sense of the terms. Nevertheless, there must indeed be some kind of ethical reward and punishment, but they must reside in the action itself.” (6.422).
[James’ Reply: That’s a quote, but I see no argument there. What I’m talking about is a community standard. I propose and live by the standards I articulated. These standards do carry penalties and rewards, not within themselves, but within the community.]
So here I agree to Scott Barber, everyone has to have his own ethics which must reside in the action itself. So ethics should be implicit in the way we work, talk, etc. If it becomes explicit “language does degenerate into moralizations” (Heinz von Foerster). A code of ethics is surely explicit. The language I speak is /my/ language, the testing I perform is /my/ approach to it. And the same applies, in my opinion, to ethics.
[James’ Reply: And yet I have articulated ethical principles. There they are. Perhaps the language you speak really is your language– I suspect you mean something different by ethics than I do.]
Hello James,
Long-time lurker, first-time poster.
“Recuse myself from any project if I feel unable to give reasonable and workman-like effort”
Could you expand a little on this? What kind of impediments do you envisage that would result in you being “unable to give reasonable and workman-like effort”, and how would you go about rescuing yourself from such a situation?
[James’ Reply: It’s not about impediments. It’s about being paid for a service that I am not rendering. That would be fraud, or theft.
I said “recuse” and not “rescue” but both words work, I guess.
How I recuse/rescue myself is by refusing to work on that project. I will quit if necessary, as I have done several times, most notably in 1999 (that’s when I went independent).]
Many thanks,
Adrian Hague
QA Manager
fish in a bottle
Hi James
It seems valuable to set out a code of ethics to base your work on, something to refer back to when you find yourself in a challenging situation (an Ethics Heuristic?)
And I agree with you and Scott that each tester should develop their own, rather than copy someone else’s. (although yours are a good start/inspiration point)
My question is, do you ever talk through your ethical code to the project team you work with?
If so, do you set it out at project engagement or only refer to it when explaining why you’ve acted in a certain way? Or is this something you keep as a silent set of principles (only sharing with the test community)?
I’m guessing the answer will be context based …but I’d be interested to hear some examples and the project teams reactions.
[James’ Reply: I find the most people don’t have the patience to talk about ethics, so I introduce my ethical code as I encounter invitations to violate them. “Just-in-time” ethics discussions, I guess.]
Hi, thank you for a great post.
Could you elaborate a bit on “I will not keep test case metrics, because they are damaging in almost any context”.
I find metrics useful, not in the sense that a percentage will be a sure sign of quality.
[James’ Reply: If you believe that test case metrics are useful, it’s not unethical to make use of them. It’s incompetent, mind you, but it’s not unethical. Doing things that mislead your clients and amount to a mere pretense of managing a project when you should bloody well know better is NOT unethical– assuming you in fact do not know better. In that case there is another ethical issue: have you taken due care to learn enough about testing and measurement to know that test case measurement is a load of crap?
I used test case counts for about five years, casually, until the day I realized how intellectually dishonest they were. At that point, ashamed, I ceased using them. That was 20 years ago this month.
Whatever you think you are getting from test case metrics you can also get much more honestly and simply using other means– such as guessing, summarizing guesses, tracking testing time, discussing or listing tasks, or talking to people on the project.
Test case metrics are popular simply because they LOOK scientific. That’s also ultimately why I say they are unethical. If I know that the only value of a number is to trick someone into thinking that I have some kind of scientific understanding of my project, then I am committing fraud. People are enchanted by lab coats, stethoscopes, and the accoutrements of science. They allow themselves to be swayed by them regardless of violations of basic logic and measurement theory.]
But if we have identified 100 test cases, and during a period (cycle) we are only able to test 20 of these, I figure this is a useful metric to show that there is not enough time (or not enough resources) to fully test through the identified cases.
[James’ Reply: That assumes that the test cases are A) necessary, B) sufficient, C) enough different from each other to be independently valuable, D) enough similar to be comparable. It assumes that each test case takes the same amount of time to perform. It assumes you haven’t mixed wildly different levels of test cases together. It assumes a one-to-one correspondence between a test case and testing effort (in other words, that there is no testing unreported in terms of test cases). It assumes that you are not adding or deleting test cases as you go (which if you are not doing means that you aren’t thinking). It assumes that your clients are not smart enough or don’t care enough to call you on your crap (this is probably true, but it’s still a contemptible attitude to have about them). It assumes you have no better way to say “the testing is going more slowly than we hoped” than by slicing it into illusory bricks of work.
It assumes you feel no pressure to inflate your test case count, or to deflate it (BTW, why did you choose “100” test cases? Why not call the same set of activities 1000 tests, or 10 tests?
It assumes that you don’t do the sort of testing that is difficult to parse into test cases. (Is a program that randomly generates data creating one test case or a million? Or is it, itself, considered a test case?)
Meanwhile, you DO have at least a few reasonable ways to talk about testing. You don’t need the pseudoscience.]
I also think it can play an important role in presenting the work done by testers as might be required or expected by certain clients. How would you attempt to visiualize the state of the project (in terms of quality), if not through case results and metrics?
[James’ Reply: That you would ask such a question is a sad commentary on the state of our craft. You should already know, man. HOW CAN YOU NOT KNOW?
Test case counts play an important role? They play no role AT ALL in my practice.
Alternatives include: trust, talking, to do lists, kanban-style status boards, structured subjectivity dashboards, session-based test metrics, thread-based test management mindmaps.
When I hear people arguing for test case metrics, I feel like I’m listening to someone defend exorcism as a treatment for cancer (“Because people want to believe that doctors can banish evil spirits!”). It’s bizarre. Wake up and learn your craft.]
Cheers
Thomas
Thanks for this James.
So I suspect it is a shared view and expectation of most organisations that the level of integrity of those undertaking the responsibility of testing and all its related activities, be impeccable and that Testers be held accountable for carrying out these functions in accordance with the documented policies and procedures that their employers choose to prescribe to, in addition to all standards (in terms of the level of ethics applied,) being met as a default. In the unlikely event that the above-mentioned policies, procedures and standards are not clearly defined and documented, Testers should endeavour to acquire detailed written requirements relating to these expectations before agreeing to commence any testing
[James’ Reply: I’m sorry, Cheri, but that’s not going to work. No one documents all policies. No one. Nowhere. Ever. Not even the biggest companies do that. So if you are saying that we should refuse to test without a comprehensive book of policies, that advice is just not going to be followed by anyone.
If we are going to deal with ethics honestly, then we must acknowledge reality.
Moreover, human social systems cannot and should not be reduced to the equivalent of the rule book for football. A big reason we want to talk about ethics is to help people navigate WITHOUT rules.]
Adherence to ethical IT testing standards should be non-negotiable as these are ultimately what determine the ability of a Company to provide quality IT products and/or services which ability, impacts its reputation and profit margins at the end of the day i.e. poor quality testing = poor quality output of product = poor Company reputation.
[James’ Reply: I think negotiation is quite important. Negotiation is partly how we come to understand and modify out policies.
Ethics is hard! We strive to behave ethically, but there are gray areas, and pressures that sometimes get the better of us.]
This being said, I have worked for IT organisations that practice commendable integrity in applying high-quality ethics and standards to testing of all IT products, where the strictest level of governance is applied and duration of (time) allocated for testing deliver “fit for purpose” IT products and/or services, ALL application code and testing results are peer reviewed by fellow independent testers and developers as well as business analysts, before being released into production.
[James’ Reply: Really? All? Isn’t that hard?]
I have observed the beneficial outcomes of intense and rigorous testing techniques being applied where extraordinary measures are taken to break code and nothing is owned by “assumption”, where testers are empowered with the necessary tools and knowledge related to the desired and expected end-to-end functionality and outputs and where effective communication leaves no room for misinterpretation of what needs to be tested.
[James’ Reply: Really? No room at all for misinterpretation?]
The difference it makes to what is delivered to market, well, astounding would be an understatement. Provision of authentic, clear scenarios and case studies together with detailed Test Plans, certainly bring limitless value to Testers and leaves nothing to chance.
[James’ Reply: You are using too many superlatives, here. Come on. Were you personally responsible for this work?]
A Testers understanding/interpretation of requirements being assessed and alignment with Stakeholder expectation of quality, standards and integrity required, being confirmed by the Test Manager/Team Lead, also seems to promote and build an overall sense of confidence between Testers and interfacing teams?
Then there is the integrity of those conducting formal sign-off of IAT and UAT testing. For me, it is ethical practice for those peer reviewing and approving testing results, to be accountable for ensuring that ALL developed code within an organisation, is appropriately tested (without exception), prior to migrating out of any testing environments or being promoted into production and customer facing environments. In conjunction, a stringent practice that should be strictly complied with, governed and audited is the checking and sign-off of retrofitting of code (that may have needed to bypass a testing environment for emergency purposes), back into the environments that were bypassed to ensure developers and testers have a solid foundation and correct versions of code from which to develop and test application code.
The above in my opinion, should always be a collective and collaborative TEAM effort undertaken by highly qualified, experienced and committed employees and stakeholders (in accordance with ITIL best practices and processes), resulting in almost zero negative impact taking place in production environments and minimal defects/incidents being present as a result of a testing related oversight. When ethical testing practices are upheld and maintained, clearly stated and defined in related Company policy, procedure and quality specification documents, Companies can only have very happy repeat and on-going business from loyal customers, enjoy high profit margins, delivery of high-quality products/service, well paid employees and therefore, ethical and high performing teams of IT Testers.
To summarise, organisational testing ethics should be clearly defined, documented and managed, adherence thereto closely governed, moreover, ethics and values of testing experts vigorously assessed at point of interview for testing roles.
[James’ Reply: This is too clean a description. I have no credible colleague who would speak this way. Why don’t you tell us how it really is?]
This is my 2c.
Take Care
Cheri Rubens
I invite you to take up a testing role in any of the top 4 commercial banks in South Africa and discover ethical testing by the book for yourself. My answer is YES, REALLY James !! 🙂
P.S. enjoyed your feedback and totally (get) where you are coming from and your points are valid – I guess it depends on what king of industry one works in at the end of the day. The banking industry are compelled for obvious legal reasons, to ensure high standards are met and maintained.
[James’ Reply: I’ve worked with some of the largest banks in the world, including Barclays, which you may have heard of, and some others that I am not allowed to name. I can imagine that a particular bank might have policies that get followed to some large degree, but I have never seen any place that follows all of their policies all the time. It would cripple them– because the policies are generally created by people who have no idea how work happens. I once worked for a bank that had me sign an IT agreement and within seconds authorized me, orally, to break that very agreement so that we could get some work done.]
About not my job to improve the product. Are you saying that when I write a bug report I should limit myself to saying something like:
“The product goes against the ‘Comparable Products’ heuristic.”
And _not_ add:
“and changing it to do the following would fix that”
because I tend to do the latter.
[James’ Reply: I assume you are simplifying, here. Just saying that a behavior violates a generic consistency heuristic is not much help. Let’s say that you explained how and why the way a competing product handles autosave feels better than the way your own product does it. That would be an application of the comparable product heuristic. The issue would be whether you need to say “…so, you should make it work like that other product does.”
First, it seems that the change request doesn’t add much. Is it really necessary? But that not really the problem. There are two problems. 1) Aren’t there many ways to solve a problem? And aren’t you just picking one of those possibilities? And isn’t it not your job to select them? And therefore isn’t someone not doing HIS job by accepting your solution without thinking it through? AND 2) Even if it’s a good solution, are you annoying the developer by pushing it? Does annoying the developer serve your purposes?
I get around these problems by developing a relationship of trust with the devs I work with. I then freely acknowledge that any suggestion I make is just a suggestion offered in the hope that it helps, and not in any way a prescription.
The fact that product design is not testing no more hinders me in offering suggestions than I feel hindered in offering suggestions about the menus in the cafeteria. That also isn’t my job, but I can be diplomatic and friendly and maybe I will be listened to.]
Actually I believe that if I can find something wrong with a product, it is also my duty to suggest how it could be improved, at least unless the fix is very obvious.
[James’ Reply: It is literally not your duty to DO other people’s jobs. But I could agree that it is your duty to be aware of and open to inexpensive ways you can help other people in your team do whatever they need to do, as a general matter of being a good employee and teammate. Making respectful and helpful suggestions can be part of that.]