Thoughts Toward The Ethics of Testing
I am thinking and talking a lot about ethics, lately. Maybe that’s because Context-Driven testing is getting better traction, now. More testers are approaching me, asking for guidance. More testers are challenging fake testing in their organizations.
Or maybe I’m just noticing it more, because Cem Kaner used to take the brunt of all this. In that respect, I have historically been more a follower than a leader in our community. Recently, I’ve stepped forward to be more of the role model that I am expected by my colleagues to be.
(Note: Ethics is a guaranteed sore subject. Whenever I talk about ethics, I can’t avoid implying that I believe some people are not as ethical as they should be. But, you know, that’s the burden of leadership. To those who avoid politically difficult subjects, enjoy your shadowy existence. Lurk on, yon lurkers.)
I have an ethical code. Actually, several! The Association for Software Testing adopted the ACM Code of Ethics, verbatim. It’s okay. I would have preferred the simpler IEEE Code, personally. I also almost completely follow Jerry Weinberg’s code. I take comfort from all of them, plus I have my own.
I need my own, because these codes above don’t directly deal with certain common testing ethics issues.
So, I’m the content-owner for the 2nd Kiwi Workshop on Software Testing, tomorrow morning, here in Wellington, New Zealand. Our topic is ethics. To get ready for it, I thought I would write out some of the ethical principles by which I strive to operate. Here they are:
- Know what a test is. Avoid labeling an activity as a “test” unless it represents a sincere effort to discover a problem in a product.
- Maintain a reasonable impartiality. The purpose of testing is to cast light on the status of the product and its context, in the service of my clients. I may play multiple roles on a project, but my purpose, insofar as I am a tester, is not to design or improve the product.
- Do not claim to assure, ensure, or control quality. I don’t control anything about the product: a tester is a witness. In that capacity, I strive to assist the quality creation process.
- Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.
- Apply test methods that are appropriate to the level of risk in the product and the context of the project.
- Alert my clients to anything that may impair my ability to test.
- Recuse myself from any project if I feel unable to give reasonable and workman-like effort.
- Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.
- Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way. (e.g. I will not keep test case metrics, because they are damaging in almost any context)
- If I do not understand or accept my mission, it shall be my urgent priority to discover it or renegotiate it.
- Do not deceive my clients about my work, nor help others to perpetrate deception.
- Do not accept tasks for which I am not reasonably prepared or possess sufficient competence to perform, unless I am under the direction and supervision of someone who can guide me.
- Study my craft. Be alert to better solutions and better ways of working.
June 14th, 2012 at 6:56 am
Hi James!
You said “–my purpose, insofar as I am a tester, is not to design or improve the product.” Do you mean that when confronted with a poor decision you just bring the solution up and say “this might be a poor decision”? If you have the competence to give improvement suggestions, should you still leave them unvoiced? “Are you then in the role of a tester” might be a question, you’d ask.
[James' Reply: Voice whatever you like. But when you voice things that are outside your role as a tester, you run the risk of neglecting your own role, confusing others about your role, and invading the turf of those people.
I give my opinions, but I am careful to couch my opinions as that of an interested amateur. When I'm talking about problems, as such, then I am speaking from within my role and expertise.]
“Report everything that I believe, in good faith, to be a threat to the product or to the user thereof, according to my understanding of the best interests of my client and the public good.” What if you report a bug that the developers aren’t willing to fix? I heard this theory that you should only report stuff that is getting fixed. How do you feel about that?
[James' Reply: That's a ridiculous theory. How can I know what they are willing to fix until I talk to them about it? How can I know if they might wish to change their minds?]
“Do not accede to requests by my client to work in a wasteful, dangerous, or deceptive way.” In the case of a client asking for a metric derived from test cases, do you just say “no” or do you consult the client to improve their measuring? Or do you just reject the case?
[James' Reply: I can do any of those things. Usually, I talk a lot.]
Sometimes the client has a process that needs to be followed. The process might be “stupid” or inefficient. Do you help the client to make the process better or would you just ignore it and do it the way you think would be most productive and efficient? Let’s say the client uses the ISTQB foundation testing process by the book. Would you change your testing approach to fit the process or would you require the process to be changed?
- Pekka
[James' Reply: I will not violate my principle. But it's a matter of judgment as to whether something that is not a very good way to do something is also a bad way to do it. There's a lot of gray area, and I have to navigate it. I'm on a project right now where I'm being asked to follow a template. I'm going to change the template, however-- but not totally. If the customer wants to change it back, they will have to hire someone else to do that.]
June 14th, 2012 at 7:19 am
With the possible exception of one requested clarification (below), this is fabulous.
*EVERYONE*… and I mean *EVERYONE*… should have and follow a personal code of ethics. Its not for me to say what the details of another persons code should or should not include (that’s the context part), but I’d be willing to consider calling “have a personal code of ethics that you actually follow” a “best practice” (right alongside the other things I’d be willing to consider calling “best practices” like, “nourish your body”, “educate your mind”, “strive for acceptable balance in your life”, and “try not to be an idiot too often”).
The requested clarification is around the second bullet. I get the important distinction about role here, and I’m pretty sure I follow (and agree with) the intent. I’d like to hear more of your thoughts on why, when you are serving in a testing role, might not periodically be operating with the purpose of improving the product.
I’m thinking specifically about experiences I’ve had “pairing” (we didn’t call it that, but that’s what it was) with developers/architects/analysts very early in the project explicitly to test their ideas/concepts/prototypes/unit test design frameworks/etc. with the sole purpose of helping to build a better product.
I could argue that isn’t “product improvement” because there was not yet a product… I could argue that what I was testing was someone’s “work product” vs. “the end product” and thus the impartiality applies to the work product and the end product is a context factor… I could also argue that what I was doing in those moments was actually serving in a coaching role and not in a tester role. So my intent is not to debate or negate the point, my intent is to hear your thinking on the matter (though, as always, I reserve the right to debate or negate based on your answer — and I don’t think you’d have much respect for me if I didn’t reserve that right .
Again, great post. I’m hard pressed to see how an ethical person could take offense to this one.
–
Scott Barber
System Performance Strategist and CTO; PerfTestPlus, Inc.
Author; Web Load Testing for Dummies
Co-Author; Performance Testing Guidance for Web Applications
Contributing Author; Beautiful Testing
Contributing Author; How to Reduce the Cost of Software Testing
http://www.perftestplus.com
http://about.me/scott.barber
“If you can see it in your mind…
you will find it in your life.”
[James' Reply: Hi Scott. I'm not trying to say that it's wrong for a goalie to score a goal, but simply that a goalie who scores a goal on purpose is probably neglecting his duty AS a goalie. A tester may help improve the product on his own time, sure. But AS a tester, it's not in your scope to improve the product, or for that matter, to do the taxes of the company you work for.
Can you recommend a modification of the language to clarify this?]
June 14th, 2012 at 7:48 am
James – this seems to not be an ethical disposition (e.g. moral philosophy, discernment between right or wrong behavior) but instead a statement of how you might personally operate (and suggest others may choose to operate) in their profession with integrity.
[James' Reply: Each one of them is an ethical issue, not simply how I operate. To violate any of them is to cross a line that harms or may reasonably be expected to harm my client.]
An ethical translation for “Make my clients aware, with alacrity, of any mistake I have made which may require expensive or disruptive correction.” would be that, “It is wrong to consciously cover-up any mistakes that have high-impact to the client.” The tone of this alternate language seems like a broadcasting of moral judgement to me; stronger and self-righteous.
[James' Reply: Oh, is THAT what you are saying? I wrote it that way to mimic the ACM Code of Ethics style.]
My moral philosophy states: “It is right to be self-righteous about behaviors that are beneficial to one’s self and society.”
[James' Reply: I agree! I have been told that I sound arrogant and sanctimonious when I talk about ethics like this. My reply is "I think it is right to invite people who would discourage ethical conversation to meet me in hell."]
Thanks for a great post!
June 15th, 2012 at 12:01 am
James, what do you mean by product? Is it just the released software that can be “tested” or the source code as well? What about non-code artifacts like requirements & specs, or the opinions & assumptions & facts held about a product in people’s minds?
[James' Reply: I mean the released product, in all its aspects; the thing we are creating for our customers.]
Many testers are given some control over the products they work on.
[James' Reply: If so, then they are not testers. Nothing in the concept of "tester" or "testing" is the idea of creating a product. Someone who is a tester may certainly create a product, of course: in that case, he is a tester AND designer. You can call him a tester, but you would be speaking of only one of his roles.
The point of my principle is that my responsibility as a tester requires me to avoid doing things that will interfere with that responsibility. If, informally or formally, I have other roles-- let's say that I am also the receptionist, or the company cook-- then I must take care that I can perform all my roles.
I once was in charge of testing AND technical support, when I worked at SmartPatents. I soon gave up my tech support management role, however, when I determined that I could not be a responsible technical support leader while at the same time take care of the testing.]
Some teams/companies regard product improvements made by testers to be an important part of assisting the quality creation process. Do you think those testers should push back on this or is it that your principles are designed with test consultant roles in mind more than permanent roles within product development teams?
[James' Reply: I think whenever a tester is expected to improve a product, that expectation will probably interrupt and diminish their ability to test. I am a testing expert. I think testing is useful and important. I am also a programmer. I know that I cannot be a good programmer and a good tester at the same time. They are very different activities. Maybe that means I have a very high standard of programming and testing. However you think about it: yes I would probably push back if my employer expected me to be competent at development as well as competent at testing.]
June 15th, 2012 at 11:15 am
Re: testing vs product development
Here at ProChain I’ve had several opportunities to do product design, including developing as a side project a set of Microsoft Project functional macros that helped our consultants speed up some tedious work. I took great pride in creating these so-called Expert Tools, and they have become part of our regular tool set. But the time I spent developing them was time I did not spend testing the rest of our products, and to this day I’m noticeably worse testing these tools when we modify them than I am testing other products that I did not design.
June 15th, 2012 at 7:27 pm
Hey Andrew,
Do you mean Acceptance Test Testers? i.e. Testers from the client that test the “fit for purpose” of the software/product? They could have the mandate to suggest improvements that are missing for proper business use and/or defects that need to be fixed. James’ question still remains though if these people really don’t go over and beyond what a tester’s mandate usually is. So we might be stretching the term tester a bit.
Oliver
June 22nd, 2012 at 3:49 am
Ethics is surely important but in my opinion not as a topic to talk about.
Of course we can talk ethically, but a “Code of Ethics” is, in my opinion, always the wrong way to go.
[James' Reply: When you say "wrong way to go" are you proposing an ethical standard? Or, are you making a pragmatic suggestion?]
Let me quote Wittgenstein’s Tractatus Logico-Philosophicus in order to explain my point of view. “It is clear that ethics cannot be articulated” (6.421) and also “The first thought in setting up an ethical law of the form “thou shalt . . .” is: And what if I do not do it? [...] It is clear however, that ethics has nothing to do with punishment and reward in the usual sense of the terms. Nevertheless, there must indeed be some kind of ethical reward and punishment, but they must reside in the action itself.” (6.422).
[James' Reply: That's a quote, but I see no argument there. What I'm talking about is a community standard. I propose and live by the standards I articulated. These standards do carry penalties and rewards, not within themselves, but within the community.]
So here I agree to Scott Barber, everyone has to have his own ethics which must reside in the action itself. So ethics should be implicit in the way we work, talk, etc. If it becomes explicit “language does degenerate into moralizations” (Heinz von Foerster). A code of ethics is surely explicit. The language I speak is /my/ language, the testing I perform is /my/ approach to it. And the same applies, in my opinion, to ethics.
[James' Reply: And yet I have articulated ethical principles. There they are. Perhaps the language you speak really is your language-- I suspect you mean something different by ethics than I do.]
June 22nd, 2012 at 3:57 am
Hello James,
Long-time lurker, first-time poster.
“Recuse myself from any project if I feel unable to give reasonable and workman-like effort”
Could you expand a little on this? What kind of impediments do you envisage that would result in you being “unable to give reasonable and workman-like effort”, and how would you go about rescuing yourself from such a situation?
[James' Reply: It's not about impediments. It's about being paid for a service that I am not rendering. That would be fraud, or theft.
I said "recuse" and not "rescue" but both words work, I guess.
How I recuse/rescue myself is by refusing to work on that project. I will quit if necessary, as I have done several times, most notably in 1999 (that's when I went independent).]
Many thanks,
Adrian Hague
QA Manager
fish in a bottle
June 27th, 2012 at 4:00 pm
Hi James
It seems valuable to set out a code of ethics to base your work on, something to refer back to when you find yourself in a challenging situation (an Ethics Heuristic?)
And I agree with you and Scott that each tester should develop their own, rather than copy someone else’s. (although yours are a good start/inspiration point)
My question is, do you ever talk through your ethical code to the project team you work with?
If so, do you set it out at project engagement or only refer to it when explaining why you’ve acted in a certain way? Or is this something you keep as a silent set of principles (only sharing with the test community)?
I’m guessing the answer will be context based …but I’d be interested to hear some examples and the project teams reactions.
[James' Reply: I find the most people don't have the patience to talk about ethics, so I introduce my ethical code as I encounter invitations to violate them. "Just-in-time" ethics discussions, I guess.]
July 10th, 2012 at 1:08 am
Hi, thank you for a great post.
Could you elaborate a bit on “I will not keep test case metrics, because they are damaging in almost any context”.
I find metrics useful, not in the sense that a percentage will be a sure sign of quality.
[James' Reply: If you believe that test case metrics are useful, it's not unethical to make use of them. It's incompetent, mind you, but it's not unethical. Doing things that mislead your clients and amount to a mere pretense of managing a project when you should bloody well know better is NOT unethical-- assuming you in fact do not know better. In that case there is another ethical issue: have you taken due care to learn enough about testing and measurement to know that test case measurement is a load of crap?
I used test case counts for about five years, casually, until the day I realized how intellectually dishonest they were. At that point, ashamed, I ceased using them. That was 20 years ago this month.
Whatever you think you are getting from test case metrics you can also get much more honestly and simply using other means-- such as guessing, summarizing guesses, tracking testing time, discussing or listing tasks, or talking to people on the project.
Test case metrics are popular simply because they LOOK scientific. That's also ultimately why I say they are unethical. If I know that the only value of a number is to trick someone into thinking that I have some kind of scientific understanding of my project, then I am committing fraud. People are enchanted by lab coats, stethoscopes, and the accoutrements of science. They allow themselves to be swayed by them regardless of violations of basic logic and measurement theory.]
But if we have identified 100 test cases, and during a period (cycle) we are only able to test 20 of these, I figure this is a useful metric to show that there is not enough time (or not enough resources) to fully test through the identified cases.
[James' Reply: That assumes that the test cases are A) necessary, B) sufficient, C) enough different from each other to be independently valuable, D) enough similar to be comparable. It assumes that each test case takes the same amount of time to perform. It assumes you haven't mixed wildly different levels of test cases together. It assumes a one-to-one correspondence between a test case and testing effort (in other words, that there is no testing unreported in terms of test cases). It assumes that you are not adding or deleting test cases as you go (which if you are not doing means that you aren't thinking). It assumes that your clients are not smart enough or don't care enough to call you on your crap (this is probably true, but it's still a contemptible attitude to have about them). It assumes you have no better way to say "the testing is going more slowly than we hoped" than by slicing it into illusory bricks of work.
It assumes you feel no pressure to inflate your test case count, or to deflate it (BTW, why did you choose "100" test cases? Why not call the same set of activities 1000 tests, or 10 tests?
It assumes that you don't do the sort of testing that is difficult to parse into test cases. (Is a program that randomly generates data creating one test case or a million? Or is it, itself, considered a test case?)
Meanwhile, you DO have at least a few reasonable ways to talk about testing. You don't need the pseudoscience.]
I also think it can play an important role in presenting the work done by testers as might be required or expected by certain clients. How would you attempt to visiualize the state of the project (in terms of quality), if not through case results and metrics?
[James' Reply: That you would ask such a question is a sad commentary on the state of our craft. You should already know, man. HOW CAN YOU NOT KNOW?
Test case counts play an important role? They play no role AT ALL in my practice.
Alternatives include: trust, talking, to do lists, kanban-style status boards, structured subjectivity dashboards, session-based test metrics, thread-based test management mindmaps.
When I hear people arguing for test case metrics, I feel like I'm listening to someone defend exorcism as a treatment for cancer ("Because people want to believe that doctors can banish evil spirits!"). It's bizarre. Wake up and learn your craft.]
Cheers
Thomas