Accountability for What You Say is Dangerous and That’s Okay

[Note: I offered Maaret Pyhäjärvi the right to review this post and suggest edits to it before I published it. She declined.]

A few days ago I was keynoting at the New Testing Conference, in New York City, and I used a slide that has offended some people on Twitter. This blog post is intended to explore that and hopefully improve the chances that if you think I’m a bad guy, you are thinking that for the right reasons and not making a mistake. It’s never fun for me to be a part of something that brings pain to other people. I believe my actions were correct, yet still I am sorry that I caused Maaret hurt, and I will try to think of ways to confer better in the future.

Here’s the theme of this post: Getting up in front of the world to speak your mind is a dangerous process. You will be misunderstood, and that will feel icky. Whether or not you think of yourself as a leader, speaking at a conference IS an act of leadership, and leadership carries certain responsibilities.

I long ago learned to let go of the outcome when I speak in public. I throw the ideas out there, and I do that as an American Aging Overweight Left-Handed Atheist Married Father-And-Father-Figure Rough-Mannered Bearded Male Combative Aggressive Assertive High School Dropout Self-Confident Freedom-Loving Sometimes-Unpleasant-To-People-On-Twitter Intellectual. I know that my ideas will not be considered in a neutral context, but rather in the context of how people feel about all that. I accept that.  But, I have been popular and successful as a speaker in the testing world, so maybe, despite all the difficulties, enough of my message and intent gets through, overall.

What I can’t let go of is my responsibility to my audience and the community at large to speak the truth and to do so in a compassionate and reasonable way. Regardless of what anyone else does with our words, I believe we speakers need to think about how our actions help or harm others. I think a lot about this.

Let me clarify. I’m not saying it’s wrong to upset people or to have disagreement. We have several different culture wars (my reviewers said “do you have to say wars?”) going on in the software development and testing worlds right now, and they must continue or be resolved organically in the marketplace of ideas. What I’m saying is that anyone who speaks out publicly must try to be cognizant of what words do and accept the right of others to react.

Although I’m surprised and certainly annoyed by the dark interpretations some people are making of what I did, the burden of such feelings is what I took on when I first put myself forward as a public scold about testing and software engineering, a quarter century ago. My annoyance about being darkly interpreted is not your problem. Your problem, assuming you are reading this and are interested in the state of the testing craft, is to feel what you feel and think what you think, then react as best fits your conscience. Then I listen and try to debug the situation, including helping you debug yourself while I debug myself. This process drives the evolution of our communities. Jay Philips, Ash Coleman, Mike Talks, Ilari Henrik Aegerter, Keith Klain, Anna Royzman, Anne-Marie Charrett, David Greenlees, Aaron Hodder, Michael Bolton, and my own wife all approached me with reactions that helped me write this post. Some others approached me with reactions that weren’t as helpful, and that’s okay, too.

Leadership and The Right of Responding to Leaders

In my code of conduct, I don’t get to say “I’m not a leader.” I can say no one works for me and no one has elected me, but there is more to leadership than that. People with strong voices and ideas gain a certain amount of influence simply by virtue of being interesting. I made myself interesting, and some people want to hear what I have to say. But that comes with an implied condition that I behave reasonably. The community, over time negotiates what “reasonable” means. I am both a participant and a subject of those negotiations. I recommend that we hold each other accountable for our public, professional words. I accept accountability for mine. I insist that this is true for everyone else. Please join me in that insistence.

People who speak at conferences are tacitly asserting that they are thought leaders– that they deserve to influence the community. If that influence comes with a rule that “you can’t talk about me without my permission” it would have a chilling effect on progress. You can keep to yourself, of course; but if you exercise your power of speech in a public forum you cannot cry foul when someone responds to you. Please join me in my affirmation that we all have the right of response when a speaker takes the microphone to keynote at a conference.

Some people have pointed out that it’s not okay to talk back to performers in a comedy show or Broadway play. Okay. So is that what a conference is to you? I guess I believe that conferences should not be for show. Conferences are places for conferring. However, I can accept that some parts of a conference might be run like infomercials or circus acts. There could be a place for that.

The Slide

Here is the slide I used the other day:

maaret

Before I explain this slide, try to think what it might mean. What might its purposes be? That’s going to be difficult, without more information about the conference and the talks that happened there. Here are some things I imagine may be going through your mind:

  • There is someone whose name is Maaret who James thinks he’s different from.
  • He doesn’t trust nice people. Nice people are false. Is Maaret nice and therefore he doesn’t trust her, or does Maaret trust nice people and therefore James worries that she’s putting herself at risk?
  • Is James saying that niceness is always false? That’s seems wrong. I have been nice to people whom I genuinely adore.
  • Is he saying that it is sometimes false? I have smiled and shook hands with people I don’t respect, so, yes, niceness can be false. But not necessarily. Why didn’t he put qualifying language there?
  • He likes debate and he thinks that Maaret doesn’t? Maybe she just doesn’t like bad debate. Did she actually say she doesn’t like debate?
  • What if I don’t like debate, does that mean I’m not part of this community?
  • He thinks excellence requires attention and energy and she doesn’t?
  • Why is James picking on Maaret?

Look, if all I saw was this slide, I might be upset, too. So, whatever your impression is, I will explain the slide.

Like I said I was speaking at a conference in NYC. Also keynoting was Maaret Pyhäjärvi. We were both speaking about the testing role. I have some strong disagreements with Maaret about the social situation of testers. But as I watched her talk, I was a little surprised at how I agreed with the text and basic concepts of most of Maaret’s actual slides, and a lot of what she said. (I was surprised because Maaret and I have a history. We have clashed in person and on Twitter.) I was a bit worried that some of what I was going to say would seem like a rehash of what she just did, and I didn’t want to seem like I was papering over the serious differences between us. That’s why I decided to add a contrast slide to make sure our differences weren’t lost in the noise. This means a slide that highlights differences, instead of points of connection. There were already too many points of connection.

The slide was designed specifically:

  • for people to see who were in a specific room at a specific time.
  • for people who had just seen a talk by Maaret which established the basis of the contrast I was making.
  • about differences between two people who are both in the spotlight of public discourse.
  • to express views related to technical culture, not general social culture.
  • to highlight the difference between two talks for people who were about to see the second talk that might seem similar to the first talk.
  • for a situation where both I and Maaret were present in the room during the only time that this slide would ever be seen (unless someone tweeted it to people who would certainly not understand the context).
  • as talking points to accompany my live explanation (which is on video and I assume will be public, someday).
  • for a situation where I had invited anyone in the audience, including Maaret, to ask me questions or make challenges.

These people had just seen Maaret’s talk and were about to see mine. In the room, I explained the slide and took questions about it. Maaret herself spoke up about it, for which I publicly thanked her for doing so. It wasn’t something I was posting with no explanation or context. Nor was it part of the normal slides of my keynote.

Now I will address some specific issues that came up on Twitter:

1. On Naming Maaret

Maaret has expressed the belief that no one should name another person in their talk without getting their permission first. I vigorously oppose that notion. It’s completely contrary to the workings of a healthy society. If that principle is acceptable, then you must agree that there should be no free press. Instead, I would say if you stand up and speak in the guise of an expert, then you must be personally accountable for what you say. You are fair game to be named and critiqued. And the weird thing is that Maaret herself, regardless of what she claims to believe, behaves according to my principle of freedom to call people out. She, herself, tweeted my slide and talked about me on Twitter without my permission. Of course, I think that is perfectly acceptable behavior, so I’m not complaining. But it does seem to illustrate that community discourse is more complicated than “be nice” or “never cause someone else trouble with your speech” or “don’t talk about people publicly unless they gave you permission.”

2. On Being Nice

Maaret had a slide in her talk about how we can be kind to each other even though we disagree. I remember her saying the word “nice” but she may have said “kind” and I translated that into “nice” because I believed that’s what she meant. I react to that because, as a person who believes in the importance of integrity and debate over getting along for the sake of appearances, I observe that exhortations to “be nice” or even to “be kind” are often used when people want to quash disturbing ideas and quash the people who offer them. “Be nice” is often code for “stop arguing.” If I stop arguing, much of my voice goes away. I’m not okay with that. No one who believes there is trouble in the world should be okay with that. Each of us gets to have a voice.

I make protests about things that matter to me, you make protests about things that matter to you.

I think we need a way of working together that encourages debate while fostering compassion for each other. I use the word compassion because I want to get away from ritualized command phrases like “be nice.” Compassion is a feeling that you cultivate, rather than a behavior that you conform to or simulate. Compassion is an antithesis of “Rules of Order” and other lists of commandments about courtesy. Compassion is real. Throughout my entire body of work you will find that I promote real craftsmanship over just following instructions. My concern about “niceness” is the same kind of thing.

Look at what I wrote: I said “I don’t trust nice people.” That’s a statement about my feelings and it is generally true, all things being equal. I said “I’m not nice.” Yet, I often behave in pleasant ways, so what did I mean? I meant I seek to behave authentically and compassionately, which looks like “nice” or “kind”, rather than to imagine what behavior would trick people into thinking I am “nice” when indeed I don’t like them. I’m saying people over process, folks.

I was actually not claiming that Maaret is untrustworthy because she is nice, and my words don’t say that. Rather, I was complaining about the implications of following Maaret’s dictum. I was offering an alternative: be authentic and compassionate, then “niceness” and acts of kindness will follow organically. Yes, I do have a worry that Maaret might say something nice to me and I’ll have to wonder “what does that mean? is she serious or just pretending?” Since I don’t want people to worry about whether I am being real, I just tell them “I’m not nice.” If I behave nicely it’s either because I feel genuine good will toward you or because I’m falling down on my responsibility to be honest with you. That second thing happens, but it’s a lapse. (I do try to stay out of rooms with people I don’t respect so that I am not forced to give them opinions they aren’t willing or able to process.)

I now see that my sentence “I want to be authentic and compassionate” could be seen as an independent statement connected to “how I differ from Maaret,” implying that I, unlike her, am authentic and compassionate. That was an errant construction and does not express my intent. The orange text on that line indicated my proposed policy, in the hope that I could persuade her to see it my way. It was not an attack on her. I apologize for that confusion.

3. Debate vs. Dialogue

Maaret had earlier said she doesn’t want debate, but rather dialogue. I have heard this from other Agilists and I find it disturbing. I believe this is code for “I want the freedom to push my ideas on other people without the burden of explaining or defending those ideas.” That’s appropriate for a brainstorming session, but at some point, the brainstorming is done and the judging begins. I believe debate is absolutely required for a healthy professional community. I’m guided in this by dialectical philosophy, the history of scientific progress, the history of civil rights (in fact, all of politics), and the modern adversarial justice system. Look around you. The world is full of heartfelt disagreement. Let’s deal with it. I helped create the culture of small invitational peer conferences in our industry which foster debate. We need those more than ever.

But if you don’t want to deal with it, that’s okay. All that means is that you accept that there is a wall between your friends and those other people whom you refuse to debate with. I will accept the walls if necessary but I would rather resolve the walls. That’s why I open myself and my ideas for debate in public forums.

Debate is not a process of sticking figurative needles into other people. Debate is the exchange of views with the goal of resolving our differences while being accountable for our words and actions. Debate is a learning process. I have occasionally heard from people I think are doing harm to the craft that they believe I debate for the purposes of hurting people instead of trying to find resolution. This is deeply insulting to me, and to anyone who takes his vocation seriously. What’s more, considering that these same people express the view that it’s important to be “nice,” it’s not even nice. Thus, they reveal themselves to be unable to follow their own values. I worry that “Dialogue not debate” is a slogan for just another power group trying to suppress its rivals. Beware the Niceness Gang.

I understand that debating with colleagues may not be fun. But I’m not doing it for fun. I’m doing it because it is my responsibility to build a respectable craft. All testing professionals share this responsibility. Debate serves another purpose, too, managing the boundaries between rival value systems. Through debate we may discover that we occupy completely different paradigms; schools of thought. Debate can’t bridge gaps between entirely different world views, and yet I have a right to my world view just as you have a right to yours.

Jay Philips said on Twitter:

I admire Jay. I called her and we had a satisfying conversation. I filled her in on the context and she advised me to write this post.

One thing that came up is something very important about debate: the status of ideas is not the only thing that gets modified when you debate someone; what also happens is an evolution of feelings.

Yes I think “I’m right.” I acted according to principles I think are eternal and essential to intellectual progress in society. I’m happy with those principles. But I also have compassion for the feelings of others, and those feelings may hold sway even though I may be technically right. For instance, Maaret tweeted my slide without my permission. That is copyright violation. She’s objectively “wrong” to have done that. But that is irrelevant.

[Note: Maaret points out that this is legal under the fair use doctrine. Of course, that is correct. I forgot about fair use. Of course, that doesn’t change the fact that though I may feel annoyed by her selective publishing of my work, that is irrelevant, because I support her option to do that. I don’t think it was wise or helpful for her to do that, but I wouldn’t seek to bar her from doing so. I believe in freedom to communicate, and I would like her to believe in that freedom, too]

I accept that she felt strongly about doing that, so I [would] choose to waive my rights. I feel that people who tweet my slides, in general, are doing a service for the community. So while I appreciate copyright law, I usually feel okay about my stuff getting tweeted.

I hope that Jay got the sense that I care about her feelings. If Maaret were willing to engage with me she would find that I care about her feelings, too. This does not mean she gets whatever she wants, but it’s a factor that influences my behavior. I did offer her the chance to help me edit this post, but again, she refused.

4. Focus and Energy

Maaret said that eliminating the testing role is a good thing. I worry it will lead to the collapse of craftsmanship. She has a slide that says “from tester to team member” which is a sentiment she has expressed on Twitter that led me to say that I no longer consider her a tester. She confirmed to me that I hurt her feelings by saying that, and indeed I felt bad saying it, except that it is an extremely relevant point. What does it mean to be a tester? This is important to debate. Maaret has confirmed publicly (when I asked a question about this during her talk) that she didn’t mean to denigrate testing by dismissing the value of a testing role on projects. But I don’t agree that we can have it both ways. The testing role, I believe, is a necessary prerequisite for maintaining a healthy testing craft. My key concern is the dilution of focus and energy that would otherwise go to improving the testing craft. This is lost when the role is lost.

This is not an attack on Maaret’s morality. I am worried she is promoting too much generalism for the good of the craft, and she is worried I am promoting too much specialism. This is a matter of professional judgment and perspective. It cannot be settled, I think, but it must be aired.

The Slide Should Not Have Been Tweeted But It’s Okay That It Was

I don’t know what Maaret was trying to accomplish by tweeting my slide out of context. Suffice it to say what is right there on my slide: I believe in authenticity and compassion. If she was acting out of authenticity and compassion then more power to her. But the slide cannot be understood in isolation. People who don’t know me, or who have any axe to grind about what I do, are going to cry “what a cruel man!” My friends contacted me to find out more information.

I want you to know that the slide was one part of a bigger picture that depicts my principled objection to several matters involving another thought leader. That bigger picture is: two talks, one room, all people present for it, a lot of oratory by me explaining the slide, as well as back and forth discussion with the audience. Yes, there were people in the room who didn’t like hearing what I had to say, but “don’t offend anyone, ever” is not a rule I can live by, and neither can you. After all, I’m offended by most of the talks I attend.

Although the slide should not have been tweeted, I accept that it was, and that doing so was within the bounds of acceptable behavior. As I announced at the beginning of my talk, I don’t need anyone to make a safe space for me. Just follow your conscience.

What About My Conscience?

  • My conscience is clean. I acted out of true conviction to discuss important matters. I used a style familiar to anyone who has ever seen a public debate, or read an opinion piece in the New York Times. I didn’t set out to hurt Maaret’s feelings and I don’t want her feelings to be hurt. I want her to engage in the debate about the future of the craft and be accountable for her ideas. I don’t agree that I was presuming too much in doing so.
  • Maaret tells me that my slide was “stupid and hurtful.” I believe she and I do not share certain fundamental values about conferring. I will no longer be conferring with her, until and unless those differences are resolved.
  • Compassion is important to me. I will continue to examine whether I am feeling and showing the compassion for my fellow humans that they are due. These conversations and debates I have with colleagues help me do that.
  • I agree that making a safe space for students is important. But industry consultants and pundits should be able to cope with the full spectrum, authentic, principled reactions by their peers. Leaders are held to a higher standard, and must be ready and willing to defend their ideas in public forums.
  • The reaction on Twitter gave me good information about a possible trend toward fragility in the Twitter-facing part of the testing world. There seems to be a significant group of people who prize complete safety over the value that comes from confrontation. In the next conference I help arrange, I will set more explicit ground rules, rather than assuming people share something close to my own sense of what is reasonable to do and expect.
  • I will also start thinking, for each slide in my presentation: “What if this gets tweeted out of context?”

(Oh, and to those who compared me to Donald Trump… Can you even imagine him writing a post like this in response to criticism? BELIEVE ME, he wouldn’t.)

The New Testing Conference

Anna Royzman is starting a new testing conference which she has decided to call the New Testing Conference. Anna has asked me to tell you about some of the ways we will be trying to live up to that bold name.

(Disclosure: although it doesn’t belong to me and I am not in charge of anything, bear in mind that I have an economic interest in it. I’m being paid to present at it and I’m advising Anna on the design of the program and invitation of speakers. I will be doing a tutorial and probably other things, too.)

Position Talks as Gentle Debate

We were talking about what it means to have a test conference that focuses on “newness” and one of the things we realized is that new is always controversial. New is always sketchy. For any new practice, there will be lots of practitioners who roll their eyes about it or scowl darkly. Therefore, if we want to talk about new things, we have to deal with the clash between “tried and comfortable” and “new and nervous.” So, this conference must help good ideas move beyond the new and nervous stage, while letting the not so good ideas fall back into obscurity (at least for a while… until the next generation unearths it again like a cursed monkey paw).

A structure we want to try is this:

  1. Hold two or more short position talks around a particular topic. For instance “Is BDD worth doing or a vain waste of time?”
  2. The speakers discuss and debate the topic BEFORE the conference. That way, at the conference, they would be able to put their best ideas forward and avoid misrepresenting the other side’s argument.
  3. They each speak for 10-15 minutes explaining their arguments.
  4. There is a 20-minute break for the audience, during which they may speak with the speakers to give them ideas or continue the debate. The speaker don’t get a break.
  5. The speakers give 5-minute follow-up lightning talks to respond to the other speakers or amend their previous statements.

Each Track Talk Includes a Demonstration, Exercise, or Experience Report

We feel that just talking about concepts isn’t enough. So, each track talk will include at least one of the following:

  • Demonstration: Show how an idea works in practice, then take questions and comments from the audience.
  • Exercise: Get the audience to try something, then debrief.
  • Experience Report: Tell a specific story about something that you experienced at a particular time and place, then take questions and comments.

“360 degree” Tutorials

I’m not sure if that’s quite the right name, but we want to do some workshops based on a particular structure I’ve been experimenting with:

  1. The instructor offers a challenge (which he has previously performed and has results ready to share).
  2. The students perform the challenge.
  3. Debrief with instructor commentary and critique.
  4. The instructor shows what he did and challenges the students to critique it.
  5. Students critique instructors work.
  6. Instructor critiques his own work.

Part of the fun of a “360” kind of workshop, as an instructor, is that I try to anticipate every criticism that the students will make of my work. I usually miss something, but then I add it to my list of critiques and I am even more prepared for the next time I run the workshop. I end up looking smarter and smarter– but of course the punchline is: I’m smarter because I opened myself to all this criticism. And when we all get comfortable hearing critical reactions to our work, our whole community grows smarter, faster.

We. Use. Tools.

Context-Driven testers use tools to help ourselves test better. But, there is no such thing as test automation.

Want details? Here’s the 10,000 word explanation that Michael Bolton and I have been working on for months.

Editor’s Note: I have just posted version 1.03 of this article. This is the third revision we have made due to typos. Isn’t it interesting how hard it is to find typos in your own work before you ship an article? We used automation to help us with spelling, of course, but most of the typos are down to properly spelled words that are in the wrong context. Spelling tools can’t help us with that. Also, Word spell-checker still thinks there are dozens of misspelled words in our article, because of all the proper nouns, terms of art, and neologisms. Of course there are the grammar checking tools, too, right? Yeah… not really. The false positive rate is very high with those tools. I just did a sweep through every grammar problem the tool reported. Out of the five it thinks it found, only one, a missing hyphen, is plausibly a problem. The rest are essentially matters of writing style.

One of the lines it complained about is this: “The more people who use a tool, the more free support will be available…” The grammar checker thinks we should not say “more free” but rather “freer.” This may be correct, in general, but we are using parallelism, a rhetorical style that we feel outweighs the general rule about comparatives. Only humans can make these judgments, because the rules of grammar are sometimes fluid.

The Man Who Opened the Door

I just heard that Ed Yourdon died.

In ’93 or early ’94, I got a strange email from him. He had heard about me in Mexico and he wanted to meet. I had never been to Mexico and I had never met or spoken to Ed. I was shocked: One of the most famous people in the software development methodology world wanted to talk to me, a test manager in Silicon Valley who had almost no writing and had spoken at only one conference! This was the very first time I realized that I had begun to build a reputation in the industry.

Ed was not the first famous guy I had met. I met Boris Beizer at that conference I mentioned, and that did not go well (we yelled at each other… he told me that I was full of shit… that kind of thing). I thought that might be the end of my ambition to rock the testing industry, if the heavy hitters were going to hate me.

Ed was a heavy hitter. I owned many of his books and I had carefully read his work on structured analysis. He was one of my idols.

So we met. We had a nice dinner at the Hyatt in Burlingame, south of San Francisco. He told me I needed to study systems thinking more deeply. He challenged me to write a book and asked me to write articles for American Programmer (later renamed to the Cutter IT Journal).

The thing that got to me was that Ed treated me with respect. He asked me many questions. He encouraged me to debate him. He pushed me to write articles on the CMM and on Good Enough Software– both subjects that got me a lot of favorable attention.

On the day of our meeting, he was 49– the same age I am now. He set me on a path to become a guy like him– because he showed me (as many others would later do, as well) that the great among us are people who help other people aspire to be great, too. I enjoy helping people, but reflecting on how I was helped reminds me that it is not just fun, it’s a moral imperative. If Ed reached out his hand to me, some stranger, how can I not do the same?

Ed saw something in me. Even now I do not want to disappoint him.

Rest in Peace, man.

 

Competent People and Conference Keynotes

My colleague and friend Anne-Marie Charrett has a thing about women. A) She is one. B) She feels that not enough of them are speaking at testing conferences. (See also Fiona Charles’ post on this subject.) I support Anne-Marie’s cause partly because I support the woman herself and it would make her happy. This is how humanity works: we are tribal creatures. Don’t deny it, you will just sound silly.

There is another reason I support their cause, though. It’s related to the fact that we people are not only tribal creatures. We are also creatures of myth, story, and principle. Each of us lives inside a story, and we want that story to “win,” whatever that may mean to us. Apart from tribal struggles, there is a larger meta-tribal struggle over what constitutes the “correct” or “good” or “moral” story.

In other words, it isn’t only whom we like that motivates us, but also what seems right. I’m not religious, so I won’t bother to talk about that aspect of things. But in the West, the professional status of women is a big part of the story of good and proper society; about what seems right.

The story I’m living, these days, is about competence. And I think most people speaking at testing conferences are not competent enough. A lot of what’s talked about at testing conferences is the muttering of idiots. By idiot, I mean functionally stupid people: people who choose not to use their minds to find excellent solutions to important problems, but instead speak ritualistically and uncritically about monsters and angels and mathematically invalid metrics and fraudulent standards and other useless or sinister tools that are designed to amaze and confuse the ignorant.

I want to see at least 50% of people speaking at conferences to be competent. That’s my goal. I think it is achievable, but it will take a lot of work. We are up against an entrenched and powerful interest: the promoters-of-ineptness (few of whom realize the damage they do) who run the world and impose themselves on our craft.

Why are there so many idiots and why do they run the world? The roots and dynamics of idiocracy are deep. It’s a good question but I don’t want to go into it here and now.

What I want to say is that Anne-Marie and Fiona, along with some others, can help me, and I can help them. I want to encourage new voices to take a place in the Great Conversation of testing because I do believe there is an under-tapped pool of talent among the women of testing. I am absolutely opposed to quotas or anything that simply forces smaller people with higher voices onto the stage for the sake of it. Instead let’s find and develop talent that leads us into a better future. This is what SpeakEasy is all about.

Maybe, if we can get more women speaking and writing in the craft, we will be able to imagine a world where more than 50% of keynote speakers are not spouting empty quotes from great thinkers and generic hearsay about projects and incoherent terminology and false dichotomies and ungrounded opinions and unworkable heuristics presented in the form of “best practices.”

I am not a feminist. I’m not going to be one. This is why I have work to do. I am not naturally biased in favor of considering women, and even if I were, can I be so sure that I’m not biased in favor of the attractive ones? Or against them? Research suggests no one can be complacent about overcoming our biology and gender identity. So, it’s a struggle. Any honest man will tell you that. And, I must engage that struggle while maintaining my implacable hostility to charlatans and quacks. The story I am living tells me this is what I must do. Also, Anne-Marie has asked for my help.

Here’s my call to action: To bring new beautiful minds forth to stand against mediocrity, we need to make the world a better, friendlier place especially for the women among us. I’m asking all you other non-feminists out there to consider working with me on this.

Exploratory Testing 3.0

[Authors’ note: Others have already made the point we make here: that exploratory testing ought to be called testing. In fact, Michael said that about tests in 2009, and James wrote a blog post in 2010 that seems to say that about testers. Aaron Hodder said it quite directly in 2011, and so did Paul Gerrard. While we have long understood and taught that all testing is exploratory (here’s an example of what James told one student, last year), we have not been ready to make the rhetorical leap away from pushing the term “exploratory testing.” Even now, we are not claiming you should NOT use the term, only that it’s time to begin assuming that testing means exploratory testing, instead of assuming that it means scripted testing that also has exploration in it to some degree.]

[Second author’s note: Some people start reading this with a narrow view of what we mean by the word “script.” We are not referring to text! By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This includes text instructions, but also any form of instructions, or even biases that are not instructions.]

By James Bach and Michael Bolton

In the beginning, there was testing. No one distinguished between exploratory and scripted testing. Jerry Weinberg’s 1961 chapter about testing in his book, Computer Programming Fundamentals, depicted testing as inherently exploratory and expressed caution about formalizing it. He wrote, “It is, of course, difficult to have the machine check how well the program matches the intent of the programmer without giving a great deal of information about that intent. If we had some simple way of presenting that kind of information to the machine for checking, we might just as well have the machine do the coding. Let us not forget that complex logical operations occur through a combination of simple instructions executed by the computer and not by the computer logically deducing or inferring what is desired.”

Jerry understood the division between human work and machine work. But, then the formalizers came and confused everyone. The formalizers—starting officially in 1972 with the publication of the first testing book, Program Test Methods—focused on the forms of testing, rather than its essences. By forms, we mean words, pictures, strings of bits, data files, tables, flowcharts and other explicit forms of modeling. These are things that we can see, read, point to, move from place to place, count, store, retrieve, etc. It is tempting to look at these artifacts and say “Lo! There be testing!” But testing is not in any artifact. Testing, at the intersection of human thought processes and activities, makes use of artifacts. Artifacts of testing without the humans are like state of the art medical clinics without doctors or nurses: at best nearly useless, at worst, a danger to the innocents who try to make use of them.

We don’t blame the innovators. At that time, they were dealing with shiny new conjectures. The sky was their oyster! But formalization and mechanization soon escaped the lab. Reckless talk about “test factories” and poorly designed IEEE standards followed. Soon all “respectable” talk about testing was script-oriented. Informal testing was equated to unprofessional testing. The role of thinking, feeling, communicating humans became displaced.

James joined the fray in 1987 and tried to make sense of all this. He discovered, just by watching testing in progress, that “ad hoc” testing worked well for finding bugs and highly scripted testing did not. (Note: We don’t mean to make this discovery sound easy. It wasn’t. We do mean to say that the non-obvious truths about testing are in evidence all around us, when we put aside folklore and look carefully at how people work each day.) He began writing and speaking about his experiences. A few years into his work as a test manager, mostly while testing compilers and other developer tools, he discovered that Cem Kaner had coined a term—”exploratory testing”—to represent the opposite of scripted testing. In that original passage, just a few pages long, Cem didn’t define the term and barely described it, but he was the first to talk directly about designing tests while performing them.

Thus emerged what we, here, call ET 1.0.

(See The History of Definitions of ET for a chronological guide to our terminology.)

ET 1.0: Rebellion

Testing with and without a script are different experiences. At first, we were mostly drawn to the quality of ideas that emerged from unscripted testing. When we did ET, we found more bugs and better bugs. It just felt like better testing. We hadn’t yet discovered why this was so. Thus, the first iteration of exploratory testing (ET) as rhetoric and theory focused on escaping the straitjacket of the script and making space for that “better testing”. We were facing the attitude that “Ad hoc testing is uncontrolled and unmanageable; something you shouldn’t do.” We were pushing against that idea, and in that context ET was a special activity. So, the crusaders for ET treated it as a technique and advocated using that technique. “Put aside your scripts and look at the product! Interact with it! Find bugs!”

Most of the world still thinks of ET in this way: as a technique and a distinct activity. But we were wrong about characterizing it that way. Doing so, we now realize, marginalizes and misrepresents it. It was okay as a start, but thinking that way leads to a dead end. Many people today, even people who have written books about ET, seem to be happy with that view.

This era of ET 1.0 began to fade in 1995. At that time, there were just a handful of people in the industry actively trying to develop exploratory testing into a discipline, despite the fact that all testers unconsciously or informally pursued it, and always have. For these few people, it was not enough to leave ET in the darkness.

ET 1.5: Explication

Through the late ‘90s, a small community of testers beginning in North America (who eventually grew into the worldwide Context-Driven community, with some jumping over into the Agile testing community) was also struggling with understanding the skills and thought processes that constitute testing work in general. To do that, they pursued two major threads of investigation. One was Jerry Weinberg’s humanist approach to software engineering, combining systems thinking with family psychology. The other was Cem Kaner’s advocacy of cognitive science and Popperian critical rationalism. This work would soon cause us to refactor our notions of scripted and exploratory testing. Why? Because our understanding of the deep structures of testing itself was evolving fast.

When James joined ST Labs in 1995, he was for the first time fully engaged in developing a vision and methodology for software testing. This was when he and Cem began their fifteen-year collaboration. This was when Rapid Software Testing methodology first formed. One of the first big innovations on that path was the introduction of guideword heuristics as one practical way of joining real-time tester thinking with a comprehensive underlying model of the testing process. Lists of test techniques or documentation templates had been around for a long time, but as we developed vocabulary and cognitive models for skilled software testing in general, we started to see exploratory testing in a new light. We began to compare and contrast the important structures of scripted and exploratory testing and the relationships between them, instead of seeing them as activities that merely felt different.

In 1996, James created the first testing class called “Exploratory Testing.”  He had been exposed to design patterns thinking and had tried to incorporate that into the class. He identified testing competencies.

Note: During this period, James distinguished between exploratory and ad hoc testing—a distinction we no longer make. ET is an ad hoc process, in the dictionary sense: ad hoc means “to this; to the purpose”. He was really trying to distinguish between skilled and unskilled testing, and today we know better ways to do that. We now recognize unskilled ad hoc testing as ET, just as unskilled cooking is cooking, and unskilled dancing is dancing. The value of the label “exploratory testing” is simply that it is more descriptive of an activity that is, among other things, ad hoc.

In 1999, James was commissioned to define a formalized process of ET for Microsoft. The idea of a “formal ad hoc process” seemed paradoxical, however, and this set up a conflict which would be resolved via a series of constructive debates between James and Cem. Those debates would lead to we here will call ET 2.0.

There was also progress on making ET more friendly to project management. In 2000, inspired by the work for Microsoft, James and Jon Bach developed “Session-Based Test Management” for a group at Hewlett-Packard. In a sense this was a generalized form of the Microsoft process, with the goal of creating a higher level of accountability around informal exploratory work. SBTM was intended to help defend exploratory work from compulsive formalizers who were used to modeling testing in terms of test cases. In one sense, SBTM was quite successful in helping people to recognize that exploratory work was entirely manageable. SBTM helped to transform attitudes from “don’t do that” to “okay, blocks of ET time are things just like test cases are things.”

By 2000, most of the testing world seemed to have heard something about exploratory testing. We were beginning to make the world safe for better testing.

ET 2.0: Integration

The era of ET 2.0 has been a long one, based on a key insight: the exploratory-scripted continuum. This is a sliding bar on which testing ranges from completely exploratory to completely scripted. All testing work falls somewhere on this scale. Having recognized this, we stopped speaking of exploratory testing as a technique, but rather as an approach that applies to techniques (or as Cem likes to say, a “style” of testing).

We could think of testing that way because, unlike ten years earlier, we now had a rich idea of the skills and elements of testing. It was no longer some “creative and mystical” act that some people are born knowing how to do “intuitively”. We saw testing as involving specific structures, models, and cognitive processes other than exploring, so we felt we could separate exploring from testing in a useful way. Much of what we had called exploratory testing in the early 90’s we now began to call “freestyle exploratory testing.”

By 2006, we settled into a simple definition of ET, simultaneous learning, test design, and test execution. To help push the field forward, James and Cem convened a meeting called the Exploratory Testing Research Summit in January 2006. (The participants were James Bach, Jonathan Bach, Scott Barber, Michael Bolton, Elisabeth Hendrickson, Cem Kaner, Mike Kelly, Jonathan Kohl, James Lyndsay, and Rob Sabourin.) As we prepared for that, we made a disturbing discovery: every single participant in the summit agreed with the definition of ET, but few of us agreed on what the definition actually meant. This is a phenomenon we had no name for at the time, but is now called shallow agreement in the CDT community. To combat shallow agreement and promote better understanding of ET, some of us decided to adopt a more evocative and descriptive definition of it, proposed originally by Cem and later edited by several others: “a style of testing that emphasizes the freedom and responsibility of the individual tester to continually optimize the quality of his work by treating test design, test execution, test result interpretation, and learning as mutually supporting activities that continue in parallel throughout the course of the project.” Independently of each other, Jon Bach and Michael had suggested the “freedom and responsibility” part to that definition.

And so we had come to a specific and nuanced idea of exploration and its role in testing. Exploration can mean many things: searching a space, being creative, working without a map, doing things no one has done before, confronting complexity, acting spontaneously, etc. With the advent of the continuum concept (which James’ brother Jon actually called the “tester freedom scale”) and the discussions at the ExTRS peer conference, we realized most of those different notions of exploration are already central to testing, in general. What the adjective “exploratory” added, and how it contrasted with “scripted,” was the dimension of agency. In other words: self-directedness.

The full implications of the new definition became clear in the years that followed, and James and Michael taught and consulted in Rapid Software Testing methodology. We now recognize that by “exploratory testing”, we had been trying to refer to rich, competent testing that is self-directed. In other words, in all respects other than agency, skilled exploratory testing is not distinguishable from skilled scripted testing. Only agency matters, not documentation, nor deliberation, nor elapsed time, nor tools, nor conscious intent. You can be doing scripted testing without any scrap of paper nearby (scripted testing does not require that you follow a literal script). You can be doing scripted testing that has not been in any way pre-planned (someone else may be telling you what to do in real-time as they think of ideas). You can be doing scripted testing at a moment’s notice (someone might have just handed you a script, or you might have just developed one yourself). You can be doing scripted testing with or without tools (tools make testing different, but not necessarily more scripted). You can be doing scripted testing even unconsciously (perhaps you feel you are making free choices, but your models and habits have made an invisible prison for you). The essence of scripted testing is that the tester is not in control, but rather is being controlled by some other agent or process. This one simple, vital idea took us years to apprehend!

In those years we worked further on our notions of the special skills of exploratory testing. James and Jon Bach created the Exploratory Skills and Tactics reference sheet to bring specificity and detail to answer the question “what specifically is exploratory about exploratory testing?”

In 2007, another big slow leap was about to happen. It started small: inspired in part by a book called The Shape of Actions, James began distinguishing between processes that required human judgment and wisdom and those which did not. He called them “sapient” vs. “non-sapient.” This represented a new frontier for us: systematic study and development of tacit knowledge.

In 2009, Michael followed that up by distinguishing between testing and checking. Testing cannot be automated, but checking can be completely automated. Checking is embedded within testing. At first, James objected that, since there was already a concept of sapient testing, the distinction was unnecessary. To him, checking was simply non-sapient testing. But after a few years of applying these ideas in our consulting and training, we came to realize (as neither of us did at first) that checking and testing was a better way to think and speak than sapience and non-sapience. This is because “non-sapience” sounds like “stupid” and therefore it sounded like we were condemning checking by calling it non-sapient.

Do you notice how fine distinctions of language and thought can take years to work out? These ideas are the tools we need to sort out our practical decisions. Yet much like new drugs on the market, it can sometimes take a lot of experience to understand not only benefits, but also potentially harmful side effects of our ideas and terms. That may explain why those of us who’ve been working in the craft a long time are not always patient with colleagues or clients who shrug and tell us that “it’s just semantics.” It is our experience that semantics like these mean the difference between clear communication that motivates action and discipline, and fragile folklore that gets displaced by the next swarm of buzzwords to capture the fancy of management.

ET 3.0: Normalization

In 2011, sociologist Harry Collins began to change everything for us. It started when Michael read Tacit and Explicit Knowledge. We were quickly hooked on Harry’s clear writing and brilliant insight. He had spent many years studying scientists in action, and his ideas about the way science works fit perfectly with what we see in the testing field.

By studying the work of Harry and his colleagues, we learned how to talk about the difference between tacit and explicit knowledge, which allows us to recognize what can and cannot be encoded in a script or other artifacts. He distinguished between behaviour (the observable, describable aspects of an activity) and actions (behaviours with intention) (which had inspired James’ distinction between sapient and non-sapient testing). He untangled the differences between mimeomorphic actions (actions that we want to copy and to perform in the same way every time) and polimorphic actions (actions that we must vary in order to deal with social conditions); in doing that, he helped to identify the extents and limits of automation’s power. He wrote a book (with Trevor Pinch) about how scientific knowledge is constructed; another (with Rob Evans) about expertise; yet another about how scientists decide to evaluate a specific experimental result.

Harry’s work helped lend structure to other ideas that we had gathered along the way.

  • McLuhan’s ideas about media and tools
  • Karl Weick’s work on sensemaking
  • Venkatesh Rao’s notions of tempo which in turn pointed us towards James C. Scott’s notion of legibility
  • The realization (brought to our attention by an innocent question from a tester at Barclays Bank) that the “exploratory-scripted continuum” is actually the “formality continuum.” In other words, to formalize an activity means to make it more scripted.
  • The realization of the important difference between spontaneous and deliberative testing, which is the degree of reflection that the tester is exercising. (This is not the same as exploratory vs. scripted, which is about the degree of agency.)
  • The concept of “responsible tester” (defined as a tester who takes full, personal, responsibility for the quality of his work).
  • The advent of the vital distinction between checking and testing, which replaced need to talk about “sapience” in our rhetoric of testing.
  • The subsequent redefinition of the term “testing” within the Rapid Software Testing namespace to make these things more explicit (see below).

About That Last Bullet Point

ET 3.0 as a term is a bit paradoxical because what we are working toward, within the Rapid Software Testing methodology, is nothing less than the deprecation of the term “exploratory testing.”

Yes, we are retiring that term, after 22 years. Why?

Because we now define all testing as exploratory.  Our definition of testing is now this:

“Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”

Where does scripted testing fit, then?  By “script” we are speaking of any control system or factor that influences your testing and lies outside of your realm of choice (even temporarily). This does not refer only to specific instructions you are given and that you must follow. Your biases script you. Your ignorance scripts you. Your organization’s culture scripts you. The choices you make and never revisit script you.

By defining testing to be exploratory, scripting becomes a guest in the house of our craft; a potentially useful but foreign element to testing, one that is interesting to talk about and apply as a tactic in specific situations. An excellent tester should not be complacent or dismissive about scripting, any more than a lumberjack can be complacent or dismissive about heavy equipment. This stuff can help you or ruin you, but no serious professional can ignore it.

Are you doing testing? Then you are already doing exploratory testing. Are you doing scripted testing? If you’re doing it responsibly, you are doing exploratory testing with scripting (and perhaps with checking).  If you’re only doing “scripted testing,” then you are just doing unmotivated checking, and we would say that you are not really testing. You are trying to behave like a machine, not a responsible tester.

ET 3.0, in a sentence, is the demotion of scripting to a technique, and the promotion of exploratory testing to, simply, testing.

Let’s test at Let’s Test

I’ve been telling people that the best conference I know for thinking testers is Let’s Test (followed closely by CAST, which I will also be at, this year, in New York). Let’s Test was created by people who experienced CAST and wanted to be even more dedicated to Context-Driven testing principles.

Now, I’m here in Stockholm once again to be with the most interesting testers in Europe. I’m not done with my presentations, yet. But I still have a couple of days.

(I will presenting a new model of what it means to be an excellent observer, together with one or two observation challenges for participants. And Pradeep Soundararajan and I will be presenting a tutorial on reviewing a specification by testing it.)

Let’s Test is not for the faint of heart. Events go on day and night. I suffer from terrible jet lag, so I probably won’t be seen after dinner. But for you crazy kids, it’s a great place to try a testing exercise, or present one.

(Note: I’m being paid to teach at Let’s Test. I don’t get a percentage of the gate, though– I get paid the same whether anyone shows up or not.)

Australia Let’s Test

I will also be in Australia for the first Let’s Test happening down there, in September. There are some interesting testers in Oz. I’m sure they will all be there. It will be the first great party of ambitious intellectual testers that I know of in the history of Australian testing.

Anne-Marie Charrett and I will be doing our Coaching Testers tutorial, which is the only time this year we will teach it together.

“Intellectual” testers?

Why do I keep saying that? Because the state of the practice in testing is for testers NOT to read about their craft, NOT to study social science or know anything about the proper use of statistics or the meaning of the word “heuristic”, and NOT to challenge the now 40 year stale ideas about making testing into factory work that lead directly to mass outsourcing of testing to lowest bidder instead of the most able tester.

Intellectual testers are not the most common type of tester.

The ISTQB and similar programs require your stupidity and your fear in order to survive. And their business model is working. They don’t debate us for the same reason that HP made billions of dollars selling bad test tools by pitching them to non-testers who had more money than wisdom. Debating us would spoil their racket.

So, don’t be like that. Be smart.

I’ll see you at Let’s Test.

Test Jumpers: One Vision of Agile Testing

Many software companies, these days, are organized around a number of small Agile teams. These teams may be working on different projects or parts of the same project. I have often toured such companies with their large open plan offices; their big tables and whiteboards festooned with colorful Post-Its occasionally fluttering to the floor like leaves in a perpetual autumn display; their too many earbuds and not nearly enough conference rooms. Sound familiar, Spotify? Skype?

(This is a picture of a smoke jumper. I wish test jumpers looked this cool.)

I have a proposal for skilled Agile testing in such places: a role called a “test jumper.” The name comes from the elite “smoke jumper” type of firefighter. A test jumper is a trained and enthusiastic test lead (see my Responsible Tester post for a description of a test lead) who “jumps” into projects and from project to project: evaluating the testing, doing testing or organizing people in other roles to do testing. A test jumper can function as test team of one (what I call an omega tester ) or join a team of other testers.

The value of a role like this arises because in a typical dedicated Agile situation, everyone is expected to help with testing, and yet having staff dedicated solely to testing may be unwarranted. In practice, that means everyone remains chronically an amateur tester, untrained and unmotivated. The test jumper role could be a role held by one person, dedicated to the mastery of testing skills and tools, who is shared among many projects. This is a role that I feel close to, because it’s sort of what I already do. I am a consulting software tester who likes to get his hands dirty doing testing and running in-house testing events. I love short-term assignments and helping other testers come up to speed.

 

 

What Does a Test Jumper Do?

A test jumper basically asks, How are my projects handling the testing? How can I contribute to a project? How can I help someone test today?

Specifically a test jumper:

  • may spend weeks on one project, acting as an ordinary responsible tester.
  • may spend a few days on one project, organizing and leading testing events, coaching people, and helping to evaluate the results.
  • may spend as little as 90 minutes on one project, reviewing a test strategy and giving suggestions to a local tester or developer.
  • may attend a sprint planning meeting to assure that testing issues are discussed.
  • may design, write, or configure a tool to help perform a certain special kind of testing.
  • may coach another tester about how to create a test strategy, use a tool, or otherwise learn to be a better tester.
  • may make sense of test coverage.
  • may work with designers to foster better testability in the product.
  • may help improve relations between testers and developers, or if there are no other testers help the developers think productively about testing.

Test jumping is a time-critical role. You must learn to triage and split your time across many task threads. You must reassess project and product risk pretty much every day. I can see calling someone a test jumper who never “jumps” out of the project, but nevertheless embodies the skills and temperament needs to work in a very flexible, agile, self-managed fashion, on an intense project.

Addendum #1: Commenter Augusto Evangelisti suggests that I emphasize the point about coaching. It is already in my list, above, but I agree it deserves more prominence. In order to safely “jump” away from a project, the test jumper must constantly lean toward nudging, coaching, or even training local helpers (who are often the developers themselves, and who are not testing specialists, even though they are super-smart and experienced in other technical realms) and local responsible testers (if there are any on that project). The ideal goal is for each team to be reasonably self-sufficient, or at least for the periodic visits of the test jumper to be enough to keep them on a good track.

What Does a Test Jumper Need?

  • The ability and the enthusiasm for plunging in and doing testing right now when necessary.
  • The ability to pull himself out of a specific test task and see the big picture.
  • The ability to recruit helpers.
  • The ability to coach and train testers, and people who can help testing.
  • A wide knowledge of tools and ability to write tools as needed.
  • A good respectful relationship with developers.
  • The ability to speak up in sprint planning meetings about testing-related issues such as testability.
  • A keen understanding of testability.
  • The ability to lead ad hoc groups of people with challenging personalities during occasional test events.
  • An ability to speak in front of people and produce useful and concise documentation as necessary.
  • The ability to manage many threads of work at once.
  • The ability to evaluate and explain testing in general, as well as with respect to particular forms of testing.

A good test jumper will listen to advice from anyone, but no one needs to tell a test jumper what to do next. Test jumpers manage their own testing missions, in consultation with such clients as arise. A test jumper must be able to discover and analyze the testing context, then adapt to it or shape it as necessary. It is a role made for the Context-Driven school of testing.

Does a Test Jumper Need to be a Programmer?

Coding skills help tremendously in this role, but being a good programmer is not absolutely required. What is required is that you learn technical things very quickly and have excellent problem-solving and social skills. Oh, and you ought to live and breathe testing, of course.

How Does a Test Jumper Come to Be?

A test jumper is mostly self-created, much as good developers are. A test jumper can start as a programmer, as I did, and then fall in love with the excitement of testing (I love the hunt for bugs). A test jumper may start as a tester, learn consulting and leadership skills, but not want to be a full-time manager. Management has its consolations and triumphs, of course, but some of us like to do technical things. Test jumping may be part of extending the career path for an experienced and valuable tester.

Finding Your Own Integrity

I have a belief that I’m not going to justify– I’m simply going to say it and challenge you to look into your own experience and your own heart and see the truth of it for yourself: Your sense of identity, as a human among humans, is the most powerful force that animates and directs your choices. It is more important than sex or food or religion. It lurks behind every neurosis (including those involving sex or food or religion). As I read history and experience life, answers to the questions “Who am I? Am I a good example of what I should be?” are the prime movers of human choice throughout all of history, and the proximal cause of every war.

There are certainly exceptions to this rule: drug addiction, mental illness, or panic over a sudden, surprising, physical threat. Maybe those things have little to do with identity. Granted. I’m talking about normal daily life (and every Shakespeare play).

“I am an American. I am a human. I am a father. I am a husband. I am lovable. I am helpful. I am a tester. I am a skeptic. I am an outsider. I am dangerous. I am safe. I am honorable. I am fallible. I am truthful. I am intellectual…”  Each of these statements, for me, are reflective shards that tumble in a kaleidoscope of my identity. The models of personhood they represent comprise my moral compass. Although the pattern formed in that kaleidoscope may seem to shift with the situation, the underlying logic of my adult identity changes little with time.

That is the context for integrity.

Integrity means wholeness; the harmony and completeness of one’s identity. Practically speaking, a person with integrity is a person to lives consistently according to their avowed moral code, as opposed to someone who has no moral code, or who changes it as a matter of convenience. A person of integrity therefore creates continuity across the events of his life, and other people feel they know who they are dealing with.

The Challenge of Finding Your Integrity

Recently, in a discussion about what is reasonable for an employer to ask of a tester, a colleague felt I was trying to impose my own values onto potential employers of my students and wrote that as teachers of new testers “employment [for the testers] should be our first priority.” I disagreed sharply, writing that “our first priority is integrity.” My correspondent seemed to take offense to that.

Now, the employment-first position might be construed to imply that we should advocate robbing banks, because it is the quickest way to get money, or perhaps we should train prostitutes, because prostitution is an old and reliable industry with lots of job security for the right people. That would be absurd, but it’s also a straw man argument. I am certain no one intends to argue that any job is better than no job. Safety, legality and morality do enter into the equation.

Conversely, the integrity-first position might be cast as requiring a tester to immediately protest or resign in the face of any ethical dilemma or systemic ethical lapse, no matter how seemingly minor. This would turn most testers into insufferable, dour lawyers on their projects. We would get very little done. Who would hire such people?

These extreme positions are not very interesting, except as tools for meditating on what might be reasonable behavior. Therefore, I’d like to describe a less extreme position that I think is more defensible and workable. It goes like this:

1. Integrity is a vital and important matter. We suffer as people and society suffers when we treat it too lightly.

2. As testers and technical people, our integrity is routinely threatened by well-meaning clients and colleagues who want us to portray ourselves and the world to be a certain way, even if that isn’t strictly the truth.

3. If we never think directly about integrity, and simply trust in the innate goodness of ourselves and others, we are definitely taking this matter too lightly.

4. Integrity is not like a vase that shatters easily, and that once shattered is irretrievable. Integrity is more like an ongoing public artwork, exposed to and buffeted by the elements, sometimes damaged but always ultimately repairable (although our reputation may be another matter). Integrity is a work in progress for all of us.

5. Integrity, like education, is both personal and social. Your society judges you. It is reasonable that it does. But it is also reasonable to negotiate limits on that judgment. We spend our lives negotiating those lines, one way or another.

6. Forgiveness, although perhaps difficult and expensive to obtain, should always be available to us. (I test this by occasionally imagining my most “depraved” enemies in testing, and then imagining what they could do that would allow me to forgive them and even collaborate with them.)

7. Although integrity is our highest priority, in general, it is not the only thing that matters. We must apply wisdom and judgment so that the maintenance of integrity does not unreasonably affect our ability to survive. There is no set formula for how to do that.

8. Therefore, our practical priority must be: to learn how to think through and solve problems of survival while maintaining reasonable integrity. This itself is an ongoing project, requiring temperance and self-forgiveness.

9. New testers need to realize that they are not necessarily responsible for the quality of their work. Sometimes you will be asked to do things you don’t understand the value of, even though there may be value. In those situations, it’s okay to be compliant, as long as you are under supervision and someone competent is taking responsibility for what you do. It’s okay to watch and learn and not necessarily to make trouble. (Although, I usually did, even as a newbie.)

10. Experienced testers? Well, much is expected of you. Your clients (your non-tester colleagues and bosses) don’t know how to test, but you are supposed to. You can’t just do what you are told. That would be like letting a drunk friend drive home. Remember, someday your clients may sober up and wonder why you agreed to their stupid plan when you were supposed to be the expert.

Having laid this hopefully reasonable and workable strategy before you… I actually think the dispute between me and my correspondent, above, was not about the importance of integrity or employment at all, but rather about the specifics of the case we were debating. I should tell you what that was: whether it is reasonable for an employer to expect an entry-level tester to “write test cases.”

From a context-driven testing perspective, no practice can be declared unreasonable outside all contexts. But I do know a lot about the typical contexts of testing. I have seen profound waste, all around the industry, due to reckless and thoughtless documenting and counting of things called “test cases.” So, I don’t think that it is reasonable, generally speaking, to require young testers to write test cases. First, because “writing test cases” is what people who don’t know how to test think testers do– so, it’s usually an indicator of incompetent management. Second, because entry-level testers do not have the skills to write test cases in such a way that won’t result in a near complete waste of their client’s time and money. And third, because there are such obviously better things to do, in most cases, including learning about the product and actually testing the product.

Many people disagree with me. But I believe their attitude on this is the single most direct and vital cause of the perpetual infancy and impotency that we see in the testing industry. In other words, it’s not just a disagreement about style, it’s something that I think threatens our integrity as sincere and thoughtful testers. Casual shrugging about test case writing must be stamped out the way transfats are being outlawed in fast food. Yes, that also took years to accomplish.

Speaking of fast food…

Here’s a metaphor that might help: eating at McDonalds.

Eating at McDonalds will not kill you (well, not outright). But what if you were forced to eat at McDonalds for your work? Every day, breakfast, lunch and dinner. Nothing but McDonalds. What if it were obvious to you that eating at McDonalds was not helping you actually succeed in your work? What if instead it was clear to you that such a diet was harming your ability to get your work done? For instance, perhaps you are a restaurant reviewer, except you are almost always full of McDonalds food so you can’t ever enjoy a proper meal at a restaurant you are supposed to review? And yet your manager, who knows nothing about restaurant reviewing, insists that you maintain a McDonalds-dominated dietary regimen.

Couldn’t someone say, hey, it’s a job and you should do what you are told? Yes, they could say that. And it might be true enough at first. But over time, that diet would hurt you; over time, you would have to cope with how poorly you were doing what you believed to be your real job. You might even be criticized for missing bugs– I mean– failing to review restaurants fully, even though it’s largely due to your employer’s own unreasonable process prescriptions.

At some point you might say “enough!!” You might refuse to eat another Big Mac. From the point of view of your management and colleagues, it might look like you were risking your job just because you didn’t want to eat a hamburger. It might look crazy to them. But from your point of view, the issue isn’t the one burger, but rather the unhealthy system, slowly killing you. This breakdown comes more quickly if you happen to have a gluten allergy.

Ethics and integrity in testing is not just about following prissy little rules that many other people flout– it’s about not making yourself sick even if other people are willing to live a sickly life. This requires that you be able to judge what health and sickness means to you. Integrity is about identity health.

A Story of Quitting Even Though I Needed the Work

In 1998, I was hired by a consulting company outside of Washington D.C. I negotiated for a $30,000 sign-on bonus, and bought a house in Virginia. I was the sole breadwinner in my family, with a wife and son to support. I bought a new car, too. In short, I was “all in.”

Six months later, I quit. I had no other job to go to. I had bills due. It took me seven years to pay back my sign-on bonus, with interest (I forfeited it because I did not stay for two years). But with the help of colleagues and family over the following weeks, I made the transition to running my own business. I am most thankful for my wife’s response when I came home that night and told her I walked out on our only source of income. She shrugged and said it was surely for the best, and that something good would come of it. (I can only recommend, gentlemen, that you marry an optimist if you can.) I am also thankful to Cem Kaner, who bought me a laptop (my only computer was then owned by my employer) and said “times like these are when you discover who your true friends are.” This was fitting because it was partly because of Cem that I had previously decided never to sacrifice my professional integrity.

This illustrates one lesson about ethics: community support helps us find and maintain our integrity.

I quit because my company was insisting that I bill hours on a project that, in my opinion, was absolutely certain not to benefit from my work. The client wanted me to create fake test cases. They didn’t call them fake test cases, of course. They claimed to want real test cases; and good ones. But no product had been designed at that time! All I had access to was a general description of requirements, which in this case were literally statements of problems the product was intended to solve, with no information on how they would be solved. It was a safety-critical factory process control system, and no one could show me what it would look like or provide any examples of what specifically it might do. The only test cases I could possibly design would therefore be vague and imaginary, based on layers of soft, fluffy assumptions. The customer told me they would be happy if I delivered a document that consisted of the text of each requirement preceded by the phrase “verify that…” I told them they didn’t need a tester for that. They needed a macro.

The integrity picture was clouded, in that case, because the client believed they had to follow the “V-Model” process, which they had interpreted as demanding that I submit a test case specification document. It was a clash between the integrity of a heuristic (the V-Model) vs. the integrity of solving the problem for which the heuristic was designed. My client might have said that I was the one violating the integrity of the process. Whereas I would have said that my client was not competent to make that judgment.

I’m not saying I won’t do bad work… I’m just saying I won’t do bad work for money. If I do bad work, I want it to be for fun or for learning, but not to anyone’s expense or detriment. Hence a line I use once in a while “I could do that for you, except that you pay me too much.” This is one reason I like being independent. I control what I bill for, and if I think a portion of my work is not acceptable, I don’t charge for it– like a chef who refuses to serve an overcooked steak.

It wasn’t as sudden as it looked…

I didn’t just lose my temper at the first sign of trouble. Things had been coming to a boil for a while. On my very first day I reviewed the RFP for that project and concluded it was doomed, but management bid on it anyway, telling me I needed to “be practical” and that surely “we could be helpful to them if they hired us.” I needed the job, so I relented against my better judgment.

During my first staff meeting, my first week on the job, I challenged the consulting staff about what they did to study testing on their own time. My challenge was met with an awkward silence, after which one of the consultants, sounding soul-wounded, told me he was offended that I would suggest that they weren’t already good enough as testers, “These are the best people I’ve ever worked with” said the twenty-something tester with little experience and no public reputation. “But how do you know they are good?” I asked, knowing that our company had just issued a press release about having hired me (a “distinguished industry pioneer” to quote it exactly). There were other murmurs of annoyance around the table, and the manager stepped in to change the subject. I could have pushed the issue, but I didn’t. I needed the job, so I relented against my better judgment.

I was later told that despite my company’s public position, the other consultants felt that I was a mere armchair expert, whereas they were practical men. I don’t know what evidence they had for that. They never showed me what they could do that I supposedly could not. Management tolerated this attitude. That means they were lying directly to their customers about me– claiming I was an expert when clearly they did not believe I was one. I could have insisted they behave in accordance with their public statements about me. But… I needed the job, so I relented against my better judgment.

I knew the day had come when I must quit because I found myself fantasizing about throwing chairs through windows. That triggered a sort of circuit-breaker of judgment: change your life now, now, now.

So what happened after that?

I suffered for this decision. First came the panic attack. I felt dizzy and it was hard to breathe for a few hours. This was followed by a few years of patching together a project here and a project there, never more than 8 weeks from completely running out of money and credit. We were twice within a week of filing for bankruptcy in the early days. During that time I walked away from a few more projects. I resigned from a dysfunctional government project, hopefully saving valuable taxpayer dollars by not writing a completely unnecessary Software Configuration Management plan that no one on the team wanted. I got myself fired from a project at Texas Instruments after about 90 minutes, because I told them a few things they didn’t want to hear (but that I felt were both true and important).

It’s not all suffering, of course. I once was fired from a project (along with the rest of the test team) and then was the only one hired back– partly because the client realized that my high standards meant that I billed far fewer hours than other consultants. In other words, saying no and being a troublemaker earned me another 500 hours of work, while the yes-sayers lost their situations. I also got some great gigs, including my very first one as an independent, specifically because I am a rabble-rousing kind of thinker.

These days, I cultivate as many clients as I can, so that I don’t rely too much on any one of them. And I have established a reputation for being honest and blunt that generally prevents the wrong sort of people from trying to hire me. It’s not easy, but it can be done: I have made integrity my top priority.

What about before I was well known?

Well, I’ve always had this attitude. It’s not some luxury to me. It’s fundamental. That’s why I had to leave high school. I’ve never been able to “play the game” at the expense of feeling like a good honest man. Like I said, I suffered for it. I wanted to go try myself at MIT, where my much more pliable best friend from high school eventually graduated. I am born to be an academic, but since I can’t stand the compliance-to-ceremony culture of the academic world, I must be an independent scholar, without access to expensive journals and fantastic libraries.

Before anybody heard of me, I treated getting work like dating: be a slightly exaggerated version of myself so that I will be rejected quickly if the relationship is not a fit (a stress testing strategy, you might say). My big break came at Apple, where I worked for a man of vision and good humor who seemed to relish being the mentor I needed. The environment was open and supportive. There was an element of luck in that my first ten years in testing I worked for people who didn’t ask me to tell lies or do bad work on purpose.

So I know it’s possible to find such people. They are out there. You don’t have to work for bozos, and if you currently do, there is yet hope.

A person who does not live true to himself feels sick and weak inside. My identity as “excellent software tester” demands that I take my craft seriously. I hope you will take this craft seriously, too.

P.S. What if my sense of identity doesn’t require me to be good at my job?

Then, technically, none of this applies to you. Your ethical code can include doing bad work. But… why are you reading my blog? How did you get in? Guards! Seize him!

 

Justifying Real Acceptance Testing

This post is not about the sort of testing people talk about when nearing a release and deciding whether it’s done. I have another word for that. I call it “testing,” or sometimes final testing or release testing. Many projects perform that testing in such a perfunctory way that it is better described as checking, according to the distinction between testing and checking I have previously written of on this blog. As Michael Bolton points out, that checking may better be described as rejection checking since a “fail” supposedly establishes a basis for saying the product is not done, whereas no amount of “passes” can show that it is done.

Acceptance testing can be defined in various ways. This post is about what I consider real acceptance testing, which I define as testing by a potential acceptor (a customer), performed for the purpose of informing a decision to accept (to purchase or rely upon) a product.

Do we need acceptance testing?

Whenever a business decides to purchase and rely upon a component or service, there is a danger that the product will fail and the business will suffer. One approach to dealing with that problem is to adopt the herd solution: follow the thickest part of the swarm; choose a popular product that is advertised or reputed to do what you want it to do and you will probably be okay. I have done that with smartphones, ruggedized laptops, file-sharing services, etc. with good results, though sometimes I am disappointed.

My business is small. I am nimble compared to almost every other company in the world. My acceptance testing usually takes the form of getting a trial subscription to service, or downloading the “basic” version of a product. Then I do some work with it and see how I feel. In this way I learned to love Dropbox, despite its troubling security situation (I can’t lock up my Dropbox files), or the fact that there is a significant chance it will corrupt very large files. (I no longer trust it with anything over half of a gig).

But what if I were advising a large company about whether to adopt a service or product that it will rely upon across dozens or hundreds or thousands of employees? What if the product has been customized or custom built specifically for them? That’s when acceptance testing becomes important.

Doesn’t the Service Level Agreement guarantee that the product will work?

There are a couple of problems with relying on vendor promises. First, the vendor probably isn’t promising total satisfaction. The service “levels” in the contract are probably narrowly and specifically drawn. That means if you don’t think of everything that matters and put that in the contract, it’s not covered. Testing is a process that helps reveal the dimensions of the service that matter.

Second, there’s an issue with timing. By the time you discover a problem with the vendor’s product, you may be already relying on it. You may already have deployed it widely. It may be too late to back out or switch to a different solution. Perhaps your company negotiated remedies in that case, but there are practical limitations to any remedy. If your vendor is very small, they may not be able to afford to fix their product quickly. If you vendor is very large, they may be able to afford to drag their feet on the fixes.

Acceptance testing protects you and makes the vendor take quality more seriously.

Acceptance testing should never be handled by the vendor. I was once hired by a vendor to do penetration testing on their product in order to appease a customer. But the vendor had no incentive to help me succeed in my assignment, nor to faithfully report the vulnerabilities I discovered. It would have been far better if the customer had hired me.

Only the accepting party has the incentive to test well. Acceptance testing should not be pre-agreed or pre-planned in any detail– otherwise the vendor will be sure that the product passes those specific tests. It should be unpredictable, so that the vendor has an incentive to make the product truly meet its requirements in a general sense. It should be adaptive (exploratory) so that any weakness you find can be examined and exploited.

The vendor wants your money. If your company is large enough, and the vendor is hungry, they will move mountains to make the product work well if they know you are paying attention. Acceptance testing, done creatively by skilled testers on a mission, keeps the vendor on its toes.

By explicitly testing in advance of your decision to accept the product, you have a fighting chance to avoid the disaster of discovering too late that the product is a lemon.

My management doesn’t think acceptance testing matters. What do I do?

1. Make the argument, above.
2. Communicate with management, formally, about this. (In writing, so that there is a record.)
It is up to management to make decisions about business risk. They may feel the risk is not worth worrying about. In that case, you must wait and watch. People are usually more persuaded by vivid experiences, rather than abstract principles, so:
1. Collect specific examples of the problems you are talking about. What bugs have you experienced in vendor products?
2. Collect news reports about bugs in products of your vendors (or other vendors) that have been disruptive.
3. In the event you get to do even a little acceptance testing, make a record of the problems you find and be ready to remind management of that history.

 

To The New Tester

About once a week, I get an email like one of these:

Hi James,

I’m from Hyderabad, India. I’m working as a Testing Engineer and doing Manual Testing from the Last 1 Year. I want to know what are things I need to follow to become a good tester. I didn’t have any programming background. I want to learn automation testing for career growth, I want to improve my test case preparing skills and documentation skills. Please suggest me in this regard.

Hi, James,

I am looking for software testing jobs. Actually I’m fresher, I don’t know much about testing but I interested too. Can you please suggest me which type of testing is good?(like manual or automation or etc..) I have got some ideas about manual testing and DB-PL/SQL.

Respected Sir,

I am a Big Fan Of You.

BTech Graduate In Electrical & Electronics. Done Software testing course in QTP and Selenium. I got job as Mobile Application Tester. I am the only tester in this company. iOS and Android Apps. Now i have 3 months experience. I am mainly doing Manual testing. I dont know , whether i am going in right path.

Can you help me please? (Advice)

This blog post shall be my standard answer to these emails…

Dear New Tester/Testing Hopeful,

Thank you for choosing testing. We need smart people in our craft, and I hope you are one of them. I know you must have something going for you because, unlike most young testers, you reached out to me. That’s not easy.

Here are some general ideas:

  • Read my blog (www.satisfice.com/blog).
  • Read the materials on my website (www.satisfice.com).
  • Consider reading either of of my books: Lessons Learned in Software Testing, or Secrets of a Buccaneer-Scholar.
  • Read the materials and blog at www.developsense.com
  • Get on Twitter and watch the conversations among the Context-Driven community. Participate in those discussions.
  • Join software-testing@yahoogroups.com. It’s a quiet, moderated email forum for Context-Driven testers.
  • I offer free coaching, too, but you have to read this blog post first.
  • Practice testing things that aren’t secret, and then post your test results online so that others can see them.
  • Consider taking the RTI Online class.
  • Consider taking the BBST class.
  • Participate in sessions conducted by www.weekendtesting.com.
  • Consider attending the Let’s Test conference or the CAST conference.
  • If you want to get good at using tools while testing, try learning Python. Personally, I use Perl, but I hear Python is good.

Here are some books to read:

  • Introduction to General Systems Thinking, by Gerald M. Weinberg
  • Quality Software Management, Vol. 1: Systems Thinking, by Gerald M. Weinberg
  • Tacit and Explicit Knowledge, by Harry Collins
  • The Black Swan, by Nassim Taleb
  • Testing Computer Software, by Cem Kaner, remains a good classic testing book.

What about certification?

  • Don’t get certified. There are no respectable commercial testing certifications.
  • If you are forced to get certified for some reason, do not take it seriously. It’s not an achievement, it’s just a conveyor belt that extracts your money and gives you nothing you couldn’t get for the price of a Google search.
  • True certification remains this: the respect of respectable people.

What kinds of testers are there?

  • Automated? Manual? There is no such thing as manual or automated testing. It’s all just testing. Testing is often supported by tools that attempt to simulate user interaction with the system. This is what people call “test automation” even though it is only automating a crude approximation of one aspect of testing. If you have the ambition to be a one-man test team, it is extremely valuable to learn how to make your own tools.
  • Exploratory? Scripted? There is no such thing as an exploratory or scripted tester. All good testing is exploratory to some degree and scripted to some degree.
  • Tester. This is a testing generalist who can contribute to any test team. Sometimes called a QA analyst, QA engineer, or test engineer. I prefer the simplicity of “tester.”
  • Omega Tester. The omega tester (which I sometimes call a test jumper, after the analogy of a paratrooper) is one who can do anything. An omega tester is equipped to be the only tester in a project team, if necessary. Omega testers can lead testing, or work with a team of other testers. I am an omega tester. I aspire to be a good one.
  • Performance Tester. The performance tester understands the mathematics and dynamics of the performance of large-scale systems. They use tools that create high loads and measure the performance envelope of systems as they scale up. Performance testers often don’t think of themselves as testers.
  • Usability Tester. The usability tester is a bit mythical. I have met only two dedicated usability testers in my entire career, but I have seen more of them from a distance. A usability tester specializes in studying how users feel about using and learning a product.
  • Security Tester. Security testers also often don’t think of themselves as testers. Security is an exciting, specialized form of testing that requires the mastery of a great many facts about a great many technologies.
  • Testing Toolsmith. A testing toolsmith is a programmer dedicated to writing and maintaining tools that help testers. This is what a lot of people would call an “automated tester” but you better not use that term around me.
  • “SDET” Software Development Engineer in Test. This means a full on programmer who does testing using his programming skills.

I want to help everyone, but at the same time, I have my own work to do. So, I limit my attention to a special kind of student. I work with people who study and practice and want to test better and understand testing more deeply than others. If that describes you, then instead of writing to me with a question, write to me with examples of your work.

Good luck!

— James

P.S. See also what Allen Johnson has to say about this.

 

Tyranny of the Innocents, Exhibit A: Gary Cohen

Gary Cohen, Deputy Administrator and Director, Center for Consumer Information and Insurance Oversight, Centers for Medicare and Medicaid
Services, is not a computer guy. He’s a lawyer. He knows the insurance industry. And yet he was put in charge of a very large and important project: Healthcare.gov.

Here’s what BusinessWeek said about him on August 22nd:

“Gary Cohen seems awfully calm for a man whose job is to make sure Obamacare doesn’t flop. As head of CCIIO (awkward pronunciation: Suh-SIE-O), he oversees the complex, politically fraught system of state health insurance exchanges that will begin signing up uninsured Americans starting on Oct. 1. It hasn’t exactly been a smooth rollout. Many Americans still have no idea the exchanges exist, and the administration has struggled to explain who’s eligible for coverage under the Affordable Care Act and how they enroll. Cohen is convinced the confusion will clear up once things are up and running. “We’re going to get to the point where the discussions we’re having today will fade into the background,” he says.

He should have known that the system wasn’t going to work, at that point. But he’s not a technology guy, so perhaps he thought some big-brained hacker from the movies was going to pull it together at the last minute?

Here’s what he was asked and what he answered at a House Committee on Oversight hearing on May 21st:

Ms. DUCKWORTH. …Could you speak a little bit on the Administration’s readiness to
reach out to this huge number of people so that they can enroll in
time? Basically, you say that you are going to be ready to go on
October 1st, and you need to be. If not, what do you need in order
to get ready and have a successful rollout of these provisions?

Mr. COHEN. So we have a plan in place that basically is timed
so that people are getting the information close to the time in
which there is something that they can do with it. So right now we
are in what we call the education phase, which began in January
and proceeds through June, where we are just putting out information.
We are in the process of re-purposing the HealthCare.gov site
to be really a consumer information site. Our call center will be
going live in June, where people will be able to call and get information
that way. And then starting in the summer we will begin
what we call the anticipation, or get ready phase. And I am not an
expert in these things, but what I understand is that if you start
too early and then people say, well, what do I do, and then there
is nothing that they can do because it is too soon, then you may
end up having people who get a little bit kind of frustrated or disappointed.
So we really are gearing towards making sure the people get the
information they need in time for October, when they actually can
take action and begin to get enrollment coverage.

Hmm. He was asked directly if he needed anything to make sure he was ready to go on October 1st. His answer was basically: no thank you.

Did he really think everything was on track? Why didn’t his people prepare him to set expectations better?

Mr. GOSAR. Mr. Cohen, how closely is HHS working with IRS on Obamacare
implementation?
Mr. COHEN. We are working closely with IRS on those aspects of
implementation where we have to work together, so, for example,
as you know, in determining whether a person is eligible for Medicaid
or CHIP on the one hand, or tax credits in the marketplaces
on the other, income is a test, and we are working with IRS on
verifying people’s income when they apply.
Mr. GOSAR. So the IRS is going to be gathering and sending this
enormous amount of taxpayer information to all the 50 exchanges.
All 50 exchanges are to be ready by October 1st, right?
Mr. COHEN. Yes.
Mr. GOSAR. So will there be any problems with this massive
amount of data sharing?
Mr. COHEN. No. And data sharing may not be exactly the right
way to look at it. Basically what will happen is people will put information
about their income in an application; that information
will be verified by data that comes from the IRS, but there is no
exchange of information from the IRS to the exchange; the information
goes out, it is verified, and it comes back.
Mr. GOSAR. But it is still from the exchange going to the IRS,
and that is where I am going.
Mr. COHEN. It is going to the data hub. Information is coming
from the IRS to the data hub and from the exchange to the data
hub, and there is a comparison and then there is an answer back.
But the tax information isn’t actually going to the exchange.

What a refreshingly blunt answer to the question of whether there will be any trouble with data exchange: No. Unfortunately, we now know there are massive problems with that. Why didn’t he give a more nuanced answer? Why didn’t he hedge? This is why I think he’s an innocent– a child put in charge of the chocolate factory. He didn’t need to be, but that’s how he played it. I guess he was distracted by other duties and trusted the technologists? Or maybe he dismissed the concerns of the technologists as mere excuses? I wonder.

 

 

 

 

Healthcare.gov and the Tyranny of the Innocents

The failure of Healthcare.gov is probably not because of sinister people. It’s probably because of the innocents who run the project. These well-intentioned people are truly as naive as little children. And they must be stopped.

They are, of course, normal intelligent adults. I’m sure they got good grades in school– if you believe in that sort of thing– and they can feed and clothe themselves. They certainly look normal, even stately and wise. It’s just that they are profoundly ignorant about technology projects while being completely oblivious to and complacent about that ignorance. That is the biggest proximal cause of this debacle. It’s called the Dunning-Kruger syndrome (which you can either look up or confidently assure yourself that you don’t need to know about): incompetence of a kind that makes you unable to assess your own lack of competence.

Who am I talking about? I’m talking to some extent about everyone above the first level of management on that project, but mostly I’m talking about anyone who was in the management chain for that project but who has never coded or tested before in their working lives. The non-technical people who created the conditions that the technical people had to work under.

I also blame the technical people in a different way. I’ll get to that, below.

How do I come to this conclusion? Well, take a look at the major possibilities:

Maybe it didn’t fail. Maybe this is normal for projects to have a few glitches? Oh my, no. Project failures are not often clear cut. But among failures, this one is cut as clearly as the Hope Diamond. This is not a near miss. This is the equivalent of sending Hans away to sell the family cow and he comes back with magic beans. It’s the equivalent of going to buy a car and coming back with a shopping cart that has a cardboard sign on which someone has written “CAR” in magic marker. It’s a swing and a miss when the batter was not even holding a bat. It’s so bad, I hope criminal charges are being considered. Make no mistake, the people who ran this project scammed the US government.

Did it fail because it’s too hard a project to do? It’s a difficult project, for sure. It may have been too hard to do under the circumstances prescribed. If so, then we should have heard that message a year ago. Loudly and publicly. We didn’t hear that? Why? Could it have been that the technical people kept their thoughts and feelings carefully shrouded? That’s not what’s being reported. It’s come out that technical people were complaining to management. Management must have quashed those complaints.

Did politics prevent the project from succeeding? No doubt that created a terrible environment in which to produce the system. So what? If it’s too hard, just laugh and say “hey this is ridiculous, we can’t commit to creating this system” UNLESS, of course, you are hoping to hide the problem forever, like a child who has wet the bed and dumps the sheets out the back window. I suppose it’s possible that Republican operatives secretly conspired to make the project fail. If so, I hope that comes out. Doesn’t matter, though. Management could still have seen it coming, unless the whole development team was in on the fix.

Were the technical people incompetent? Probably. It’s likely that many of the programmers were little better than novices, from what I can tell by looking at the bug reports coming through. It was a Children’s Crusade, I guess. But again, so what? The purpose of management, at each of the contracting agencies and above them, is to assess and assure the general competence and performance of the people working on the job. That comes first. I’m sure there were good people mixed in there, somewhere. I have harsh feelings for them, however. I would say to them: Why didn’t you go public? Why didn’t you resign? You like money that much? Your integrity matters that little to you?

Management created the conditions whereby this project was “delivered” in a non-working state. Not like the Dreamliner. The 787 had some serious glitches, and Boeing needs to shape that up. What I’m talking about is boarding an aircraft for a long trip only to be told by the captain “Well, folks it looks like we will be stuck here at the gate for a little while. Maintenance needs to install our wings and engines. I don’t know much about aircraft building, but I promise we will be flying by November 30th. Have some pretzels while you wait.”

Management must bear the prime responsibility for this. I’m not sure that Obama himself is to blame. Everyone under him though? Absolutely.

What About Testing?

Little testing happened on the site. The testing that happened seems to have confirmed what everyone knew. Now this article has come out, about what’s happening behind the scenes. I sure hope they have excellent Rapid Testers working on that, because there is no time for TDD or much of any unit testing and certainly no time to write bloated nonsensical “test case specs” that usually infect government efforts like so much botfly larvae.

Notice the bit at the end?

“It’s a lot of work but people are committed to it. I haven’t heard anyone say it’s not a doable job,” the source said of the November 30th deadline to fix the online portal to purchase insurance on the federal exchange.

Exactly. That’s exactly the problem, Mr. Source. This is what I mean by the tyranny of the innocents. If no one is telling you that the November 30th deadline is not doable, and you think that’s a good sign, then you are an innocent. If you are managing to that expectation then you are a tyrant. It’s probably not doable. I already know that this can’t possibly leave enough time for reasonable testing of the system. Even if it is doable, only a completely dysfunctional project has no one on it speaking openly about whether it is doable.

What Can Be Done?

Politics will ruin everything. I have no institutional solution for this kind of problem. “Best practices” won’t help. Oversight committees won’t help. I can only say that each of us can and should foster a culture of personal ethical behavior. I was on a government project, briefly, years ago. I concluded it was an outlandish waste of taxpayer money and I resigned. I wanted the money. But I resigned anyway. It wasn’t easy. I had car payments and house payments to make. Integrity can be hard. Integrity can be lonely. I don’t always live up to my highest ideals for my own behavior, and when that happens I feel shame. The shame I feel spurs me to be better. That’s all I’m hoping for, really. I hope the people who knew better on this project feel shame. I hope they listen to that shame and go on to be better people.

I do have advice for the innocents. I’ll speak directly to you, Kathy Sebelius, since you are the most public example of who I am talking about…

Hi Kathy,

You’re not a technology person. You shouldn’t have to be. But you need people working for you who are, because technology is opaque. It may surprise you to know that unlike building bridges and monuments, the status of software can be effectively hidden from anyone more than one level above (or sideways from) the programmer or tester who is actually working on that particular piece of it. It’s like managing a gold mine without being able to go down into the mine yourself.

This means you are in a weak position, as an executive. You can pound the table and threaten to fire people, sure. It won’t help. The way in which an executive can use direct power will only make a late software project even later. Every use of direct power weakens your influence. Use indirect power, instead. Imagine that you are taming wild birds. I used to do that as a kid in Vermont. It requires quietness and patience. The first part is to stand for an hour holding birdseed in your hand. Stand quietly and eventually they are landing in your hand.

To have managed this project well, you needed to have created an environment where people could speak without fear. You needed to work with your direct reports to make sure they weren’t filtering out too much of the bad news. You needed to visit the project on a regular basis, and talk to the lowest level people. Then you needed to forgive their managers for not telling you all the bad news. It’s a maddeningly slow process. If you notice, the Pope is currently doing something very similar. Hey, I’m an atheist and yet I find myself listening to that guy. He’s a master of indirect leadership.

You did have the direct power to set expectations. I’m sure you realize you could have done a much better job of that, but perhaps you felt fear, yourself. As your employer (a taxpaying citizen), I bear a little of that responsibility. The country is getting the Healthcare.gov site that it deserves, in a sense.

If you are going to continue in public service, please do yourself and all of us a favor and take a class on software project management. Attend a few lectures. Get smart about what kinds of dodges and syndromes contractors use.

Don’t be an innocent, marching to the slaughter, while millions of dollars line the pockets of the people who run CGI and all those other parasite companies.

— Sincerely, James

My Political Agenda

I have $200,000 of unpaid medical bills due to the crazy jacked up prices and terrible insurance situation for individual citizens in the United States. I am definitely a supporter of the concept of health care reform, even the flawed Obamacare system, if that’s the best we can do for now.

I was pleased to see the failure of the Healthcare.gov website, at first. A little failure helps me make my arguments about how hard it is to do technology well; how getting it right means striving to better ourselves, and no formula or textual incantation will do that for us.

This is too much failure! I want it to stop now. Still, I’m an adult, a software project expert and not in any way an innocent. I know it’s not going to be resolved soon. No Virginia, there won’t be a Healthcare.gov website this Christmas.

Addendum:

From cnn.com:

Summers wrote a memo to the President in 2010 suggesting that HealthCare.gov was not something the government could handle and he needed to bring in experts.

While Summers would not provide details about internal discussions, he said Tuesday, “You need experts. You need to trust but you need to verify. You can’t go rushing the schedule when you get behind or you end up making more errors.”

Damn straight. If this is true then I’m sure glad someone around Obama had basic wisdom. I guess nobody listened to him.

A Public Service Announcement About Exploratory Testing

[Updated: I revamped and added some more examples to the list.]

I got this message from Oliver Vilson, today:

Oliver V.: hi James. Just had a chat with Helena_JM. She reminded me something… don’t know if you’ve written blog about it or pushed it into  RST.. One Test lead from another company mentioned me he has problems with his testers. Some of them are saying that they don’t have to do test plans, since your teaching seems to align that…
James Bach: Any more details?
Oliver V.: rough translation from test team lead : “ET seems to have reputation as “excuse for shitty testing”. People can’t explain what they did and why. If you ask them for test plan or explanation, all you get is “but Bach said…”.

I have, from time to time, heard rumors that some people cite my writings and teachings as an excuse to do bad testing. I think it would help to do a public service announcement…

Attention Testers and Managers of Testers

If a tester claims he is justified in doing bad work because of something I’ve published or said, please email me at james@satisfice.com, or Skype me, and I will help you stop that silliness.

I teach skilled software testing for people who intend to do an excellent job. That process is necessarily exploratory in nature. It also necessarily will have some scripted elements– partly due to the nature of thinking and partly due to the requirements of excellent intellectual work.

I do not teach evasiveness or obscurantism. I do not ever tell a tester that he can get away with refusing to explain his test process. Explaining testing is an important part of being a professional.

Why People Get Confused

I reinvented software testing for myself, from first principles. So, I teach from a very different set of premises. This is necessary, because common ideas about testing are so idiotic. But it does result in confusion when my ideas are taken out of context and “mixed in” to the idiocy. Consider: “I won’t create a detailed test plan document” is a perfectly ordinary and potentially reasonable thing to say in RST. It is a statement about things made explicit in a document, not a statement about lack of planning. Yet Factory School methodology confuses documents for content. If you say that to one of them, it may be mistaken for a refusal to apply appropriate rigor to your work.

Here are some examples of how someone might misapply my teachings:

  1. Rapid Software Testing methodology (RST) is not the same thing as exploratory testing. ET is very simple. Anyone can do ET, just as anyone can look at a painting. But there’s a huge difference between a skilled appraisal of a painting by an expert and a bored glance by a schoolkid. RST is a methodology for doing testing (including scripted and exploratory testing) well. Therefore, anyone doing ET badly is not doing my methodology.
  2. In RST, a plan is not a document, it’s a set of ideas. Therefore, I say you don’t need to have a test plan template, or any sort of written test plan document in order to have a good test plan. I often document my test ideas, though, in different ways, when that helps. Therefore, the lack of a test plan (a guiding set of ideas) probably represents an immature and possibly inadequate test process, but the lack of a test plan document is not necessarily a problem.
  3. In RST, a test is not a document, it’s a performance. Therefore the lack of documented tests is not necessarily a problem, but poor testing (which can be determined by direct observation by a skilled tester or test manager, just as poor carpentry or poor doctoring can be detected) is a problem.
  4. In RST, we have no templates for reporting. But reporting is crucial. Reporting skills are crucial. Accountability is crucial. Credibility is crucial. We teach the art of telling a testing story. Therefore, anyone who declines to explain himself when asked about his testing is not practicing RST. I disavow such testers. (However, just because explaining oneself is an important part of testing doesn’t mean a manager can insist on arbitrarily voluminous documentation or arbitrary metrics. I suspect that, in some cases, managers who complain about testers refusing to document or explain themselves are really just obsessed with a specific method of documentation and refusing to accept other viable solutions to the same problem.)
  5. In RST we say that testing cannot be automated, and that tools can become an obsession. This leads some to think I am against tools. No, I am against bad work. Unfortunately, some tools, such as expensive HP/Mercury tools, are often used to wastefully automate weak fact checking at he expense of good testing. Yes., tools and the technical skills to create and apply them play an important role in great testing. It’s not automating testing when I use tools, because testing is whatever testers do, not what tools do. Therefore a tester who refuses to learn and use tools in general is not practicing RST.
  6. In RST we distinguish between checking and testing. This allows us to distinguish between a test process that is appropriately thoughtful and deep, and one (based solely on checking) that would be reckless and shallow. But when we criticize a checking-only test strategy, some people get confused and think we are criticizing the presence of checking rather than the lack of testing. Therefore, a tester who refuses to design or perform checks that are actually economical and helpful is not doing RST.
  7. In RST, we ban unscientific, abusive attempts at using metrics to control the test process. But when some people hear us attack, say, the counting of test cases, they assume that means we don’t believe in even the concept or principle of measurement. Instead, we support using inquiry-focused metrics (which inspire questions rather than dictating decisions), we promote active skepticism about numbers applied to social systems, and we promote the development of observation, reasoning, and social skills that limit the need for quantification. Therefore any tester who simply refuses to consider using metrics of any kind is not doing RST.
  8. Some people hear about the freedom of exploratory testing, and they confuse that with irresponsibility. But that’s silly. If you drive a car, you are free to run over pedestrians or smash into buildings– except you don’t, because you are responsible! Also, it’s against the law. Freedom is not the same thing as having a right. Therefore, anyone who accepts the freedom of exploratory testing and cannot or will not manage that testing appropriately is an incompetent or irresponsible tester.

A Satisfying News Story

Here is the difference between happiness and satisfaction:

Consider this quote is from How Zynga went from social gaming powerhouse to has-been.

“During my five-month mark, it started turning sour when we were pushing a lot of code that was destroying the ecosystem—they were not fixing bugs,” he said. “At one time, I had 10,000 players trapped inside a quest. 10,000! The attitude was ‘Don’t worry about them.’ [Management] would rather grab new players, keep them for three months or so, get $5 to $10 from them, and those players would quit and leave.”

When I read this, I am not happy. It’s sad to see hundreds of jobs lost because– among other things– a company loses interest in producing a product that works well.

But I am satisfied when yet another example of a high flying wipeout shows that testing (and bug fixing) matters.