Six Things That Go Wrong With Discussions About Testing

Talking about software testing is not easy. It’s not natural! Testing is a “meta” activity. It’s not just a task, but a task that generates new tasks (by finding bugs that should be fixed or finding new risks that must be examined). It’s a task that can never be “completed” yet must get “done.”

Confusion about testing leads to ineffective conversations that focus on unimportant issues while ignoring the things that matter. Here are some specific ways that testing conversations fail:

  1. When people care about how many test cases they have instead of what their testing actually does. The number of test cases (e.g. 500, 257, 39345) tells nothing to anyone about “how much testing” you are doing. The reason that developers don’t brag about how many files they created today while developing their product is that everyone knows that it’s silly to count files, or keystrokes, or anything like that. For the same reasons, it is silly to count test cases. The same test activity can be represented as one test case or one million test cases. What if a tester writes software that automatically creates 100,000 variations of a single test case? Is that really “100,000” test cases, or is it one big test case, or is it no test case at all? The next time someone gives you a test case count, practice saying to yourself “that tells me nothing at all.” Then ask a question about what the tests actually do: What do they cover? What bugs can they detect? What risks are they motivated by?
  2. When people speak of a test as an object rather than an event. A test is not a physical object, although physical things such as documentation, data, and code can be a part of tests. A test is a performance; an activity; it’s something that you do. By speaking of a test as an object rather than a performance, you skip right over the most important part of a test: the attention, motivation, integrity, and skill of the tester. No two different testers ever perform the “same test” in the “same way” in all the ways that matter. Technically, you can’t take a test case and give it to someone else without changing the resulting test in some way (just as no quarterback or baseball player will execute the same play in the same way twice) although the changes don’t necessarily matter.
  3. When people can’t describe their test strategy as it evolves. Test strategy is the set of ideas that guide your choices about what tests to design and what tests to perform in any given situation. Test strategy could also be called the reasoning behind the actions that comprise each test. Test strategy is the answer to questions such as “why are these tests worth doing?” “why not do different tests instead?” “what could we change if we wanted to test more deeply?” “what would we change if we wanted to test more quickly?” “why are we doing testing this way?” These questions arise not just after the testing, but right at the start of the process. The ability to design and discuss test strategy is a hallmark of professional testing. Otherwise, testing would just be a matter of habit and intuition.
  4. When people talk as if automation does testing instead of humans. If developers spoke of development the way that so many people speak of testing, they would say that their compiler created their product, and that all they do is operate the compiler. They would say that the product was created “automatically” rather than by particular people who worked hard and smart to write the code. And management would become obsessed with “automating development” by getting ever better tools instead of hiring and training excellent developers. A better way to speak about testing is the same way we speak about development: it’s something that people do, not tools. Tools help, but tools do not do testing.There is no such thing as an automated test. The most a tool can do is operate a product according to a script and check for specific output according to a script. That would not be a test, but rather a fact check about the product. Tools can do fact checking very well. But testing is more than fact checking because testers must use technical judgment and ingenuity to create the checks and evaluate them and maintain and improve them. The name for that entire human process (supported by tools) is testing. When you focus on “automated tests” you usually defocus from the skills, judgment, problem-solving, and motivation that actually controls the quality of the testing. And then you are not dealing with the important factors that control the quality of testing.
  5. When people talk as if there is only one kind of test coverage. There are many ways you can cover the product when you test it. Each method of assessing coverage is different and has its own dynamics. No one way of talking about it (e.g. code coverage) gives you enough of the story. Just as one example, if you test a page that provides search results for a query, you have covered the functionality represented by the kind of query that you just did (function coverage), and you have covered it with the particular data set of items that existed at that time (data coverage). If you change the query to invoke a different kind of search, you will get new functional coverage. If you change the data set, you will get new data coverage. Either way, you may find a new bug with that new coverage. Functions interact with data; therefore good testing involves covering not just one or the other but also with both together in different combinations.
  6. When people talk as if testing is a static task that is easily formalized. Testing is a learning task; it is fundamentally about learning. If you tell me you are testing, but not learning anything, I say you are not testing at all. And the nature of any true learning is that you can’t know what you will discover next– it is an exploratory enterprise.It’s the same way with many things we do in life, from driving a car to managing a company. There are indeed things that we can predict will happen and patterns we might use to organize our actions, but none of that means you can sleepwalk through it by putting your head down and following a script. To test is to continually question what you are doing and seeing.

    The process of professional testing is not design test cases and then follow the test cases. No responsible tester works this way. Responsible testing is a constant process of investigation and experiment design. This may involve designing procedures and automation that systematically collects data about the product, but all of that must be done with the understanding that we respond to the situation in front of us as it unfolds. We deviate frequently from procedures we establish because software is complicated and surprising; and because the organization has shifting needs; and because we learn of better ways to test as we go.

Through these and other failures in testing conversations, people persist in the belief that good testing is just a matter of writing ever more “test cases” (regardless of what they do); automating them (regardless of what automation can’t do); passing them from one untrained tester to another; all the while fetishizing the files and scripts themselves instead of looking at what the testers are doing with them from day to day.

We. Use. Tools.

Context-Driven testers use tools to help ourselves test better. But, there is no such thing as test automation.

Want details? Here’s the 10,000 word explanation that Michael Bolton and I have been working on for months.

Editor’s Note: I have just posted version 1.03 of this article. This is the third revision we have made due to typos. Isn’t it interesting how hard it is to find typos in your own work before you ship an article? We used automation to help us with spelling, of course, but most of the typos are down to properly spelled words that are in the wrong context. Spelling tools can’t help us with that. Also, Word spell-checker still thinks there are dozens of misspelled words in our article, because of all the proper nouns, terms of art, and neologisms. Of course there are the grammar checking tools, too, right? Yeah… not really. The false positive rate is very high with those tools. I just did a sweep through every grammar problem the tool reported. Out of the five it thinks it found, only one, a missing hyphen, is plausibly a problem. The rest are essentially matters of writing style.

One of the lines it complained about is this: “The more people who use a tool, the more free support will be available…” The grammar checker thinks we should not say “more free” but rather “freer.” This may be correct, in general, but we are using parallelism, a rhetorical style that we feel outweighs the general rule about comparatives. Only humans can make these judgments, because the rules of grammar are sometimes fluid.

TestInsane’s Mindmaps Are Crazy Cool

Most testing companies offer nothing to the community or the field of testing. They all seem to say they hire only the best experts, but only a very few of them are willing to back that up with evidence. Testing companies, by and large, are all the same, and the sameness is one of mediocrity and mendacity.

But there are a few exceptions. One of them is TestInsane, founded by ex-Moolyan co-founder Santosh Tuppad. This is a company to watch.

The wonderful thing about TestInsane is their mindmaps. More than 100 of them. What lovelies! Check them out. They are a fantastic public contribution! Each mindmap tackles some testing-related subject and lists many useful ideas that will help you test in that area.

I am working on a guide to bug reporting, and I found three maps on their site that are helping me cover all the issues that matter. Thank you TestInsane!

I challenge other testing companies to contribute to the craft, as well.

Note: Santosh offered me money to help promote his company. That is a reasonable request, but I don’t do those kinds of deals. If I did that even once I would lose vital credibility. I tell everyone the same thing: I am happy to work for you if you pay me, but I cannot promote you unless I believe in you, and if I believe in you I will promote you for free. As of this writing, I have not done any work for TestInsane, paid or otherwise, but it could happen in the future.

I have done paid work for Moolya, and Per Scholas, both of which I gush about on a regular basis. I believe in those guys. Neither of them pay me to say good things about them, but remember, anyone who works for a company will never say bad things. There are some other testing companies I have worked for that I don’t feel comfortable endorsing, but neither will I complain about them in public (usually… mostly).

A Test is a Performance

Testing is a performance, not an artifact.

Artifacts may be produced before, during, or after the act of testing. Whatever they are, they are not tests. They may be test instructions, test results, or test tools. They cannot be tests.

Note: I am speaking a) authoritatively about how we use terms in Rapid Testing Methodology, b) non-authoritatively of my best knowledge of how testing is thought of more broadly within the Context-Driven school, and c) of my belief about how anyone, anywhere should think of testing if they want a clean and powerful way to talk about it.

I may informally say “I created a test.” What I mean by that is that I designed an experience, or I made a plan for a testing event. That plan itself is not the test, anymore than a picture of a car is a car. Therefore, strictly speaking, the only way to create a test is to perform a test. As Michael Bolton likes to say, there’s a world of difference between sheet music and a musical performance, even though we might commonly refer to either one as “music.” Consider these sentences: “The music at the symphony last night was amazing.” vs. “Oh no, I left the music on my desk at home.”

We don’t always have to speak strictly, but we should know how and know why we might want to.

Why can’t a test be an artifact?

Because artifacts don’t think or learn in the full human sense of that word, that’s why, and thinking is central to the test process. So to claim that an artifact is a test is like wearing a sock puppet on your hand and claiming that it’s a little creature talking to you. That would be no more than you talking to yourself, obviously, and if you removed yourself from that equation the puppet wouldn’t be a little creature, would it? It would be a decorated sock lying on the floor. The testing value of an artifact can be delivered only in concert with an appropriately skilled and motivated tester.

With procedures or code you can create a check. See here for a detailed look at the difference between checking and testing. Checking is part of testing, of course. Anyone who runs checks that fail knows that the next step is figuring out what the failures mean. A tester must also evaluate whether the checks are working properly and whether there are enough of them, or too many, or the wrong kind. All of that is part of the performance of testing.

When a “check engine” light goes on in your car, or any strange alert, you can’t know until you go to a mechanic whether that represents a big problem or a little problem. The check is not testing. The testing is more than the check itself.

But I’ve seen people follow test scripts and only do what the test document tells them to do!

Have you really witnessed that? I think the most you could possibly have witnessed is…

EITHER:

a tester who appeared to do “only” what the test document tells him, while constantly and perhaps unconsciously adjusting and reacting to what’s happening with the system under test. (Such a tester may find bugs, but does so by contributing interpretation, judgment, and analysis; by performing.)

OR:

a tester who necessarily missed a lot of bugs that he could have found, either because the test instructions were far too complex, or far too vague, or there was far too little of it (because that documentation is darn expensive) and the tester failed to perform as a tester to compensate.

In either case, the explicitly written or coded “test” artifact can only be an inanimate sock, or a sock puppet animated by the tester. You can choose to suffer without a tester, or to cover up the presence of the tester. Reality will assert itself either way.

What danger could there be in speaking informally about writing “tests?”

It’s not necessarily dangerous to speak informally. However, a possible danger is that non-testing managers and clients of our work will think of testers as “test case writers” instead of as people who perform the skilled process of testing. This may cause them to treat testers as fungible commodities producing “tests” that are comprised solely of explicit rules. Such a theory of testing– which is what we call the Factory school of testing thought– leads to expensive artifacts that uncover few bugs. Their value is mainly in that they look impressive to ignorant people.

If you are talking to people who fully understand that testing is a performance, it is fine to speak informally. Just be on your guard when you hear people say “Where are your tests?” “Have you written any tests?” or “Should you automate those tests?” (I would rather hear “How do you test this?” “Where are you focusing you testing?” or “Are you using tools to help your testing?”)

Thanks to Michael Bolton and Aleksander Simic for reviewing and improving this post.

 

Finding Your Own Integrity

I have a belief that I’m not going to justify– I’m simply going to say it and challenge you to look into your own experience and your own heart and see the truth of it for yourself: Your sense of identity, as a human among humans, is the most powerful force that animates and directs your choices. It is more important than sex or food or religion. It lurks behind every neurosis (including those involving sex or food or religion). As I read history and experience life, answers to the questions “Who am I? Am I a good example of what I should be?” are the prime movers of human choice throughout all of history, and the proximal cause of every war.

There are certainly exceptions to this rule: drug addiction, mental illness, or panic over a sudden, surprising, physical threat. Maybe those things have little to do with identity. Granted. I’m talking about normal daily life (and every Shakespeare play).

“I am an American. I am a human. I am a father. I am a husband. I am lovable. I am helpful. I am a tester. I am a skeptic. I am an outsider. I am dangerous. I am safe. I am honorable. I am fallible. I am truthful. I am intellectual…”  Each of these statements, for me, are reflective shards that tumble in a kaleidoscope of my identity. The models of personhood they represent comprise my moral compass. Although the pattern formed in that kaleidoscope may seem to shift with the situation, the underlying logic of my adult identity changes little with time.

That is the context for integrity.

Integrity means wholeness; the harmony and completeness of one’s identity. Practically speaking, a person with integrity is a person to lives consistently according to their avowed moral code, as opposed to someone who has no moral code, or who changes it as a matter of convenience. A person of integrity therefore creates continuity across the events of his life, and other people feel they know who they are dealing with.

The Challenge of Finding Your Integrity

Recently, in a discussion about what is reasonable for an employer to ask of a tester, a colleague felt I was trying to impose my own values onto potential employers of my students and wrote that as teachers of new testers “employment [for the testers] should be our first priority.” I disagreed sharply, writing that “our first priority is integrity.” My correspondent seemed to take offense to that.

Now, the employment-first position might be construed to imply that we should advocate robbing banks, because it is the quickest way to get money, or perhaps we should train prostitutes, because prostitution is an old and reliable industry with lots of job security for the right people. That would be absurd, but it’s also a straw man argument. I am certain no one intends to argue that any job is better than no job. Safety, legality and morality do enter into the equation.

Conversely, the integrity-first position might be cast as requiring a tester to immediately protest or resign in the face of any ethical dilemma or systemic ethical lapse, no matter how seemingly minor. This would turn most testers into insufferable, dour lawyers on their projects. We would get very little done. Who would hire such people?

These extreme positions are not very interesting, except as tools for meditating on what might be reasonable behavior. Therefore, I’d like to describe a less extreme position that I think is more defensible and workable. It goes like this:

1. Integrity is a vital and important matter. We suffer as people and society suffers when we treat it too lightly.

2. As testers and technical people, our integrity is routinely threatened by well-meaning clients and colleagues who want us to portray ourselves and the world to be a certain way, even if that isn’t strictly the truth.

3. If we never think directly about integrity, and simply trust in the innate goodness of ourselves and others, we are definitely taking this matter too lightly.

4. Integrity is not like a vase that shatters easily, and that once shattered is irretrievable. Integrity is more like an ongoing public artwork, exposed to and buffeted by the elements, sometimes damaged but always ultimately repairable (although our reputation may be another matter). Integrity is a work in progress for all of us.

5. Integrity, like education, is both personal and social. Your society judges you. It is reasonable that it does. But it is also reasonable to negotiate limits on that judgment. We spend our lives negotiating those lines, one way or another.

6. Forgiveness, although perhaps difficult and expensive to obtain, should always be available to us. (I test this by occasionally imagining my most “depraved” enemies in testing, and then imagining what they could do that would allow me to forgive them and even collaborate with them.)

7. Although integrity is our highest priority, in general, it is not the only thing that matters. We must apply wisdom and judgment so that the maintenance of integrity does not unreasonably affect our ability to survive. There is no set formula for how to do that.

8. Therefore, our practical priority must be: to learn how to think through and solve problems of survival while maintaining reasonable integrity. This itself is an ongoing project, requiring temperance and self-forgiveness.

9. New testers need to realize that they are not necessarily responsible for the quality of their work. Sometimes you will be asked to do things you don’t understand the value of, even though there may be value. In those situations, it’s okay to be compliant, as long as you are under supervision and someone competent is taking responsibility for what you do. It’s okay to watch and learn and not necessarily to make trouble. (Although, I usually did, even as a newbie.)

10. Experienced testers? Well, much is expected of you. Your clients (your non-tester colleagues and bosses) don’t know how to test, but you are supposed to. You can’t just do what you are told. That would be like letting a drunk friend drive home. Remember, someday your clients may sober up and wonder why you agreed to their stupid plan when you were supposed to be the expert.

Having laid this hopefully reasonable and workable strategy before you… I actually think the dispute between me and my correspondent, above, was not about the importance of integrity or employment at all, but rather about the specifics of the case we were debating. I should tell you what that was: whether it is reasonable for an employer to expect an entry-level tester to “write test cases.”

From a context-driven testing perspective, no practice can be declared unreasonable outside all contexts. But I do know a lot about the typical contexts of testing. I have seen profound waste, all around the industry, due to reckless and thoughtless documenting and counting of things called “test cases.” So, I don’t think that it is reasonable, generally speaking, to require young testers to write test cases. First, because “writing test cases” is what people who don’t know how to test think testers do– so, it’s usually an indicator of incompetent management. Second, because entry-level testers do not have the skills to write test cases in such a way that won’t result in a near complete waste of their client’s time and money. And third, because there are such obviously better things to do, in most cases, including learning about the product and actually testing the product.

Many people disagree with me. But I believe their attitude on this is the single most direct and vital cause of the perpetual infancy and impotency that we see in the testing industry. In other words, it’s not just a disagreement about style, it’s something that I think threatens our integrity as sincere and thoughtful testers. Casual shrugging about test case writing must be stamped out the way transfats are being outlawed in fast food. Yes, that also took years to accomplish.

Speaking of fast food…

Here’s a metaphor that might help: eating at McDonalds.

Eating at McDonalds will not kill you (well, not outright). But what if you were forced to eat at McDonalds for your work? Every day, breakfast, lunch and dinner. Nothing but McDonalds. What if it were obvious to you that eating at McDonalds was not helping you actually succeed in your work? What if instead it was clear to you that such a diet was harming your ability to get your work done? For instance, perhaps you are a restaurant reviewer, except you are almost always full of McDonalds food so you can’t ever enjoy a proper meal at a restaurant you are supposed to review? And yet your manager, who knows nothing about restaurant reviewing, insists that you maintain a McDonalds-dominated dietary regimen.

Couldn’t someone say, hey, it’s a job and you should do what you are told? Yes, they could say that. And it might be true enough at first. But over time, that diet would hurt you; over time, you would have to cope with how poorly you were doing what you believed to be your real job. You might even be criticized for missing bugs– I mean– failing to review restaurants fully, even though it’s largely due to your employer’s own unreasonable process prescriptions.

At some point you might say “enough!!” You might refuse to eat another Big Mac. From the point of view of your management and colleagues, it might look like you were risking your job just because you didn’t want to eat a hamburger. It might look crazy to them. But from your point of view, the issue isn’t the one burger, but rather the unhealthy system, slowly killing you. This breakdown comes more quickly if you happen to have a gluten allergy.

Ethics and integrity in testing is not just about following prissy little rules that many other people flout– it’s about not making yourself sick even if other people are willing to live a sickly life. This requires that you be able to judge what health and sickness means to you. Integrity is about identity health.

A Story of Quitting Even Though I Needed the Work

In 1998, I was hired by a consulting company outside of Washington D.C. I negotiated for a $30,000 sign-on bonus, and bought a house in Virginia. I was the sole breadwinner in my family, with a wife and son to support. I bought a new car, too. In short, I was “all in.”

Six months later, I quit. I had no other job to go to. I had bills due. It took me seven years to pay back my sign-on bonus, with interest (I forfeited it because I did not stay for two years). But with the help of colleagues and family over the following weeks, I made the transition to running my own business. I am most thankful for my wife’s response when I came home that night and told her I walked out on our only source of income. She shrugged and said it was surely for the best, and that something good would come of it. (I can only recommend, gentlemen, that you marry an optimist if you can.) I am also thankful to Cem Kaner, who bought me a laptop (my only computer was then owned by my employer) and said “times like these are when you discover who your true friends are.” This was fitting because it was partly because of Cem that I had previously decided never to sacrifice my professional integrity.

This illustrates one lesson about ethics: community support helps us find and maintain our integrity.

I quit because my company was insisting that I bill hours on a project that, in my opinion, was absolutely certain not to benefit from my work. The client wanted me to create fake test cases. They didn’t call them fake test cases, of course. They claimed to want real test cases; and good ones. But no product had been designed at that time! All I had access to was a general description of requirements, which in this case were literally statements of problems the product was intended to solve, with no information on how they would be solved. It was a safety-critical factory process control system, and no one could show me what it would look like or provide any examples of what specifically it might do. The only test cases I could possibly design would therefore be vague and imaginary, based on layers of soft, fluffy assumptions. The customer told me they would be happy if I delivered a document that consisted of the text of each requirement preceded by the phrase “verify that…” I told them they didn’t need a tester for that. They needed a macro.

The integrity picture was clouded, in that case, because the client believed they had to follow the “V-Model” process, which they had interpreted as demanding that I submit a test case specification document. It was a clash between the integrity of a heuristic (the V-Model) vs. the integrity of solving the problem for which the heuristic was designed. My client might have said that I was the one violating the integrity of the process. Whereas I would have said that my client was not competent to make that judgment.

I’m not saying I won’t do bad work… I’m just saying I won’t do bad work for money. If I do bad work, I want it to be for fun or for learning, but not to anyone’s expense or detriment. Hence a line I use once in a while “I could do that for you, except that you pay me too much.” This is one reason I like being independent. I control what I bill for, and if I think a portion of my work is not acceptable, I don’t charge for it– like a chef who refuses to serve an overcooked steak.

It wasn’t as sudden as it looked…

I didn’t just lose my temper at the first sign of trouble. Things had been coming to a boil for a while. On my very first day I reviewed the RFP for that project and concluded it was doomed, but management bid on it anyway, telling me I needed to “be practical” and that surely “we could be helpful to them if they hired us.” I needed the job, so I relented against my better judgment.

During my first staff meeting, my first week on the job, I challenged the consulting staff about what they did to study testing on their own time. My challenge was met with an awkward silence, after which one of the consultants, sounding soul-wounded, told me he was offended that I would suggest that they weren’t already good enough as testers, “These are the best people I’ve ever worked with” said the twenty-something tester with little experience and no public reputation. “But how do you know they are good?” I asked, knowing that our company had just issued a press release about having hired me (a “distinguished industry pioneer” to quote it exactly). There were other murmurs of annoyance around the table, and the manager stepped in to change the subject. I could have pushed the issue, but I didn’t. I needed the job, so I relented against my better judgment.

I was later told that despite my company’s public position, the other consultants felt that I was a mere armchair expert, whereas they were practical men. I don’t know what evidence they had for that. They never showed me what they could do that I supposedly could not. Management tolerated this attitude. That means they were lying directly to their customers about me– claiming I was an expert when clearly they did not believe I was one. I could have insisted they behave in accordance with their public statements about me. But… I needed the job, so I relented against my better judgment.

I knew the day had come when I must quit because I found myself fantasizing about throwing chairs through windows. That triggered a sort of circuit-breaker of judgment: change your life now, now, now.

So what happened after that?

I suffered for this decision. First came the panic attack. I felt dizzy and it was hard to breathe for a few hours. This was followed by a few years of patching together a project here and a project there, never more than 8 weeks from completely running out of money and credit. We were twice within a week of filing for bankruptcy in the early days. During that time I walked away from a few more projects. I resigned from a dysfunctional government project, hopefully saving valuable taxpayer dollars by not writing a completely unnecessary Software Configuration Management plan that no one on the team wanted. I got myself fired from a project at Texas Instruments after about 90 minutes, because I told them a few things they didn’t want to hear (but that I felt were both true and important).

It’s not all suffering, of course. I once was fired from a project (along with the rest of the test team) and then was the only one hired back– partly because the client realized that my high standards meant that I billed far fewer hours than other consultants. In other words, saying no and being a troublemaker earned me another 500 hours of work, while the yes-sayers lost their situations. I also got some great gigs, including my very first one as an independent, specifically because I am a rabble-rousing kind of thinker.

These days, I cultivate as many clients as I can, so that I don’t rely too much on any one of them. And I have established a reputation for being honest and blunt that generally prevents the wrong sort of people from trying to hire me. It’s not easy, but it can be done: I have made integrity my top priority.

What about before I was well known?

Well, I’ve always had this attitude. It’s not some luxury to me. It’s fundamental. That’s why I had to leave high school. I’ve never been able to “play the game” at the expense of feeling like a good honest man. Like I said, I suffered for it. I wanted to go try myself at MIT, where my much more pliable best friend from high school eventually graduated. I am born to be an academic, but since I can’t stand the compliance-to-ceremony culture of the academic world, I must be an independent scholar, without access to expensive journals and fantastic libraries.

Before anybody heard of me, I treated getting work like dating: be a slightly exaggerated version of myself so that I will be rejected quickly if the relationship is not a fit (a stress testing strategy, you might say). My big break came at Apple, where I worked for a man of vision and good humor who seemed to relish being the mentor I needed. The environment was open and supportive. There was an element of luck in that my first ten years in testing I worked for people who didn’t ask me to tell lies or do bad work on purpose.

So I know it’s possible to find such people. They are out there. You don’t have to work for bozos, and if you currently do, there is yet hope.

A person who does not live true to himself feels sick and weak inside. My identity as “excellent software tester” demands that I take my craft seriously. I hope you will take this craft seriously, too.

P.S. What if my sense of identity doesn’t require me to be good at my job?

Then, technically, none of this applies to you. Your ethical code can include doing bad work. But… why are you reading my blog? How did you get in? Guards! Seize him!

 

To The New Tester

About once a week, I get an email like one of these:

Hi James,

I’m from Hyderabad, India. I’m working as a Testing Engineer and doing Manual Testing from the Last 1 Year. I want to know what are things I need to follow to become a good tester. I didn’t have any programming background. I want to learn automation testing for career growth, I want to improve my test case preparing skills and documentation skills. Please suggest me in this regard.

Hi, James,

I am looking for software testing jobs. Actually I’m fresher, I don’t know much about testing but I interested too. Can you please suggest me which type of testing is good?(like manual or automation or etc..) I have got some ideas about manual testing and DB-PL/SQL.

Respected Sir,

I am a Big Fan Of You.

BTech Graduate In Electrical & Electronics. Done Software testing course in QTP and Selenium. I got job as Mobile Application Tester. I am the only tester in this company. iOS and Android Apps. Now i have 3 months experience. I am mainly doing Manual testing. I dont know , whether i am going in right path.

Can you help me please? (Advice)

This blog post shall be my standard answer to these emails…

Dear New Tester/Testing Hopeful,

Thank you for choosing testing. We need smart people in our craft, and I hope you are one of them. I know you must have something going for you because, unlike most young testers, you reached out to me. That’s not easy.

Here are some general ideas:

  • Read my blog (www.satisfice.com/blog).
  • Read the materials on my website (www.satisfice.com).
  • Consider reading either of of my books: Lessons Learned in Software Testing, or Secrets of a Buccaneer-Scholar.
  • Read the materials and blog at www.developsense.com
  • Get on Twitter and watch the conversations among the Context-Driven community. Participate in those discussions.
  • Join software-testing@yahoogroups.com. It’s a quiet, moderated email forum for Context-Driven testers.
  • I offer free coaching, too, but you have to read this blog post first.
  • Practice testing things that aren’t secret, and then post your test results online so that others can see them.
  • Consider taking the RTI Online class.
  • Consider taking the BBST class.
  • Participate in sessions conducted by www.weekendtesting.com.
  • Consider attending the Let’s Test conference or the CAST conference.
  • If you want to get good at using tools while testing, try learning Python. Personally, I use Perl, but I hear Python is good.

Here are some books to read:

  • Introduction to General Systems Thinking, by Gerald M. Weinberg
  • Quality Software Management, Vol. 1: Systems Thinking, by Gerald M. Weinberg
  • Tacit and Explicit Knowledge, by Harry Collins
  • The Black Swan, by Nassim Taleb
  • Testing Computer Software, by Cem Kaner, remains a good classic testing book.

What about certification?

  • Don’t get certified. There are no respectable commercial testing certifications.
  • If you are forced to get certified for some reason, do not take it seriously. It’s not an achievement, it’s just a conveyor belt that extracts your money and gives you nothing you couldn’t get for the price of a Google search.
  • True certification remains this: the respect of respectable people.

What kinds of testers are there?

  • Automated? Manual? There is no such thing as manual or automated testing. It’s all just testing. Testing is often supported by tools that attempt to simulate user interaction with the system. This is what people call “test automation” even though it is only automating a crude approximation of one aspect of testing. If you have the ambition to be a one-man test team, it is extremely valuable to learn how to make your own tools.
  • Exploratory? Scripted? There is no such thing as an exploratory or scripted tester. All good testing is exploratory to some degree and scripted to some degree.
  • Tester. This is a testing generalist who can contribute to any test team. Sometimes called a QA analyst, QA engineer, or test engineer. I prefer the simplicity of “tester.”
  • Omega Tester. The omega tester (which I sometimes call a test jumper, after the analogy of a paratrooper) is one who can do anything. An omega tester is equipped to be the only tester in a project team, if necessary. Omega testers can lead testing, or work with a team of other testers. I am an omega tester. I aspire to be a good one.
  • Performance Tester. The performance tester understands the mathematics and dynamics of the performance of large-scale systems. They use tools that create high loads and measure the performance envelope of systems as they scale up. Performance testers often don’t think of themselves as testers.
  • Usability Tester. The usability tester is a bit mythical. I have met only two dedicated usability testers in my entire career, but I have seen more of them from a distance. A usability tester specializes in studying how users feel about using and learning a product.
  • Security Tester. Security testers also often don’t think of themselves as testers. Security is an exciting, specialized form of testing that requires the mastery of a great many facts about a great many technologies.
  • Testing Toolsmith. A testing toolsmith is a programmer dedicated to writing and maintaining tools that help testers. This is what a lot of people would call an “automated tester” but you better not use that term around me.
  • “SDET” Software Development Engineer in Test. This means a full on programmer who does testing using his programming skills.

I want to help everyone, but at the same time, I have my own work to do. So, I limit my attention to a special kind of student. I work with people who study and practice and want to test better and understand testing more deeply than others. If that describes you, then instead of writing to me with a question, write to me with examples of your work.

Good luck!

— James

P.S. See also what Allen Johnson has to say about this.

 

A Public Service Announcement About Exploratory Testing

[Updated: I revamped and added some more examples to the list.]

I got this message from Oliver Vilson, today:

Oliver V.: hi James. Just had a chat with Helena_JM. She reminded me something… don’t know if you’ve written blog about it or pushed it into  RST.. One Test lead from another company mentioned me he has problems with his testers. Some of them are saying that they don’t have to do test plans, since your teaching seems to align that…
James Bach: Any more details?
Oliver V.: rough translation from test team lead : “ET seems to have reputation as “excuse for shitty testing”. People can’t explain what they did and why. If you ask them for test plan or explanation, all you get is “but Bach said…”.

I have, from time to time, heard rumors that some people cite my writings and teachings as an excuse to do bad testing. I think it would help to do a public service announcement…

Attention Testers and Managers of Testers

If a tester claims he is justified in doing bad work because of something I’ve published or said, please email me at james@satisfice.com, or Skype me, and I will help you stop that silliness.

I teach skilled software testing for people who intend to do an excellent job. That process is necessarily exploratory in nature. It also necessarily will have some scripted elements– partly due to the nature of thinking and partly due to the requirements of excellent intellectual work.

I do not teach evasiveness or obscurantism. I do not ever tell a tester that he can get away with refusing to explain his test process. Explaining testing is an important part of being a professional.

Why People Get Confused

I reinvented software testing for myself, from first principles. So, I teach from a very different set of premises. This is necessary, because common ideas about testing are so idiotic. But it does result in confusion when my ideas are taken out of context and “mixed in” to the idiocy. Consider: “I won’t create a detailed test plan document” is a perfectly ordinary and potentially reasonable thing to say in RST. It is a statement about things made explicit in a document, not a statement about lack of planning. Yet Factory School methodology confuses documents for content. If you say that to one of them, it may be mistaken for a refusal to apply appropriate rigor to your work.

Here are some examples of how someone might misapply my teachings:

  1. Rapid Software Testing methodology (RST) is not the same thing as exploratory testing. ET is very simple. Anyone can do ET, just as anyone can look at a painting. But there’s a huge difference between a skilled appraisal of a painting by an expert and a bored glance by a schoolkid. RST is a methodology for doing testing (including scripted and exploratory testing) well. Therefore, anyone doing ET badly is not doing my methodology.
  2. In RST, a plan is not a document, it’s a set of ideas. Therefore, I say you don’t need to have a test plan template, or any sort of written test plan document in order to have a good test plan. I often document my test ideas, though, in different ways, when that helps. Therefore, the lack of a test plan (a guiding set of ideas) probably represents an immature and possibly inadequate test process, but the lack of a test plan document is not necessarily a problem.
  3. In RST, a test is not a document, it’s a performance. Therefore the lack of documented tests is not necessarily a problem, but poor testing (which can be determined by direct observation by a skilled tester or test manager, just as poor carpentry or poor doctoring can be detected) is a problem.
  4. In RST, we have no templates for reporting. But reporting is crucial. Reporting skills are crucial. Accountability is crucial. Credibility is crucial. We teach the art of telling a testing story. Therefore, anyone who declines to explain himself when asked about his testing is not practicing RST. I disavow such testers. (However, just because explaining oneself is an important part of testing doesn’t mean a manager can insist on arbitrarily voluminous documentation or arbitrary metrics. I suspect that, in some cases, managers who complain about testers refusing to document or explain themselves are really just obsessed with a specific method of documentation and refusing to accept other viable solutions to the same problem.)
  5. In RST we say that testing cannot be automated, and that tools can become an obsession. This leads some to think I am against tools. No, I am against bad work. Unfortunately, some tools, such as expensive HP/Mercury tools, are often used to wastefully automate weak fact checking at he expense of good testing. Yes., tools and the technical skills to create and apply them play an important role in great testing. It’s not automating testing when I use tools, because testing is whatever testers do, not what tools do. Therefore a tester who refuses to learn and use tools in general is not practicing RST.
  6. In RST we distinguish between checking and testing. This allows us to distinguish between a test process that is appropriately thoughtful and deep, and one (based solely on checking) that would be reckless and shallow. But when we criticize a checking-only test strategy, some people get confused and think we are criticizing the presence of checking rather than the lack of testing. Therefore, a tester who refuses to design or perform checks that are actually economical and helpful is not doing RST.
  7. In RST, we ban unscientific, abusive attempts at using metrics to control the test process. But when some people hear us attack, say, the counting of test cases, they assume that means we don’t believe in even the concept or principle of measurement. Instead, we support using inquiry-focused metrics (which inspire questions rather than dictating decisions), we promote active skepticism about numbers applied to social systems, and we promote the development of observation, reasoning, and social skills that limit the need for quantification. Therefore any tester who simply refuses to consider using metrics of any kind is not doing RST.
  8. Some people hear about the freedom of exploratory testing, and they confuse that with irresponsibility. But that’s silly. If you drive a car, you are free to run over pedestrians or smash into buildings– except you don’t, because you are responsible! Also, it’s against the law. Freedom is not the same thing as having a right. Therefore, anyone who accepts the freedom of exploratory testing and cannot or will not manage that testing appropriately is an incompetent or irresponsible tester.

Seven Kinds of Testers

Most of my work is teaching, coaching, and evaluating testers. But as a humanist, I want to apply the Diversity Heuristic: our differences can make us a stronger team. That means I can’t pick one comfortable kind of tester and grade people against that template. On the other hand, I do see interesting patterns of skill and temperament among testers, and it seems reasonable to talk about those patterns in a broad sense. Even though snowflakes are unique, it’s also true that snowflakes are all alike.

So, I propose that there are at least seven different types of testers: administrative tester, technical tester, analytical tester, social tester, empathic tester, user, and developer. As I explain each type, I want you to understand this:

These types are patterns, not prisons. They are clusters of heuristics; or in some cases, roles. Your style or situation may fit more than one of these patterns.

  • Administrative Tester. The administrative tester wants to move things along. Do the task, clear the obstacles, get to “done.” High level administrative testers want to be in the meetings, track the agreements, get the resources, update the dashboards. They are coordinators; managers.  Low level administrative testers often enjoy the paperwork aspect of testing: checking off boxes on spreadsheets, etc. (I was a test manager for years and did a lot of administrative work.) Warning: Administrative testers often are tempted to “fake” the test process. This pattern does not focus on the intellectual details of testing, but more the visible apparatus.
  • Technical Tester. The technical tester builds tools, uses tools, and in general thinks in terms of code. They are great as advocates for testability because they speak the language of developers. The people called SDETs are technical testers. Google and Microsoft love technical testers. (As a programmer I have one foot in this pattern at all times.) Warning: Technical testers are often tempted not to test things that can’t easily be tested with the tools they have. And they often don’t study testing, as such, preferring to learn more about tools.
  • Analytical Tester. The analytical tester loves models and typically enjoys mathematics (although not necessarily). Analytical testers create diagrams, matrices, and outlines. They read long specs. They gravitate to combination testing. (If I had to choose one category to be, I would have to say I am more analytical than anything else.) Warning: Analytical testers are prone to planning paralysis. They often dream of optimal test sets instead of good enough. If they can’t easily model it, they may ignore it.
  • Social Tester. The social tester wants you! Social testers discover all the people who can help them and prefer working in teams to being alone. Social testers understand that other people often have already done the work that needs to be done, and that no one person needs to have the whole solution. A social tester knows that you don’t have to be a coder to test– but it sure helps to know one. A good social tester cultivates social capital: credibility and services to offer others. (I follow a lot of the social tester pattern. My brother, Jon, is the classic social tester.) Warning: Social testers can get lazy and seem like they are mooching off of other people’s hard work. Also, they can socialize too much, at the expense of the work.
  • Empathic Tester. Empathic testers immerse themselves in the product. Their primary method is to empathize with the users. This is not quite the same as being a user expert, since there’s an important difference between being a tester who advocates for users and a user who happens to test. This is so different from my style that I have not recognized, nor respected, this pattern until recently. People with a non-technical background often adopt this pattern, and sometimes also the administrative or social tester pattern, too. Warning: Empathic testers typically have a difficult time putting into words what they do and how they do it.
  • User Expert. Notice I did not say “user tester.” User experts may be called domain experts or subject matter experts. They do not see themselves as testers, but as potential users who are helping out in a testing role. An expert tester can make tremendous use of user experts. Warning: User experts, not having a tester identity, tend not to study or develop deep testing skills.
  • Developer. Developers often test. They are ideally situated for unit testing, and they create testability in the products they design. A technical tester can benefit by spending time as a developer, and when a developer comes into testing, he is usually a technical tester. Warning: Developers, not having a tester identity, tend not to study or develop deep testing skills.

When I’m sizing up a tester during coaching. I find it useful to think in terms of these categories, so that I can more efficiently guess his strengths and weaknesses and be of service.

Do you think I have missed a category? Do you think I have de-composed them poorly? Make your case in the comments.

 

Bharani Asks About Daily Practice

Bharani from India is a novice tester who appears to have a lot of energy. She recently emailed me this:

Sir, I need one small tips. As a, new tester how should they improve to became sharp and clever in the testing field.
 I mean that, What is their homework. what they should do daily…without skipping
For ex: like normal daily activities. we have to do exercise  breakfast..so….so..
I mean that, what is the testers daily activities?

I am asked many questions by testers. This particular one reveals a lot about the questioner. For one thing, she thinks differently, because I have never been asked this. And it’s a question about discipline, so perhaps she’s a tester who is truly serious about developing herself.

My first answer is the same as the one my world famous writer of a father so often gives: write.

Write every day.

For me this takes the form of carrying my Moleskine notebook everywhere I go (now with a bandolier for my pens!). Whenever I find myself with a few moments, I make notes of my thoughts about testing and technical life. My notebooks serve as one source of great new ideas for my consulting and teaching.

But for testers, there are a few more practices to keep in mind…

Watch yourself think ever day.

While you are working, notice how you think. Notice where your ideas come from. Try to trace your thoughts. This is not easy. You have to practice to improve your skill of self-observation.

As a tester, you must become an artist of your thoughts, you must learn to see the structure and gain control of the structures by which you notice what no one else notices. These structures are not simply “intuition” or “common sense.” We can do better than that.

Question something about how you work every day.

Testers question things, of course. That’s what testing is. But too few testers questions how they work. Too few testers question why testing is the way it is.

For example: Why do we talk about tests “passing” and “failing”, instead of talking about whether there are problems? Do all bugs need to have steps to reproduce? How do we decide when to write a test down and what steps need to be written?

Explain testing every day.

The ability to explain testing is linked to nearly every other testing skill. Even if no one makes you explain your methodology, you can explain it to yourself. Do that. Practice doing it by voice and in writing. Don’t rest with buzzwords. Don’t tell yourself that you did “black box testing.” Break it down. Explain what you did specifically and explain why you did it.

 

Why Scripted Testing is Not for Novices

…Unless you want bad testing.

Claire Moss writes:

I am surprised that you say that scripted testing is harder for novice
testers. I would have expected that having so much structure around
the tests would make getting into testing easier for someone with less
experience and that the scripted instructions would make up for a lack
of discipline on the part of the tester.

Structure != “being told what to do”
First, you are misusing the word “structure.” All testing is structured. If what you mean by structure is “externally imposed structure” then say that. But even if you are not aware of a structure in your testing, it is there. When I tell a novice tester to test, and don’t tell him how to test, he will be dominated by certain structures he is largely unaware of– or if aware he cannot verbalize or control them much. For instance: the user interface look and feel is a guiding structure for novice testers. They test what they see.

Cognitive science offer plenty of ideas and insights about the structures that guide our thinking and behavior. See the book Predictably Irrational by Dan Ariely for more on this.

Scripted testing always has at least two distinct parts: test design and test execution. They must be considered independently.

Scripted test execution is quite a bit more difficult than exploratory testing, unless you are assuming that the tester following the script has exactly the same knowledge and skill as the test designer (even then it is a qualitatively different sort of cognitive process than designing). An exploratory tester is following (indeed forming as he goes) his own intentions and ideas. But, a scripted tester, to do well, must apprehend the intent of the one who wrote the script. Moreover, the scripted tester must go beyond the stated intent and honor the tacit intent, as well– otherwise it’s just shallow, bad testing.

Try using a script to guide a 10 year-old to drive a car safely on a busy city street. I don’t believe it can be done. You can’t overcome lack of basic skills with written instructions.

And sure, yeah, there is also the discipline issue, but that’s a minor thing, compared to the other things.

As for scripted test design, that also is a special skill. I can ask my son to put together a computer. He knows how to do that. But if I were to ask him for a comprehensive step-by-step set of instructions to allow me to do it, I doubt the result would help me much. Writing a script requires patience, judgment, and lots of empathy for the person who will execute it. He doesn’t yet have those qualities.

Most people don’t like to write. They aren’t motivated. Now give them a task that requires excellent writing. Bad work generally results.

Both on the design side and the execution side, scripted testing done adequately is harder than exploratory testing done adequately. It’s hard to separate an integrated cognitive activity into two pieces and still make it work.

The reason managers assume it’s simpler and easier is that they have low standards for the quality of testing and yet a strong desire for the appearances of order and productivity.

When I am training a new tester, I begin with highly exploratory testing. Eventually, I will introduce elements of scripting. All skilled testers must feel comfortable with scripted testing, for those rare times when it’s quite important.

Examples

1. Start browser

2. Go to CNN.com

3. Test CNN.com and report any problems you find.

This looks like a script, and it is sort of a script, but the interesting details of the testing are left unspecified. One of the elements of good test scripting is to match the instructions to the level of the tester as well as to the design goal of the test. In this case, no design goal is apparent.

This script does not necessarily represent bad testing– because it doesn’t represent any testing whatsoever.

1. Open Notepad

2. Type “hello”

3. Verify that “hello” appears on the screen.

This script has the opposite problem. It specifies what is completely unnecessary to specify. If the tester follows this script, he is probably dumbing himself down. There may be some real good reason for these steps, but again, the design goal is not apparent. The tester’s mind is therefore not being effectively engaged. Congratulations, designer, you’ve managed to treat a sophisticated miracle of human procreation, gestation, mothering, socializing, educating, etc. as if he were the equivalent of an animated poking stick. That’s like buying an iPad, then using it as a serving tray for a platter of cheese.

A Tester’s Commitments

This is the latest version of the commitments I make when I work with a programmer.

Dear Programmer,

 

My job is to help you look good. My job is to support you as you create quality; to ease that burden instead of adding to it. In that spirit, I make the following commitments to you.

 

Sincerely,

 

Tester

  1. I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied.
  2. I am not the gatekeeper of quality. I don’t “own” quality. Shipping a good product is a goal shared by all of us.
  3. I will test your code as soon as I can after you deliver it to me. I know that you need my test results quickly (especially for fixes and new features).
  4. I will strive to test in a way that allows you to be fully productive. I will not be a bottleneck.
  5. I’ll make every reasonable effort to test, even if I have only partial information about the product.
  6. I will learn the product quickly, and make use of that knowledge to test more cleverly.
  7. I will test important things first, and try to find important problems. (I will also report things you might consider unimportant, just in case they turn out to be important after all, but I will spend less time on those.)
  8. I will strive to test in the interests of everyone whose opinions matter, including you, so that you can make better decisions about the product.
  9. I will write clear, concise, thoughtful, and respectful problem reports. (I may make suggestions about design, but I will never presume to be the designer.)
  10. I will let you know how I’m testing, and invite your comments. And I will confer with you about little things you can do to make the product much easier to test.
  11. I invite your special requests, such as if you need me to spot check something for you, help you document something, or run a special kind of test.
  12. I will not carelessly waste your time. Or if I do, I will learn from that mistake.

(This is cool! Yong Goo Yeo has created a Prezi of this.)

An Important Comment

This comment came in from Stuart Taylor. I think it’s important enough that I should add this to the post:

Hi James,

I’m not sure if this is meant to be a little tongue in cheek, but that’s how I read it. That said, I fell at point 1. “I provide a service” really?

Yes, really. This is not a joke. And point #1 is the most important point.

This implies there may be alternative service providers, and if my customer relationship management isn’t up to snuff, I could lose this “client”. Already I feel subservient, I’m an option.

Of course there are alternative service providers. (There are alternative programmers as well, but I’m not concerned with that, here. I’m making commitments that have to do with me and my role, here.) And yes, of COURSE you may lose your client. Actually that’s a lot of what “Agile” has done: fired the testers. In their place, we often find quasi-tester/quasi-programmers who are more concerned with fitting in and poking around tools than doing testing.

Testing is not programming. Programmers are not testers. If you mix that, you do a sorry job of both. In other words: I am a programmer and a tester, but not a good programmer at the same time that I’m a good tester.

Please meditate on the difference between service and subservience. I am a servant and I am proud of that. I am support crew. I spent my time as a production programmer and I’m glad I don’t do that any more. I don’t like that sort of pressure. I like to serve people who will take that pressure on my behalf.

This doesn’t make me a doormat. Nobody wipes their feet on me– I clean their feet. There’s a world of difference. Good mothers know this difference better than anyone.

Personally, when I work on a software  project, I want to feel like I’m part of the team. I need the developer, and the developer wants me. We work together to bake quality in from the start, and not try and sprinkle it on at the end as part of a “service” that I offer.

What strange ideas you have about service. You think people on the same team don’t serve each other? You think services are something “sprinkled” on the end? Please re-think this.

I want to be on the same team, too. The difference, maybe, is that I require my colleagues to choose me. I don’t rap on the door and say “let me in I belong here! I like to be on the team!” (You can’t just “join” a team. The easier it is to join a team, the less likely that it is actually a team, I think. Excellent teams must gel together over time. I want to speed that process– hence my commitments) Instead, I am invited. I insist that it be an invitation. And how I get invited quickly is by immediately dissolving any power struggle that the programmers may worry about, then earning their respect through the fantastic quality of my work.

You can, of course, demand respect. Then you will fail. Then five years later, you will realize true respect cannot be demanded. (That’s what I did. Ask anyone about what a terror I was when I worked at Apple Computer. If you are less combative or more intelligent than I am, you may make this transition in less time.)

I may have actually agreed with your points before I was exposed to Continuous Integration, because that’s how my relationship used to be; hence me interpreting this as a light hearted piece. However I know that this kind of relationship still exists today (its here at this place). When I began working with continuous integration, the traditional role of the tester, with a discrete testing phase became blurred, as the test effort is pushed back up stream towards the source. If the tester and developer share a conversation about “how can we be sure we build the right thing, and how can we ensure we built it right” before the code gets cut, and the resulting tests from that are used to help write the code (TDD, BDD, SBE), then both of us can have a nice warm fuzzy feeling about the code being of good quality. The automation removes the repetition (and reduces the feedback) to maintain that assertion.

First, your experience has nothing to do with “continuous integration.” Continuous integration is a concept. Your experience is actually related to people. Specific people are collaborating with you in specific satisfying ways, probably because it is their pleasure to do so. I have had wonderful and satisfying professional relationships on a number of projects– mostly at Borland, where we were not doing continuous integration, and where testers and programmers cooperated freely, despite being organized into separate administrative entities.

(Besides, I have to say… Testing cannot be automated. Automation doesn’t remove repetition, it changes it. Continuous integration, as I see it, is another way of forcing your customers do the testing, while placing naive trust in the magic of bewildering machinery. I have yet to hear about continuous integration from anyone who seemed to me to be a student of software testing. Perhaps that will happen someday.)

In any case, nothing I’m saying here prevents or discourages collaboration. What it addresses are some of the unhealthy things that discourage collaboration between a tester and a programmer. I’m not sure if in your role you are doing testing. If you want to be a programmer or a designer or manager or cheerleader or hanger-on, then do that. However, testing is inherently about critical thinking, and therefore it always carries a risk (in the same way that being a dentist does) that we may touch a nerve with our criticism. This is the major risk I’m trying to mitigate.

[Note: Michael Bolton pointed out to me the “warm fuzzy” line. I can’t believe I let that go by when I wrote this reply, earlier. Dude! Stuart! If you are in the warm fuzzy business, you are NOT in the testing business. My goal is not anything to do with warm fuzzy thinking. My goal is to patrol out in the cold and shine my bright bright torches into the darkness. On some level, I am never satisfied as a tester. I’m never sure of anything. That’s what it MEANS to be a professional tester, rather than an amateur or a marketer. Other people think positively. Other people believe. The warmth and fuzziness of belief is not for us, man. Do not pollute your testing mentality with that.]

My confusion over this being a tongue in cheek post is further compounded by point 2. Too many testers do believe that they own quality. They become the quality gate keeper, and i think they enjoy it. The question “can this go live” is answered by the tester. The tester may be unwilling to relinquish that control, because they perceive that as a loss of power.

Just as excellent testers must understand they provide a service, they must also understand that they are not “quality” people. They simply hold the light so that other people can work. Testers do not create quality. If you create quality, you are not being a tester. You are being an amateur programmer/designer/manager. That’s okay, but my commitment list is meant for people who see themselves as testers.

I guess what i’m fumbling about with here, is that some testers will read this, and sign up to it. Maybe even print it out and put it up on the wall.

I hope they do. This list has served me well.

Either way, you have written another thought provoking piece James, but i cant help wondering about the back story. What prompted you to write it?

Stuart – sat in the coffee shop, but with laptop and wifi, not a notepad 😉

This piece was originally prompted when I took over a testing and build team for SmartPatents, in 1997. I was told that I would decide when the software was good enough. I called a meeting of the programmers and testers and distributed the first version of this document in order to clarify to the programmers that I was not some obstacle or buzzing horsefly, but rather a partner with them in a SERVICE role.

That was also the first time I gave up my office and opted to sit in a low cubicle next to the entrance that no one else wanted– the receptionist’s cubicle. It was perfect for my role. Programmers streamed by all day and couldn’t avoid me.

I think this piece is also a reaction to the Tester’s Bill of Rights (the one by Gilb, not the one by Bernie Berger, although I have my concerns about that one, too), which is one of the most arrogant pieces of crap I’ve ever seen written on behalf of testers. I will not link to it.

And now that I think of it, I wrote this a year after I hired a personal assistant, and two months later had her promoted to be my boss, even though she retained almost the same duties. The experience of working for someone whose actual day-to-day role was to serve me– but who felt powerful and grateful as my manager– helped me understand the nature of service and leadership in a new way.

Finally, a current that runs through my life is my experience as a husband and father. I am very powerful. I can make my family miserable if I so choose. I choose to serve them, instead.

Get Thee to the Konditori

I didn’t see you at the coffee shop, today.

Where were you, tester who told me that you need “certification” because, unlike me, you don’t have a public reputation? Where were you, tester who fears the big machine won’t hire you unless you conform, head bowed, to some lowest common standard? How about you, tester who told me that you dare not stop bullshitting your management about test case counts because “he likes those numbers” and, unlike me, you don’t have the credibility (and therefore the power) to say no without being clapped in chains and thrown into the brig?

I was there as usual, with my hardcover Moleskine and my current favorite pens (the Pigma Micron 01 by Sakura Color Products Corporation and the Mitsubishi Pencil Company Unipin Fineline).

I was working on my intellectual property. What were you doing?

Okay I get it: you were busy with your day job. And when the day is over you are tired and you want to go home. Maybe you have children. Children are important. But don’t tell me you are powerless to change things; that you can’t upgrade yourself. You have the power, but it’s not yet a priority for you. If you have reasonable passion and ambition in this field, you will soar above the problems that so many others feel trapped by. You must take time to develop your craft; educate yourself. Punching the time clock is not enough to obtain a better income and better job security.

One place to start is the Konditori (that’s Swedish for coffee shop… sounds cool and kind of mysterious, don’t you think?).

(This photo, above, is one of two Konditori’s I worked at today, in Stockholm.)

I have to keep developing myself and my intellectual property. It’s the engine by which I feed my family. So, today I worked on Rapid Testing Estimation. I want to roll out my new estimation methodology in time for my next test management class, next week.

(Note: I mostly drink green tea, these days… but it’s Stockholm! You gotta drink coffee in Stockholm!)

But who am I kidding? I work in coffee shops, furiously scribbling in my little notebooks, mainly because I love it. It is practical and effective, sure. Mainly, though, it’s peaceful and… hopeful. I am thrilled to feel that, at any moment, I might experience a breakthrough. And you know those breakthroughs, when they happen, can lead to a new hour or two of cutting edge class material, for which people will pay handsome sums. Last year we had the first conference on Session-based Test Management. My brother and I invented that at a coffee shop (well, it was a Denny’s… that’s pretty close).

We can wipe out bad testing. We can dump commercial certification programs into the dustbin. We can create a powerful craft where testers are well paid and respected. And this is how to do it: Meet me at the Konditori. Bring your Moleskine.

(Only if you love the craft, please. If you don’t, geez man, get a different job!)

Immaturity of Maturity Models

Maturities models (TMMi, CMM, CMMi, etc.) are a dumb idea. They are evidence of immaturity in our craft. Insecure managers sometimes cling to them as they might a treasured baby blanket. In so doing, they retard their own growth and learning.

A client of mine recently asked for some arguments against the concept of maturity models in testing. My reply makes for a nice post, so here goes…

First, here are two articles attacking the general idea of “maturity” models:

Maturity Models Have it Backwards

The Immaturity of the CMM

Here is another article that attacks one of the central tenets of most maturity models, which is the idea that there are universal “best practices”:

No Best Practices

And of course commercial tester certification, which I often ridicule on this blog, is a related phenomenon.
Here are some talking points:

1. I suggest this definition of maturity: “Maturity is the degree to which a system has realized its potential and adapted to its context.”

In support of this definition, consider these relevant snippets from the Oxford English Dictionary.

* Fullness or perfection of growth or development.
* Deliberateness of action; mature consideration, due deliberation.
* The state of being complete, perfect, or ready; fullness of development.
* The stage at which substantial growth no longer occurs.

2. I suggest this definition of maturity model: “a maturity model is plan for achieving maturity.”

By this definition, I know of nothing that is called a “maturity model” that actually is a maturity model. This is because maturity cannot be achieved by mimicking the “look” of mature organizations. Maturity is achieved through growing and learning as you encounter and deal with natural problems.

3. Maturity is not our goal in engineering. Our goal is to achieve success, satisfaction, security, and respect through the mechanism of doing good work.

No one gains success through maturity. It is not our goal. Some businesses benefit by the appearance of maturity, but that is a matter of marketing, not engineering. And regardless of how we achieve maturity, not all maturity is desirable. A creature approaching death is also mature.

Hey, blacksmithing is a mature craft, and yet look around… where are the blacksmiths? The world has moved on. We are in a period of growth, study, and creativity in testing. No one can say what the state of the art of our craft will be in 50 years. It will evolve, and we– our minds and experiences– are the engine of that evolution.

4. The behaviors of a healthy mature organization cannot be a template for success.

We achieve maturity by learning and growing as a testing organization, not by aiming at or emulating “mature” behaviors.

Maturity is a dependent variable. We don’t manipulate our maturity directly. We simply learn and grow, wherever that takes us. As any parent knows, you cannot speed up the maturation of your children by entreating them to “be mature.” Their very immaturity is partly the means by which they will mature. Immature creatures play and experiment. Research in rats, for instance, documents severe developmental problems in rats that were prevented from playing. Rats who don’t play as juveniles are much less able to deal with unexpected situations as adults. They cannot deal effectively with stress, compared to normal rats.

There can NEVER be one ultimate form or process that we declare to be mature, UNLESS our context never changes. This is because, in engineering, we are in the business of solving problems in context. However, our context changes regularly, because people change, technology changes, and because we are continuing to experiment and innovate.

Darwin’s theory of the origin of species is also a theory of the maturation and continual re-generation of species. As he understood, maturity is always a relative matter.

5. The “maturity model” of any outsider is essentially a propaganda tool. It is a marketing tool, not an engineering tool.

Every attempt to formalize testing constitutes a claim, on some person’s part, that he knows what testing should be, and that other people cannot be trusted to know this (otherwise, why not let them decide for themselves how to test?).

I have formalized testing, myself, and that’s exactly what I am thinking when I do so. But I do not impose my view of testing on any other organization or tester, unless they work for me. My formalizations are specific to my experiences and analysis of specific situations. I offer these ideas to my clients as input to a process of ongoing study and adaptation. To make any best practice claims would be irresponsible.

6. If you want to get better try this: create an environment where learning and innovation is encouraged; institutionalize mechanisms that promote this, such as internal training, peer conferences, pilot projects, and mentoring.

As my colleague Michael Bolton likes to say “no mature person, involved in a serious matter, lets any other mature person do their thinking for them.”

Mature people take responsibility for themselves. Therefore, don’t adopt anyone else’s “maturity model” of testing. Let your own people lead you.

Methodology Debates: Traps and Transformations

(This article is adapted from work I did with Johanna Rothman, at the 1st Amplifying Your Effectiveness conference. It’s never been widely published, so here you go.)

As a context-driven testing methodologist, I am required to think through the methods I use. Sometimes that means debating methodology with people who have a different view about what should be done. Over time, I’ve gained a lot of experience in debate. One thing I’ve learned is that most people have good ideas, but few people know how to debate them. This is too bad, because a successful debate can make a community stronger, while avoiding debates creates a nurturing environment for weak ideas. Let’s look at how to avoid the traps that make debates fail, and how to transform disagreement into powerful consensus.

Sometimes a debate is really part of a war. The advice below won’t help much if that is the case. This advice is more for situations where you are highly motivated to create or maintain a working relationship with someone you disagree with– such as when you work in the next cubicle from the guy.

Traps

  • Conflicting Terminology: Be alert to how you are using technical terms. A common term like “bug” has different meanings to different people. If someone says “Unit testing is absolutely essential to good software quality” among your first concerns should be “What does he mean by ‘unit testing’, ‘essential’, and ‘quality’?” Beware, sometimes a debate about definitions bears important fruit, but it can also be another trap. You can spend all your energy on them without necessarily touching the marrow of the subject. On the other hand, you can allow yourself to understand and even use someone else’s terminology in your debate without committing yourself to changing your preferred terminology in general.
  • Paradigm Conflict: A paradigm is an all-inclusive way of explaining the world, generally tied into terminology and assumptions about practices and contexts. Two different paradigms may explain the same phenomena in totally different ways. When two people from different paradigms come together, each may seem insane to the other. Whenever you feel that your opponent is insane, maybe that’s time to stop and consider that you are trying to cross a paradigmatic boundary. In which case, you should talk about that, first.
  • Ambiguous Metrics: Don’t be seduced by numbers. They can mean anything. The problem is knowing what they do, in fact, mean. When someone quotes numbers at me, I wonder how the metric was collected, and what influenced the people who collected them. I wonder if the numbers were sanitized in any way. For instance, when someone tells me that he performed 1000 test cases, I wonder if he’s talking about trivial test cases, or vital ones. There’s no way to know unless I personally review the tests, or conduct a detailed interview of the tester.
  • Confusing Feeling and Rationality: Beware of confusing feeling communication with rational communication. Be alert to the intensity of the feelings associated with the ideas being presented. Many debates that seem to be about ideas may indeed be about loyalty, trust, respect, and other fundamental issues. A statement like “C++ is the best language in the world. All other languages are garbage” may actually mean “C++ is the only language I know. I am comfortable with what I know. I don’t want to concern myself with languages I don’t already know, because then I feel like a beginner, again.” There’s an old saying that you can’t use logic to refute a conclusion that wasn’t arrived at through logic. That may not be strictly true, but it’s a helpful guideline. So, if you sense a strange intensity around the debate, your best bet may be to stop talking about ideas and start exploring the feelings.
  • Confusing Outcome and Understanding: Sometimes one person can be debating for the purpose of getting a particular outcome, while the other person is debating to understand the subject better, or help you understand them. Confusing these approaches can lead to a lot of unnecessary pain. So, consider saying what your goal is, and ask the other person what they want to get out of the debate.
  • Hidden Context: You may not know enough about the world the other person lives in. Maybe work life for them is completely different than it is for you. Maybe they live under a different set of requirements and challenges. Try saying “I want to understand better why you feel the way you do. Can you tell me more about your [life, situation, work, company, etc.]?”
  • Hidden History: You may not know enough about other debates and other struggles that shaped the other person’s position. If you notice that the other person seems to be making many incorrect assumptions about what you mean, or putting words in your mouth, consider asking something like “Have you ever had this argument with someone else?”
  • Hidden Goals: Not knowing what the other person wants from you. You might try learning about that by asking “Are we talking about the right things?”or “What would you like me to do?” Keep any hint of sarcasm out of your voice when you say that. Your intent should be to learn about what they want, because maybe you can give it to them without compromising anything that’s important to you.
  • False Urgency: Feeling like you are trapped and have to debate right now. It’s always fair to get prepared to discuss a difficult subject. You don’t have to debate someone at a particular time just because that person feels like doing it right then. One way to get out of this trap is just to say “This subject is important to me, but I’m not prepared to debate it right now.”
  • Flipping the Bozo Bit: If you question the sanity, good faith, experience, or intelligence of the person who disagrees with you, then the debate will probably end right there. You’ll have a war, instead. So, if you do that, in the heat of the moment, your best bet for recovery may be to take a break. When you come back, ask questions and listen carefully to be sure you understand what the other guy is trying to say.
  • Short-Term Focus: Hey, think of the future. Successful spouses know that the ability to lose an argument gracefully can help strengthen the marriage. I lose arguments to my wife so often that she gives me anything I want. The same goes for teams. Consider a longer term view of the debate. For instance, if you sense an impasse, you could say “I’m worried that we’re arguing too much. Let’s do it your way.” or “Tell you what: let’s try it your way as an experiment, and see what happens.” or “Maybe we need to get some more information before we can come to agreement on this.”

Transforming Disagreement

An important part of transforming disagreement is to synchronize your terminology and frames of reference, so that you’re talking about the same thing (avoiding the “pro-life vs. pro-choice” type of impasse). Another big part is changing a view of the situation that allows only one choice into one that allows many reasonable choices (the “reasonable people can bet on different horses” position). Here are some ideas for how to do that:

  • Transform absolute statements into context-specific statements. Consider changing “X is true” to “In situation Y, X is true.” In other words, make your assumptions explicit. That allows the other person to say “I’m talking about a different situation.”
  • Transform certainties into probabilities and alternatives. Consider changing “X is true” to “X is usually true” or”X, Y, or Z can be true, but X is the most likely” That allows the other person to question the basis of your probability estimate, but it also opens the door to the possibility of resolving the disagreement as a simpler matter of differing opinions on probability rather than the more fundamental problem of what is possible.
  • Transform rules into heuristics. Consider changing “You should do X” to something like”If you have problem Y and want to solve it, doing something like X might help.” The first statement is probably a suggestion in the clothing of a moral imperative. But in technical works, we are usually not dealing with morals, but rather with problems. If someone tells me that I should write a test plan according to the IEEE-829 template, then I wonder what problem that will solve, whether I indeed have that problem, how important that problem is, whether 829 would solve it, and what other ways that same problem might be solved.
  • Transform implicit stakeholders and concerns into explicit stakeholders and concerns. Consider changing “X is bad” to “I don’t like X” or”I’m worried about X” or “Stakeholder Y doesn’t like X.” There are no judgments without a judger. Bring the judger out into the open, instead of using language that make an opinion sound like a law of physics. This opens the door to talk about who matters and who gets to decide, which can be a more important issue than the decision itself. Another response you can make to “X is bad” is to” ask compared to what?” which will bring out the unspecified standard.
  • Translate the other person’s story into your terms and check for accuracy. Consider saying something like “I want to make sure I understand what you’re telling me. You’re saying that…” then follow with “Does that sound right?” and listen for agreement. If you sense a developing impasse, try suspending your part of the argument and become an interviewer, asking questions to make sure the other person’s story is fully told. Sometimes that’s a good last resort option. If they challenge you to prove them wrong or demand a reply, you can say “It’s a difficult issue. I need to think about it some more.”
  • Translate the ideas into a diagram. Try drawing a picture that shows both views of the problem. Sometimes that helps put a disagreement into perspective (literally). This can help especially in a “blind men and the elephant” situation, where people are arguing because they are looking at different parts of the same thing, without realizing it. For instance, if I argue that testing should start late, and someone else argues that testing should start early, we can draw a timeline and put things on the timeline that represent the various issues we’re debating. We may discover that we are making different assumptions about the cost of bugs curve, and which point we can draw several curves and discuss the forces that affect them.
  • Translate total disagreement into shades of agreement. Do you completely disagree with the other person, or disagree just a little? Consider looking at it as shades of agreement. Is it total opposition or is it just discomfort. This is important because I know, sometimes, I begin an argument with a vague unease about someone’s point of view. If they then react defensively to that, as if I’ve attacked them, then I might feel driven firmly to the other side of the debate. Sometimes when looking for shades of agreement, you discover that you’ve been in violent agreement all along.
  • Transform your goal from being right to being a team. Is there a way to look at the issue being debated as related to the goal of being a strong team? This is something you can do in your own mind to reframe the debate. Is it possible that the other person is arguing less from the force of logic and more from the fear of being ignored? If so, then being a good listener may do more to resolve the debate than being a good thinker. Every debate is a chance to strengthen a relationship. If you’re on the “right” side, you can strengthen it by being a gracious winner and avoiding I-told-you-so behavior. If you’re on the “wrong” side, you can strengthen the team by publicly acknowledging that you have changed you mind, that you have been persuaded. When you don’t know who is right, you can still respect feelings and consider how the outcome and style of the debate might harm your ability to work together.
  • Transform conclusions to processes. If the other person is holding onto a conclusion you disagree with, consider addressing the process by which they came to adopt that conclusion. Talk about whether that process was appropriate and whether it could be revisited.
  • Express faith in the other person. If the debate gets tense, pause and remind the other person that you respect his good faith and intentions. But only say that if it’s true. If it’s not true, then you should stop debating about the idea immediately, and deal instead with your feelings of mistrust. Any debate that’s not based on trust is doomed from the start, unless of course it’s not really a debate, but a war, a game, or a performance put on for an audience.
  • Wait and listen. Sometimes, a conversation looks like a debate, and feels like a debate, but is actually something else. Sometimes we just need to vent for a bit, and be heard. That’s one reason why being a good listener is not only polite, but eminently practical.
  • Express appreciation when someone tries to transform your position. When you notice someone making an effort to use these transformations in a conversation with you, thank them. This is a good thing. It’s a sign that they are trying to connect with you and help you express you ideas.

Context-Driven Methodology

Today I am writing about the Context-Driven school of software testing. This is a community that initially coalesced around Cem Kaner’s Los Altos Workshops on Software Testing (the LAWST meetings), in the late 90’s. I am a founding member of that community.

In the last two years, more people have joined the community of active context-driven thinkers, helping to develop the notion of what “context-driven” means for testing. I feel like we have a critical mass.

Schools of Thought

In this context, “school” relates to this definition from Merriam-Webster’s Dictionary:

4 a: a group of persons who hold a common doctrine or follow the same teacher (as in philosophy, theology, or medicine) the Aristotelian school; also: the doctrine or practice of such a group b: a group of artists under a common influence c: a group of persons of similar opinions or behavior; also: the shared opinions or behavior of such a group other schools of thought

This definition is a good start, but we are referring to a stronger sense of school: a paradigm; an ontology; a way of organizing, experiencing, and thinking about the entire field. In other words, in the way I use the term, it’s not possible to belong to more than one school of testing at a time. If you tried to be in two schools at once, the school I think you would actually belong to would be a new school called Confused.

The value of the school concept is that it allows us to make progress in our work without constantly having the same fruitless arguments with people who are pursuing a fundamentally different view of the field.

I see the testing world in terms of six major schools of testing. These include the “breakthrough� schools of Analytical, Control, and Test-Driven Design, and the “pragmatic� schools of Oblivious, Factory, and Context-Driven. The breakthrough schools generally hope to reduce or eliminate the need for independent testing. The pragmatic schools generally accept the idea of dedicated software testers and deal with the problem of how testers should get the job done.

To the best of my knowledge, only my school has named itself. Bret Pettichord, Cem Kaner, and I, came up with the names for the other schools. More on the first five in another post, right now I want to focus on the world from the Context-Driven school perspective.

The Context Driven Basics

Cem Kaner came up with the term “context-driven”, in 1999. Cem and I drafted the principles of the Context-Driven School in 2001. Context-driven is all about people solving problems.

“Context-driven” means to match your solutions to your problems as your problems change. The principles of my particular flavor of Context-Driven methodology can be found here.

Consider this rant:

Stop memorizing things people say are solutions, then applying them without understanding how or if they actually solve the problems you face.Don’t pretend to solve problems– really solve them. Don’t solve pretend problems– really solve real problems. And you know what? Those problems are going to shift and change, sometimes day by day or minute by minute.

If you think you can do good work by following some guru’s scripted Best Practice, you might as well blindfold yourself, take a dozen sleeping pills, and drive the Daytona 500. But the nice thing about driving blind in the Daytona 500 is that you know when you’ve crashed and burned. The hard thing about software testing is that you can do terrible work for years and not realize it until some uber-bug that you could have and should have found nails some poor user right in his “GUI components.” Even then, you’ll probably just blame the developers for putting the bug there in the first place.

If you want to take responsibility for the quality of your own work, practices won’t save you: what saves you is your skill, plus your ambition to use your skill. I’m not saying you should ignore practices, nor should you hate them. Just don’t worship them.

To learn to test is to learn to solve testing problems in every situation (we call them contexts) that you encounter. If you do that, your process will unfold naturally, your methods will be fluid, your practice balanced and flexible. So ask yourself, do you have the skills you need? Ask yourself, is your testing fitting your context?

(That’s the end of the rant.)

I like ranting, sometimes. In a rant, I can bring the world into sharp focus. I can say bold, definite things. Unfortunately, the rant above, while advocating the context-driven attitude, is not itself an example of that attitude. Do you see why?

Let me explain…

Four Attitudes Toward Context

When I say “context” I mean the totality of a situation that influences the success or failure of an enterprise. If I want my testing to be successful, I need to behave, somehow, in harmony with the aspects of my world that make testing success possible. For instance, I’m not going to introduce the test technique of cause-effect graphing into the context of a team that has no idea what that means or how to do it, and no time or resources to figure it out.

Everybody copes with context. I see four interesting flavors of coping:

    1. Context-Oblivious

To be context-oblivious is to be unaware of the relationships between the way you work and the many aspects of the context in which you work. A context-oblivious practitioner is aware of doing things, but can’t tell us why it’s done that way. This is typical of novices, of course, but also of experienced people who do not like to think about methodology (a methodology being a system of methods; a method being some way of doing something).

Completely context-oblivious testers simply don’t read blogs like this. What would be the point?

    1. Context-Specific

To be context-specific is to be adapted to a context and to know that you are adapted. You know what problems you face, and you solve them. But those problems don’t change, and therefore you don’t need to adjust your solutions.

Most of us Americans would say we know how to drive a car. When we say that we are thinking of a particular context: driving on ordinary public roads in ordinary weather in ordinary cars. Any ordinary car driver in America would know that our driving skill in that context probably does not carry over to driving a race car in the Daytona 500 (even without a blindfold). Still less does it carry over to “driving” an airplane, even though there are many elements in common between airplanes and cars.

I don’t need to learn how to drive a race car, because I’m not in that context, and I don’t even need to learn how to drive in unfamiliar situations, because I simply don’t drive in unfamiliar situations.

It’s easy to believe that being context-specific is good enough, but beware: What if you suddenly lose half your team, or part of your testing get outsourced, or features of your product change radically, or you have to adapt to use a new tool. You may not have the option of adapting to one context only.

    1. Context-Imperial

To be context-imperial is to keep your solutions the same while striving to change the context around you to fit those solutions.

A classic example is when a tester refuses to test without a “complete specification” instead of learning how to test with or without a spec.

Another classic example is the typical testing consultant who asserts we should all use some Best Practice he has in his mind. When it doesn’t work, he says we must not be using “engineering discipline” to follow the golden path to success, or maybe we don’t have “upper management support” to force us to do the right thing. The solution can’t be wrong, so the context must be.

A modern example is the Agilist who insists that test-driven design is the only good way to write software, and the all software must therefore be written in an object-oriented language, and that 100% of all tests must be automated, because otherwise… well… it’s just not Agile! Kent Beck has spoken! Don’t anger the gods of Agile!

It’s the context-imperialists who bug me most. They are like doctors who prescribe medicine to patients they’ve never examined.

    1. Context-Driven

To be context-driven is to behave as if there is literally no such thing as a best practice. All we can hope for is good practices in context. Context-driven methodologists are very interested in learning about practices. Then they decompose and recompose them to fit the situation at hand.

To summarize the attitudes of practitioners…

  • The context-oblivious simply follow convention.
  • The context-specific cultivate the skill to succeed in one stable corner of the world.
  • The context-imperial insist that the world change to suit their own skill.
  • The context-driven cultivate the skill to adapt to a changing world.

These Attitudes Are Not Mutually Exclusive

I said the four types above were attitudes, not schools. (Note that I’m using the same label “context-driven” as both the name of an attitude toward context and the name of my school of testing thought which is centered on that attitude.) That’s because these attitudes, unlike schools, can mix together.

Even the most ardent context-driven methodologist (and I suspect I am that person) cannot be aware of all the aspects of context that matter in every situation. For instance, if a tester is secretly planning to sabotage the project, I don’t necessarily take that into account when planning that project. In that respect, I am context-oblivious.

I am also partly context-specific, because all my thinking about testing is colored and biased by the kinds of projects I have worked on in the past. Since I have done little military work, for instance, my ideas and calculations are probably different than they would be if I had grown up as a tester in that world.

I am also context-imperial, in some ways. If you want me to design a test process for a commercial product with the requirement that every single test be procedurally specified and placed into a test case management system, I would probably tell you that it sounds like a formula for bad testing and I would try to get you to change your mind. I’m not confident I can be successful in that context.

Despite all that, my self-image and my pride as a tester lies mostly with my belief that I can and I should be able to see a wide range of solutions to nearly any problem. That’s what being skilled in this craft of testing means to me.

Look Again at the Rant

See the unqualified assertions in the rant (e.g. “Those problems are going to shift and change…�)? When directed to a person in a context I know nothing of, they are the hallmarks of the context-imperial attitude. The context-driven way is to make speak in conditionals (“If your problems shift and change…�) and qualifiers (“…then you might find it useful to learn how to adapt your solutions to fit.�), or else stick to speaking about my own experience (“I find it rewarding to study the dynamics of testing problems in general, so that I can solve them no matter how they shift and change. This is because, as a consultant, I am asked to give advice in a wide range of situations.�)

If you look at my writing on this blog closely, you will notice that I use conditionals, qualifiers and personal experiences a great deal. Now you know why. It’s the price of belonging to the Context-Driven school.

Context-Driven Challenge

Any decent context-driven thinker can cite at least three scenarios where taking the context-driven attitude would be a bad idea. Can you think of them?