The CAST testing conference is happening in Colorado Springs, July 13-16. I mention this for two reasons:

1. I will be teaching a testing tutorial there. I also will be wandering around with my various testing games and challenges hoping to do them with anyone who wants to see what I mean by “testing skills.”

2. CAST is the only conference that is organized in the true spirit of the Context-Driven School of Testing. It’s kind of our home conference. If you find my thoughts interesting, or those of Cem Kaner, Michael Bolton, Ben Simo, etc. Then you’ll want to be there, if you can.

This comes from an ISTQB advertisement they spammed me with, today:

“To ensure the quality of any software system, testers and QA professionals must thoroughly test the product. But how do you know that these tests are effective? If your team is conducting ad hoc, informal tests with little guidance or planning, the quality of the end product can be severely jeopardized—negatively affecting your bottom line.”

I don’t like to say things like this, nor am I comfortable supporting people who do. It’s not that it’s untrue– it is not necessarily untrue. But it is the kind of statement that fans the flames of a certain sort of Factory School bigotry in our industry. “Oh, you can’t trust testing unless it is pre-planned, pre-packaged, pre-approved, formalized, etc.”

Notice they say nothing about skill. It’s all about methodology, here, not skill. This kind of setup suggests that the next statement will be about the importance of factory-like test methodology. But that’s not what happens.

“The best way to be certain that you are providing customers with quality software is to make sure your team of testers is certified.”

Friends, I’m aware of no one in the industry– not even my worst enemies, not even Rex Black or Stuart Reid– who would publicly assert this or defend it. In fact, in debates against those of us who think certification is consumer fraud, the most typical move is for certificationists to say that certification isn’t even about skill, but rather about basic knowledge. “It’s a start” they say. “It’s a foundation.” (I reply that it’s a bad start and a bad foundation. Much worse than what it tries to replace.)

But then they allow this sort of advertising to go out! Completely undercutting their innocent-sounding plea! And they wonder why I complain that they don’t have the best interests of the testing craft at heart.

Notice that the “ad hoc and unplanned” stuff doesn’t even logically connect to certification. In fact, wouldn’t a highly skilled tester be far more likely to succeed with an ad hoc testing regime? When Roger Federer plays ad hoc tennis, I bet he still wins.

I think the reasons they start talking about methodology and end up talking about certification is A) their potential customers don’t understand the difference between skill and method, B) method is more concrete than skill, thus easier to evoke, and C) they know that what they say doesn’t have to be true or even logical, as long as it evokes horror and promises hope.

Oh, but there’s more…

“By taking the Software Tester Certification course and earning an internationally recognized certification in software testing, your team will gain the expertise needed to handle your greatest testing challenges; earn credibility and recognition as competent quality assurance professionals; and provide greater value to your organization.”

It’s internationally recognized? By whom? Some people who don’t study testing and some people who study testing and financially benefit from certification. Okay, but it is also internationally ridiculed by serious testers of many nations who wish to raise themselves to a level of skill that can’t be obtained in just a couple of days of training.

I recently encountered Dot Graham, now semi-retired, who told me that it hurts her feelings when people like me suggest that certificationists are only in it for the money. Dot is a sweet person. I don’t want to hurt her feelings. But I point her to advertising like this and I challenge her to explain it in any other terms. If not greed, then what, Dot? Stupidity? Pride?

Dot doesn’t want to argue with me about this. Of course she doesn’t. Rex Black doesn’t want to argue, either. Naturally. What answer could there be? Lois Koslowski once told me that “big dogs” don’t need to debate (in fairness, Lois Koslowski claims not to be a tester. I agree that she showed no testing competence or knowledge in the conversation we had. I just mention her because she did claim to be in charge of the ASTQB organization. Yikes!) This is capitalism in its ugly form– harvesting the ignorance and fear of others. Debate has no place here.

Is there no one in that self-declared professional community who reviews the advertising and stands up for professional temperance and humility?

A salesman from G2 Test Labs just called me. He said he was from India. He wanted to know if my testing company needed to partner with an offshore lab like his. I’m writing this now, while the memory of the conversation is fresh.

After he made his brief brief opening monologue I asked him “I’m a testing company. Why are you calling me?”

“Maybe you want to have an offshore arm.” He said.

“Well that depends on the skills of your testers. How do you train your testers?” I asked.

“Oh… we don’t do any training. But our testers are certified by other organizations.”

“Which organizations certify your testers?”

“Uh… I will have to check on that and get back to you.”

“Yes, that’s important information. Are ALL your testers certified?”

“Probably… most of them are.”

“Sounds like you don’t know.”

“…”

“Hey, this will make a funny post. Check my blog in about an hour. Goodbye!”

Now, in fairness, the salesman sounded like he was about 22 years old. Perhaps they sent him to call me as part of some hazing ritual.

[Oh, I just remembered, in his opening statement, he mentioned that his company was ISO-9001 certified, too. Wow. That takes me back to 1992, when I was fighting ISO-9001 certification. That certification program turned out not to amount to anything, either.]

Matt Heusser wrote an interesting post about “boutique testers.” I like the idea of boutique testers (boutique intellectuals of all kinds, actually, which is why I wrote my new book.) And I am an example of one. The testing I’ve done in recent years has been mostly on court cases or part of coaching testers, though. I want to do more ordinary testing. I need that in order to keep up my practice.

I live on an island, making travel expensive and annoying. But I have a great Internet connection.

There are important challenges to being a remote tester, though. The main technical problem, in my experience, is acquiring and configuring the product to test. Getting up to speed fast benefits from being onsite. The main social problem is trust. Hiring a remote tester, especially one who’s supposed to be high powered, is a little like hiring a therapist. In my experience, developers feel more sensitive about a well-paid, high status outsider poking at their work, than they do about an internal tester who probably will disappear in a few weeks.

Then there’s communication. Despite all the tools for modern communication, we haven’t yet developed a culture of remote interaction that lets us use those tools effectively. Even though I’m always on Skype, and people can see I’m on Skype, they still ask permission to call me on Skype! And I also feel nervous about calling other people on Skype. They might be annoyed with me. I do use GoToMeeting, and that helps a lot. Michael Bolton and I have been collaboratively writing, recently, using it. We like it.

Finally, there is one big logistical problem: availability. You can call me up and have me test for you. But, being a fully independent consultant, my time is chopped up. I have a week here and a few days there, usually. This is the main reason I go in for short-term consulting and coaching. Unless a rich client comes along and induces me to clear my schedule, I can’t afford to have only that one client.

Still, with the price of travel so high, I think this is the direction we need to go: each of us developing ourselves into unique thinkers with strong brands, and then remotely connecting (interesting oxymoron, eh?) with our clients.

Phone: 360-440-1435
Email: james@satisfice.com
Skype: satisfice

I’ve released a new video, which is a whimsical look at a serious subject: explaining exploratory testing.

In the video, my brother and I independently test an “Easy Button” for 10 minutes. Neither of us had seen the other’s test session. Then I edited the 20 minutes of total testing down to a 4 minute highlight reel and added subtitles.

The subtitles are important. One of the core skills of excellent testing is being able to reflect upon, describe, explain, and defend your work. The rhetoric of testing is a big part of Rapid Testing methodology.

So, everything we did, we can explain. If someone stops me when I’m testing, I can give a report on the spot, in oral or written form, and I can put specific technical terminology to it. In my experience, most testers are not able to do that, and there’s one major reason– they don’t practice. It does take practice, friends. While you were enjoying your Sunday, my brother and I were challenging each other to a testing duel.

You might quibble with me about the specific terminology that I used in the video. Indeed, there is a great deal of leeway. One single test activity might simultaneously be a function test, a happy path test, a scenario test, a claims test, and a state-transition test! There’s no clean orthogonality to be found. And as you already know if you read my blog, I reject any “official” lexicon of testing. But I’m not just throwing these terms around, I can explain each one, and say what is and is not an example of it.

What about the Easy Button?

Our principal finding is that the Easy Button is extremely durable. I’m surprised at the high quality of the fit and finish. Also it feels solid (I discovered why when I disassembled it and found apparently lead weights inside. Plus, the button surface is amazingly resilient to repeated hard blows with a rock hammer).

But I’m also surprised that it claims not to be a “toy.” Of course it’s a toy. Of course little kids will play with it.

If I were seriously consulting about testing it, I would probably suggest that its physical qualities were more important to validate than its functional qualities. There appears to be little risk associated with its functionality. On the other hand, there appears to be little risk with its physical qualities either.

I would suggest that it’s far more important to test the web version of the “Easy Button” than the physical version. I would move on to that.

(Since this pertains to both self-education AND technical work, I’m posting this on both of my blogs)

Randy Ingermanson has an interesting approach to writing fiction. It’s called the Snowflake Method. It looks interesting, but I won’t be following it in my work.

First, Don’t Follow
I only use my own methods. That is to say I’m happy to use anyone else’s ideas, but only if they become mine, first. I can learn from other people, but I don’t follow anyone. See the difference? The only way I can responsibly follow someone as a thinker is if they are supervising my work. For instance, when Captain Ben taught me to sail, I used his methods because he was right there to correct me. Also it was his boat, and he answered my questions and let me experiment with alternative ideas to see why they were inferior. As he trained me, his methods became my methods. I began to do them based on my sense of their logic– which means I also came to understand under what circumstances I might need to change them. That’s the difference between learning and numb indoctrination.

When Jerry Weinberg taught me the Fieldstone Method of writing, I formed my own interpretation of it, and now it’s the James Bach version of Weinberg’s Fieldstone Method. And when I teach Rapid Software Testing, my methodology ideally becomes personal to each student, morphing to their own preferences and patterns, or else they should not be using it.

“Composting” Good?
In describing the Snowflake Method, Ingermanson discusses something that he says every writer does: composting. That’s where you actually dream up the story. He writes that

“It’s an informal process and every writer does it differently. I’m going to assume that you know how to compost your story ideas and that you have already got a novel well-composted in your mind and that you’re ready to sit down and start writing that novel.He says how you do that is a personal creative matter.”

Okay. Interesting that he says nothing about how to do that, though, since for me that’s almost all that writing is. But now, the actual Snowflake Method, he says, kicks in after composting is done with. It’s a way of progressive outlining of the book so that you can write it in an organized way.

Wait, did he say that happens after composting?

AFTER composting? Seriously?

This is a problem for me, because I’m nearly always doing that thing he calls composting. For me, writing is an exploratory activity. I’m constructing my ideas before I write them down and also as I’m writing them down. I’ve written many articles and two books that way. I have not yet written much fiction, but I have a hard time believing my method will be or should be different for fiction.

“Seat of the Pants” Bad?
Here Ingermanson makes a tiresome rhetorical move: He contrasts his approach with the “seat of the pants” method. He believes his method is better. I agree that it’s probably better for him, because it’s his own personal method. But on what basis can he say that his method is better than the alternatives for anyone else? Besides, it sounds like “composting” is just “seat of the pants” that happens to be Ingermanson-approved.

This is typical best practices rhetoric, and the pattern generally goes as follows:

1. I conceive my method as figure and everything else as ground. I won’t talk about how my method blends into and is supported by any other methods or skills or talents or preferences. I won’t talk about how it may go horribly wrong. The method is an island.

2. Since I like my method better than the other not-the-method thing I once did, [cite anecdotal and cherry-picked evidence here], it is probably better.

3. Since I taught someone else to use my method and they said they like my method better than whatever unexamined way of working they once had, [did they actually use my method? Well, they said they did, but I didn't actually watch them do it], it is even more probably better.

There are a few problems with this pattern of reasoning. One is that it is not necessarily a comparison of one method to another. It’s more likely a comparison of a state of confusion to a state of decision. Decision usually does win over confusion. The people who are out looking for a method may not already have a sound understanding of the methods they already use. So they leap on any method offered as if it were a life buoy. This of course is no indication that the method itself is better than any other method, but merely that people hate feeling confused and incompetent.

Another problem is that even when it is a comparison of methods, it’s generally a comparison between an ineffable method and one that sounds good when explained. Things that are ineffable, no matter how useful, get a bad reputation. That’s why you’ve met at least one person in your life who has claimed that you need to “learn to breathe” or “remember to breathe.” In fact, you already have a method of breathing, and unless your eyes have just gone so fuzzy that you can’t read this at all, you are probably breathing pretty well right this moment. An effective way to present a method of breathing could be to say “If you are having problem X, one solution might be to try a special kind of breathing called Y. Let’s try it now so you can see what I mean…” This way offers the practice without implicitly or explicitly denying other ways of working.

Yet another problem is that all methods rest on a certain way of organizing the world. If you don’t accept that foundation, then the method won’t satisfy you. Ingermanson seems to find it easy to segment heavy creative work from the light creative work. Hence composting is good, but seat-of-the-pants writing is bad. Since I don’t accept that distinction, to use the Snowflake Method as presented would force me to become alienated from my creative process. I would not be in direct touch with my own mind, but all thoughts would be mediated through the controlling outline of the Snowflake. Ick!

A Rhetoric for Pushing Back
It’s not “seat of the pants”, I say. It’s not merely “ad hoc.”

It’s thoughtful and responsible, rather than mindless and robotic. It’s exploratory, rather than pre-scripted. It’s agile rather than rigid. It’s constructive and generative, rather than a mere conditioned response.

Want more? Try breaking the method down into sub-parts. In exploratory work, I might cite such tasks as:

• overproduce ideas and abandon them (think “brainstorming”)
• recover previously abandoned ideas (think “boneyard”)
• pursue lines of inquiry
• conduct thought experiments
• alternate my tactics for better progress
• dynamically manage my focus (from very focused to de-focused)
• charter my own work in light of my mission as I understand it
• view my work from different perspectives
• produce results, then reproduce them differently based on what I learned (cyclic learning)
• construct a new and better version of myself as I work

Seat of the pants? That sounds like a put-down. Why don’t they call it dynamic control and development? Because that doesn’t sound like a put-down.

Reclaim Your Personal Method
As Adam Savage says, “I reject your reality, and substitute my own.” Yes, indeedy.

You don’t have to accept someone else’s intermediating artifice between you and your thoughts. Whether that’s a book outline, or a test plan document, TPI, or some method of artificial breathing you can say no. You can say “that would be irresponsible, because I must remain attached to the source of my own methods of working. I can’t drive a car safely from the BACK SEAT!”

Having said all that, I found Randy’s Snowflake Method interesting and I think I will try it. I will meld it with my exploratory style of working, of course, and claim it for my own.

Pradeep Soundararajan got threatened with lawyers when he criticized Testing Experience magazine for being under the thumb of the ISTQB (for those who don’t yet know, the ISTQB are the guys who want to prevent you from getting work as a tester unless you first pass their silly test. They also plagiarized my definition of exploratory testing, while subtly changing that definition to alter part of its fundamental meaning).

The editor of that magazine could have said “Look, we believe in the ISTQB. That’s just how it is.” Instead he hinted that he would sue Pradeep if he blogged his criticism. Pradeep blogged anyway.

A couple of authors of testing textbooks have threatened to sue me, in the past. I don’t know what they think they were accomplishing by that. I just turned around and blogged about them. It’s not illegal to criticize bad work and the people who do it.

The ISTQB is not part of any community of software testers. They are a business that ignores the rest of the testing world while pursuing their own agenda to line their pockets and promote themselves with misleading advertisements. That’s my opinion, which I have reached through a variety of experiences and investigations as part of being in this craft. One of those experiences is the time that the ISTQB approached me to run their American operation. They spent 30 minutes telling me how great it would be to take advantage of the American market for certification, before they realized I thought it was a terrible idea. After that, I guess they decided that I’m not qualified to have an opinion, since they’ve never paid attention to me since.

I once read Rex Black’s own advertising (promoting ISTQB certification) as part of a keynote speech at the CAST conference– his exact words, mind you– and after reading it I explained why I thought it was misleading. Rex then demanded the return of the money he paid to sponsor the conference.

You might think, yeah, well, of course he should get his money back, until you remember that this was a conference dedicated to free investigation of testing ideas, and not a get-out-of-criticism-if-you-pay-a-fee show. CAST is a free speech zone.

I hope that testers will recognize these opportunists for what they are and begin to fight back. I’m glad that Pradeep is doing just that.

I’m well known for promoting philosophy. By philosopher, I mean someone skilled in the exploration, analysis and clarification of meaningful ideas and the processes by which we arrive at those ideas. Since software is made of ideas, that sounds like testing to me. So, if you want to be an excellent all-around tester, I think you need to think like a philosopher.

(Of course, I have critics who dispute this, but there’s a funny thing– in order to form a coherent argument against my point of view you would have to do philosophy, and that very process would tend to undermine your credibility. Kinda like beating someone to death to prove that pacificism is best!

If you really believe you don’t need philosophy to be a tester, a more effective move would be to refuse to argue. Just use coercion and manipulation to get your way. This is what most of the certificationists choose to do. The harsh light of reason and open debate is a kind of acid to them.)

Promoting the practice of philosophy in daily work is a somewhat lonely job, because our schools manage to turn it into drab and awful subject. I persist for only one reason: I have a drive to be an excellent tester, and I want to help other testers be excellent, too. Otherwise, I would have given up long ago.

Enter Tawney Gowan. Tawney is a somewhat hyperactive young woman from Montreal who attended my STAR East tutorial called “How to Teach Yourself Testing.” She’s been a tester for a few years, but she recently went back to school to study the history ideas. After the class, she came up to me and complained that my citation of Thomas Kuhn’s theory of paradigms in scientific revolutions was out of date. I was tickled that she had an opinion on that, since so few testers have read anything about it.

After a few minutes of hearing her rant, I thought I would buy myself some time. So I asked, “Can you put together a reading list of the things you think I need to stufy?”

She replied, “I already have.”

I just think this is cool. It’s the kind of service I need from a colleague. I hope that Tawney stays in testing and starts a blog.

Here’s to testers who are fearless learners!

Take a look at the screen shot, below. This is from my first five minutes of playing with Wolfram/Alpha. Do you see what’s wrong with it? I’ll tell you in a minute…

Wolfram/Alpha is the new search engine that isn’t so much a search engine as a find-interesting-ways-to-analyze-data-and-show-it-to-me engine. It’s a closed system, as far as I can tell. It does some cool things. But I don’t understand how they will keep up with the data quality problem.

This worries me because the output from Wolfram/Alpha looks authoritative. I want to be able to trust it. But look at this slightly disturbing problem. I searched for Francis Bacon, but instead of getting a page about the various Francis Bacons of history and having an opportunity to disambiguate, I got the output, below. As you see, it combines information from two different men: Francis Bacon, 1st Viscount St. Alban and Lord Chancellor of England under Elizabeth I, and Francis Bacon, the painter. Furthermore, there appears to be no way to focus the search. Adding search terms that should distinguish between the two men appears to do nothing.

This tells me that there isn’t a lot of data in the system, yet, and that the data that is there may be mangled in ways that I may not notice unless I already know the thing I asked to learn about.

At least with Google and Wikipedia, it’s a relatively open system where I get a variety of results. So, beware, folks.

That said, I’m going back to playing with Wolfram/Alpha some more… Because it’s cool.

As you may know I’ve written a book about learning called Secrets of a Buccaneer-Scholar that will come out in September. It’s about learning in general, not just testing. I have a site and blog devoted to just that stuff.

As part of developing the ideas in the book, I’m also publishing videos. The first one was not related so much to testing, so I didn’t post here about it.

The video I just posted, however, does relate more to testing. It’s a video about describing things, and seeing things in terms of systems, connected to the world around. You may find it relevant. This is all part of my strong philosophy that testing is not just a little task. To test well is to behave like a scientist. Testing is science by another name.

Here’s the link.

I hope you will notice that this is an experiment in how we might create entertaining, yet informative teaching videos about otherwise dry intellectual subjects.

I’m delighted to read Michelle Smith’s play-by-play description of how she is coaching new testers. Take a look.

Let me catalog the coolnesses:

1. “The team I work with was previously exposed to Rapid Software Testing. This exposure caused me to wonder what would happen if these new folks were exposed to some of these ideas early on?”

Notice that she uses the word “wonder”? That’s the attitude I hope to foster in people who take my class. It’s an attitude of curiosity and personal responsibility. She doesn’t speak of applying practices as if the people on the team were milling machines waiting to be programmed. She implies that her testers are learners under their own control. Her attitude is one of establishing a productive but not coercive  relationship.

I don’t know if she got this from my class– probably she had it beforehand– but it’s an attitude I share with her.

2. “I went in their shared office and opened up a five minute conversation with them by asking “What is a bug?” and following that with “who are the people that matter?”.”

Michelle mentions “five minute conversations” a few times. And notice how most of her interactions were in the form of puzzles and questions. It speaks of a light touch with coaching. Light touch is good. Especially with novice testers, experience speaks louder than lecture. Introduce an idea then try it, or try something first, then talk about the idea. Either way, I like how she was getting them working.

3. She had them practice explaining themselves, both in writing and by voice.

4. She was concerned more with the cognitive parts of testing than with the artifacts. That’s good because excellent testing artifacts, such as bug reports and test cases, come from the thinking of the testers. Think well and the rest will follow.

5. She has them work on several aspects of testing. Notice how she deals with oracles, tools, mission, the social process and gaining product knowledge.

I bet what Michelle is doing will lead to better, more passionate testers, and more dynamic, flexible testing. Compare this to what we see so often in our industry: testers simply told to sit down and create test cases. Look, even if you think pre-defined test cases make for great testing, I think to be successful with that you have to base it on skilled and knowledgeable testers. Michelle is creating that foundation with her team.

Overall, what I’m most happy with is that Michelle has made Rapid Testing her own thing. This is vital. This is fundamental to the spirit of my teaching. I want to grow colleagues who confidently think for themselves. Hats off to you, Michelle, for doing that and blogging about it.

Finally, Michelle writes, “I have no idea if what I am doing is going to produce any benefits to them, to the team, or to the stakeholders. Time will tell.”

No best practices nonsense, here. No certification mentality. Just healthy skepticism. Thank you, Michelle!

One of the things that makes it hard to talk about quality software is that we first must overcome the dominating myth about quality, which goes like this: The quality of a product is built into it by its development team. They create quality by following disciplined engineering practices to engineer the source code so that it will fulfill the requirements of the user.

This is a myth, not a lie. It’s a simplified story that helps us make sense of our experience. Myths like this can serve a useful purpose, but we  must take care not to believe in them as if they were the great and hoary truth.

Here are some of the limitations of the myth:

1. Quality is not a thing and it is not built. To think of it as a thing is to commit the “reification fallacy” that my colleague Michael Bolton loves to hate. Instead, quality is a relationship. Excellent quality is a wonderful sort of relationship. Instead of “building” quality, it’s more coherent to say we arrange for it. Of course you are thinking “what’s the difference between arrange and build? A carpenter could be said to arrange wood into the form of a cabinet. So what?” I like the word arrange because it shifts our attention to relationships and because arrangement suggests less permanence. This is important because in technology we are obliged to work with many elements that are subject to imprecision, ambiguity and drift.
2. A “practice” is not the whole story of how things get done. To say that we accomplish things by following “practices” or “methods” is to use a figure of speech called a synecdoche– the substitution of a part for the whole. What we call practices are the public face of a lot of shadowy behavior that we don’t normally count as part of the way we work. For instance, joking around, or eating a salad at your desk, or choosing which email to read next, and which to ignore. A social researcher examining a project in progress would look carefully at who talks to whom, how they talk and what they talk about. How is status gained or lost? How do people decide what to do next? What are the dominant beliefs about how to behave in the office? How are documents created and marketed around the team? In what ways do people on the team exert or accept control?
3. Source code is not the product. The product is the experience that the user receives. That experience comes from the source code in conjunction with numerous other components that are outside the control and sometimes even the knowledge of product developers. It also comes from documentation and support. And that experience plays out over time on what is probably a chaotic multi-tasking computing environment.
4. “Requirements” are not the requirements, and the “users” are not the users. I don’t know what my requirements are for any of the software I have ever used. I mean, I do know some things. But for anything I think I know, I’m aware that someone else may suggest something that is different that might please me better. Or maybe they will show me how something I thought was important is actually harmful. I don’t know my own requirements for certain. Instead, I make good guesses. Everyone tries to do that. People learn, as they see and work with products, more about what they want. Furthermore, what they want actually changes with their experiences. People change. The users you think you are targeting may not be the users you get.
5. Fulfillment is not forever and everywhere. The state of the world drifts. A requirement fulfilled today may no longer be fulfilled tomorrow, because  of a new patch to the operating system, or because a new competing product has been released.  Another reason we can’t count on a requirement being fulfilled is that can does not mean will. What I see working with one data set on one computer may not work with other data on another computer.

These factors make certain conversations about quality unhelpful. For instance, I’m impatient when someone claims that unit testing or review will guarantee a great product, because unit testing and review do not account for system level effects, or transient data occurring in the field, or long chains of connected transactions, or intermittent failure of third-party components. Unit testing and review focus on source code. But source code is not the product. So they can be useful, but they are still mere heuristic devices. They provide no guarantee.

Once in a while, I come across a yoho who thinks that a logical specification language like “Z” is the great solution. Because then your specification can be “proven correct.” The big problems with that, of course, is that correctness in this case simply means self-consistency. It does not mean that the specification corresponds to the needs of the customer, nor that it corresponds to the product that is ultimately built.

I’m taking an expansive view of products and projects and quality, because I believe my job is to help people get what they want. Some people, mainly those who go on and on about “disciplined engineering processes” and wish to quantify quality, take a narrower view of their job. I think that’s because their overriding wish is that any problems not be “their fault” but rather YOUR fault. As in, “Hey, I followed the formal spec. If you put the wrong things in the formal spec, that’s YOUR problem, stupid.”

My Take on the Quality Story

Let me offer a more nuanced version of the quality story– still a myth, yes– but one more useful to professionals:

A product is a dynamic arrangement, like a garden that is subject to the elements. A high quality product takes skillful tending and weeding over time. Just like real gardeners, we are not all powerful or all knowing as we grow our crop. We review the conditions and the status of our product as we go. We try to anticipate problems, and we react to solve the problems that occur. We try to understand what our art can and cannot do, and we manage the expectations of our customers accordingly. We know that our product is always subject to decay, and that the tastes of our customers vary. We also know that even the most perfect crop can be spoiled later by a bad chef. Quality, to a significant degree, is out of our hands.

After many years of seeing things work and fail (or work and THEN fail), I think of quality as ephemeral. It may be good enough, at times. It may be better than good enough. But it fades; it always fades, like something natural.

Or like sculpture by Andy Goldsworthy.  (Check out this video.)

This is true for all software, but the degree to which it is a problem will vary. Some systems have been built that work well over time. That is the result of excellent thinking and problem solving on the part of the development team. But I would argue it is also the result of favorable conditions in the surrounding environment. Those conditions are subject to change without notice.

After Adam White’s test of of Rypple.com, I decided to try it myself. I soon ran into a fairly serious problem. I was able to try the service without registering, but when I tried to register, the system claimed I already was registered. Then when I tried to reset my password, it claimed I was not registered. Seemed like someone’s database tables were in a bunch.

(Oh well, quality is dead…)

But since I was so hard on WebGreeter about their creepy customer service technology. I thought I’d call Rypple’s toll free line and take my chances.

Aside: I hate calling tech support. I hate the stonewalling and being patronized by poorly trained anonymous dunderheads. Even if I were to lose most of my fingers in a data mining accident I could still count on one hand the good experiences I’ve had with tech support.

Anyway, I had a really good experience. After one ring, James Tam answered. That’s right no voicemail menu! I had tapped the man Tam himself!

The first thing I said was “I want to report a bug on your system.” I was expecting a defensive or brusque reply. Instead he asked me to tell him about it, and I did.

Well here’s the thing. He couldn’t solve my problem right away. He tried. I believe he’s working on it right now. But one thing he said struck me: “This looks like a problem on our side.”

Let me savor those words again…

This looks like a problem on our side.

Somebody taking responsibility?? How often do you hear that from a website support guy? Assuming you ever get to talk to one?

Look what happened. On paper, I had a bad quality experience. Yet I feel good about Rypple. Go Rypple.

The Factory School is that community of process people who believe testing benefits from eliminating the human element as much as possible. They wish to mechanize testing, and to condition the humans within it to see themselves as machines and emulate machines as much as possible. It’s an idea that has a number of advantages, with the important caveat that it makes good software testing impossible by leaving no room in the process for skill and thinking.

Sometimes when I complain about the Factory School of software testing, people think I’m exaggerating or making it up.

But check out this quote from Yaron Sinai, CEO of Elementools:

“With Test Case, the team doesn’t think,” Sinai said. “They just need to follow the steps. And for you, as a testing manager, you know that once they completed a set of tests, you know they followed the steps that needed to be followed.”

At first when I saw this, I thought it was a joke news story. Apparently not. (For the sake of Mr. Sinai, I hope he was misquoted. I will gladly post a retraction if that is the case.)

The man’s tool is called Test Case, which is emblematic right there. It’s focus is on test cases, not good testing. His view of test cases appears to be test procedure steps, and he wants testers to stop thinking, dammit, and just follow steps. Like factory robots. Robots that don’t question or talk back. Nice. Saaafe. Rooooooboootttts.

I haven’t found much information about Mr. Sinai online. He apparently hasn’t written about testing, per se. At least he is consistent: he has contributed no ideas to the testing craft, and now he sells an idea prevention system in the guise of a test management tool.

I want to ask Mr. Sinai whether, as a CEO, he faithfully follows his “CEO cases” each day, written for him by other, smarter CEO’s. Or does he, gasp, think for himself? I bet he would reply that although he is a smart CEO, not all CEO’s are smart enough to make their own decisions, and thus it’s only reasonable that their work should be scripted. When I suggest that a minimum requirement to be a CEO should be the ability to think about business problems and make decisions on their own behalf, I’m sure he will say “that’s just not practical.”  By which he will mean, of course, that HE doesn’t know how to do it.

The Factory School promotes the antithesis of engineering, while often using the word engineering as if it were some corpse impaled on a stick. Their approach to managing an engineering process is to kill it. Indeed, a dead process, like a dead horse, is much easier to manage once you get used to the smell.

And a lot of top managers buy this crap because the demos are simple, they are unaware of alternatives that would actually help, and the perfume on their cravats tends to mask the stench of tool vendors who don’t know anything about the craft.

Michael Bolton reported on our quick test of IMVU, whose development team brags about having no human-mediated test process before deploying their software to the field.

Some commentors have pointed out that the bugs we found in our twenty minute review weren’t serious– or couldn’t have been– because the IMVU  developers feel successful in what they have produced, and apparently, there are satisfied users of the service.

Hearing that, I’m reminded of the Silver Bridge, which fell down suddenly, one day, after forty years of standing up. Up to that day, it must have seemed quite reasonable to claim that the bridge was a high quality bridge, because– look!– it’s still standing! But lack of information is not proof of excellence, it turns out. That’s why we test. Testing doesn’t provide all possible information, but it provides some. Good testing will provide lots of useful information.

I don’t know if the IMVU system is good enough. I do know that IMVU has no basis to claim that their “continuous integration” process, with all their “automated test cases” has anything to do with their success. By exactly the same “not dead yet” argument, they could justify not running any test cases at all. I can’t help but mention that the finance industry used the same logic to defend deregulation and a weak enforcement of the existing laws that allowed Ponzi schemes and credit swap orders to cripple the world economy. Oops, there goes a few trillion dollars– hey maybe we should have been doing better oversight all these years!

It may be that no possible problem that could be revealed by competent testing would be considered a bad problem byIMVU. If that is the case, then the true reason they are successful is that they have chosen to offer a product that doesn’t matter to people who will accept anything they are offered. Of course, they could use ANY set of practices to do that.

Clearly, what they think they’ve done is establish a test process through automation that will probably discover any important problem that could happen before they release. That’s why Michael and I tested it, and we quickly verified what we expected to find: several problems that materially interfered with the claimed functionality of IMVU, and numerous glitches that suggested the presence of more serious problems nearby. Maybe its present users are willing to put up with it, or maybe they are willing to put up with it for now. But that’s not the point.

The point is that IMVU is not doing certain ordinary and obvious things that would reveal problems in their product and they promote that approach to doing business as if it’s an innovation instead of an evasion of responsibility.

The IMVU people can’t know whether there are, in fact, serious problems in their product because they have chosen not to discover them. That they promote this as a good practice (and that manual testing doesn’t scale, which is also bullshit) tells me that they don’t know what testing is for and they don’t know the difference between testing and a set of computerized behaviors called “test cases”.

They are setting themselves up to rediscover what many others have before them– why we test. Their own experiences will be the best teacher. I predict they will have some doozies.