Monthly Archives: June 2009

The Drunken Gold Rush

Posted on by
29

This comes from an ISTQB advertisement they spammed me with, today:

“To ensure the quality of any software system, testers and QA professionals must thoroughly test the product. But how do you know that these tests are effective? If your team is conducting ad hoc, informal tests with little guidance or planning, the quality of the end product can be severely jeopardized—negatively affecting your bottom line.”

I don’t like to say things like this, nor am I comfortable supporting people who do. It’s not that it’s untrue– it is not necessarily untrue. But it is the kind of statement that fans the flames of a certain sort of Factory School bigotry in our industry. “Oh, you can’t trust testing unless it is pre-planned, pre-packaged, pre-approved, formalized, etc.”

Notice they say nothing about skill. It’s all about methodology, here, not skill. This kind of setup suggests that the next statement will be about the importance of factory-like test methodology. But that’s not what happens.

“The best way to be certain that you are providing customers with quality software is to make sure your team of testers is certified.”

Friends, I’m aware of no one in the industry– not even my worst enemies, not even Rex Black or Stuart Reid– who would publicly assert this or defend it. In fact, in debates against those of us who think certification is consumer fraud, the most typical move is for certificationists to say that certification isn’t even about skill, but rather about basic knowledge. “It’s a start” they say. “It’s a foundation.” (I reply that it’s a bad start and a bad foundation. Much worse than what it tries to replace.)

But then they allow this sort of advertising to go out! Completely undercutting their innocent-sounding plea! And they wonder why I complain that they don’t have the best interests of the testing craft at heart.

Notice that the “ad hoc and unplanned” stuff doesn’t even logically connect to certification. In fact, wouldn’t a highly skilled tester be far more likely to succeed with an ad hoc testing regime? When Roger Federer plays ad hoc tennis, I bet he still wins.

I think the reasons they start talking about methodology and end up talking about certification is A) their potential customers don’t understand the difference between skill and method, B) method is more concrete than skill, thus easier to evoke, and C) they know that what they say doesn’t have to be true or even logical, as long as it evokes horror and promises hope.

Oh, but there’s more…

“By taking the Software Tester Certification course and earning an internationally recognized certification in software testing, your team will gain the expertise needed to handle your greatest testing challenges; earn credibility and recognition as competent quality assurance professionals; and provide greater value to your organization.”

It’s internationally recognized? By whom? Some people who don’t study testing and some people who study testing and financially benefit from certification. Okay, but it is also internationally ridiculed by serious testers of many nations who wish to raise themselves to a level of skill that can’t be obtained in just a couple of days of training.

I recently encountered Dot Graham, now semi-retired, who told me that it hurts her feelings when people like me suggest that certificationists are only in it for the money. Dot is a sweet person. I don’t want to hurt her feelings. But I point her to advertising like this and I challenge her to explain it in any other terms. If not greed, then what, Dot? Stupidity? Pride?

Dot doesn’t want to argue with me about this. Of course she doesn’t. Rex Black doesn’t want to argue, either. Naturally. What answer could there be? Lois Koslowski once told me that “big dogs” don’t need to debate (in fairness, Lois Koslowski claims not to be a tester. I agree that she showed no testing competence or knowledge in the conversation we had. I just mention her because she did claim to be in charge of the ASTQB organization. Yikes!) This is capitalism in its ugly form– harvesting the ignorance and fear of others. Debate has no place here.

Is there no one in that self-declared professional community who reviews the advertising and stands up for professional temperance and humility?

G2 Test Labs: Cry “Certification!”

Posted on by
14

A salesman from G2 Test Labs just called me. He said he was from India. He wanted to know if my testing company needed to partner with an offshore lab like his. I’m writing this now, while the memory of the conversation is fresh.

After he made his brief brief opening monologue I asked him “I’m a testing company. Why are you calling me?”

“Maybe you want to have an offshore arm.” He said.

“Well that depends on the skills of your testers. How do you train your testers?” I asked.

“Oh… we don’t do any training. But our testers are certified by other organizations.”

“Which organizations certify your testers?”

“Uh… I will have to check on that and get back to you.”

“Yes, that’s important information. Are ALL your testers certified?”

“Probably… most of them are.”

“Sounds like you don’t know.”

“…”

“Hey, this will make a funny post. Check my blog in about an hour. Goodbye!”

Now, in fairness, the salesman sounded like he was about 22 years old. Perhaps they sent him to call me as part of some hazing ritual.

[Oh, I just remembered, in his opening statement, he mentioned that his company was ISO-9001 certified, too. Wow. That takes me back to 1992, when I was fighting ISO-9001 certification. That certification program turned out not to amount to anything, either.]

Have Internet, Will Test

Posted on by
7

Matt Heusser wrote an interesting post about “boutique testers.” I like the idea of boutique testers (boutique intellectuals of all kinds, actually, which is why I wrote my new book.) And I am an example of one. The testing I’ve done in recent years has been mostly on court cases or part of coaching testers, though. I want to do more ordinary testing. I need that in order to keep up my practice.

I live on an island, making travel expensive and annoying. But I have a great Internet connection.

There are important challenges to being a remote tester, though. The main technical problem, in my experience, is acquiring and configuring the product to test. Getting up to speed fast benefits from being onsite. The main social problem is trust. Hiring a remote tester, especially one who’s supposed to be high powered, is a little like hiring a therapist. In my experience, developers feel more sensitive about a well-paid, high status outsider poking at their work, than they do about an internal tester who probably will disappear in a few weeks.

Then there’s communication. Despite all the tools for modern communication, we haven’t yet developed a culture of remote interaction that lets us use those tools effectively. Even though I’m always on Skype, and people can see I’m on Skype, they still ask permission to call me on Skype! And I also feel nervous about calling other people on Skype. They might be annoyed with me. I do use GoToMeeting, and that helps a lot. Michael Bolton and I have been collaboratively writing, recently, using it. We like it.

Finally, there is one big logistical problem: availability. You can call me up and have me test for you. But, being a fully independent consultant, my time is chopped up. I have a week here and a few days there, usually. This is the main reason I go in for short-term consulting and coaching. Unless a rich client comes along and induces me to clear my schedule, I can’t afford to have only that one client.

Still, with the price of travel so high, I think this is the direction we need to go: each of us developing ourselves into unique thinkers with strong brands, and then remotely connecting (interesting oxymoron, eh?) with our clients.

Phone: 360-440-1435
Email: james@satisfice.com
Skype: satisfice

Putting Subtitles to Testing

Posted on by
7

I’ve released a new video, which is a whimsical look at a serious subject: explaining exploratory testing.

In the video, my brother and I independently test an “Easy Button” for 10 minutes. Neither of us had seen the other’s test session. Then I edited the 20 minutes of total testing down to a 4 minute highlight reel and added subtitles.

The subtitles are important. One of the core skills of excellent testing is being able to reflect upon, describe, explain, and defend your work. The rhetoric of testing is a big part of Rapid Testing methodology.

So, everything we did, we can explain. If someone stops me when I’m testing, I can give a report on the spot, in oral or written form, and I can put specific technical terminology to it. In my experience, most testers are not able to do that, and there’s one major reason– they don’t practice. It does take practice, friends. While you were enjoying your Sunday, my brother and I were challenging each other to a testing duel.

You might quibble with me about the specific terminology that I used in the video. Indeed, there is a great deal of leeway. One single test activity might simultaneously be a function test, a happy path test, a scenario test, a claims test, and a state-transition test! There’s no clean orthogonality to be found. And as you already know if you read my blog, I reject any “official” lexicon of testing. But I’m not just throwing these terms around, I can explain each one, and say what is and is not an example of it.

What about the Easy Button?

Our principal finding is that the Easy Button is extremely durable. I’m surprised at the high quality of the fit and finish. Also it feels solid (I discovered why when I disassembled it and found apparently lead weights inside. Plus, the button surface is amazingly resilient to repeated hard blows with a rock hammer).

But I’m also surprised that it claims not to be a “toy.” Of course it’s a toy. Of course little kids will play with it.

If I were seriously consulting about testing it, I would probably suggest that its physical qualities were more important to validate than its functional qualities. There appears to be little risk associated with its functionality. On the other hand, there appears to be little risk with its physical qualities either.

I would suggest that it’s far more important to test the web version of the “Easy Button” than the physical version. I would move on to that.

Reclaim Your Personal Method

Posted on by
3

(Since this pertains to both self-education AND technical work, I’m posting this on both of my blogs)

Randy Ingermanson has an interesting approach to writing fiction. It’s called the Snowflake Method. It looks interesting, but I won’t be following it in my work.

First, Don’t Follow
I only use my own methods. That is to say I’m happy to use anyone else’s ideas, but only if they become mine, first. I can learn from other people, but I don’t follow anyone. See the difference? The only way I can responsibly follow someone as a thinker is if they are supervising my work. For instance, when Captain Ben taught me to sail, I used his methods because he was right there to correct me. Also it was his boat, and he answered my questions and let me experiment with alternative ideas to see why they were inferior. As he trained me, his methods became my methods. I began to do them based on my sense of their logic– which means I also came to understand under what circumstances I might need to change them. That’s the difference between learning and numb indoctrination.

When Jerry Weinberg taught me the Fieldstone Method of writing, I formed my own interpretation of it, and now it’s the James Bach version of Weinberg’s Fieldstone Method. And when I teach Rapid Software Testing, my methodology ideally becomes personal to each student, morphing to their own preferences and patterns, or else they should not be using it.

“Composting” Good?
In describing the Snowflake Method, Ingermanson discusses something that he says every writer does: composting. That’s where you actually dream up the story. He writes that

“It’s an informal process and every writer does it differently. I’m going to assume that you know how to compost your story ideas and that you have already got a novel well-composted in your mind and that you’re ready to sit down and start writing that novel.He says how you do that is a personal creative matter.”

Okay. Interesting that he says nothing about how to do that, though, since for me that’s almost all that writing is. But now, the actual Snowflake Method, he says, kicks in after composting is done with. It’s a way of progressive outlining of the book so that you can write it in an organized way.

Wait, did he say that happens after composting?

AFTER composting? Seriously?

This is a problem for me, because I’m nearly always doing that thing he calls composting. For me, writing is an exploratory activity. I’m constructing my ideas before I write them down and also as I’m writing them down. I’ve written many articles and two books that way. I have not yet written much fiction, but I have a hard time believing my method will be or should be different for fiction.

“Seat of the Pants” Bad?
Here Ingermanson makes a tiresome rhetorical move: He contrasts his approach with the “seat of the pants” method. He believes his method is better. I agree that it’s probably better for him, because it’s his own personal method. But on what basis can he say that his method is better than the alternatives for anyone else? Besides, it sounds like “composting” is just “seat of the pants” that happens to be Ingermanson-approved.

This is typical best practices rhetoric, and the pattern generally goes as follows:

1. I conceive my method as figure and everything else as ground. I won’t talk about how my method blends into and is supported by any other methods or skills or talents or preferences. I won’t talk about how it may go horribly wrong. The method is an island.

2. Since I like my method better than the other not-the-method thing I once did, [cite anecdotal and cherry-picked evidence here], it is probably better.

3. Since I taught someone else to use my method and they said they like my method better than whatever unexamined way of working they once had, [did they actually use my method? Well, they said they did, but I didn’t actually watch them do it], it is even more probably better.

There are a few problems with this pattern of reasoning. One is that it is not necessarily a comparison of one method to another. It’s more likely a comparison of a state of confusion to a state of decision. Decision usually does win over confusion. The people who are out looking for a method may not already have a sound understanding of the methods they already use. So they leap on any method offered as if it were a life buoy. This of course is no indication that the method itself is better than any other method, but merely that people hate feeling confused and incompetent.

Another problem is that even when it is a comparison of methods, it’s generally a comparison between an ineffable method and one that sounds good when explained. Things that are ineffable, no matter how useful, get a bad reputation. That’s why you’ve met at least one person in your life who has claimed that you need to “learn to breathe” or “remember to breathe.” In fact, you already have a method of breathing, and unless your eyes have just gone so fuzzy that you can’t read this at all, you are probably breathing pretty well right this moment. An effective way to present a method of breathing could be to say “If you are having problem X, one solution might be to try a special kind of breathing called Y. Let’s try it now so you can see what I mean…” This way offers the practice without implicitly or explicitly denying other ways of working.

Yet another problem is that all methods rest on a certain way of organizing the world. If you don’t accept that foundation, then the method won’t satisfy you. Ingermanson seems to find it easy to segment heavy creative work from the light creative work. Hence composting is good, but seat-of-the-pants writing is bad. Since I don’t accept that distinction, to use the Snowflake Method as presented would force me to become alienated from my creative process. I would not be in direct touch with my own mind, but all thoughts would be mediated through the controlling outline of the Snowflake. Ick!

A Rhetoric for Pushing Back
It’s not “seat of the pants”, I say. It’s not merely “ad hoc.”

It’s thoughtful and responsible, rather than mindless and robotic. It’s exploratory, rather than pre-scripted. It’s agile rather than rigid. It’s constructive and generative, rather than a mere conditioned response.

Want more? Try breaking the method down into sub-parts. In exploratory work, I might cite such tasks as:

  • overproduce ideas and abandon them (think “brainstorming”)
  • recover previously abandoned ideas (think “boneyard”)
  • pursue lines of inquiry
  • conduct thought experiments
  • alternate my tactics for better progress
  • dynamically manage my focus (from very focused to de-focused)
  • charter my own work in light of my mission as I understand it
  • view my work from different perspectives
  • produce results, then reproduce them differently based on what I learned (cyclic learning)
  • construct a new and better version of myself as I work

Seat of the pants? That sounds like a put-down. Why don’t they call it dynamic control and development? Because that doesn’t sound like a put-down.

Reclaim Your Personal Method
As Adam Savage says, “I reject your reality, and substitute my own.” Yes, indeedy.

You don’t have to accept someone else’s intermediating artifice between you and your thoughts. Whether that’s a book outline, or a test plan document, TPI, or some method of artificial breathing you can say no. You can say “that would be irresponsible, because I must remain attached to the source of my own methods of working. I can’t drive a car safely from the BACK SEAT!”

Having said all that, I found Randy’s Snowflake Method interesting and I think I will try it. I will meld it with my exploratory style of working, of course, and claim it for my own.